h2^ddlmZddlZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZmZmZmZmZddlmZmZddlmZe rdd lmZmZmZdd lmZej<eZ ed d Z!Gdd eeZ"Gdde"Z#Gdde$e Z%edGddZ&ddZ'y)) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyCallableLiteralOptionalTypeVarUnion)BaseDocumentTransformerDocument)Self) CollectionIterableSequence)SetTS TextSplitter)boundceZdZdZddedddf ddZeddZ d dd Zdd Z dd Z dd Z e dd Z e ddedf ddZ ddZy)rz)Interface for splitting text into chunks.iFTc|dkrd|}t||dkrd|}t|||kDrd|d|d}t|||_||_||_||_||_||_y)adCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacemsgs [/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_text_splitters/base.py__init__zTextSplitter.__init__!s* ?0 =CS/ ! 1 4]ODCS/ ! : %.}o><46 S/ !%+ /- /!1cy)z$Split text into multiple components.N)r#texts r+ split_textzTextSplitter.split_textIsr-Nc |xsigt|z}g}t|D]\}}d}d}|j|D]} tj||} |j r>||z|j z } |j| td| }|| d<t| }t| | } |j| |S)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater1copydeepcopyr!rfindmaxrappend) r#texts metadatas _metadatas documentsir0indexprevious_chunk_lenchunkr5offsetnew_docs r+create_documentszTextSplitter.create_documentsMs32$U"3   ' *GAtE!" . *==A7(("%77$:M:MMF IIeSF^)r<r4r5rG)r#r@r=r>docs r+split_documentszTextSplitter.split_documentsasVry +C LL)) *   S\\ * +$$Ui$@@r-cl|j|}|jr|j}|dk(ry|S)N)joinr"strip)r#docs separatorr0s r+ _join_docszTextSplitter._join_docsis3~~d#  ! !::>B>N>N=OQ{#a'//+yAC C( $"5"55 [9IA9M STU**+!AI!6!6{1~!F-0-=-AMq"'2!"o  $"5"55 [9IA9M STU**+!AI   q ! Tc+.>.B]J JE9 K:ook95 ? KK  r-c  ddlm}t|s d}t|dfd }|d d|i|S#t$r}d}t||d}~wwxYw) z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBasec8tj|SN)r6tokenizer0 tokenizers r+_huggingface_tokenizer_lengthzNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengths9--d344r-z`Could not import transformers python package. Please install it with `pip install transformers`.Nr&r0strreturnintr/)$transformers.tokenization_utils_baser^ isinstancer ImportError)clsrckwargsr^r*rderrs ` r+from_huggingface_tokenizerz'TextSplitter.from_huggingface_tokenizersp + Ti)@AW!o% 5K#@KFKK  +E S/s *  +s%2 AA  Agpt2allc  ddl}||j| n|j| d fd } t |t r ||d} i|| }|dd| i|S#t$r}d}t||d}~wwxYw) z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c>tj|SN)allowed_specialdisallowed_special)r6encode)r0rurvencs r+_tiktoken_encoderz=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r-) encoding_name model_namerurvr&rer/)tiktokenrkencoding_for_model get_encoding issubclassTokenTextSplitter) rlrzr{rurvrmr|rnr*ry extra_kwargsrxs `` @r+from_tiktoken_encoderz"TextSplitter.from_tiktoken_encoders ,   !--j9C'' 6C  c, -!.(#2&8 L 0/,/F?#4???? ,A  c" +  ,sA A9&A44A9c 6|jt|S)z2Transform sequence of documents by splitting them.)rJlist)r#r@rms r+transform_documentsz TextSplitter.transform_documentss##DO44r-)r$rhr%rhr&zCallable[[str], int]r'z$Union[bool, Literal['start', 'end']]r(boolr)rrgNoner0rfrg list[str]r`)r=rr>zOptional[list[dict[Any, Any]]]rglist[Document])r@zIterable[Document]rgr)rOrrPrfrg Optional[str])rVz Iterable[str]rPrfrgr)rcr rmr rgr) rzrfr{rru'Union[Literal['all'], AbstractSet[str]]rv&Union[Literal['all'], Collection[str]]rmr rgr)r@Sequence[Document]rmr rgr)__name__ __module__ __qualname____doc__r6r,rr1rGrJrQr\ classmethodrosetrrr/r-r+rrsH3 03?D %!%&2&2&2. &2 = &2  &2&2 &2P33MQ+I (A(TLL,$$(CF5EJ *@*@"*@A *@ C *@  *@ *@*@X5+57:5 5r-cVeZdZdZddedf dfd ZddZxZS) rz/Splitting text to tokens using model tokenizer.rpNrqc t |di| ddl}||j |} n|j |} | |_||_||_y#t$r}d}t||d}~wwxYw)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r/) superr,r|rkr}r~ _tokenizer_allowed_special_disallowed_special) r#rzr{rurvrmr|rnr*rx __class__s r+r,zTokenTextSplitter.__init__s "6" ,   !--j9C'' 6C /#5  ,A  c" +  ,sA A-A((A-cdfd }tjjjj|}t ||S)aSplits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. chjj|jjSrt)rrwrr)_textr#s r+_encodez-TokenTextSplitter.split_text.._encodes4??)) $ 5 5#'#;#;* r-)r%tokens_per_chunkdecoderwrb)rrfrgz list[int]) Tokenizerrrrrsplit_text_on_tokens)r#r0rrcs` r+r1zTokenTextSplitter.split_textsE  --!--??))  $CCr-) rzrfr{rrurrvrrmr rgrr)rrrrrr,r1 __classcell__)rs@r+rrs]9$$(CF5EJ 66"6A 6 C 6  6 68Dr-rc|eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZy)Languagez"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6N)rrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6r/r-r+rr&s, C B D F B B C E F C D D E EH E D C F E A C DG FJ!Lr-rT)frozenc@eZdZUdZded< ded< ded< ded<y ) rzTokenizer data class.rhr%rzCallable[[list[int]], str]rzCallable[[str], list[int]]rwN)rrrr__annotations__r/r-r+rrFs)*, &&= &&=r-rcg}|j|}d}t||jzt|}|||}|t|kr|j |j ||t|k(r |S||j|j z z }t||jzt|}|||}|t|kr|S)z6Split incoming text and return chunks using tokenizer.r)rwminrr6r<rr%)r0rcrV input_ids start_idxcur_idx chunk_idss r+rrTsF  &II)i888#i.IG)G,I c)n $ i&&y12 c)n $  M Y//)2I2III i)"<"<rs" #!G">>2   8 $ T(E5*CE5P=D =D@"sD"@ $ > > >r-