Li7 dZddlZddlmZmZddlmZmZddlmZddl m Z ddl m Z dd l m Zdd l mZej d Zgd ZGd dZdedej*j,j.defdZdedefdZGddZy)z)[Experimental] Text Only Local Tokenizer.N)AnyIterable)OptionalUnion)sentencepiece_model_pb2)_common)_local_tokenizer_loader) _transformers)typeszgoogle_genai.local_tokenizer)_parse_hex_byte_token_str_to_bytesLocalTokenizer_TextsAccumulatorceZdZdZddZdeefdZdeejddfdZ dejddfd Z d ejddfd Z d ejdejfd ZdeejddfdZdeej"ddfdZdej"ddfdZdej(dej(fdZdej,dej,fdZdeeefdeeeffdZdedefdZy)raAccumulates countable texts from `Content` and `Tool` objects. This class is responsible for traversing complex `Content` and `Tool` objects and extracting all the text content that should be included when calculating token counts. A key feature of this class is its ability to detect unsupported fields in `Content` objects. If a user provides a `Content` object with fields that this local tokenizer doesn't recognize (e.g., new fields added in a future API update), this class will log a warning. The detection mechanism for `Content` objects works by recursively building a "counted" version of the input object. This "counted" object only contains the data that was successfully processed and added to the text list for tokenization. After traversing the input, the original `Content` object is compared to the "counted" object. If they don't match, it signifies the presence of unsupported fields, and a warning is logged. returnNcg|_yN_textsselfs Z/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/google/genai/local_tokenizer.py__init__z_TextsAccumulator.__init__;s DKc|jSrrrs r get_textsz_TextsAccumulator.get_texts>s ;;rcontentsc4|D]}|j|yr) add_content)rrcontents r add_contentsz_TextsAccumulator.add_contentsAs  w rr!chtjg|j}|jrB|jD]2}|jJtj}|j |j td|j|j|_|j,|j|j|j|_ |j,|j|j|j|_ |j6|j|_ |jj|j|jj|5|j!d|j!dk7rt"j%d|d|dyy)N)partsrolez6LocalTokenizers do not support non-text content types.T) exclude_nonezHContent contains unsupported types for token counting. Supported fields z. Got .)r Contentr%r$Part file_data inline_data ValueErrorvideo_metadata function_calladd_function_callfunction_responseadd_function_responsetextrappend model_dumploggerwarning)rr!counted_contentpart counted_parts rr z_TextsAccumulator.add_contentEsmm"7<<@O}}--3$$$000zz| >> %)9)9)EF      *(,(;(;, %    )  !3!3 4'+'9'9, $  ! ! -  $ $T%;%; <+/+A+A, ( 99 "ii,  ++  TYY '$$\2%3(t,0J0J1K1 nn $%VG9A 7rr.c|jr%|jj|jtj|j}|j r#|j |j }||_yy)zProcesses a function call and adds relevant text to the accumulator. Args: function_call: The function call to process. )nameN)r;rr3r FunctionCallargs_dict_traverse)rr.counted_function_call counted_argss rr/z#_TextsAccumulator.add_function_calldsi  kk++,!..M4F4FG((););)rr0counted_function_responsecounted_responses rr1z'_TextsAccumulator.add_function_responsess!& 6 6 8 kk*//0'8'='=$!!,,->-G-GH+;("rrGctj}|jr6|jj |j|j|_|j r6|jj |j |j |_|j r"|j|j }||_|jr"|j|j}||_|Sr) r FunctionDeclarationr;rr3 description parameters add_schemarR)rrGrHcounted_parametersrTs rrEz0_TextsAccumulator._function_declaration_traverses$)#<#<#>    kk-223*>*C*C"''' kk-99:  * *#.&&??+?+J+JK0B"-$$)=)F)FG.>"+ ''rschemactj}|jr|j|_|jr|j|_|j|j|_|j r6|j j|j |j |_|jr6|j j|j|j|_|jr6|j j|j|j|_ |jr6|j j|j|j|_ |jr|j|_ |jr"|j|j}||_ |jr\i}|jjD]6\}}|j j||j|}|||<8||_|j r"|j#|j }||_|S)zProcesses a schema and adds relevant text to the accumulator. Args: schema: The schema to process. Returns: The new schema object with only countable fields. )r Schematypetitledefaultformatrr3rWenumextendrequiredproperty_orderingitemsrY propertiesexample _any_traverse) rr[counted_schemacounted_schema_itemsdkeyvalue counted_valuecounted_schema_examples rrYz_TextsAccumulator.add_schemas\\^N {{"KKn ||#\\n ~~!%~~n }} kk'$mmn  kk++,#)#5#5n  {{ kk%"KKn  kk) &n )/)A)An& ||!__V\\:1n  a))//1*#u 3. ##$n ~~#11&..A5n rrlci}|jjt|j|j D]\}}|j |||<|S)zProcesses a dict and adds relevant text to the accumulator. Args: d: The dict to process. Returns: The new dict object with only countable fields. )rrclistkeysrfri)rrl counted_dictrmvals rr>z _TextsAccumulator._dict_traversesYLKKtAFFH~&GGI2S,,S1l32 rrnc t|tr|jj||St|tr|j |St|t r|Dcgc]}|j|c}S|Scc}w)zProcesses a value and adds relevant text to the accumulator. Args: value: The value to process. Returns: The new value with only countable fields. ) isinstancestrrr3dictr>rrri)rrnitems rriz_TextsAccumulator._any_traversesp% kk l E4   '' E4 38 94d  & 99 l:s#B)rN)__name__ __module__ __qualname____doc__rrrxrr r(r"r r<r/rDrIrLrQrOr1rVrEr]rYryrr>rirrrr'sM& # 8EMM#: t 4> 0U-?-? 0D 0 5:: %** Xejj1d4 ()?)? @4 4 <$55 <  <("'";";(   ((+u||+ +Z d38n c3h rrtokenr^rc|tjjjjk(rt |j ddS|jddjdS)Nrbig)length byteorderu▁ zutf-8) r ModelProto SentencePieceTypeBYTEr to_bytesreplaceencode)rr^s rrrs\ $ / / = = B B G GG 5 ! * *!u * EE == $ + +G 44rct|dk7rtd||jdr|jdstd| t |ddd}|d k\rtd ||S#t$rtd |wxYw) zParses a hex byte string of the form '<0xXX>' and returns the integer value. Raises ValueError if the input is malformed or the byte value is invalid. zInvalid byte length: z<0x>zInvalid byte format: zInvalid hex value: zByte value out of range: )lenr, startswithendswithint)rrus rr r s  Z1_ ,UG4 55   % s(; ,UG4 554 eAaj" C CZ 08 99 * 4 *5'2 334s A22B c@eZdZdZdefdZejddddee je jfde e jd e jfd Zejddee je jfd e j fd Zy) ra[Experimental] Text Only Local Tokenizer. This class provides a local tokenizer for text only token counting. LIMITATIONS: - Only supports text based tokenization and no multimodal tokenization. - Forward compatibility depends on the open-source tokenizer models for future Gemini versions. - For token counting of tools and response schemas, the `LocalTokenizer` only supports `types.Tool` and `types.Schema` objects. Python functions or Pydantic models cannot be passed directly. model_namectj||_tj|j|_tj |j|_yr)loaderget_tokenizer_name_tokenizer_nameload_model_proto _model_protoget_sentencepiece _tokenizer)rrs rrzLocalTokenizer.__init__$sF!44Z@D//0D0DED..t/C/CDDOrzThe SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.N)configrrrctj|}t}tjj |xsi}|j ||jr|j|j|jr;|jjr%|j|jj|jr/|j tj|jg|jjt|j!}tj"t%d|DS)aCounts the number of tokens in a given text. Args: contents: The contents to tokenize. config: The configuration for counting tokens. Returns: A `CountTokensResult` containing the total number of tokens. Usage: .. code-block:: python from google import genai tokenizer = genai.LocalTokenizer(model_name='gemini-2.0-flash-001') result = tokenizer.count_tokens("What is your name?") print(result) # total_tokens=5 c32K|]}t|ywr)r).0tokenss r z.LocalTokenizer.count_tokens..Rs?V?s) total_tokens)t t_contentsrr CountTokensConfigmodel_validater"rJrLgeneration_configresponse_schemarYsystem_instructionrrrrrCountTokensResultsum)rrrprocessed_contentstext_accumulator tokens_lists r count_tokenszLocalTokenizer.count_tokens)s:h/(*  $ $ 3 3FLb AF!!"45 ||  . F$<$<$L$L!!&":":"J"JK   ##ALL&2K2K1L$MN//((.>.H.H.J)KLK  " "?;?? rctj|}t}|D]}|j||jj |j }g}|D];}|js|jD]}|j|j=g}t||D]\} } |jtj| jD cgc]} | jc} | jD cgc]C} t| j |j"j| jj$Ec} | tj&|Scc} wcc} w)a,Computes the tokens ids and string pieces in the input. Args: contents: The contents to tokenize. Returns: A `ComputeTokensResult` containing the token information. Usage: .. code-block:: python from google import genai tokenizer = genai.LocalTokenizer(model_name='gemini-2.0-flash-001') result = tokenizer.compute_tokens("What is your name?") print(result) # tokens_info=[TokensInfo(token_ids=[279, 329, 1313, 2508, 13], tokens=[b' What', b' is', b' your', b' name', b'?'], role='user')] ) token_idsrr%) tokens_info)rrrr rEncodeAsImmutableProtorr$r3r%zipr TokensInfopiecesidrpiecerr^ComputeTokensResult) rrrrr! tokens_protosroles_ token_infos tokens_protor%rs rcompute_tokenszLocalTokenizer.compute_tokensUs[4h/(*%,""7+,OO::""$M E%%  %A ,,w|| $ %% K!-7  d   /;/B/BCeC ,22 &kk4#4#4#;#;EHH#E#J#J      $ $ ==Ds E/AE4)r{r|r}r~rxrr experimental_warningrr ContentListUnionContentListUnionDictrCountTokensConfigOrDictrrrrrrrrrs EE  7B9= &e,,e.H.HHI&u445 &   & &P 7B2>e,,e.H.HHI2>   2> 2>rr)r~loggingtypingrrrr sentencepiecerr r rr rr getLoggerr5__all__rrxrrrbytesrrr rrrrrs 0 "1/    9 : MM`5 5-88FFKK5 5 3 3 ,u>u>r