Li 8ddlZddlZddlZddlZddlZddlmZmZddlZddl Z ddl Z ddl m Z ddddddZ dddddddddd ZejdGd d Zed d edd dZdedefdZdededefdZdeddfdZdededeefdZdedededdfdZdededefdZdededefd Zd!edefd"Zej:d!ede j<fd#Zd$edefd%Z ej:d!ede jBfd&Z"y)'N)Optionalcast)sentencepiece_model_pb2gemma3)zgemini-2.5-prozgemini-2.5-flashzgemini-2.5-flash-litezgemini-2.0-flashzgemini-2.0-flash-lite) zgemini-2.5-pro-preview-06-05zgemini-2.5-pro-preview-05-06zgemini-2.5-pro-exp-03-25zgemini-live-2.5-flashzgemini-2.5-flash-preview-05-20zgemini-2.5-flash-preview-04-17z#gemini-2.5-flash-lite-preview-06-17zgemini-2.0-flash-001zgemini-2.0-flash-lite-001T)frozenc"eZdZUeed<eed<y)_TokenizerConfig model_url model_hashN)__name__ __module__ __qualname__str__annotations__b/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/google/genai/_local_tokenizer_loader.pyr r 2s ./rr zyhttps://raw.githubusercontent.com/google/gemma_pytorch/33b652c465537c6158f9a472ea5700e5e770ad3f/tokenizer/tokenizer.model@61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2)r r zhttps://raw.githubusercontent.com/google/gemma_pytorch/014acb7ac4563a5f77c76d7ff98f31b568c16508/tokenizer/gemma3_cleaned_262144_v2.spiece.model@1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c)gemma2r file_url_pathreturnctj|}|jtt|j S)z.Loads file bytes from the given file url path.)requestsgetraise_for_statusrbytescontent)rresps r _load_filer Is- m $$ eT\\ ""r model_data expected_hashch|s tdtj|j|k(S)z:Returns true if the content is valid by checking the hash.zexpected_hash is required) ValueErrorhashlibsha256 hexdigestr!r"s r_is_valid_modelr)Ps.  0 11  # - - /= @@r file_pathctjj|sy tj|y#t$rYywxYw)zRemoves the file if exists.N)ospathexistsremoveOSError)r*s r_maybe_remove_filer1Ws8  "  IIi   s8 AActjj|syt|d5}|j }dddt |r|St |y#1swY$xYw)z&Loads the content from the cache path.Nrbr()r,r-r.openreadr)r1)r*r"frs r_maybe_load_from_cacher7bs_  "  ItffhG}E NY s A!!A* cache_dir cache_pathrc. tj|d|dzttjzdz}t |d5}|j |dddtj||y#1swY xYw#t$rYywxYw)z$Saves the content to the cache path.T)exist_ok.z.tmpwbN) r,makedirsruuiduuid4r4writerenamer0)r8r9rtmp_pathr6s r_maybe_save_to_cacherDrs KK D)3TZZ\!22V;H h gggIIh #   s*A B A<B<BB BBfile_urlct|}t||s5tj|j }t d|d|d|S)z*Loads model bytes from the given file url.r(z2Downloaded model file is corrupted. Expected hash z. Got file hash r<)r r)r%r&r'r$)rEr"r actual_hashs r_load_from_urlrHsZ x ' G= I..)335K  '(8 Q H  .rc`tjjtjd}t j |jj}tjj||}t||}|s t||}t||||S)aYLoads model bytes from the given file url. 1. If the find local cached file for the given url and the cached file hash matches the expected hash, the cached file is returned. 2. If local cached file is not found or the hash does not match, the file is downloaded from the given url. And write to local cache and return the file bytes. 3. If the file downloaded from the given url does not match the expected hash, raise ValueError. Args: file_url: The url of the file to load. expected_hash: The expected hash of the file. Returns: The file bytes. vertexai_tokenizer_model)r*r"rEr")r8r9r) r,r-jointempfile gettempdirr%sha1encoder'r7rHrD)rEr" model_dirfilename model_pathr!s r_loadrTs$ggll8..02LM) \\(//+ , 6 6 8(ww||Ix0*%-* OJj* rtokenizer_namec |tvr,td|dttjt t|j t|j S)z6Loads model proto bytes from the given tokenizer name.z Tokenizer z( is not supported.Supported tokenizers: rK) _TOKENIZERSr$listkeysrTr r )rUs r_load_model_proto_bytesrZsh;&  ^$%!!%k&6&6&8!9 : <  >*44/:: rcbtj}|jt||S)z0Loads model proto from the given tokenizer name.)r ModelProtoParseFromStringrZ)rU model_protos rload_model_protor_s, (224+ 5nEF r model_namec ,|tjvr t|S|tjvr t|Std|ddj tjddj tjd)z1Gets the tokenizer name for the given model name.zModel z% is not supported. Supported models: z, z. )!_GEMINI_MODELS_TO_TOKENIZER_NAMESrY(_GEMINI_STABLE_MODELS_TO_TOKENIZER_NAMESr$rL)r`s rget_tokenizer_namerds499;; ,Z 88;@@BB 3J ??zl? JkJpJpJr@s?ttvw{xAxABjBoBoBqxrwssvw rcbtj}|jt||S)zrrs  ! 1  % % %!%-$, (%&.&.+3$!) ,(d#$N N d N  #c#e#A5AAA # $   &)  e_     #&  16       C E s35D C E ''33cc.H.Hr