h;8UdZddlmZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZddlmZddlmZmZddlmZdd lmZej0d Zdd Zdd ZddZddZdaded<ddZ GddeZ!y)a!Module contains code for a cache backed embedder. The cache backed embedder is a wrapper around an embedder that caches embeddings in a key-value store. The cache is used to avoid recomputing embeddings for the same text. The text is hashed and the hash is used as the key in the cache. ) annotationsN)Sequence)CallableLiteralOptionalUnioncast) Embeddings) BaseStore ByteStore) batch_iterate)EncoderBackedStorei)intctj|jddj}t j t |S)aReturn a UUID derived from *text* using SHA-1 (deterministic). Deterministic and fast, **but not collision-resistant**. A malicious attacker could try to create two different texts that hash to the same UUID. This may not necessarily be an issue in the context of caching embeddings, but new applications should swap this out for a stronger hash function like xxHash, BLAKE2 or SHA-256, which are collision-resistant. utf-8F)usedforsecurity)hashlibsha1encode hexdigestuuiduuid5NAMESPACE_UUID)textsha1_hexs X/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain/embeddings/cache.py_sha1_hash_to_uuidrs7||DKK0%HRRTH ::nh //c6dk(r tdfd }|S)aCreate a default key encoder function. Args: namespace: Prefix that segregates keys from different embedding models. algorithm: * ``'sha1'`` - fast but not collision-resistant * ``'blake2b'`` - cryptographically strong, faster than SHA-1 * ``'sha256'`` - cryptographically strong, slower than SHA-1 * ``'sha512'`` - cryptographically strong, slower than SHA-1 Returns: A function that encodes a key using the specified algorithm. rcdk(rt|Sdk(r6tj|jdj Sdk(r6tj |jdj Sdk(r6tj |jdj Sd}t|)z+Encode a key using the specified algorithm.rblake2brsha256sha512zUnsupported algorithm: )rrr!rrr"r# ValueError)keymsg algorithm namespaces r _key_encoderz/_make_default_key_encoder.._key_encoder<s  [!3C!8 9: :  ![G1D!E!O!O!Q RS S  [ 70C!D!N!N!P QR R  [ 70C!D!N!N!P QR R' {3or)r%strreturnr*)_warn_about_sha1_encoder)r(r'r)s`` r_make_default_key_encoderr-+s F "  rcHtj|jS)zSerialize a value.)jsondumpsr)values r_value_serializerr2Ls ::e  # # %%rcrttttj|j S)zDeserialize a value.)r listfloatr/loadsdecode)serialized_values r_value_deserializerr9Qs& U TZZ(8(?(?(AB CCrFbool_warned_about_sha1cNtstjdtddayy)z9Emit a one-time warning about SHA-1 collision weaknesses.aUsing default key encoder: SHA-1 is *not* collision-resistant. While acceptable for most cache scenarios, a motivated attacker can craft two different payloads that map to the same cache key. If that risk matters in your environment, supply a stronger encoder (e.g. SHA-256 or BLAKE2) via the `key_encoder` argument. If you change the key encoder, consider also creating a new cache, to avoid (the potential for) collisions with existing keys.)category stacklevelTN)r;warningswarn UserWarningrrr,r,Zs.   J! " rceZdZdZddd dfdZddZddZddZddZe d dd d d  dd Z xZ S)CacheBackedEmbeddingsaInterface for caching results from embedding models. The interface allows works with any store that implements the abstract store interface accepting keys of type str and values of list of floats. If need be, the interface can be extended to accept other implementations of the value serializer and deserializer, as well as the key encoder. Note that by default only document embeddings are cached. To cache query embeddings too, pass in a query_embedding_store to constructor. Examples: .. code-block: python from langchain.embeddings import CacheBackedEmbeddings from langchain.storage import LocalFileStore from langchain_community.embeddings import OpenAIEmbeddings store = LocalFileStore('./my_cache') underlying_embedder = OpenAIEmbeddings() embedder = CacheBackedEmbeddings.from_bytes_store( underlying_embedder, store, namespace=underlying_embedder.model ) # Embedding is computed and cached embeddings = embedder.embed_documents(["hello", "goodbye"]) # Embeddings are retrieved from the cache, no computation is done embeddings = embedder.embed_documents(["hello", "goodbye"]) N batch_sizequery_embedding_storecZt|||_||_||_||_y)aInitialize the embedder. Args: underlying_embeddings: the embedder to use for computing embeddings. document_embedding_store: The store to use for caching document embeddings. batch_size: The number of documents to embed between store updates. query_embedding_store: The store to use for caching query embeddings. If ``None``, query embeddings are not cached. N)super__init__document_embedding_storerHunderlying_embeddingsrG)selfrMrLrGrH __class__s rrKzCacheBackedEmbeddings.__init__s/" (@%%:"%:"$rc |jj|}t|Dcgc] \}}| | }}}t|j|D]v}|Dcgc]}|| }}|j j |}|jjtt||t||D] \} } | || < xtttt|Scc}}wcc}w)wEmbed a list of texts. The method first checks the cache for the embeddings. If the embeddings are not found, the method uses the underlying embedder to embed the documents and stores the results in the cache. Args: texts: A list of texts to embed. Returns: A list of embeddings for the given texts. ) rLmget enumerater rGrMembed_documentsmsetr4zipr r5 rNtextsvectorsivectorall_missing_indicesmissing_indices missing_textsmissing_vectorsindexupdated_vectors rrTz%CacheBackedEmbeddings.embed_documentss372O2O2T2T 3  )1* !VV^A* *  -T__>QR 0O/>?!U1X?M?"88HHWO  ) ) . .S89 *-_o)N 0%~!/ 0  0 e    * @s C)C) C/c ,K|jj|d{}t|Dcgc] \}}| | }}}t|j|D]}|Dcgc]}|| }}|j j |d{}|jjtt||d{t||D] \} } | || < tttt|S7cc}}wcc}w77Lw)rQN) rLamgetrSr rGrMaembed_documentsamsetr4rVr r5rWs rrdz&CacheBackedEmbeddings.aembed_documentss.//55e< <  )1* !VV^A* * -T__>QR 0O/>?!U1X?M?$($>$>$O$O%O//55S89  *-_o)N 0%~!/ 0 0 e    % =* @ sODDD DD D# D / DD5DD>D DDc |js|jj|S|jj|g\}||S|jj|}|jj ||fg|S)0Embed query text. By default, this method does not cache queries. To enable caching, set the ``cache_query`` parameter to ``True`` when initializing the embedder. Args: text: The text to embed. Returns: The embedding for the given text. )rHrM embed_queryrRrUrNrcachedr[s rrhz!CacheBackedEmbeddings.embed_querys))--99$? ?..33TF;   M++77= ""''$(89 rcbK|js#|jj|d{S|jj|gd{\}||S|jj|d{}|jj ||fgd{|S7y7W707 w)rgN)rHrM aembed_queryrcreris rrlz"CacheBackedEmbeddings.aembed_querys))33@@FF F44::D6BB   M11>>tDD((..v/?@@@ GBE@sE+B/B'#B/B)(B/:B+;%B/ B-!B/)B/+B/-B/Fr)r(rGquery_embedding_cache key_encodercft|tr t||}n't|r|rd}t |d}t |t tt tf||tt}|dur|} n1|durd} n*t tt tf||tt} ||||| S)a7On-ramp that adds the necessary serialization and encoding to the store. Args: underlying_embeddings: The embedder to use for embedding. document_embedding_cache: The cache to use for storing document embeddings. *, namespace: The namespace to use for document cache. This namespace is used to avoid collisions with other caches. For example, set it to the name of the embedding model used. batch_size: The number of documents to embed between store updates. query_embedding_cache: The cache to use for storing query embeddings. True to use the same cache as document embeddings. False to not cache query embeddings. key_encoder: Optional callable to encode keys. If not provided, a default encoder using SHA-1 will be used. SHA-1 is not collision-resistant, and a motivated attacker could craft two different texts that hash to the same cache key. New applications should use one of the alternative encoders or provide a custom and strong key encoder function to avoid this risk. If you change a key encoder in an existing cache, consider just creating a new cache, to avoid (the potential for) collisions with existing keys or having duplicate keys for the same text in the cache. Returns: An instance of CacheBackedEmbeddings that uses the provided cache. zgDo not supply `namespace` when using a custom key_encoder; add any prefixing inside the encoder itself.zakey_encoder must be either 'blake2b', 'sha1', 'sha256', 'sha512' or a callable that encodes keys.TFNrF) isinstancer*r-callabler$rr4r5r2r9) clsrMdocument_embedding_cacher(rGrnror&rLrHs rfrom_bytes_storez&CacheBackedEmbeddings.from_bytes_storesV k3 '3I{KK k "C!o%3 S/ !#5c4;6F#G $    $  !D ($< ! "e +$( !$6sDK7G$H%!# % ! ! $!"7   r) rMr rLzBaseStore[str, list[float]]rG Optional[int]rHz%Optional[BaseStore[str, list[float]]]r+None)rXz list[str]r+zlist[list[float]])rr*r+ list[float])rMr rtr r(r*rGrvrnzUnion[bool, ByteStore]rozKUnion[Callable[[str], str], Literal['sha1', 'blake2b', 'sha256', 'sha512']]r+rE) __name__ __module__ __qualname____doc__rKrTrdrhrl classmethodru __classcell__)rOs@rrErElsL%)GK %)%#>% " % E % %.  D$ L.. $(8= T )T #,T  T " T  6T  T  T T rrE)rr*r+z uuid.UUID)r(r*r'r*r+zCallable[[str], str])r1zSequence[float]r+bytes)r8rr+rx)r+rw)"r| __future__rrr/rr@collections.abcrtypingrrrrr langchain_core.embeddingsr langchain_core.storesr r langchain_core.utils.iterr langchain.storage.encoder_backedrUUIDrrr-r2r9r;__annotations__r,rErCrrrsv# $;;063?t$ 0B& D !D "$D JD r