hQddlmZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZdd Z dd Z!GddeZ"y)) annotationsN)Path)AnyCallableDictIterableListOptionalTuple)Document) Embeddings guard_import) VectorStore) AddableMixinDocstore)InMemoryDocstore)DistanceStrategycz|tjtjj|ddddz}|S)z!Normalize vectors to unit length.T)axiskeepdimsg-q=N)npcliplinalgnorm)xs d/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/scann.py normalizers1T:E4 HHA HctdS)z= Import `scann` if available, otherwise raise error. scannrr rdependable_scann_importr$s   r cXeZdZdZddej df ddZ d ddZ d ddZ d ddZ dddZ d dd Z d dd Z d dd Z d d d Ze d! d"d Ze d d#dZe d d$dZd%d&dZe d%dd d'dZd(dZ d ddZy))ScaNNa`ScaNN` vector store. To use, you should have the ``scann`` python package installed. Example: .. code-block:: python from langchain_community.embeddings import HuggingFaceEmbeddings from langchain_community.vectorstores import ScaNN model_name = "sentence-transformers/all-mpnet-base-v2" db = ScaNN.from_texts( ['foo', 'bar', 'barz', 'qux'], HuggingFaceEmbeddings(model_name=model_name)) db.similarity_search('foo?', k=1) NFc t||_||_||_||_||_||_||_||_y)z%Initialize with necessary components.N) embeddingindexdocstoreindex_to_docstore_iddistance_strategyoverride_relevance_score_fn _normalize_L2 _scann_config) selfr(r)r*r+relevance_score_fn normalize_L2r, scann_configs r__init__zScaNN.__init__3sA#   $8!!2+=())r c ~t|jtstd|jdt d)NSIf trying to add texts, the underlying docstore should support adding items, which does notz(Updates are not available in ScaNN, yet.) isinstancer*r ValueErrorNotImplementedError)r0texts embeddings metadatasidskwargss r__addz ScaNN.__addHsC$--6''+}}oY@ ""LMMr c x|jjt|}|j||f||d|S)alRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. r=r>)r(embed_documentslist _ScaNN__add)r0r;r=r>r?r<s r add_textszScaNN.add_textsWs;$^^33DK@ tzz%TycTVTTr c t|jtstd|jdt |\}}|j ||f||d|S)aRun more texts through the embeddings and add to the vectorstore. Args: text_embeddings: Iterable pairs of string and embedding to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. r6r7rB)r8r*rr9ziprE)r0text_embeddingsr=r>r?r;r<s radd_embeddingszScaNN.add_embeddingslse$$--6''+}}oY@   1ztzz%TycTVTTr c td)a3Delete by vector ID or other criteria. Args: ids: List of ids to delete. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. z*Deletions are not available in ScaNN, yet.)r:)r0r>r?s rdeletez ScaNN.deletes""NOOr c tj|gtj}|jr t |}|j j |||n|\}}g} t|dD]\} } | dk(r |j| } |jj| ttstd| d|s|jD cic]\} }| t|ts|gn|}} }t!fd|jDs| j#|d| f| j#|d| f|j%d}|k|j&t(j*t(j,fvrt.j0nt.j2}| Dcgc]\}}|||r||f} }}| d|Scc}} wcc}}w) aReturn docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: score_threshold: Optional, a floating point value between 0 to 1 to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and L2 distance in float for each. Lower score represents more similarity. dtypeNrrzCould not find document for id z, got c3^K|]$\}}jj||v&ywN)metadataget).0keyvaluedocs r z?ScaNN.similarity_search_with_score_by_vector..s)W*#us||'',5Ws*-score_threshold)rarrayfloat32r.rr)search_batched enumerater+r*searchr8r r9itemsrDallappendrSr,rMAX_INNER_PRODUCTJACCARDoperatorgele)r0r(kfilterfetch_kr?vectorindicesscoresdocsji_idrUrVrYcmprW similaritys ` r&similarity_search_with_score_by_vectorz,ScaNN.similarity_search_with_score_by_vectors09+RZZ8   v&F**33 AW gaj) 1DAqBw++A.C--&&s+Cc8, #B3%vcU!STT!'-lln"U 5$(?%UJW WWKKfQil 34 S&)A,/0! 1$!**%67  &))$668H8P8PQR [[ (,#Cz?3j!D BQx-"s  G=G"c j|jj|}|j||f||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with L2 distance in float. Lower score represents more similarity. rhri)r( embed_queryrs)r0queryrgrhrir?r(rms rsimilarity_search_with_scorez"ScaNN.similarity_search_with_scoresN*NN..u5 :t::        r c f|j||f||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. ru)rs) r0r(rgrhrir?docs_and_scoresrW_s rsimilarity_search_by_vectorz!ScaNN.similarity_search_by_vectorsN(F$EE       #22Q222 -c f|j||f||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. ru)rx) r0rwrgrhrir?rzrWr{s rsimilarity_searchzScaNN.similarity_searchsG(<$;; 1 #W 8> #22Q222r}c td}|jdtj} |jdd} t j |tj } |r t| } | |jj| | } n| tjk(r:|jj| ddjj} n9|jj| ddjj} g} |*|Dcgc]}ttj !}}t#|D]*\}}|r||ni}| j%t'||,t)t#|}t+|t+| k7r#t-t+|d t+| d t/t)t1|j3| }||| ||fd |i|Scc}w) Nr"r,r3rN dot_product squared_l2) page_contentrRz ids provided for z, documents. Each document should have an id.r2)rrSrEUCLIDEAN_DISTANCErrZr[rscann_ops_pybindcreate_searcherrbbuilderscore_brute_forcebuildstruuiduuid4r]rar dictlen ExceptionrrHvalues)clsr;r<r(r=r>r2r?r"r,r3rjr) documentsr{rotextrR index_to_idr*s r__fromz ScaNN.__from+sW%"JJ !1!D!D zz.$7 *BJJ7 v&F  #**::6<PE $4$F$FF**2261mL&&(UW**2261lK&&(UW  ;.343tzz|$4C4 ' MGAt'0y|bH   X4(K L M9S>* { s9~ -{#$$6s9~6FG44  $D[-?-?-A9)M$NO      &     5s$Hc T|j|}|j|||f||d|S)aNConstruct ScaNN wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the ScaNN database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import ScaNN from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() scann = ScaNN.from_texts(texts, embeddings) rB)rC _ScaNN__from)rr;r(r=r>r?r<s r from_textszScaNN.from_textsgsG4..u5 szz          r c |Dcgc]}|d }}|Dcgc]}|d }}|j|||f||d|Scc}wcc}w)aConstruct ScaNN wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the ScaNN database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import ScaNN from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) scann = ScaNN.from_embeddings(text_embedding_pairs, embeddings) rrrB)r) rrIr(r=r>r?tr;r<s rfrom_embeddingszScaNN.from_embeddingssm8 //!1//$34qad4 4szz          04s = Acrt|}|dj|z }|jdd|jj t |t |dj|z d5}tj|j|jf|dddy#1swYyxYw)zSave ScaNN index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to save index, docstore, and index_to_docstore_id to. {index_name}.scann index_nameTexist_okparents{index_name}.pklwbN) rformatmkdirr) serializeropenpickledumpr*r+)r0 folder_pathrpath scann_pathfs r save_localzScaNN.save_localsK 077:7NN $5 S_-$+22j2II4 P GTU KK(A(ABA F G G Gs 7-B--B6)allow_dangerous_deserializationc |s tdt|}|dj|z }|jddt d}|j j t|} t|dj|z d5} tj| \} } d d d |||   fi|S#1swYxYw) aLoad ScaNN index, docstore, and index_to_docstore_id from disk. Args: folder_path: folder path to load index, docstore, and index_to_docstore_id from. embedding: Embeddings to use when generating queries index_name: for saving with a specific index file name allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading a pickle file. Pickle files can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. aBThe de-serialization relies loading a pickle file. Pickle files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to enable deserialization. If you do this, make sure that you trust the source of the data. For example, if you are loading a file that you created, and know that no one else has modified the file, then this is safe to do. Do not set this to `True` if you are loading a file from an untrusted source (e.g., some random site on the internet.).rrTrr"rrbN) r9rrrrr load_searcherrrrload) rrr(rrr?rrr"r)rr*r+s r load_localzScaNN.load_locals./ "  K 077:7NN $5W%&&44S_E$+22j2II4 P TU  $ 9eX/CNvNN  s B<<Cc|j |jS|jtjk(r |jS|jtj k(r |j Std)a8 The 'correct' relevance function may differ depending on a few things, including: - the distance / similarity metric used by the VectorStore - the scale of your embeddings (OpenAI's are unit normed. Many others are not!) - embedding dimensionality - etc. zJUnknown distance strategy, must be cosine, max_inner_product, or euclidean)r-r,rrb%_max_inner_product_relevance_score_fnr_euclidean_relevance_score_fnr9)r0s r_select_relevance_score_fnz ScaNN._select_relevance_score_fnsr  + + 733 3  ! !%5%G%G G== =  # #'7'J'J J55 5  r c |jdd}|j}| td|j|f|||d|}|D cgc]\} } | || f} } } || D cgc]\} } | |k\r| | f} } } | Scc} } wcc} } w)z?Return docs and their similarity scores on a scale from 0 to 1.rYNzLnormalize_score_fn must be provided to ScaNN constructor to normalize scores)rgrhri)poprr9rx) r0rwrgrhrir?rYr1rzrWscoredocs_and_rel_scoresrrs r(_similarity_search_with_relevance_scoresz.ScaNN._similarity_search_with_relevance_scoress!**%6=!<<>  %9 <$;;      @O 1;eS$U+ ,    &(;##C0j!# # #" #s B*B)r(r r)rr*rr+zDict[int, str]r1z"Optional[Callable[[float], float]]r2boolr,rr3z Optional[str])NN) r; Iterable[str]r<zIterable[List[float]]r=Optional[List[dict]]r>Optional[List[str]]r?rreturn List[str]) r;rr=rr>rr?rrr) rIz!Iterable[Tuple[str, List[float]]]r=rr>rr?rrrrQ)r>rr?rrzOptional[bool])N) r( List[float]rgintrhOptional[Dict[str, Any]]rirr?rrList[Tuple[Document, float]]) rwrrgrrhrrirr?rrr) r(rrgrrhrrirr?rrList[Document]) rwrrgrrhrrirr?rrr)NNF)r;rr<zList[List[float]]r(r r=rr>rr2rr?rrr&) r;rr(r r=rr>rr?rrr&) rIzList[Tuple[str, List[float]]]r(r r=rr>rr?rrr&)r))rrrrrNone) rrr(r rrrrr?rrr&)rzCallable[[float], float])__name__ __module__ __qualname____doc__rrr4rErFrJrLrsrxr|r classmethodrrrrrrrr#r rr&r&!sM.BF".>.Q.Q&**** * - * ? **,*$*2+/#' N N* N( N ! N  N  N$+/#' UU(U! U  U  U0+/#' U:U(U! U  U  U8 P"+/ >> >) >  >  > &>F+/  )      &D+/ 33 3) 3  3  3 3@+/ 33 3) 3  3  3 32 +/#'"9 9 &9  9 ( 9 ! 9 9 9  9 9 v +/#' ! ! ! ( ! ! !  !  ! ! F +/#' $ 6$ $ ( $ ! $  $  $ $ LG$ " 3O 16 3O3O3O 3O *. 3O3O 3O3Oj8+/ "#"# "#) "#  "#  "# &"#r r&)r np.ndarrayrr)rr)# __future__rrdrrpathlibrtypingrrrrr r r numpyrlangchain_core.documentsr langchain_core.embeddingsr langchain_core.utilsrlangchain_core.vectorstoresr!langchain_community.docstore.baserr&langchain_community.docstore.in_memoryr&langchain_community.vectorstores.utilsrrr$r&r#r rrsM" GGG-0-3DCC !T#KT#r