h,ddlmZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%m&Z&ejNe(Z)ddd Z*dd Z+GddeZ,y)) annotationsN)Path) AnyCallableDictIterableListOptionalSequenceSizedTupleUnion)Document) Embeddings)run_in_executor) VectorStore) AddableMixinDocstore)InMemoryDocstore)DistanceStrategymaximal_marginal_relevancec|0dtjvrttjd} |rddlm}|Sddl} |S#t $r t dwxYw)aM Import faiss if available, otherwise raise error. If FAISS_NO_AVX2 environment variable is set, it will be considered to load FAISS with no AVX2 optimization. Args: no_avx2: Load FAISS strictly with no AVX2 optimization so that the vectorstore is portable and compatible with other devices. N FAISS_NO_AVX2r) swigfaisszCould not import faiss python package. Please install it with `pip install faiss-gpu` (for CUDA supported GPU) or `pip install faiss-cpu` (depending on Python version).)osenvironboolgetenvfaissr ImportError)no_avx2rs d/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/faiss.pydependable_faiss_importr#'si?bjj8ryy12   0 L  L   H   sAAAct|trUt|trEt|t|k7r.t|d|d|dt|d|dt| y)Nz and z% expected to be equal length but len(z)=z and len() isinstancer len ValueError)xyx_namey_names r"_len_check_if_sizedr,Bsk!U 1e 4Q3q69IheF8$("SVHIfXRAx A   c4eZdZdZddej f d1dZed2dZd3dZ d3dZ d4dZ d4d Z d5 d6d Z d5 d7d Z d5 d7d Z d5 d8d Z d9 d:dZ d9 d:dZ d9 d;dZ d9 d;dZ d9 ddZ d9 d>dZddddd d?dZddddd d?dZ d@ dAdZ d@ dAdZ d@ dBdZ d@ dBdZdCdDd ZdEd!Z e!dddej f dFd"Z"e! d5 dGd#Z#e! d5 dHd$Z$e! d5 dId%Z%e! d5 dId&Z&dJdKd'Z'e! dJdd( dLd)Z(dMd*Z)e!dd( dNd+Z*dOd,Z+ d9 d;d-Z, d9 d;d.Z-e. dPd/Z/dQd0Z0y)RFAISSuFAISS vector store integration. See [The FAISS Library](https://arxiv.org/pdf/2401.08281) paper. Setup: Install ``langchain_community`` and ``faiss-cpu`` python packages. .. code-block:: bash pip install -qU langchain_community faiss-cpu Key init args — indexing params: embedding_function: Embeddings Embedding function to use. Key init args — client params: index: Any FAISS index to use. docstore: Docstore Docstore to use. index_to_docstore_id: Dict[int, str] Mapping of index to docstore id. Instantiate: .. code-block:: python import faiss from langchain_community.vectorstores import FAISS from langchain_community.docstore.in_memory import InMemoryDocstore from langchain_openai import OpenAIEmbeddings index = faiss.IndexFlatL2(len(OpenAIEmbeddings().embed_query("hello world"))) vector_store = FAISS( embedding_function=OpenAIEmbeddings(), index=index, docstore= InMemoryDocstore(), index_to_docstore_id={} ) Add Documents: .. code-block:: python from langchain_core.documents import Document document_1 = Document(page_content="foo", metadata={"baz": "bar"}) document_2 = Document(page_content="thud", metadata={"bar": "baz"}) document_3 = Document(page_content="i will be deleted :(") documents = [document_1, document_2, document_3] ids = ["1", "2", "3"] vector_store.add_documents(documents=documents, ids=ids) Delete Documents: .. code-block:: python vector_store.delete(ids=["3"]) Search: .. code-block:: python results = vector_store.similarity_search(query="thud",k=1) for doc in results: print(f"* {doc.page_content} [{doc.metadata}]") .. code-block:: python * thud [{'bar': 'baz'}] Search with filter: .. code-block:: python results = vector_store.similarity_search(query="thud",k=1,filter={"bar": "baz"}) for doc in results: print(f"* {doc.page_content} [{doc.metadata}]") .. code-block:: python * thud [{'bar': 'baz'}] Search with score: .. code-block:: python results = vector_store.similarity_search_with_score(query="qux",k=1) for doc, score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") .. code-block:: python * [SIM=0.335304] foo [{'baz': 'bar'}] Async: .. code-block:: python # add documents # await vector_store.aadd_documents(documents=documents, ids=ids) # delete documents # await vector_store.adelete(ids=["3"]) # search # results = vector_store.asimilarity_search(query="thud",k=1) # search with score results = await vector_store.asimilarity_search_with_score(query="qux",k=1) for doc,score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") .. code-block:: python * [SIM=0.335304] foo [{'baz': 'bar'}] Use as Retriever: .. code-block:: python retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={"k": 1, "fetch_k": 2, "lambda_mult": 0.5}, ) retriever.invoke("thud") .. code-block:: python [Document(metadata={'bar': 'baz'}, page_content='thud')] NFcJt|tstjd||_||_||_||_||_||_ ||_ |jtjk7r0|jr#tjd|jyyy)z%Initialize with necessary components.t`embedding_function` is expected to be an Embeddings object, support for passing in a function will soon be removed.z2Normalizing L2 is not applicable for metric type: N)r%rloggerwarningembedding_functionindexdocstoreindex_to_docstore_iddistance_strategyoverride_relevance_score_fn _normalize_L2rEUCLIDEAN_DISTANCEwarningswarn)selfr4r5r6r7relevance_score_fn normalize_L2r8s r"__init__zFAISS.__init__s,j9 NNB #5   $8!!2+=()  " "&6&I&I I"" MM $ 6 679 # Jr-cRt|jtr |jSdSN)r%r4rr>s r" embeddingszFAISS.embeddingss.$11:>  # #  r-ct|jtr|jj|S|Dcgc]}|j|c}Scc}wrC)r%r4rembed_documents)r>textstexts r"_embed_documentszFAISS._embed_documentssH d--z :**::5A A>CDdD++D1D DDsAcKt|jtr#|jj|d{St d7wNr1)r%r4raembed_documents Exception)r>rHs r"_aembed_documentszFAISS._aembed_documentssK d--z :00AA%HH H B  I9A A A ct|jtr|jj|S|j|SrC)r%r4r embed_queryr>rIs r" _embed_queryzFAISS._embed_querys: d--z :**66t< <**40 0r-cKt|jtr#|jj|d{St d7wrL)r%r4r aembed_queryrNrSs r" _aembed_queryzFAISS._aembed_querysK d--z :00==dCC CB DrPc t}t|jtst d|jdt ||dd|xs+|Dcgc]}t tj!c}}t ||dd|xs d|D}t|||D cgc]\}} } t|| | } } }} t | |dd |r+t|tt|k7r t d tj|tj } |j r|j#| |j$j'| |jj't|| D cic]\}} ||  c} }t|j(}t+|Dcic] \}}||z| }}}|j(j-||Scc}wcc} } }wcc} }wcc}}w) NzSIf trying to add texts, the underlying docstore should support adding items, which z does notrH metadatasidsc3 K|]}iywrC).0_s r" zFAISS.__add..+s"5!2"5s )id page_contentmetadata documentsrEz$Duplicate ids found in the ids list.dtype)r#r%r6rr'r,struuiduuid4ziprr&setnparrayfloat32r:r@r5addr7 enumerateupdate)r>rHrErYrZrr^ _metadatasid_tmrcvectordoc starting_lenj index_to_ids r"__addz FAISS.__adds()$--6''+}}oY@  E9g{C77Ac$**,'7E375"5u"5 !eZ8  Q !a 8  Iz; M 3s8s3s8},CD D*BJJ7      v & v CY4GHS38HI4445 ;DS>JC|a',J J !!((5 18 IJs$G('G- G4 8G:c dt|}|j|}|j||||S)alRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. rYrZ)listrJ _FAISS__addr>rHrYrZkwargsrEs r" add_textszFAISS.add_textsBs4"U **51 zz%yczJJr-cKt|}|j|d{}|j||||S7w)aRun more texts through the embeddings and add to the vectorstore asynchronously. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. Nr|)r}rOr~rs r" aadd_textszFAISS.aadd_textsWs@$U 11%88 zz%yczJJ9s ><>c Bt|\}}|j||||S)aAdd the given texts and embeddings to the vectorstore. Args: text_embeddings: Iterable pairs of string and embedding to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. r|)rir~)r>text_embeddingsrYrZrrHrEs r"add_embeddingszFAISS.add_embeddingsms*& 1zzz%yczJJr-c |t}tj|gtj}|jr|j ||j j|||n|\}} g} ||j|} t| dD]\} } | dk(r |j| }|jj|}t|tstd|d||- |jsl| j!||d| f| j!||d| f|j#d}|k|j$t&j(t&j*fvrt,j.nt,j0}| Dcgc]\}}|||r||f} }}| d|Scc}}w)aReturn docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Union[Callable, Dict[str, Any]]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: score_threshold: Optional, a floating point value between 0 to 1 to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and L2 distance in float for each. Lower score represents more similarity. rdNrCould not find document for id , got score_threshold)r#rkrlrmr:r@r5search_create_filter_funcror7r6r%rr'rbappendgetr8rMAX_INNER_PRODUCTJACCARDoperatorgele)r> embeddingkfilterfetch_krrruscoresindicesdocs filter_funcrxi_idrvrcmp similaritys r"&similarity_search_with_score_by_vectorz,FAISS.similarity_search_with_score_by_vectors4()9+RZZ8      v &**++FAWU  226:Kgaj) 1DAqBw++A.C--&&s+Cc8, #B3%vcU!STT!s||,KKfQil 34 S&)A,/0 1!**%67  &))$668H8P8PQR [[ (,#Cz?3j!D BQx sF8cVKtd|j|f|||d|d{S7w)aReturn docs most similar to query asynchronously. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: score_threshold: Optional, a floating point value between 0 to 1 to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and L2 distance in float for each. Lower score represents more similarity. Nrrr)rr)r>rrrrrs r"'asimilarity_search_with_score_by_vectorz-FAISS.asimilarity_search_with_score_by_vectorsH:%   7 7        )')c V|j|}|j||f||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with L2 distance in float. Lower score represents more similarity. rr)rTrr>queryrrrrrrs r"similarity_search_with_scorez"FAISS.similarity_search_with_scoresJ0%%e, :t::        r-cK|j|d{}|j||f||d|d{}|S7%7w)aReturn docs most similar to query asynchronously. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with L2 distance in float. Lower score represents more similarity. Nr)rWrrs r"asimilarity_search_with_scorez#FAISS.asimilarity_search_with_score sa0,,U33 ATAA         4 sA=A?AAc f|j||f||d|}|Dcgc]\}}| c}}Scc}}w)aYReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. r)r r>rrrrrdocs_and_scoresrvr^s r"similarity_search_by_vectorz!FAISS.similarity_search_by_vector/sN.F$EE       #22Q222 -cK|j||f||d|d{}|Dcgc]\}}| c}}S7cc}}ww)ahReturn docs most similar to embedding vector asynchronously. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. rN)rrs r"asimilarity_search_by_vectorz"FAISS.asimilarity_search_by_vectorOs^.!M L L  !  !  !  #22Q22 3?7 ? 9??c f|j||f||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. r)r r>rrrrrrrvr^s r"similarity_searchzFAISS.similarity_searchosG(<$;; 1 #W 8> #22Q222rcK|j||f||d|d{}|Dcgc]\}}| c}}S7cc}}ww)aReturn docs most similar to query asynchronously. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. rN)rrs r"asimilarity_searchzFAISS.asimilarity_searchsX(!C B B 1! #W! 8>!  #22Q22 3r?rr lambda_multrc|jjtj|gtj||n|dz\}}||j |}g} |dD]w} | dk(r |j | } |jj| } t| tstd| d| || jsg| j| ytj| g}|dD cgc],} | dk7s |jjt| .} } ttj|gtj| ||}g}|D]x} |d| dk(r|j |d| } |jj| } t| tstd| d| |j| |d| fz|Scc} w)azReturn docs and their similarity scores selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. rdrrrr)rr)r5rrkrlrmrr7r6r%rr'rbr reconstructintr)r>rrrrrrrrfiltered_indicesrrrvrE mmr_selectedrs r"2max_marginal_relevance_search_with_score_by_vectorz8FAISS.max_marginal_relevance_search_with_score_by_vectors8**++ HHi[ 3~G7Q;   226:K! QZ /7//2mm**3/!#x0$'Fse6RUQV%WXXs||,$++A. /hh 012G>EajTAQSGdjj,,SV4T T1 HHi[ 3 #   8Aqz!}"++GAJqM:C--&&s+Cc8, #B3%vcU!STT  " "C1#6 7 8'Us < G#&G#c VKtd|j|||||d{S7w)aReturn docs and their similarity scores selected using the maximal marginal relevance asynchronously. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. Nr)rr)r>rrrrrs r"3amax_marginal_relevance_search_with_score_by_vectorz9FAISS.amax_marginal_relevance_search_with_score_by_vectors::%   C C #    rc d|j|||||}|D cgc]\}} | c} }Scc} }w)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)r r>rrrrrrrrvr^s r"'max_marginal_relevance_search_by_vectorz-FAISS.max_marginal_relevance_search_by_vector s?4QQ GVR #22Q222s ,cK|j|||||d{}|D cgc]\}} | c} }S7cc} }ww)a(Return docs selected using the maximal marginal relevance asynchronously. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rN)rrs r"(amax_marginal_relevance_search_by_vectorz.FAISS.amax_marginal_relevance_search_by_vector+sS6JJQ[QWK   #22Q22  3s>6 > 8>>c X|j|}|j|f||||d|}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)rTr r>rrrrrrrrs r"max_marginal_relevance_searchz#FAISS.max_marginal_relevance_searchLsK4%%e, ;t;;  #     r-cK|j|d{}|j|f||||d|d{}|S7&7w)a+Return docs selected using the maximal marginal relevance asynchronously. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. Nr)rWrrs r"amax_marginal_relevance_searchz$FAISS.amax_marginal_relevance_searchqsb4,,U33 BTBB  #      4 sA>AAAAc | tdt|j|jj }|rtd||jj Dcic]\}}|| }}}|Dchc]}|| }}|j jtj|tj|jj|t|jj Dcgc] \}}||vr| } }}t| Dcic]\}}|| c}}|_ycc}}wcc}wcc}}wcc}}w)zDelete by ID. These are the IDs in the vectorstore. Args: ids: List of ids to delete. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. zNo ids provided to delete.zESome specified ids do not exist in the current store. Ids not found: rdT)r'rj differencer7valuesitemsr5 remove_idsrkfromiterint64r6deletesortedro) r>rZr missing_idsidxrrreversed_indexindex_to_deleter remaining_idss r"rz FAISS.deletesE ;9: :#h))$*C*C*J*J*LM W-!  483L3L3R3R3TUxsC#s(UU:=>3>#.>> bkk/JK S!!!:!:!@!@!BC 3'   ;DM:R$S3QV$S!V>  %Ts- E E E/ Ec t|jts tdt |j }|j j|j g}|j jD]R\}}|jj|}t|ts td|j||z||fT|jj|Dcic] \}}}|| c}}}|D cic] \} }}| | } }} }|j j| ycc}}}wcc}}} w)zMerge another FAISS object with the current one. Add the target FAISS to the current one. Args: target: FAISS object you wish to merge into the current one Returns: None. z'Cannot merge with this type of docstorezDocument should be returnedN)r%r6rr'r&r7r5 merge_fromrrrrrnrp) r>targetrw full_infor target_idrvr^rr5rys r"rzFAISS.merge_froms$--6FG G4445  fll+ "77==? ALAy//((3Cc8, !>??   lQ. 3? @  A yAA 338AB7@AAmeS!uczA A !!((5BAs ,D7 D>c Vt} |tjk(r| jt |d} n| j t |d} |j dt} |j di} ||| | | f||d|} | j||||| S)Nrr6r7)r@r8r|) r#rr IndexFlatIPr& IndexFlatL2poprr~)clsrHrErrYrZr@r8rrr5r6r7vecstores r"__fromz FAISS.__froms()  0 B B B%%c*Q-&89E%%c*Q-&89E::j*:*<=%zz*@"E     &/    ujI3Gr-c T|j|}|j|||f||d|S)aOConstruct FAISS wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the FAISS database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import FAISS from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() faiss = FAISS.from_texts(texts, embeddings) r|)rG _FAISS__fromrrHrrYrZrrEs r" from_textszFAISS.from_textssG6..u5 szz          r-cpK|j|d{}|j|||f||d|S7w)aeConstruct FAISS wrapper from raw documents asynchronously. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the FAISS database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import FAISS from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() faiss = await FAISS.afrom_texts(texts, embeddings) Nr|)rMrrs r" afrom_textszFAISS.afrom_textssS6%55e<< szz          =s 646c lt|\}}|jt|t||f||d|S)aConstruct FAISS wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the FAISS database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import FAISS from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = zip(texts, text_embeddings) faiss = FAISS.from_embeddings(text_embedding_pairs, embeddings) r|)rirr})rrrrYrZrrHrEs r"from_embeddingszFAISS.from_embeddingsBsO: 1zszz K          r-c8K|j||f||d|Sw)z:Construct FAISS wrapper from raw documents asynchronously.r|)r)rrrrYrZrs r"afrom_embeddingszFAISS.afrom_embeddingsis:#s""         scPt|}|jddt}|j|jt ||dz t ||dz d5}tj|j|jf|dddy#1swYyxYw)aSave FAISS index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to save index, docstore, and index_to_docstore_id to. index_name: for saving with a specific index file name T)exist_okparents.faiss.pklwbN) rmkdirr# write_indexr5rfopenpickledumpr6r7)r> folder_path index_namepathrfs r" save_localzFAISS.save_local{sK  D$ /() $**c$J|j |jS|jtjk(r |jS|jtj k(r |j S|jtjk(r |jStd)a8 The 'correct' relevance function may differ depending on a few things, including: - the distance / similarity metric used by the VectorStore - the scale of your embeddings (OpenAI's are unit normed. Many others are not!) - embedding dimensionality - etc. zJUnknown distance strategy, must be cosine, max_inner_product, or euclidean) r9r8rr%_max_inner_product_relevance_score_fnr;_euclidean_relevance_score_fnCOSINE_cosine_relevance_score_fnr'rDs r"_select_relevance_score_fnz FAISS._select_relevance_score_fns  + + 733 3  ! !%5%G%G G== =  # #'7'J'J J55 5  # #'7'>'> >22 2  r-c |j}| td|j|f|||d|}|D cgc]\}} ||| f} }} | Scc} }w)?Return docs and their similarity scores on a scale from 0 to 1.Lrelevance_score_fn must be provided to FAISS constructor to normalize scoresr)rr'r r>rrrrrr?rrvscoredocs_and_rel_scoress r"(_similarity_search_with_relevance_scoresz.FAISS._similarity_search_with_relevance_scoress"<<>  %9 <$;;      @O 1;eS$U+ ,  #" sAcK|j}| td|j|f|||d|d{}|D cgc]\}} ||| f} }} | S7"cc} }ww)rNrr)rr'rrs r")_asimilarity_search_with_relevance_scoresz/FAISS._asimilarity_search_with_relevance_scoress"<<>  %9 !C B B !  !  !  @O 1;eS$U+ ,  #"  s!8A%A A%AA%A%c t|r|St|tstdt |ddlm}m}m}m }m }m }||||||d}ddd}||z tt gdz} d |D])} | s| jd s| | vstd |  d fd d fd  |S)a Create a filter function based on the provided filter. Args: filter: A callable or a dictionary representing the filter conditions for documents. Returns: A function that takes Document's metadata and returns True if it satisfies the filter conditions, otherwise False. Raises: ValueError: If the filter is invalid or contains unsupported operators. z5filter must be a dict of metadata or a callable, not r)eqrgtrltne)z$eqz$neqz$gtz$ltz$gtez$ltec ||vSrCr\abs r"z+FAISS._create_filter_func..`s Qr-c ||vSrCr\r)s r"r,z+FAISS._create_filter_func..as !r-)z$inz$nin)$and$or$not $&filter contains unsupported operator: c8ttrKgjD]-\}}|vrtd|j ||f/dfd }|Stt r%t kDrtfdSfdSfdS)a Creates a filter function based on field and condition. Args: field: The document field to filter on condition: Filter condition (dict for operators, list for in, or direct value for equality) Returns: A filter function that takes a document and returns boolean r3cP|jtfdDS)aW Evaluates a document against a set of predefined operators and their values. This function applies multiple comparison/sequence operators to a specific field value from the document. All conditions must be satisfied for the function to return True. Args: doc (Dict[str, Any]): The document to evaluate, containing key-value pairs where keys are field names and values are the field values. The document must contain the field being filtered. Returns: bool: True if the document's field value satisfies all operator conditions, False otherwise. c36K|]\}}||ywrCr\)r]opvalue doc_values r"r_zYFAISS._create_filter_func..filter_func_cond..filter_fn..sO Er)U3Os)rall)rvr9field operatorss @r" filter_fnzFFAISS._create_filter_func..filter_func_cond..filter_fns#$!$IOYOOOr-c*|jvSrCr)rv condition_setr;s r"r,zEFAISS._create_filter_func..filter_func_cond..sswwu~'Fr-c*|jvSrCr?rv conditionr;s r"r,zEFAISS._create_filter_func..filter_func_cond..s3775>Y#>r-c,|jk(SrCr?rBs r"r,zEFAISS._create_filter_func..filter_func_cond..sswwu~:r-)rvDict[str, Any]returnr)r%dictrr'rr}r& frozenset) r;rCr7r8r=r@r< OPERATIONSSET_CONVERT_THRESHOLDs `` @@r"filter_func_condz3FAISS._create_filter_func..filter_func_condls)T* !*!2>IB+(+QRTQU)VWW$$jne%<=> P*! )T*y>$99$-i$8MFF>>: :r-c8d|vr|dDcgc] }| c}fdSd|vr|dDcgc] }| c}fdSd|vr|dfdS|jDcgc]\}}||c}}fdScc}wcc}wcc}}w)a Creates a filter function that evaluates documents against specified filter conditions. This function processes a dictionary of filter conditions and returns a callable that can evaluate documents against these conditions. It supports logical operators ($and, $or, $not) and field-level filtering. Args: filter (Dict[str, Any]): A dictionary containing filter conditions. Can include: - Logical operators ($and, $or, $not) with lists of sub-filters - Field-level conditions with comparison or sequence operators - Direct field-value mappings for equality comparison Returns: Callable[[Dict[str, Any]], bool]: A function that takes a document (as a dictionary) and returns True if the document matches all filter conditions, False otherwise. r.c.tfdDS)Nc3.K|] }|ywrCr\r]rrvs r"r_zSFAISS._create_filter_func..filter_func....&?!qv&?r:rvfilterss`r"r,z@FAISS._create_filter_func..filter_func..3&?w&?#?r-r/c.tfdDS)Nc3.K|] }|ywrCr\rOs r"r_zSFAISS._create_filter_func..filter_func....rPrQ)anyrSs`r"r,z@FAISS._create_filter_func..filter_func..rUr-r0c| SrCr\)rvconds r"r,z@FAISS._create_filter_func..filter_func..s tCy=r-c.tfdDS)Nc3.K|] }|ywrCr\)r]rCrvs r"r_zSFAISS._create_filter_func..filter_func....s"Ni9S>"NrQrR)rv conditionss`r"r,z@FAISS._create_filter_func..filter_func..ss"N:"NNr-)r) r sub_filterr;rCrZr]rTrrKs @@@r"rz.FAISS._create_filter_func..filter_funcs*EKF^Tz;z2T??EKE]Sz;z2S??"6&>200)/ $E9! 2JO NUTsB B0B)r;rfrCz%Union[Dict[str, Any], List[Any], Any]rF Callable[[Dict[str, Any]], bool])rrErFr_)callabler%rGr'typerr$rr%rr&r'rHr} startswith)rr$rr%rr&r'COMPARISON_OPERATORSSEQUENCE_OPERATORSVALID_OPERATORSr7rIrJrrKs @@@@r"rzFAISS._create_filter_func;s$ F M&$'GV ~V  43  '+ *,>> #D$47N$NO " PBbmmC(R-F #I"!NOO P2 ;2 ;#H2 ; -2 ;h% ON6""r-c|Dcgc]}|jj|}}|Dcgc]}t|ts|c}Scc}wcc}wrC)r6rr%r)r>rZrrrrvs r" get_by_idszFAISS.get_by_idssF589c $$S)99#Az#x'@AA:As"A AA)r4z/Union[Callable[[str], List[float]], Embeddings]r5rr6rr7zDict[int, str]r?z"Optional[Callable[[float], float]]r@rr8r)rFzOptional[Embeddings])rH List[str]rFList[List[float]])rIrfrF List[float])NN) rH Iterable[str]rEzIterable[List[float]]rYOptional[Iterable[dict]]rZOptional[List[str]]rFrh) rHrkrYOptional[List[dict]]rZrmrrrFrh) r!Iterable[Tuple[str, List[float]]]rYrnrZrmrrrFrh)rNr) rrjrrr)Optional[Union[Callable, Dict[str, Any]]]rrrrrFList[Tuple[Document, float]]) rrfrrrrprrrrrFrq) rrjrrrzOptional[Dict[str, Any]]rrrrrFList[Document]) rrjrrrrprrrrrFrr) rrfrrrrprrrrrFrr) rrjrrrrrfloatrrprFrq)rrrN)rrjrrrrrrsrrprrrFrr)rrfrrrrrrsrrprrrFrrrC)rZrmrrrFOptional[bool])rr/rFNone)rHrkrErirrrYrlrZrmr@rr8rrrrFr/) rHrhrrrYrnrZrmrrrFr/) rHz list[str]rrrYrnrZrmrrrFr/) rrorrrYrlrZrmrrrFr/)r5)rrfrrfrFru) rrfrErrrfrrrrrFr/)rFbytes) rrvrErrrrrrFr/)rFzCallable[[float], float])rrprFr_)rZz Sequence[str]rFzlist[Document])1__name__ __module__ __qualname____doc__rr;rApropertyrErJrOrTrWr~rrrrrrrrrrrrrrrrrrr classmethodrrrrrrr rrrr r" staticmethodrrgr\r-r"r/r/Ksd }PBF".>.Q.Q! !  !!-!?!!,!F  E 1 /3#' ((*(, ( ! (  (Z+/#' KK(K! K  K  K0+/#' KK(K! K  K  K2+/#' K:K(K! K  K  K2<@ ?? ?: ?  ?  ? &?H<@ % %  % : %  %  %  &% T<@    :      & J<@    :      & J+/ 33 3) 3  3  3 3F<@ 33 3: 3  3  3 3F<@ 33 3: 3  3  3 38<@ 33 3: 3  3  3 3: <@BB  B  B  B:B &BP <@% %   %  %  % :%  &% T <@ 33 3 3  3 : 33 3D <@ 33 3 3  3 : 33 3H <@ ## # #  # : ## #P <@ ## # #  # : ## #J D6@ /3#'".>.Q.Q&  ,  ! , > +/#' " " " ( " ! "  "  " " H +/#' " " " ( " ! "  "  " " H /3#' $ :$ $ , $ ! $  $  $ $ L /3#'  :  ,  !       "G& " 1P 16 1P1P1P 1P *. 1P1P 1P1PfT 16 PPP *. P  P PP><<@ ## #: #  #  # &#@<@ ## #: #  #  # &#:K#9K# )K#K#ZBr-r/rC)r!rtrFr) r(rr)rr*rfr+rfrFru)- __future__rloggingrrrrgr<pathlibrtypingrrrrr r r r r rnumpyrklangchain_core.documentsrlangchain_core.embeddingsrlangchain_core.runnables.configrlangchain_core.vectorstoresr!langchain_community.docstore.baserr&langchain_community.docstore.in_memoryr&langchain_community.vectorstores.utilsrr getLoggerrwr2r#r,r/r\r-r"rsv"    -0;3DC   8 $6 @BK@Br-