htdZddlmZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZddlmZdd lmZed gZd Zd Zd Zej>ej@d jBZ"ejFej@djBZ$ejJjBZ&ddZ'ddZ(ddZ)ddZ*ddZ+GddeZ,y)z&Wrapper around TileDB vector database.) annotationsN)AnyDictIterableListMappingOptionalTuple)Document) Embeddings guard_import) VectorStore)maximal_marginal_relevance euclidean documentsvectorsuint64float32c.tdtdfS)z@Import tiledb-vector-search if available, otherwise raise error.tiledb.vector_searchtiledbr e/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/tiledb.pydependable_tiledb_importrs +,X rc(|tjS) Get the URI of the vector index.)VECTOR_INDEX_NAMEurigroups rget_vector_index_uri_from_groupr##s " # ' ''rc(|tjS)zGet the URI of the documents array from group. Args: group: TileDB group object. Returns: URI of the documents array. )DOCUMENTS_ARRAY_NAMEr r!s r"get_documents_array_uri_from_groupr&(s % & * **rc|dtS)r/)rr s rget_vector_index_urir*4sU!%& ''rc|dtS)z#Get the URI of the documents array.r()r%r)s rget_documents_array_urir,9sU!() **rc>eZdZdZdddddd d%dZed&dZdded  d'd Zddd d  d(d Z ddd d  d)dZ d* d+dZ d* d,dZ dd ddd d-dZ d. d/dZ d. d0dZeddd d1dZeddedddd d2dZ d3 d4dZ d5 d6dZeddeddddf d7d Zeddedddd d8d!Zeeddd" d9d#Zd:d$Zy);TileDBa2TileDB vector store. To use, you should have the ``tiledb-vector-search`` python package installed. Example: .. code-block:: python from langchain_community import TileDB embeddings = OpenAIEmbeddings() db = TileDB(embeddings, index_uri, metric) NF)vector_index_uridocs_array_uriconfig timestampallow_dangerous_deserializationc |s td||_|j|_||_||_||_tdtd} } | j|5| j|jd} |dk7r|n t| |_ |dk7r|n t| |_ | j| j|jd} | jj!d|_| j||_|j"dk(rD| j&j(d |j|j |j$d | |_nR|j"d k(rC| j,j.d |j|j |j$d | |_d d d y #1swYy xYw) aInitialize with necessary components. Args: allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading data using pickle. data can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. aTileDB relies on pickle for serialization and deserialization. This can be dangerous if the data is intercepted and/or modified by malicious actors prior to being de-serialized. If you are sure that the data is safe from modification, you can set allow_dangerous_deserialization=True to proceed. Loading of compromised data using pickle can result in execution of arbitrary code on your machine.rr ctx_or_configrr/ index_typeFLAT)r r2r3IVF_FLATNr) ValueError embedding embed_queryembedding_function index_urimetricr2r scope_ctxGroupr#r0r&r1closemetagetr9r3 flat_index FlatIndex vector_indexivf_flat_index IVFFlatIndex)selfr=r@rAr0r1r2r3r4kwargs tiledb_vsr index_groupr"s r__init__zTileDB.__init__Ls,/2 #"+"7"7"   / 0  "   F  3  ,,t~~s;K$r)!4[A  !"R'7 D      LL!6!6z/TileDB.process_index_results..s3"U''++C0E9s*-)ropenr1r3r2zip MAX_UINT64 MAX_FLOAT_32lenr<rFr strpickleloadsnparraytolistastypeuint8tobytesr[items isinstancelistallappendrD)rLidsscoresrUrVrWrdocs docs_arrayidxscoredocpickled_metadatar[r_r`ras @rprocess_index_resultszTileDB.process_index_resultss*h'[[   t{{! c6* 1JCaxEQJj Ul%:S/C{c#f+.!3 #B3%vcU!STT"wwz2 !s3v;q>/BCJ+!<<HH-4467>>rxxHPPR'/ #!'-lln"U 5$(?%UJ&,llnKKU 34 Z/05 16 /3Peu7Oe PPBQxQs G* G0G0rUrVfetch_kc d|vr|jd}nt}|jjt j t j |j tjgj tjfd||n|i|\}}|j|d|d|||S)a[Return docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: nprobe: Optional, number of partitions to check if using IVF_FLAT index score_threshold: Optional, a floating point value to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and distance in float for each. Lower score represents more similarity. rWrUrrvrwrVrUrW) pop MAX_FLOATrIqueryrkrlrnrr~) rLr=rUrVrrMrWdis r&similarity_search_with_score_by_vectorz-TileDB.similarity_search_with_score_by_vectors4  &$jj):;O'O&t  && HHbhhy)00<= > E Ebjj Q >aw  1 ))!QqT&A*  rc V|j|}|j|f|||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with Distance as float. Lower score represents more similarity. r)r?r)rLrrUrVrrMr=rxs rsimilarity_search_with_scorez#TileDB.similarity_search_with_scoresH,++E2 :t::       rc f|j|f|||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. r)r) rLr=rUrVrrMdocs_and_scoresr|_s rsimilarity_search_by_vectorz"TileDB.similarity_search_by_vectorsL(F$EE      #22Q222 -c f|j|f|||d|}|Dcgc]\}}| c}}Scc}}w)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. r)r) rLrrUrVrrMrr|rs rsimilarity_searchzTileDB.similarity_search7sG(<$;;  vw :@ #22Q222r?rUr lambda_multrVc d|vr|jd}nt}|jjt j t j |j tjgj tjfd||n|dzi|\}} |j| d|d|||n|dz|} | D cgc].\} } |jj| jgd0} } } tt j |gtj| ||}g}|D]}|j| ||Scc} } w)azReturn docs and their similarity scores selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. rWrUrrdtype)rUr)rrrIrrkrlrnrr~r=embed_documentsr\rru)rLr=rUrrrVrMrWrwindicesresultsr|rrS mmr_selectedrrs r2max_marginal_relevance_search_with_score_by_vectorz9TileDB.max_marginal_relevance_search_with_score_by_vectorPsa:  &$jj):;O'O1$++11 HHbhhy)00<= > E Ebjj Q gWq[   ,, !9gWq[+ - QX FLc1DNN * *C,<,<+= >q A  2 HHi[ 3 #   /A  " "71: . / s3Ec h|j|f||||d|}|D cgc]\}} | c} }Scc} }w)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)r) rLr=rUrrrVrMrr|rs r'max_marginal_relevance_search_by_vectorz.TileDB.max_marginal_relevance_search_by_vectorsO4R$QQ  #    #22Q222s .c X|j|}|j|f||||d|}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)r?r) rLrrUrrrVrMr=rxs rmax_marginal_relevance_searchz$TileDB.max_marginal_relevance_searchsK4++E2 ;t;;  #     rT) metadatasr2ctdtd}}|j|5 |j||j |d} t | j } t| j } |dk(r |jj| |||n$|dk(r|jj| |||| j| t|jd d td z ftj tj" } |j%| }|j'd tj dd}|g}|r3|j'dtj(d}|j+||j-|dd|}|j.j| || j| t0| j3dddy#|j$r} | d} ~ wwxYw#1swYyxYw)Nrrr6wr:)r dimensions vector_typer2r;)nameidr)rdomainrrZU1T)rrvarr[F)rsparseallows_duplicatesattrs)rrB group_create TileDBErrorrCr*r r,rGcreaterJaddrDimrerkrrDomainAttrroru ArraySchemaArrayr%rD)clsr@r9rrrr2rNrerrr"r0docs_uridimdom text_attrr metadata_attrschemas rrz TileDB.creates / 0  "   F  3/  ##I.LLC0E3EII> .uyy9HV#$$++() +! , z)((//() +! 0 II&->I ? **:>*hhryy)C --$C rxx~4 PIKE & 288QU V  ]+''"' (F LL  & 1 IIh%9I : KKM_/ / %%   / / s.HG'F"H'G=6G88G==HH r:r)rrvrAr9r2index_timestampc *|tvrtd|dtttdtd} } t j |j tj}|j|||jd|j|du| | j| 5|s tdt|}t|}|3|Dcgc](}ttj d t"dz *}}t j |j tj$}| j&j(d||||| d k7r| nd| d | | j+|d 5}|Lt j,t/|tj$ }t1t/|D]}|||< i}t j ||d <|st j2t/|gt4 }d }|D]B}t j6t9j:|tj< ||<|dz }D||d<|||<dddddd|d|||| d| Scc}w#1swY#xYw#1swY'xYw)NzUnsupported distance metric: z. Expected one of rrr)r@r9rrrr2r6z3embeddings must be provided to build a TileDB indexr)r9r@ input_vectors external_idsrr2rrrZr[)r=r@rAr2r) INDEX_METRICSr<rsrrkrlrnrrshaperrBr*r,rhrandomrandintrer ingestioningestrczerosrgrangeemptyobject frombufferridumpsro)rtextsrSr=r@rrvrAr9r2rrMrNrrr0rrrArdatarr[s r__fromz TileDB.__froms  &3F8<''+M':&;=  / 0  " ,33BJJ?  !$**1-%++t+    F  3$ ' !VWW3I> .y9H{GLM!s6>>!Z!^<=MM88C=// :L &I   & & %*+)3Ba3GT   Xs+ 'q'#%88CJbii#HL"3u:.,*+ Q,!xxV ($&HHc)n-=V$LMA$-+-=="LL2"((, a(Q  (5D$"&,# ''$ 'J      =N ' ''$ '$ 's81*J -I8A0J 8C#I=J 8J =J J  Jc tj|jtj}|jj ||dk7r|ydy)amDelete by vector ID or other criteria. Args: ids: List of ids to delete. timestamp: Optional timestamp to delete with. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. rN)rr3T)rkrlrnrrI delete_batch)rLrvr3rMrs rdeletez TileDB.deletees_xx}++BII6  &&%i1n ' SW ' rc td}|jjt|}|3|Dcgc](}t t j dtdz *}}tj|jtj} tjt|d} tt|D]-} tj|| tj| | </|j j#| | |dk7r|ndi} tj|| d<|stjt|gt$} d} |D]B}tj&t)j*|tj,| | <| dz } D| | d <|j/|j0d |dk7r|nd|j2 }| || <|j5|Scc}w) aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional ids of each text object. timestamp: Optional timestamp to write new texts with. kwargs: vectorstore specific parameters Returns: List of ids from adding the texts into the vectorstore. rNrrOr)rrr3rZr[rrY)rr=rrsrhrrrerkrlrnrrrgrrrI update_batchrrrirrorcr1r2rD)rLrrrvr3rMrrSrrrrrxrr[rys r add_textszTileDB.add_textszs(h'^^33DK@ ;CHIa3v~~aa89ICIxx}++BII6 ((C OC8s:' CA*Q-rzzBGAJ C &&%#,>it ' xxV  HHc)n%5VDMA% #%==h1Grxx#X a Q  -D [[    #,>it;; ! $( <  =Js-G6z/tmp/tiledb_arrayc  bg} |j|} |jd|| |||||||| d | S)aConstruct a TileDB index from raw documents. Args: texts: List of documents to index. embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. ids: Optional ids of each text object. metric: Metric to use for indexing. Defaults to "euclidean". index_uri: The URI to write the TileDB arrays index_type: Optional, Vector index type ("FLAT", IVF_FLAT") config: Optional, TileDB config index_timestamp: Optional, timestamp to write new texts with. Example: .. code-block:: python from langchain_community import TileDB from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() index = TileDB.from_texts(texts, embeddings) rrSr=rrvrAr@r9r2rr)r _TileDB__from) rrr=rrvrAr@r9r2rrMrSs r from_textszTileDB.from_textssZF ..u5 szz  !!+    rc |D cgc]} | d } } |D cgc]} | d } } |jd| | |||||||| d | Scc} wcc} w)aConstruct TileDB index from embeddings. Args: text_embeddings: List of tuples of (text, embedding) embedding: Embedding function to use. index_uri: The URI to write the TileDB arrays metadatas: List of metadata dictionaries to associate with documents. metric: Optional, Metric to use for indexing. Defaults to "euclidean". index_type: Optional, Vector index type ("FLAT", IVF_FLAT") config: Optional, TileDB config index_timestamp: Optional, timestamp to write new texts with. Example: .. code-block:: python from langchain_community import TileDB from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) db = TileDB.from_embeddings(text_embedding_pairs, embeddings) rrrr)r)rtext_embeddingsr=r@rrvrAr9r2rrMtrrSs rfrom_embeddingszTileDB.from_embeddingss{J //!1//$34qad4 4szz  !!+    04s A A)rAr2r3c |d|||||d|S)a}Load a TileDB index from a URI. Args: index_uri: The URI of the TileDB vector index. embedding: Embeddings to use when generating queries. metric: Optional, Metric to use for indexing. Defaults to "euclidean". config: Optional, TileDB config timestamp: Optional, timestamp to use for opening the arrays. )r=r@rAr2r3rr)rr@r=rAr2r3rMs rloadz TileDB.loads/(      rc F|jjdi||_y)Nr)rIconsolidate_updates)rLrMs rrzTileDB.consolidate_updates7s AD--AAKFKr)r=r r@rhrArhr0rhr1rhr2Optional[Mapping[str, Any]]r3rr4boolrMr)returnzOptional[Embeddings]) rvz List[int]rw List[float]rUintrVOptional[Dict[str, Any]]rWfloatrList[Tuple[Document, float]]) r=rrUrrVrrrrMrrr) rrhrUrrVrrrrMrrr)rTNr) r=rrUrrVrrrrMrrList[Document]) rrhrUrrVrrrrMrrr)r=rrUrrrrrrVrrMrrr)rTrrN)r=rrUrrrrrrVrrMrrr)rrhrUrrrrrrVrrMrrr)r@rhr9rhrrrznp.dtyperrr2rrNone)r List[str]rSzList[List[float]]r=r r@rhrOptional[List[dict]]rvOptional[List[str]]rArhr9rhr2rrrrMrrr.)Nr)rvrr3rrMrrzOptional[bool])NNr) rz Iterable[str]rrrvrr3rrMrrr)rrr=r rrrvrrArhr@rhr9rhr2rrrrMrrr.)rzList[Tuple[str, List[float]]]r=r r@rhrrrvrrArhr9rhr2rrrrMrrr.)r@rhr=r rArhr2rr3rrMrrr.)rMrrr)__name__ __module__ __qualname____doc__rPpropertyrSrr~rrrrrrr classmethodrDEFAULT_METRICrrrrrrrrrrr.r.>s- &!# .205HHH H  HH,HH*.HHT+/!*7 77  7 ) 77 &7z+/ % %   % ) %  % %  &% V+/    )    &F+/ 33 3) 3  3  3 3@+/ 33 3) 3  3  3 3: +/99  9  9  9)99 &9| +/ "3"3 "3 "3  "3 ) "3"3 "3N +/ ## # #  # ) ## #J.2=== =  ==,= ==~+/#'$ .2 N N &N  N  N (N !N N N ,N N N  N N bAB&:=MP 0+/#' 55(5! 5  5  5 5n +/#'$, .2 0 0 0 ( 0 ! 0  0 0 0 ,0 0 0  0 0 d+/#'$ .2 3 63 3  3 ( 3 !3 3 3 ,3 3 3  3 3 j %.2     ,       8Lrr.)rr)r"rrrh)r rhrrh)-r __future__rrirsystypingrrrrrr r numpyrklangchain_core.documentsr langchain_core.embeddingsr langchain_core.utilsrlangchain_core.vectorstoresr&langchain_community.vectorstores.utilsr frozensetrrr%riinformaxrefinforf float_inforrr#r&r*r,r.rrrr s," FFF-0-3M;-( " RXXhbhhx( ) - - rxx+,00 NN   ( +( + z L[z Lr