hUddlmZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZdd l m!Z!e rddl"Z"ed Z#e$ee#Z%d e&d <ed Z'e$ee'Z(de&d<dZ)dZ*dZ+e+dzZ,gdZ-gdZ.ddigZ/ej`e1Z2d/dZ3d0dZ4d1dZ5d2d3dZ6edddGdd eZ7 d4 d5d!Z8 d6 d7d"Z9 d8d#Z:d$gdf d9d%Z;d:d&Zd=d)Z? d>d*Z@d?d+ZA d@ dAd,ZBdBd-ZCdCd.ZDy)D) annotationsN)deepcopy) TYPE_CHECKINGAnyCallableDictIterableListLiteralOptionalSizedTupleTypeUnionget_args) deprecated)Document) Embeddings) VectorStore)maximal_marginal_relevance)L2IPzList[DISTANCE_METRICS]AVAILABLE_DISTANCE_METRICS) TileDBDense TileDBSparse FaissFlat FaissIVFFlatFlinngz List[ENGINES]AVAILABLE_ENGINES langchain ) _distanceidcontent)r$r&blobzMissing propertyct|trUt|trEt|t|k7r.t|d|d|dt|d|dt| y)z Check that sizes of two variables are the same Args: x: Variable to compare y: Variable to compare x_name: Name for variable x y_name: Name for variable y z and z% expected to be equal length but len(z)=z and len(N) isinstancer len ValueError)xyx_namey_names c/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/vdms.py_len_check_if_sizedr1?sm!U 1e 4Q3q69IheF8$("SVHIfXRAx A   cJt|Dcgc]\}}| c}}Scc}}wN)_results_to_docs_and_scores)resultsdoc_s r0_results_to_docsr9Qs9'B CFCC CC Cs cg} |d\}}t|dkDrd|dvrd|ddvr|ddd}|D]r}t|dd}|d}tD] }||vs||= |jD cic]\} } | tvr| | } } } |j t || |ft|Scc} } w#t$r#} tjd| Yd} ~ |Sd} ~ wwxYw) NrFindDescriptorentitiesr$ r&) page_contentmetadataz2No results returned. Error while parsing results: ) r*roundINVALID_DOC_METADATA_KEYSitemsINVALID_METADATA_VALUEappendr Exceptionloggerwarning) r6 final_res responsesblobsresult_entitiesentdistance txt_contentspmkeymvalpropses r0r5r5Us5IQ"1: 5  NQ  IaL0il+;<<'l+;CC!c ddl}|j}|j|||S#t$r tdwxYw)zVDMS client for the VDMS server. Args: host: IP or hostname of VDMS server port: Port to connect to VDMS server rNzOCould not import vdms python package. Please install it with `pip install vdms.)vdms ImportErrorconnect)hostportrUclients r0 VDMS_Clientr[wsO TYY[F NN4 M   8   s +Az0.3.18z1.0.0zlangchain_vdms.VDMS)sinceremovalalternative_importcDeZdZdZdeddddd d3dZed4dZd5dZd6d Z d7d Z d8d Z d9d Z e edf d:d Z d; ddZ d; d?dZ d@ dAdZ dB dCdZgdf dDdZ dE dFdZ dGdZddedf dHdZ dI dJdZddef dKdZdef dLdZ dMdZdNdZdOdZ dP dQd Z dR dSd!Z!e eddddf dTd"Z"dUd#Z#e$ddeef dVd$Z%e$dddeef dWd%Z&dddgf dXd&Z'e ed'df dYd(Z(e ed'df dZd)Z)e ed'df d[d*Z*e ed'df d\d+Z+dde edddf d]d,Z,e edf d^d-Z-e edf d_d.Z.e edf d`d/Z/e edf dad0Z0 dbd1Z1 dcd2Z2y)dVDMSaIntel Lab's VDMS for vector-store workloads. To use, you should have both: - the ``vdms`` python package installed - a host (str) and port (int) associated with a deployed VDMS Server Visit https://github.com/IntelLabs/vdms/wiki more information. IT IS HIGHLY SUGGESTED TO NORMALIZE YOUR DATA. Args: client: VDMS Client used to connect to VDMS server collection_name: Name of data collection [Default: langchain] distance_strategy: Method used to calculate distances. VDMS supports "L2" (euclidean distance) or "IP" (inner product) [Default: L2] engine: Underlying implementation for indexing and computing distances. VDMS supports TileDBDense, TileDBSparse, FaissFlat, FaissIVFFlat, and Flinng [Default: FaissFlat] embedding: Any embedding function implementing `langchain_core.embeddings.Embeddings` interface. relevance_score_fn: Function for obtaining relevance score Example: .. code-block:: python from langchain_huggingface import HuggingFaceEmbeddings from langchain_community.vectorstores.vdms import VDMS, VDMS_Client model_name = "sentence-transformers/all-mpnet-base-v2" vectorstore = VDMS( client=VDMS_Client("localhost", 55555), embedding=HuggingFaceEmbeddings(model_name=model_name), collection_name="langchain-demo", distance_strategy="L2", engine="FaissFlat", ) Nrr) embeddingcollection_namedistance_strategyenginerelevance_score_fnembedding_dimensionsc||_||_||_||_|j ||||_|j ||j|j|_y)Nrdmetric)_clientsimilarity_search_enginercra_check_required_inputsoverride_relevance_score_fnadd_set_collection_name)selfrZrarbrcrdrerfs r0__init__z VDMS.__init__sj (.%!2" ##O5IJ,>(!% 00))!-! r2c|jSr4)rarps r0 embeddingszVDMS.embeddingss ~~r2ct|jtr|jj|Sd}|dz }t |)Nz*Must provide `embedding` which is expectedz to be an Embeddings object)r)rarembed_documentsr+)rptextsp_strs r0_embed_documentszVDMS._embed_documentss@ dnnj 1>>11%8 8@E 2 2EU# #r2c |j4t|jdr|jjdd|i|Std)N embed_videopathsz:Must provide `embedding` which has attribute `embed_video`)rahasattrr{r+)rpr|kwargss r0 _embed_videozVDMS._embed_videosH >> %'$..-*P-4>>--DEDVD DL r2c|j2t|jdr|jj|Std)N embed_imageurisz:Must provide `embedding` which has attribute `embed_image`)rar~rr+)rprs r0 _embed_imagezVDMS._embed_imagesA >> %'$..-*P>>--4-8 8L r2ct|jtr|jj|St d)NzEMust provide `embedding` which is expected to be an Embeddings object)r)rar embed_queryr+)rptexts r0 _embed_queryzVDMS._embed_querys5 dnnj 1>>--d3 3W r2c|j |jS|jjdvrdStd|jd)a8 The 'correct' relevance function may differ depending on a few things, including: - the distance / similarity metric used by the VectorStore - the scale of your embeddings (OpenAI's are unit normed. Many others are not!) - embedding dimensionality - etc. )ipl2c|Sr4r})r,s r0z1VDMS._select_relevance_score_fn..sQr2z=No supported normalization function for distance_strategy of z;.Consider providing relevance_score_fn to VDMS constructor.)rmrclowerr+rss r0_select_relevance_score_fnzVDMS._select_relevance_score_fnsf  + + 733 3  ! ! ' ' )\ 9 --1-C-C,DEMM r2c |jd|d<|jd||||d|}g}|D]G\}} |j|j|| f&|j||j| fI|S)z?Return docs and their similarity scores on a scale from 0 to 1.Tnormalize_distance)querykfetch_kfilterr})rmsimilarity_search_with_scorerD) rprrrrrdocs_and_scoresdocs_and_rel_scoresr7scores r0(_similarity_search_with_relevance_scoresz-VDMS._similarity_search_with_relevance_scores s  + + 3+/F' (;$;;     *,) JC//7#**C<8#**88? #"r2ct||dd||n|Dcgc]}dc}}t||dd||n*|Dcgc]}ttj!c}}t||ddg}g}g} t ||||D]V\} } } } |j || | | | \}}|$|j ||j || j | X|j||\}}| Scc}wcc}w)Nrwrt metadatasidsr?radocumentr%)r1struuiduuid4zip_VDMS__get_add_queryrD_VDMS__run_vdms_query)rprbrwrtrrr8 all_queries all_blobs inserted_idsmetaembr7r%rr'responseresponse_arrays r0addzVDMS.add*s E:w E!*!6I5?d{"  LL " : %'E)  5A222**11!4 5    y)    G!sC8Fct|||}|j|g\}}t|dkDrt|dj d}|St t }|S)N) unique_entitydeletionr,)_find_property_entityrr* _bytes2strsplitrDEFAULT_PROPERTIES)rprbrr find_queryr response_blobrs r0__get_propertieszVDMS.__get_propertiessr + =8 #'"7"7 "E- }  !$.}Q/?$@$F$Fs$K !%$%--?$@ !$$r2c|jj||\}}t||}|r|jj||fSr4)rjr_check_valid_responseprint_last_response)rprrrrrr8s r0__run_vdms_queryzVDMS.__run_vdms_querysI $(<<#5#5k9#M . !+x 8  LL , , .''r2c .t||ddt||dd||n|Dcgc]}dc}}t||dd|j|}g}t||||D]\} } } } d|ji} ddd gi}| d| g|d <t d |ddddd|| }|j |g\}}|j || | | | \}}|j|j |g|g\}}|j| |j|||jycc}w)z Updates (find, delete, add) a collection based on id. If more than one collection returned with id, error occuers r documentsrtNrrrrrr%r;rr) r1rrrrrrrD_VDMS__update_properties)rprbrrrtrr8 orig_props updated_idsrrr7r%r6rrrrr's r0__updatez VDMS.__updatesj CE;?CULA!*!6I3 ?-0II(=#!)#@)% X #33K(L ! J 1r2Tc |Dcgc]}|j|}}|r|rt|D] \} }||| d<n|rg}|D]}|jd|i||n*|D cgc]} tt j !c} }|j |} ||D cgc]} i}} n|D cgc] } t| }} |jd|| |||d||Scc}wcc} wcc} wcc} w)aRun more images through the embeddings and add to the vectorstore. Images are added as embeddings (AddDescriptor) instead of separate entity (AddImage) within VDMS to leverage similarity search capability Args: uris: List of paths to the images to add to the vectorstore. metadatas: Optional list of metadatas associated with the images. ids: Optional list of unique IDs. batch_size (int): Number of concurrent requests to send to the server. add_path: Bool to add image path as metadata Returns: List of ids from adding images into the vectorstore. ) image_pathrrrwrtrr batch_sizer}) encode_image enumeraterDrrrr_validate_vdms_propertiesadd_from) rprrrradd_pathruri b64_textsmidxr8rtms r0 add_imageszVDMS.add_images1s,2CGG3T&&#&6G G  &t_ 4 c03 $ - 4 I 6  ,!45 6_cd*K3tzz|+<*K&&D&1  %)**I*?HI!215III  !!     ;H+L +IsC#&$C(% C-5C2c ||Dcgc]}d}}|r|rt|D] \} } | || d<n|rg}|D]} |jd| i||n*|Dcgc]}ttj!c}}|j dd|i|} ||Dcgc]}i}}|j d|| |||d||Scc}wcc}wcc}w)aRun videos through the embeddings and add to the vectorstore. Videos are added as embeddings (AddDescriptor) instead of separate entity (AddVideo) within VDMS to leverage similarity search capability Args: paths: List of paths to the videos to add to the vectorstore. metadatas: Optional list of text associated with the videos. metadatas: Optional list of metadatas associated with the videos. ids: Optional list of unique IDs. batch_size (int): Number of concurrent requests to send to the server. add_path: Bool to add video path as metadata Returns: List of ids from adding videos into the vectorstore. r video_pathr|rr})rrDrrrrr) rpr|rwrrrrrr8rpathrts r0 add_videoszVDMS.add_videosis 4 =!&'AR'E'  '. 5 d04 $ - 5 I 7  ,!56 7_ce*L3tzz|+<*L'T&&=U=f=  %*++I+  !!     7(+M ,s B>$C Cc 8t|}|*|Dcgc]}ttj!}}|j |}||Dcgc]}i}}n|Dcgc] }t |}}|j d|||||d|} | Scc}wcc}wcc}w)aRun more texts through the embeddings and add to the vectorstore. Args: texts: List of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. batch_size (int): Number of concurrent requests to send to the server. Returns: List of ids from adding the texts into the vectorstore. rr})rrrrryrr) rprwrrrrr8rtrrs r0 add_textszVDMS.add_textss(U  ;.343tzz|$4C4**51  %*++I+?HI!215III$t}} !!    #5 ,Is$B  BBc |j|j}g}tdt||D]a} t | |zt|} || | } || | } || | } |r|| | }|j |j| | | }|j |c|j|j||j|S)Nr)rtrwrr) rroranger*minrextendrr)rprwrtrrrrrr start_idxend_idx batch_textsbatch_embedding_vectors batch_idsbatch_metadatasresults r0rz VDMS.add_froms**4+@+@A "$ q#e*j9 (I)j0#e*=G '2K&07&C #Ig.I"+Ig">XX%%2!) F    '# ((    ! !:t/I/I r2cL|jjs td|jtvr td|j t vr td|j td|||_n|j6t|jdr t|jd|_n|j~t|jdst|jdrRt|jd r1 |jjjj|_n td |j|}t|d r|j j#|y||_y#t$r td wxYw) Nz_VDMS client must be connected to a VDMS server.Please use VDMS_Client to establish a connectionz-distance_strategy must be either 'L2' or 'IP'z]engine must be either 'TileDBDense', 'TileDBSparse', 'FaissFlat', 'FaissIVFFlat', or 'Flinng'Must provide embedding functionrzThis is a sample sentence.rr{modelz>Embedding dimension needed. Please define embedding_dimensionsr)rj is_connectedr+rcrrkrrarr~r*rrtoken_embedding embedding_dimrrr )rprbrf current_propss r0rlzVDMS._check_required_inputss||((*E   ! !)C CLM M  ( (0A A=  >> !>? ?  +';D $ ^^ 'GDNNM,R'*!!">?(D $^^ ' DNNM 2t~~}5t~~w/,,<<JJ,!T --o> 40 1  & & - -m <4AD &"$Xs /FF#c g}g}ddgd}td|dddddd| }|j||j||\}}|dddS)Nrr%)countrr;rrreturned)rrDr)rprbrrr6rrrs r0rz VDMS.count'sy!# ! /     5!#'#8#8i#P .{+,Z88r2c,tj|Sr4)base64 b64decode)rp base64_images r0 decode_imagezVDMS.decode_image=s --r2c J||n |j}|j|||S)zDelete by ID. These are the IDs in the vectorstore. Args: ids: List of ids to delete. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. )rr)ro _VDMS__delete)rprrbrrnames r0deletez VDMS.delete@s+ #2"=4CXCX}}Ts }DDr2cd}d}t||||}|j|g|\} } |r|| dvr| d|ddd}| | |fS)Nrr;)rr6rr<rr$)rr) rpsetnamerr6r normalizemax_dist command_strrrrs r0get_k_candidateszVDMS.get_k_candidatesSsw&      $(#8#8%)#L .  3{;/ ;B? LH11r2c g} t|} | | j| ||j|||| |\} } } n5|ddgi}n&d|vrdg|d<nd|dvr|djdt||||}|j |g\} } || dvr*| d|ddkDr| d|dDcgc]}|d }}nggfS|j|||| |\} } } || dvs|| dvr| d|ddk(rggfSg}| d|dD]*}|d|vr|j|t ||k(s*n|| d|d<t || d|d<t ||kr d}t ||rS| dtjfvrd n| } t| d|dD]$\}}|d | z |d <|d | d|d|d <&| | fScc}w) N)r*rr%rr6rrr<z4Returned items < k_neighbors; Try increasing fetch_kg?r$) rrDr-rrr*rnpinfr)rpr,r)rrrr6query_embeddingrrr'rrr+rrLids_of_interest new_entitiesrxent_idxs r0get_descriptor_responsezVDMS.get_descriptor_responsejs " /     T "  151F1FgyDV2G2 .Hnh!D6*w&#'&WV_,&&t,$' E (,'<'K MPN K(4W=kJ  ''I#s3 G ct|d5}|j}tj|j dcdddS#1swYyxYw)Nrbzutf-8)openreadr  b64encodedecode)rprfr's r0rzVDMS.encode_imagesG *d # :q668D##D)009 : : :s 4A  Ac |d}|j||Dcgc]}|jc}|Dcgc]}|jc}||||Scc}wcc}w)aCreate a VDMS vectorstore from a list of documents. Args: collection_name (str): Name of the collection to create. documents (List[Document]): List of documents to add to vectorstore. embedding (Embeddings): Embedding function. Defaults to None. ids (Optional[List[str]]): List of document IDs. Defaults to None. batch_size (int): Number of concurrent requests to send to the server. Returns: VDMS: VDMS vectorstore. rZ)rZrwrrarrrb) from_textsr>r?) clsrrarrrbrrZr7s r0from_documentszVDMS.from_documentss_,#8,~~/893##9/89s||9!+  99s A A c |d}||||} |*|D cgc]} ttj!}} | j||||| Scc} w)aHCreate a VDMS vectorstore from a raw documents. Args: texts (List[str]): List of texts to add to the collection. embedding (Embeddings): Embedding function. Defaults to None. metadatas (Optional[List[dict]]): List of metadatas. Defaults to None. ids (Optional[List[str]]): List of document IDs. Defaults to None. batch_size (int): Number of concurrent requests to send to the server. collection_name (str): Name of the collection to create. Returns: VDMS: VDMS vectorstore. rZ)rbrarZ)rwrrr)rrrr ) r@rwrarrrrbrrZvdms_collectionr8s r0r?zVDMS.from_textssr0#8,+  ;.343tzz|$4C4!!! " 5s$Acg}g}ddi}|||d<d|vr|j|}||d<d|vrd|d <td |d|| } |j| |j||\} } | | fS) aGets the collection. Get embeddings and their associated data from the data store. If no constraints provided returns all embeddings up to limit. Args: constraints: A dict used to filter results by. E.g. `{"color" : ["==", "red"], "price": [">", 4.00]}`. Optional. limit: The number of documents to return. Optional. include: A list of what to include in the results. Can contain `"embeddings"`, `"metadatas"`, `"documents"`. Ids are always included. Defaults to `["metadatas", "documents"]`. Optional. rrNlimitr?rrtTr'r;)rrr6)rrrDr) rprbrrEincluderrr6rrrrs r0getzVDMS.gets("$ ! #*B-  $GG   $($9$9/$J !3GFO 7 ""GFO  #   5!#'#8#8i#P .''r2g?c T|j tdtjj |s(t |jdr|j |}ntjj |r-t |jdr|j|gd}ngtjj |r-t |jdr|j|gd}nd|d }|d z }|d z }t||j||||| } | S aRReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query (str): Query to look up. Text or path for image or video. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List of Documents selected by maximal marginal relevance. zBFor MMR search, you must specify an embedding function oncreation.rrrrr{r|(Could not generate embedding for query ''.9If using path for image or video, verify embedding model 6has callable functions 'embed_image' or 'embed_video'.) lambda_multr) rar+osrisfiler~rrr'max_marginal_relevance_search_by_vector) rprrrrOrrembedding_vector error_msgdocss r0max_marginal_relevance_searchz"VDMS.max_marginal_relevance_search9s4 >> !T  ww~~e$)O#007  WW^^E "wt~~}'M#00ug0>qA  WW^^E "wt~~}'M#00w0?B B5'LI T TI Q QIY' ';;  # <  r2c |j|g||gd}t|dddk(rgS|ddDcgc]}tt|} }t t j |t j| ||} t|} t| D cgc] \} } | | vs | }} } |Scc}wcc} } waHReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List of Documents selected by maximal marginal relevance. )rr distancesrt)query_embeddings n_resultsrrFrrdtype)rrO) query_collection_embeddingsr*r_bytes2embeddingrr0arrayfloat32r9r)rprarrrOrrr6rembedding_list mmr_selected candidatesirselected_resultss r0rRz,VDMS.max_marginal_relevance_search_by_vectoros422'[I 3  wqz!}  "I>EQZ]39%f-.N6"**5' L*'2J( 3 aqL7H   $ #! sB7! B<.B<c T|j tdtjj |s(t |jdr|j |}ntjj |r-t |jdr|j|gd}ngtjj |r-t |jdr|j|gd}nd|d }|d z }|d z }t||j||||| } | SrI) rar+rPrrQr~rrr2max_marginal_relevance_search_with_score_by_vector) rprrrrOrrrarTrUs r0(max_marginal_relevance_search_with_scorez-VDMS.max_marginal_relevance_search_with_scores4 >> !T ww~~e$)O))%0I WW^^E "wt~~}'M))w)7:I WW^^E "wt~~}'M)))8;IB5'LI T TI Q QIY' 'FF  # G  r2c |j|g||gd}t|dddk(rgS|ddDcgc]}tt|} }t t j |t j| ||} t|} t| D cgc]\} \} }| | vs| |f}} } }|Scc}wcc}} } wrX) r^r*rr_rr0r`rar5r)rprarrrOrrr6rrbrcrdrerfsrgs r0riz7VDMS.max_marginal_relevance_search_with_score_by_vectors422'[I 3  wqz!}  "I>EQZ]39%f-.N6"**5' L5W=J)2*(=  $1fq!lARA   $ #! sB>"C2Cc g} | |j}|| S|jddg} |d| vr|jd| vd}|D]1} |jd||||||| \} } | j | | g3| S)NrFrrt)rr'r;)rrrr6rr2)rorGrr6rD)rprZrbr[rrr6rr all_responsesrFqembrrs r0r^z VDMS.query_collection_embeddingss$&  ""33O  # **Y 6 ?{g522$/G % =D'+'C'C %"#5 $(D ( $Hn  (N!; < =r2c f|j|f|||d|}|Dcgc]\}}| c}}Scc}}w)aRun similarity search with VDMS. Args: query (str): Query to look up. Text or path for image or video. k (int): Number of results to return. Defaults to 3. fetch_k (int): Number of candidates to fetch for knn (>= k). filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List[Document]: List of documents most similar to the query text. )rrr)r) rprrrrrrr7r8s r0similarity_searchzVDMS.similarity_search:sG&<$;;   :@ #22Q222s -c H|jd|g|||d|}t|S)aReturn docs most similar to embedding vector. Args: embedding (List[float]): Embedding to look up documents similar to. k (int): Number of Documents to return. Defaults to 3. fetch_k (int): Number of candidates to fetch for knn (>= k). filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List of Documents most similar to the query vector. rZr[rrr})r^r9rprarrrrr6s r0similarity_search_by_vectorz VDMS.similarity_search_by_vectorRs@"3$22 '[     ((r2c j|j tdtjj |s(t |jdr|j |}ntjj |r-t |jdr|j|gd}ngtjj |r-t |jdr|j|gd}nd|d }|d z }|d z }t||jd |g|||d |}t|S)aERun similarity search with VDMS with distance. Args: query (str): Query to look up. Text or path for image or video. k (int): Number of results to return. Defaults to 3. fetch_k (int): Number of candidates to fetch for knn (>= k). filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List[Tuple[Document, float]]: List of documents most similar to the query text and cosine distance in float for each. Lower score represents more similarity. rrrrrr{rJrKrLrMrNrsr}) rar+rPrrQr~rrrr^r5) rprrrrrr2rTr6s r0rz!VDMS.similarity_search_with_scorems!* >> !>? ?77>>%(WT^^]-S/3/@/@/G&74>>=+Q"&"3"3%"3"A!"D&74>>=+Q"&"3"35'"3"B1"EFugRP XX UU  ++6d66"1!2   G+733r2c H|jd|g|||d|}t|S)a6 Return docs most similar to embedding vector and similarity score. Args: embedding (List[float]): Embedding to look up documents similar to. k (int): Number of Documents to return. Defaults to 3. fetch_k (int): Number of candidates to fetch for knn (>= k). filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. Returns: List[Tuple[Document, float]]: List of documents most similar to the query text. Lower score represents more similarity. rsr})r^r5rts r0&similarity_search_with_score_by_vectorz+VDMS.similarity_search_with_score_by_vectors@03$22 '[    +733r2c,|j||g|gS)zUpdate a document in the collection. Args: document_id (str): ID of the document to update. document (Document): Document to update. )update_documents)rprb document_idrs r0update_documentzVDMS.update_documents$$_{mhZPPr2c|Dcgc]}|j}}|Dcgc]}t|j}}|j|}|j |||||ycc}wcc}w)zUpdate a document in the collection. Args: ids (List[str]): List of ids of the document to update. documents (List[Document]): List of documents to update. )rrtrN)r>rr?ry _VDMS__update)rprbrrrrr?rts r0rzzVDMS.update_documentss7@@(%%@@IR =E %h&7&7 8  **40   !  A s A$A))rZ vdms.vdmsraOptional[Embeddings]rbrrcDISTANCE_METRICSrdENGINESrez"Optional[Callable[[float], float]]rf Optional[int]returnNone)rr)rw List[str]rList[List[float]])r|rrrrr)rrrr)rrr List[float])rzCallable[[float], float]) rrrintrrrOptional[Dict[str, Any]]rrrList[Tuple[Document, float]])NN) rbrrwrrtrr1Optional[Union[List[None], List[Dict[str, Any]]]]rOptional[List[str]]rr )rr)rbrrdrrirrr)rbrrzUnion[None, List[str]]rUnion[None, Dict[str, Any]]rbool)NNNN) rbrr? Optional[Any]raUnion[List[float], None]rrr% Optional[str]rz4Tuple[Dict[str, Dict[str, Any]], Union[bytes, None]]FF)rbrrOptional[bool]rrrr)rz List[Dict]rOptional[List]rrrTuple[Any, Any]r4) rbrrrrrrtrrrrr)rbrrr rrrr)rrrOptional[List[dict]]rrrrrrrrrr)NNNrT)r|rrwrrrrrrrrrrrrr) rwz Iterable[str]rrrrrrrrrr)rwrrtrrrrrrrrrrr)rbrrfzUnion[int, None]rr)rbrrr)r"rrbytes)NNN) rrrbrrOptional[Dict]rrrr)NNF) r)rrrr6rrrr*rrz(Tuple[List[Dict[str, Any]], List, float])r,rr)rrrrrrOptional[dict]r6rr2zOptional[List[float]]rrrz!Tuple[List[Dict[str, Any]], List])rrrr)r@ Type[VDMS]rList[Document]rarrrrrrbrrrrr`)r@rrwrrarrrrrrrrbrrrrr`) rbrrrrErrFrrr)rrrrrrrOfloatrOptional[Dict[str, List]]rrrr)rarrrrrrOrrrrrrr)rrrrrrrOrrrrrrr)rarrrrrrOrrrrrrr)rZzOptional[List[List[float]]]rbrr[rrrrrr6rrrrrrz!List[Tuple[Dict[str, Any], List]]) rrrrrrrrrrrr) rarrrrrrrrrrr) rrrrrrrrrrrr) rarrrrrrrrrrr)rbrr{rrrrr)rbrrrrrrr)3__name__ __module__ __qualname____doc__DEFAULT_COLLECTION_NAMErqpropertyrtryrrrr DEFAULT_KDEFAULT_FETCH_Krrrnr%rrrr~rDEFAULT_INSERT_BATCH_SIZErrr rrlrr#r'r-r6r classmethodrAr?rGrVrRrjrir^rqrurrxr|rzr}r2r0r`r`s $T+/6.2%AE.2  (    ,   ? ,   8$4&+/ ## # # ) #  # &#JHL#' &  E  !  H&#' !  0'+37 +/+/$+/1 +/  +/`#'.2"& 6 6  6 , 6  6  6  >6 v).#( %%&%! %  %&%'.3 ( (" (, (  (&HL 9 9 9  9 & 9 E 9  9 vMM(,M$2 M  M.+/#'3#' 66(6! 6  6 ! 66 6v&**.#'#'66#6( 6 ! 6  6!66 6v+/#'3 ''('! '  '  ' '\+/3 ##&# # ( #  ## #J6B"6B:J6B 6Bp9,. $()-&* E E'E$ E  E  E.-1$($) 222* 2 " 2 " 2 226%&&*,015#(J(J(J( J(  J( $ J(*J(/J(!J( +J(X: +/#'36    !  (  !             D+/*.#'36& &&(&( & ! &  &&& &&V'+#(\ 0(0($0( 0(  0(  0(j& ,0 44 4 4  4 * 44 4r& ,0 5$5$ 5$ 5$  5$ * 5$5$ 5$t& ,0 22 2 2  2 * 22 &2n& ,0 5$5$ 5$ 5$  5$ * 5$5$ &5$r9=)-"&.2/3#('5''' '  ' , '-'!'' +'X&,0 33 3 3 * 3  3 36&,0 )) ) ) * )  ) )<&,0 ,4,4 ,4 ,4 * ,4  ,4 &,4b&,0 44 4 4 * 4  4 &4B Q" Q14 Q@H Q  Q " )2 ?M  r2r`c d|i} d|vr|r|| d<||| d<|tvr|| d<d|vr||| d<d|vr|t|| d<d|vr |tvr|| d <d|vr |tvr|| d <|| i} | S) NsetAddr_ref propertiesrFindrrr6)rCr) r,r)rrrRrrrr6entityrs r0rrs$W-F w v **$|  0v !8 #K 0} 4J!J +} 0F!F#y & !E Lr2c |dk(rAtd||fDr-||d} ||| d<||| d<||| d<|difvr|| d<|7|| d<n1|d k(rd |i} |r|| d <| difvr| | d <| | | d <ntd||| i} | S)Nrc3$K|]}|du ywr4r}).0vars r0 z%_add_descriptorset..s141s)r& dimensionsrdrirrrrrrrr6zUnknown command: )allr+) r,r&num_dimsrdrirrRrrrr6rrs r0rrs((S1$((#31.""   %F8   %F8  ? F6N r "#(F<  !F6N + + #-F< tRj ($/F= !   'F9 ,[M:;; & !E Lr2ct|dkDrdj|nd}d}i}d|d<d|d<d |i}d |d <||d <||d<t|}i}|||<||fS) Nrrr AddEntityrclassTr'r&zqueryable propertiestyper&)r*join _str2bytes)rbrall_properties_str querytyperrR byte_datars r0_add_entity_with_blobr?s695H15L.1RTIF"F7OF6N#_5E*E&M)E) F<-.IEE) ) r2findcNg}g}gd}|j|vr)tdjdj||jdk(r"t |d}|j |||fS|jdk(r5t ||\}}|j ||j |||fS|jdk(rOt |d }|j |t ||\}}|j ||j |||fS) N)rrrz"[!] Invalid type. Choices are : {}rrT)rrr)r)rr+formatrrrDr) rbrrrrrchoicesrrs r0rrUs,  KH'G7*=DDSXXgEVWXXv%%oTJ5!"        &0.Qy5! "        )%oE5!1.Qy5! "   r2c4tj|d}|S)Nrar\)r0 frombuffer)r'rs r0r_r_xs --I .C Jr2c"|jSr4)r<)in_bytess r0rr}s ?? r2c tt|Dcgc]}|jD]}|c}}Scc}}wr4)rrr)rqrs r0_get_cmds_from_queryrs2  >1QVVX>Q>Q>? @@>s:cjt|}ttxrtfd|D}|S)Nc3dK|]'}|dvxrdd|vxrd|ddkD)yw)rrNr})rcmdrs r0rz(_check_valid_response..sZ3  x{ - (1+c* * - QK Z (1 , -3s-0)rr)rany)rrcmd_list valid_ress ` r0rrs<#K0H8T*s3 30I r2cdd|gi}td||dgdd}|g}|j|\}}t||}||fS)Nr%rr;r)rrr/)rrr) rZr)r%rfindDescriptorrresr8rs r0rrsb $$K$"- N ""K \\+ &FC%k37I n $$r2c\d}|'tj|d}|j}|S)zConvert embedding to bytes.Nrar\)r0r`tobytes)rar'rs r0rrs/ Dhhy 2{{} Kr2cd}i}d|d<|r||d<i}d|d<d|d<d g|d <||d <i}|rd d g|d<d |g|d<||d<i}|||<|S)N FindEntityrruniqueTr'rrr&rr6rrrr&rr})rbrrrrr6rrs r0rrs IF"F7O(x GGFOGG kGFOF9"$K$(!9 K 1K'F=EE) Lr2c,tj|Sr4)rencode)in_strs r0rrs ::f r2cxi}|jD]$\}}t|tr||t|<&|Sr4)rBr)rr)r? new_metadatakeyrs r0rrsA#%Lnn&+ U%&%*LS "+ r2) r,rr-rr.rr/rrr)r6rrr)r6rrr) localhosti)rXrrYrrr)NNNNNNN)r,rr)rrrrrrRrrrrrrrr6rrDict[str, Dict[str, Any]]) NNNNNNFNN)r,rr&rrrrdrrirrrrRrrrrrrrr6rrDict[str, Any])rbrrr rzTuple[Dict[str, Any], bytes]) rbrrrrr rrrr)r'rrr)rrrr)rrrr)rz List[dict]rrrr)rZrr)rr%rrzTuple[bool, Any])rarrzUnion[bytes, None]r)rbrrrrrrr)rrrr)r?rrr)E __future__rr loggingrPrcopyrtypingrrrrr r r r r rrrrnumpyr0langchain_core._api.deprecationrlangchain_core.documentsrlangchain_core.embeddingsrlangchain_core.vectorstoresr&langchain_community.vectorstores.utilsrrUrrr__annotations__rrrrrrrrArC getLoggerrrFr1r9r5r[r`rrrrr_rrrrrrrrr}r2r0rs "   6-03M 6:(CS:T5U2U  $((9#:=:% a-3<,dB7   8 $ $DD( (G@UVP ;P WP r*  !%"&"## # #  #  #  ## ###R#   "&"44 44  4  4  4 4 44 444n*.!0 ! ! ! !  !  !F A% % % % %&%*$! 8r2