hhddlmZddlZddlmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZerddlmZdd d d d Zej(eZGd d eZy)) annotationsN) TYPE_CHECKINGAnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore HippoClient localhost7788admin)hostportusernamepasswordceZdZdZ d ddZddZ d ddZ d dd Zdd Zdd Z dd Z d dd Z d ddZ d ddZ d d dZedddedddf d!dZy)"Hippoa`Hippo` vector store. You need to install `hippo-api` and run Hippo. Please visit our official website for how to run a Hippo instance: https://www.transwarp.cn/starwarp Args: embedding_function (Embeddings): Function used to embed the text. table_name (str): Which Hippo table to use. Defaults to "test". database_name (str): Which Hippo database to use. Defaults to "default". number_of_shards (int): The number of shards for the Hippo table.Defaults to 1. number_of_replicas (int): The number of replicas for the Hippo table.Defaults to 1. connection_args (Optional[dict[str, any]]): The connection args used for this class comes in the form of a dict. index_params (Optional[dict]): Which index params to use. Defaults to IVF_FLAT. drop_old (Optional[bool]): Whether to drop the current collection. Defaults to False. primary_field (str): Name of the primary key field. Defaults to "pk". text_field (str): Name of the text field. Defaults to "text". vector_field (str): Name of the vector field. Defaults to "vector". The connection args used for this class comes in the form of a dict, here are a few of the options: host (str): The host of Hippo instance. Default at "localhost". port (str/int): The port of Hippo instance. Default at 7788. user (str): Use which user to connect to Hippo instance. If user and password are provided, we will add related header in every RPC call. password (str): Required when user is provided. The password corresponding to the user. Example: .. code-block:: python from langchain_community.vectorstores import Hippo from langchain_community.embeddings import OpenAIEmbeddings embedding = OpenAIEmbeddings() # Connect to a hippo instance on localhost vector_store = Hippo.from_documents( docs, embedding=embeddings, table_name="langchain_test", connection_args=HIPPO_CONNECTION ) Raises: ValueError: If the hippo-api python package is not installed. testdefaultNFc h||_||_||_||_||_||_d|_d|_d|_g|_ |t}|j||_ d|_ |jj|j|jr2|r0|jj|j|j |jj|j|jr5|jj'|j|j|_ |j)y#t $r+} t#j$d|jd| d} ~ wwxYw#t $r+} t#j$d|jd| d} ~ wwxYw)Npktextvectorz+An error occurred while deleting the table z: z*An error occurred while getting the table )number_of_shardsnumber_of_replicasembedding_func table_name database_name index_params_primary_field _text_field _vector_fieldfieldsDEFAULT_HIPPO_CONNECTION_create_connection_aliashccolcheck_table_exists delete_table Exceptionloggingerror get_table_get_env) selfembedding_functionr"r#rr connection_argsr$drop_oldes d/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/hippo.py__init__zHippo.__init__Ps~!1"40$*(#!%!#  "6O//@ **4??DP>PQ !  MM=doo=NbQRPST      MM ? xP P/ +  *H  LL: ;G s5C8C75$C< C4# C//C4< D!DD!ctjd|'tjd|j|||j|j y)Nzinit ...zcreate collection)rHrI_create_collection_extract_fields _create_index)r4 embeddings metadatass r9r3zHippo._get_envsK  J  ! KK+ ,  # #J :  c ddlm}ddlm}t |d}t j d|g}|j||jd|j|j||jd|j|j||jd|jd|i|r|djD]k\}}t|tr3t |} |j||d|jd| iI|j||d|jmt j d ||j j#|j$d||j&|j(|j* |j j-|j$|j&|_t j1d |j$d |j&d y)Nr) HippoField) HippoTypez[_create_collection] dim: TF dimension) type_paramsz[_create_collection] fields: )nameauto_idr(r#rr z$[_create_collection] : create table z in z successfully)rArUtranswarp_hippo_api.hippo_typerVlenrHdebugappendr%STRINGr&r' FLOAT_VECTORitems isinstancelistr+ create_tabler"r#rr r2r,rI) r4rQrRrUrVdimr(keyvalue value_dims r9rNzHippo._create_collections @<*Q-  1#78  j!4!4dI ,,!22#66  77$$T__d6H6HI  OO,D1C1C0DM S rSc6ddlm}t|j|r}|jj}t j d||D]'}|jj|j)t j d|jyy)z,Grab the existing fields from the Collectionr HippoTablez[_extract_fields] schema:z04 [_extract_fields] fields:N) rArkrbr,schemarHr]r(r^rY)r4rkrlxs r9rOzHippo._extract_fieldssv? dhh +XX__F LL4VH= > + ""166* + LL7 }E F ,rScddlm}t|j|r|jj |j |jj|j i}|jdd}|y|jj |j |j|j dD]0}tjd||d|jk(s.|cSy)z0Return the vector index information if it existsrrjembedding_indexesNz[_get_index] embedding_indexes column) rArkrbr,r+get_table_infor"r#rCrHr]r')r4rk table_infororms r9 _get_indexzHippo._get_indexs? dhh +//!3!3c$//2& !+/BD I  (//ASASTOO%'!ALL#BCTBU!VW{d&8&88  ! rSc \ ddlm}ddlm}m}t |j |r|ju|jc|jd|j|jdd|_|j j|j|jd|jd |jd |jd  tj|j j|jdtj!d y|j|j"|j$|j&|j(d}|j*|j*|j|jd}||jd |jd <|jd dk(r||jd |jd <|j j|j|jd|jd |jd tj|j j|jdy|jd dk(s|jd dk(r||jd |jd <|j j|j|jd|jd |jd |jj-d d|jj-ddtj|j j|jdy|jd dk(r||jd |jd <|j j|j|jd|jd |jd |jj-d d|jj-dd|jj-dd|jj-dtj|j j|jdy|jd dk(r||jd |jd <|j j|j|jd|jd |jd |jj-d|jj-d|jj-dtj|j j|jdyt/dyyy) z Create a index on the collectionrrj) IndexType MetricTypeNlangchain_auto_create ) index_name metric_type index_typenlistryr{rzr|)r|zcreate index successfully)IVF_FLATFLATIVF_SQIVF_PQHNSW)ipIPl2L2r~r}rnprobe)r|rrnbitsm)r|rrrrMef_construction ef_search)rrrzeIndex name does not match, please enter the correct index name. (FLAT, IVF_FLAT, IVF_PQ,IVF_SQ, HNSW))rArkr[rurvrbr,rsr$rr} create_indexr'rHr]activate_indexrIr~rrrrrCrG)r4rkrurv index_dict metric_dicts r9rPzHippo._create_indexs?H dhh +0A0I ($$,&='1}}&/&8&8!# )D%HH))**)),7)),7))-8"//8 *LL//0A0A,0OPKK ;<%.$6$6 )"+"2"2"+"2"2 ) "J)mm(mm(mm(mm #K 8C))-88D%%m4((6&@:D --l;;)),7-- .. --l; --l; --m<    HH33D4E4El4ST)),7:E,,\:hF:D --l;;)),7-- .. --l; --l; --m<"&"3"3"7"7"D#'#4#4#8#82#F .  HH33D4E4El4ST**<8HD:D --l;;)),7-- .. --l; --l; --m<"&"3"3"7"7"D#'#4#4#8#82#F"&"3"3"7"7"C"//33C8.   HH33D4E4El4ST**<8FB:D --l;;)),7-- .. --l; --l; --m<"//33C8,0,=,=,A,ABS,T&*&7&7&;&;K&H.  HH33D4E4El4ST)DQ)1J +rSc ddlm}|rtd|Drtj dgSt |}tj d| |j j|}t|dk(rtj dgStj dt|t|j|s|j|||j||j|i} tj d|tj d|j |O|D]J} | j#D]5\} } | |j vs| j%| gj'| 7Ltj | |j| |j} t| }d |j vr|j j)d tj d |t+d||D]d}t-||z|}|j Dcgc] }| ||| }} |jj/|}tj1d |fdgS#t$r2|Dcgc]}|j j|ncc}w}}YAwxYwcc}w#t2$r}tj5d |||d }~wwxYw)a Add text to the collection. Args: texts: An iterable that contains the text to be added. metadatas: An optional list of dictionaries, each dictionary contains the metadata associated with a text. timeout: Optional timeout, in seconds. batch_size: The number of texts inserted in each batch, defaults to 1000. **kwargs: Other optional parameters. Returns: A list of strings, containing the unique identifiers of the inserted texts. Note: If the collection has not yet been created, this method will create a new collection. rrjc3&K|] }|dk( yw)N).0ts r9 z"Hippo.add_texts..s3AG3szNothing to insert, skipping.z[add_texts] texts: zdd_texts] len_embeddings:z[add_texts] metadatas:z[add_texts] fields:Nrz[add_texts] total_count:z05 [add_texts] insert z0Failed to insert batch starting at entity: %s/%sr)rArkallrHr]rcr!embed_documentsNotImplementedError embed_queryr\rbr,r3r&r'r(ra setdefaultr^removerangemin insert_rowsrIr/r1)r4textsrRtimeout batch_sizekwargsrkrQrm insert_dictdrfrgvectors total_countiend insert_listresr8s r9 add_textszHippo.add_textss4 @3U33 LL7 8IU  *5'23 M,,<>uEF F  [!1!123$D$6$67'l 4;;  KK  t $ / }=>q+z2 Aa*nk2C:>++FQ;q>!C0FKF hh**;7 4SE:; t c# MFKL$--99!<LLJL MPG  F;  s<I8.J63J;8 J3"J('J32J3; K"KK"c |jtjdgS|jd|||||d|}|D cgc]\}} | c} }Scc} }w)a Perform a similarity search on the query string. Args: query (str): The text to search for. k (int, optional): The number of results to return. Default is 4. param (dict, optional): Specifies the search parameters for the index. Defaults to None. expr (str, optional): Filtering expression. Defaults to None. timeout (int, optional): Time to wait before a timeout error. Defaults to None. kwargs: Keyword arguments for Collection.search(). Returns: List[Document]: The document results of the search. !No existing collection to search.)querykparamexprrr)r,rHr]similarity_search_with_score) r4rrrrrrrdoc_s r9similarity_searchzHippo.similarity_searchsd4 88  LL< =I/d// 1Eg IO #&&Q&&&s Ac |jtjdgS|jj |}|j d|||||d|}|S)a Performs a search on the query string and returns results with scores. Args: query (str): The text being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[float], List[Tuple[Document, any, any]]: r) embeddingrrrrr)r,rHr]r!r&similarity_search_with_score_by_vector) r4rrrrrrrrets r9rz"Hippo.similarity_search_with_scoresi6 88  LL< =I''33E: 9d99 1Eg QW  rSc |jtjdgS|jdd}|j |j tjd|j tjd|gtjd|tjd|tjd||jj |j |g|||}tjd ||jd z} g} d } t|D cgc] } |d |  c} D]} t|| D cic]\} }| | }} }t|j|j| }tjd |d | |d | | }| dz } | j||f| Scc} wcc}} w)a Performs a search on the query string and returns results with scores. Args: embedding (List[float]): The embedding vector being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[Tuple[Document, float]]: Resulting documents and scores. Nrz search_field:zvectors:zoutput_fields:ztopk:zdsl:) search_fieldr output_fieldstopkdslz-[similarity_search_with_score_by_vector] res:z%scoresr) page_contentmetadataz;[similarity_search_with_score_by_vector] res[0][score_col]:) r,rHr]r(rr'rr&zipr popr^)r4rrrrrrrr score_colrcountfieldrargmetarscores r9rz,Hippo.similarity_search_with_score_by_vector.s4 88  LL< =I  A T//0  }T%7%7$89: x }-. ~m_56 uQC[! tD6]#hhnn++K'    DSEJK$$y0 mDU3q6%=DE %E585NO\UEE5LODO1A1A(BTRC LL%%(VI%6$79 F9%e,E QJE JJU| $ % EOs 0G Gc |i}tjd|d|||||| d| } tjd|tjd|| j||| S)a Creates an instance of the VST class from the given texts. Args: texts (List[str]): List of texts to be added. embedding (Embeddings): Embedding model for the texts. metadatas (List[dict], optional): List of metadata dictionaries for each text.Defaults to None. table_name (str): Name of the table. Defaults to "test". database_name (str): Name of the database. Defaults to "default". connection_args (dict[str, Any]): Connection parameters. Defaults to DEFAULT_HIPPO_CONNECTION. index_params (dict): Indexing parameters. Defaults to None. search_params (dict): Search parameters. Defaults to an empty dictionary. drop_old (bool): Whether to drop the old collection. Defaults to False. kwargs: Other arguments. Returns: Hippo: An instance of the VST class. z'00 [from_texts] init the class of Hippo)r5r"r#r6r$r7z[from_texts] texts:z[from_texts] metadatas:)rrRr)rHrIr]r) clsrrrRr"r#r6r$ search_paramsr7r vector_dbs r9 from_textszHippo.from_textsssF  M => (!'+%     *5'23 .yk:;%9=rS)rrrrNNF)r5r r"rFr#rFrintr rr6Optional[Dict[str, Any]]r$Optional[dict]r7zOptional[bool])r6dictreturnr)NN)rQzOptional[list]rROptional[List[dict]]rNone)N)rQrcrRrrr)rr)rr)NNi) rz Iterable[str]rRrr Optional[int]rrrrr List[str])NNN)rrFrrrrr Optional[str]rrrrrzList[Document])rrFrrrrrrrrrrrList[Tuple[Document, float]])rz List[float]rrrrrrrrrrrr)rrrr rRrr"rFr#rFr6zDict[str, Any]r$zOptional[Dict[Any, Any]]rrr7boolrrrz'Hippo')__name__ __module__ __qualname____doc__r:r*r3rNrOrsrPrrrr classmethodr)rrrSr9rrs5t!& !"#48'+#(7&77 7  7  727%7!7rDTX(rs["LLL-03<      8 $M KM rS