hYddlmZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlmZddlmZddlmZdd lmZmZej0Zdd ZGd d eZGd deZGddeZy)) annotationsN)sha1)Thread)AnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore) BaseSettingsSettingsConfigDictc|D]}||vsyy)z Check if a string contains multiple substrings. Args: s: string to check. *args: substrings to check. Returns: True if all substrings are in the string, False otherwise. FT)sargsas f/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/myscale.pyhas_mul_sub_strrs" A: ceZdZUdZdZded<dZded<dZd ed <dZd ed <d Z ded <dZ ded<dddddZ ded<dZ ded<dZ ded<dZded<d#dZeddd d!"Zy)$MyScaleSettingsaMyScale client configuration. Attribute: myscale_host (str) : An URL to connect to MyScale backend. Defaults to 'localhost'. myscale_port (int) : URL port to connect with HTTP. Defaults to 8443. username (str) : Username to login. Defaults to None. password (str) : Password to login. Defaults to None. index_type (str): index type string. index_param (dict): index build parameter. database (str) : Database name to find the table. Defaults to 'default'. table (str) : Table name to operate on. Defaults to 'vector_table'. metric (str) : Metric to compute distance, supported are ('L2', 'Cosine', 'IP'). Defaults to 'Cosine'. column_map (Dict) : Column type map to project column name onto langchain semantics. Must have keys: `text`, `id`, `vector`, must be same size to number of columns. For example: .. code-block:: python { 'id': 'text_id', 'vector': 'text_embedding', 'text': 'text_plain', 'metadata': 'metadata_dictionary_in_json', } Defaults to identity map. localhoststrhosti intportN Optional[str]usernamepasswordMSTG index_typezOptional[Dict[str, str]] index_paramidtextvectormetadata)r&r'r(r)zDict[str, str] column_mapdefaultdatabase langchaintableCosinemetricct||SN)getattr)selfitems r __getitem__zMyScaleSettings.__getitem__UstT""rz.envutf-8myscale_ignore)env_fileenv_file_encoding env_prefixextra)r5rreturnr)__name__ __module__ __qualname____doc__r__annotations__rr!r"r$r%r*r,r.r0r6r model_configrrrrr!s>D#D#"Hm""Hm"J,0K)0 "JHcE3FC#&! LrrcteZdZdZ d dfd ZeddZddZddZddZ d ddZ e d ddZ dd Z d dd Z d dd Z d dd Z d d d Zd!dZ d" d#dZeddZxZS)$MyScalea`MyScale` vector store. You need a `clickhouse-connect` python package, and a valid account to connect to MyScale. MyScale can not only search with simple vector indexes. It also supports a complex query with multiple conditions, constraints and even sub-queries. For more information, please visit [myscale official site](https://docs.myscale.com/en/overview/) c  ddlm} ddlm}||_t ||||_nt|_|jsJ|jjr|jjsJ|jjrB|jjr,|jjr|jjsJdD]}||jjvrJ|jjjdvsJ|jjd vrt j#d t%|j'd }|jj(rPd d j+|jj(j-Dcgc]\}}d|d|dc}}znd} d|jjd|jjd|jjdd|jjdd|jjdd|jjdd|jjdd|d|jjdd|jj.d|jjd| d|jjdd } ||_d!|_d"|_||_|jjjd#vrd$nd%|_|d+|jj|jj|jj:|jj<d&||_ |j>jAd'|j>jAd*|j>jA| y#t$r tdwxYw#t$r d|_YwxYwcc}}w#tB$r7} t jEd(|j>jFd)Yd} ~ d} ~ wwxYw),zMyScale Wrapper to LangChain embedding (Embeddings): config (MyScaleSettings): Configuration to MyScale Client Other keyword arguments will pass into [clickhouse-connect](https://docs.myscale.com/) r) get_clientzlCould not import clickhouse connect python package. Please install it with `pip install clickhouse-connect`.)tqdmc|Sr2r)xs rz"MyScale.__init__..s1rN)r&r(r'r))IPCOSINEL2)ipcosinel2z_Lower case metric types will be deprecated the future. Please use one of ('IP', 'Cosine', 'L2')z try this out, ,'=z( CREATE TABLE IF NOT EXISTS .z( r&z String, r'r(z! Array(Float32), r)zP JSON, CONSTRAINT cons_vec_len CHECK length( z) = z$, VECTOR INDEX vidx z TYPE z&( 'metric_type=z,) ) ENGINE = MergeTree ORDER BY z \)rYrU)rNrOASCDESC)rrr!r"z"SET allow_experimental_json_type=1zClickhouse version=z6 - There is no allow_experimental_json_type parameter.z$SET allow_experimental_object_type=1r)$clickhouse_connectrH ImportErrorrIpgbarsuper__init__configrrrr*r,r.r0upperloggerwarninglen embed_queryr%joinitemsr$dimBS must_escape _embeddings dist_orderr!r"clientcommand Exceptiondebugserver_version) r4 embeddingrakwargsrHrIkriv index_paramsschema__ __class__s rr`zMyScale.__init__ns  5  % !DJ    DK)+DK{{{{{DKK$4$444 KK " " $$ !! ""   # 6 /A ... .. /{{!!'')-CCCC ;;  !7 7 NNG  )''78{{&& 388dkk6M6M6S6S6UVdaq1QCq\VW W  ((, (<(<'=Qt{{?P?P>QR''-./''/01''123'' 345[[++H56d3%@##';;#9#9(#C"DE++0012&&*kk&8&8%9<.I++/;;+A+A$+G*HI   &$[[''--/3CCE  ! !!!![[))[[))       KK   D E BC G$_ K   %$DJ %:WF  LL%dkk&@&@%ABFF   s:O' O?-P P'O<?PP Q&-QQc|jSr2)rlr4s r embeddingszMyScale.embeddingssrc8djfd|DS)NrWc3^K|]$}|jvrj|n|&ywr2)rkrj).0cr4s r z%MyScale.escape_str..s/V1!t/?/?*?$''1#QFVs*-)rg)r4values` r escape_strzMyScale.escape_strswwVPUVVVrc pdj|}g}|D]R}dj|Dcgc] }d|jt|d"c}}|jd|dTd|jj d|jj d|ddj|d }|Scc}w) NrTrU()z8 INSERT INTO TABLE rXz)) VALUES z )rgrrappendrar,r.)r4transac column_namesks_datan_ni_strs r _build_istrzMyScale._build_istrs XXl # #AAFbAdooc"g67q9FGA LL1QCq " #[[))*!DKK,=,=+>atD%!"   Gs%B3 c^|j||}|jj|yr2)rrnro)r4rr_i_strs r_insertzMyScale._inserts&!!'<8 F#rc |xs6|Dcgc]*}t|jdj,c}}|jj}g}|d||d||dt |j j|i} |xs|D cgc]} ic} }t tj|| |d<tt|t| z dk\sJt| j\} } d}|jt| dt| D]} t| | j|jjd|j k(sJ|j#| t||k(sf|r|j%t'|j(|| g }|j+g}t|dkDr$|r|j%|j)|| |Dcgc]}|c}Scc}wcc} wcc}w#t,$r:}t.j1d t3|d t5|d gcYd}~Sd}~wwxYw)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. ids: Optional list of ids to associate with the texts. batch_size: Batch size of insertion metadata: Optional column data to be inserted Returns: List of ids from adding the texts into the vectorstore. r7r&r'r(r)rNzInserting data...)desctotal)targetr   )rencode hexdigestrar*maprlrfjsondumpsresetziprhr^indexrirrgrrstartrprcerrortyper)r4texts metadatas batch_sizeidsrttcolmap_rrrykeysvaluesrvies r add_textszMyScale.add_textss*I5Iad188G,-779I++(( DM3 FOU H s4#3#3#?#?G 4e!4"!4 ,/ I,F WZ()3w<#l"334999L..01 f AZZV #6c)n  !1TZZ (>(>x(HIJKtxxWWWq!w<:-dll'4IAGGI G !7|aFFH Wd+"#!A# #=J"5*$  LL?47)3CCF87S TI sC/H H5B HA5H5 H >H H I/IIIc D|||fi|}|j|||||S)aZCreate Myscale wrapper with existing texts Args: texts (Iterable[str]): List or tuple of strings to be added embedding (Embeddings): Function to extract text embedding config (MyScaleSettings, Optional): Myscale configuration text_ids (Optional[Iterable], optional): IDs for the texts. Defaults to None. batch_size (int, optional): Batchsize when transmitting data to MyScale. Defaults to 32. metadata (List[dict], optional): metadata to texts. Defaults to None. Other keyword arguments will pass into [clickhouse-connect](https://clickhouse.com/docs/en/integrations/python#clickhouse-connect-driver-api) Returns: MyScale Index )rrr)r) clsrrsrratext_idsrrtctxs r from_textszMyScale.from_textss.6)V.v. ejI V rcd|jjd|jjd}||jjd|jjdz }|d|jj dz }|dz }|j jd |jjd|jjjD]}|d |d d d |dd dz }|dz }|S)zText representation for myscale, prints backends, username and schemas. Easy to use with `str(Myscale())` Returns: repr: string to show connection info and data schema z rXz @ :z zusername: z Table Schema: z4--------------------------------------------------- zDESC z|name24sz |rz| ) rar,r.rrr!rnquery named_results)r4_reprrs r__repr__zMyScale.__repr__>s "$++"6"6!7q9J9J8K3O DKK$$%Qt{{'7'7&8 DD $T[[%9%9$::TUU  ""DKK(()4;;+<+<*= > -/ A AfIc?*:1V9S/T E    rcdjtt|}|rd|}nd}d|jjdd|jjdd|jj d |jj d |d |jjd d |d|jd|d }|S)NrT PREWHERE rW SELECT r'z, r)z, dist FROM rX  ORDER BY distance(r(, []) AS dist  LIMIT )rgrrrar*r,r.rmr4q_embtopk where_str q_emb_strq_strs r _build_qstrzMyScale._build_qstrRsHHSe_- #I;/IIKK**6234'' 345++&&'q):):(;< K#{{55h?@I;O)*&  rc ^|j|jj|||fi|S)aPerform a similarity search with MyScale Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of Documents )similarity_search_by_vectorrlrf)r4rrurrts rsimilarity_searchzMyScale.similarity_searchfs9&0t//    ( ( /I AG  rc |j|||} |jj|jDcgc]C}t ||j j d||j j dEc}Scc}w#t$r:}tjdt|dt|dgcYd}~Sd}~wwxYw)Perform a similarity search with MyScale by vectors Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of (Document, similarity) r'r) page_contentr)rrrN) rrnrrr rar*rprcrrrr4rsrurrtrrrs rrz#MyScale.similarity_search_by_vector}s.  Ay9  **51??A   !"4;;#9#9&#A!Bt{{55jAB   LL?47)3CCF87S TI s0+BAB B B C/CCCc |j|jj|||} |jj |j Dcgc]H}t ||jjd||jjd|dfJc}Scc}w#t$r:}tjdt|dt|dgcYd}~Sd}~wwxYw) /Perform a similarity search with MyScale Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of documents most similar to the query text and cosine distance in float for each. Lower score represents more similarity. r'r)rdistrrrN)rrlrfrnrrr rar*rprcrrrr4rrurrtrrrs r'similarity_search_with_relevance_scoresz/MyScale.similarity_search_with_relevance_scoress*  !1!1!=!=e!DaS **51??A  %&t{{'='=f'E%F!"4;;#9#9*#E!FfI      LL?47)3CCF87S TI s0+B.A B)&B.)B.. C17/C,&C1,C1c|jjd|jjd|jjy)z, Helper function: Drop data zDROP TABLE IF EXISTS rXN)rnrorar,r.r|s rdropz MyScale.drops< #DKK$8$8#94;;;L;L:M N rc X| |Jdg}|r_t|dkDrQdj|Dcgc]}d|d c}}|j|jjdd|d|r|j|t|dkDsJd j|}d |jj d |jj d |} |jj|y cc}w#t$r(} tjt| Yd} ~ yd} ~ wwxYw)a3Delete by vector ID or other criteria. Args: ids: List of ids to delete. **kwargs: Other keyword arguments that subclasses might use. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. NzIYou need to specify where to be deleted! Either with `ids` or `where_str`rrSrUr&z IN (rz AND z DELETE FROM rXz WHERE TF) rergrrar*r,r.rnrorprcrr) r4rrrtcondsr&id_listwhere_str_finalqstrrs rdeletezMyScale.deletes& KI$5 W 6 3s8a<iiS 9r1RD 9:G LLDKK22489wiqI J  LL #5zA~~!,,u-4;;//0$++2C2C1DE$% '   KK   %!:  LLQ  s C3C88 D)D$$D)c4|jjdS)Nr))rar*r|s rmetadata_columnzMyScale.metadata_columns{{%%j11rr2)rsr raOptional[MyScaleSettings]rtrr>None)r>r )rrr>r)rrr Iterable[str]r>r)rrrrr>r)N N) rrrzOptional[List[dict]]rrrOptional[Iterable[str]]rtrr> List[str])NNNr)rrrsr rzOptional[List[Dict[Any, Any]]]rarrrrrrtrr>rFr>rr List[float]rrrr r>rN) rrrurrr rtrr>List[Document] rsrrurrr rtrr>r rrrurrr rtrr>zList[Tuple[Document, float]])r>r)NN)rzOptional[List[str]]rr rtrr>zOptional[bool])r?r@rArBr`propertyr}rrrr classmethodrrrrrrrrr __classcell__rzs@rrFrF`sB  -1^%^%*^% ^%  ^%@  W $+/'+ 66(6 6 % 6  6 6p 59,0,02  *  *  <*IM (+8E *BF    1> QT  4#' "" "! "  "  "JBF## #1>#QT# %#J $(#'$ $!$ $  $L22rrFceZdZdZdgf dfd Z d d dZ d d dZ d d dZeddZ xZ S)MyScaleWithoutJSONzsMyScale vector store without metadata column This is super handy if you are working to a SQL-native table Nc 6t|||fi|||_y)agBuilding a myscale vector store without metadata column embedding (Embeddings): embedding model config (MyScaleSettings): Configuration to MyScale Client must_have_cols (List[str]): column names to be included in query Other keyword arguments will pass into [clickhouse-connect](https://docs.myscale.com/) N)r_r`must_have_cols)r4rsrarrtrzs rr`zMyScaleWithoutJSON.__init__s! F5f5)7rcdjtt|}|rd|}nd}d|jjdddj|j d|jj d|jjd |d |jjd d |d |jd|d }|S)NrTrrWrr'z, dist, z FROM rXrrr(rrr) rgrrrar*rr,r.rmrs rrzMyScaleWithoutJSON._build_qstrsHHSe_- #I;/IIKK**6234$--./0++&&'q):):(;< K#{{55h?@I;O)*&  rc |j|||} |jj|jDcgc]E}t ||j j d|jDcic]}||| c}Gc}}Scc}wcc}}w#t$r:}tjdt|dt|dgcYd}~Sd}~wwxYw)rr'rrrrN) rrnrrr rar*rrprcrrrrs rrz.MyScaleWithoutJSON.similarity_search_by_vector%s.  Ay9  **51??A   !"4;;#9#9&#A!B/3/B/BC!a1gC D  LL?47)3CCF87S TI sA,B3B4 B B BBB C#/CCCc |j|jj|||} |jj |j Dcgc]J}t ||jjd|jDcic]}||| c}|dfLc}}Scc}wcc}}w#t$r:}tjdt|dt|dgcYd}~Sd}~wwxYw)rr'rrrrrN)rrlrfrnrrr rar*rrprcrrrrs rrz:MyScaleWithoutJSON.similarity_search_with_relevance_scoresIs*  !1!1!=!=e!DaS **51??A  %&t{{'='=f'E%F373F3F!Ga!QqT'!GfI   "H   LL?47)3CCF87S TI sA,B83B2 B-B2)B8-B22B88 C;/C60C;6C;cy)NrWrr|s rrz"MyScaleWithoutJSON.metadata_columnnsr) rsr rarrrrtrr>rr2rrrrr) r?r@rArBr`rrrrrrrs@rrrs-1$& 88*8" 8  8  8&IM (+8E .#' "" "! "  "  "JBF## #1>#QT# %#Jrr)rrrrr>bool) __future__rrlogginghashlibr threadingrtypingrrrr r r langchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresrpydantic_settingsrr getLoggerrcrrrFrrrrr se" ==-03>      <l<~V2kV2r wwr