hddlmZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z m Z mZmZmZmZmZddlZddlmZddlmZddlmZdd lmZmZdd lmZmZdd l m!Z!ejDe#Z$d Z%d Z&dZ'eGddZ(eGddZ)eGddZ*eGddZ+GddeZ,GddeZ-GddeZ.y)) annotationsN) dataclassfield)md5)AnyIterableIteratorListOptionalTupleType)CallbackManagerForRetrieverRun)Document) Embeddings)RunnableRunnableConfig) VectorStoreVectorStoreRetriever) ConfigDictivAivAivAcXeZdZUdZdZded<dZded<dZd ed <d Zd ed <dZ ded <y) SummaryConfigajConfiguration for summary generation. is_enabled: True if summary is enabled, False otherwise max_results: maximum number of results to summarize response_lang: requested language for the summary prompt_name: name of the prompt to use for summarization (see https://docs.vectara.com/docs/learn/grounded-generation/select-a-summarizer) Fbool is_enabledint max_resultsengstr response_langz"vectara-summary-ext-24-05-med-omni prompt_namestreamN) __name__ __module__ __qualname____doc__r__annotations__rrr r!f/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/vectorstores/vectara.pyrrs;JKM3;K;FDr(rc<eZdZUdZdZded<dZded<dZd ed <y ) MMRConfigaConfiguration for Maximal Marginal Relevance (MMR) search. This will soon be deprated in favor of RerankConfig. is_enabled: True if MMR is enabled, False otherwise mmr_k: number of results to fetch for MMR, defaults to 50 diversity_bias: number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to minimum diversity and 1 to maximum diversity. Defaults to 0.3. Note: diversity_bias is equivalent 1-lambda_mult where lambda_mult is the value often used in max_marginal_relevance_search() We chose to use that since we believe it's more intuitive to the user. Frr2rmmr_k333333?floatdiversity_biasN)r"r#r$r%rr&r-r0r'r(r)r+r+.s& JE3ONEr(r+cJeZdZUdZdZded<dZded<dZd ed <d Zded <y ) RerankConfigaConfiguration for Reranker. reranker: "mmr", "rerank_multilingual_v1", "udf" or "none" rerank_k: number of results to fetch before reranking, defaults to 50 mmr_diversity_bias: for MMR only - a number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to minimum diversity and 1 to maximum diversity. Defaults to 0.3. Note: mmr_diversity_bias is equivalent 1-lambda_mult where lambda_mult is the value often used in max_marginal_relevance_search() We chose to use that since we believe it's more intuitive to the user. user_function: for UDF only - the user function to use for reranking. nonerrerankerr,rrerank_kr.r/mmr_diversity_bias user_functionN) r"r#r$r%r4r&r5r6r8r'r(r)r2r2Cs2 HcHc ##M3r(r2ceZdZUdZdZded<dZded<dZd ed <d Zd ed <dZ ded<dZ ded<e e Z ded<e eZded< d ddZy )VectaraQueryConfigaWConfiguration for Vectara query. k: Number of Documents to return. Defaults to 10. lambda_val: lexical match parameter for hybrid search. filter Dictionary of argument(s) to filter on metadata. For example a filter can be "doc.rating > 3.0 and part.lang = 'deu'"} see https://docs.vectara.com/docs/search-apis/sql/filter-overview for more details. score_threshold: minimal score threshold for the result. If defined, results with score less than this value will be filtered out. n_sentence_before: number of sentences before the matching segment to add, defaults to 2 n_sentence_after: number of sentences before the matching segment to add, defaults to 2 rerank_config: RerankConfig configuration dataclass summary_config: SummaryConfig configuration dataclass rkr/ lambda_valr7rfilterNOptional[float]score_thresholdn_sentence_beforen_sentence_after)default_factoryr2 rerank_configrsummary_configc ||_||_||_||_| r| |_nt |_|r)||_||_tjdtn||_||_| r| |_ y|rBtd|j|j|_ tjdtyt|_ y)Nz[n_sentence_context is deprecated. Please use n_sentence_before and n_sentence_after insteadmmrr4r5r6z9MMRConfig is deprecated. Please use RerankConfig instead.)r<r>r?rArGrrCrDwarningswarnDeprecationWarningrFr2r-r0) selfr<r>r?rArCrDn_sentence_context mmr_configrGrFs r)__init__zVectaraQueryConfig.__init__ws$ . "0D "//D  %7D "$6D ! MML"  &7D "$4D ! !.D  !-#))#-#<#<"D  MMK"  ".D r() r;r=r7NrBrBNNNN)r<rr>r/r?rrAr@rCrrDrrOz Optional[int]rPzOptional[MMRConfig]rGzOptional[SummaryConfig]rFzOptional[RerankConfig])r"r#r$r%r<r&r>r?rArCrDrr2rFrrGrQr'r(r)r:r:Ys&AsKJFC'+O_+sc"' "EM<E$)-$HNMH+/!" !,0*.260420 2020 20 ) 20  2020*20(20020.20r(r:ceZdZdZ d ddZeddZddZddZdddZ dddZ d dd Z d d d Z d! d"d Z d#d Z d$d Z d%dZ d& d'dZe d d(dZe d d)dZd*dZd*dZd+dZy),Vectaraa`Vectara API` vector store. See (https://vectara.com). Example: .. code-block:: python from langchain_community.vectorstores import Vectara vectorstore = Vectara( vectara_customer_id=vectara_customer_id, vectara_corpus_id=vectara_corpus_id, vectara_api_key=vectara_api_key ) Ncr|xstjjd|_|xstjjd|_|xstjjd|_|j|j |j t jdn"t jd|j||_ tj|_ tjjd}|jjd |||_y) zInitialize with Vectara API.VECTARA_CUSTOMER_IDVECTARA_CORPUS_IDVECTARA_API_KEYNzHCan't find Vectara credentials, customer_id or corpus_id in environment.zUsing corpus id ) max_retrieszhttp://)osenvironget_vectara_customer_id_vectara_corpus_id_vectara_api_keyloggerwarningdebug_sourcerequestsSession_sessionadapters HTTPAdaptermountvectara_api_timeout)rNvectara_customer_idvectara_corpus_idvectara_api_keyrjsourceadapters r)rQzVectara.__init__s%8% 2::>> !< !#4# rzz~~ 8 !0 T2::>>BS3T  % % -&&.$$, NN  LL+D,C,C+DE F  ((* ##//A/> Iw/#6 r(cyNr'rNs r) embeddingszVectara.embeddingssr(cL|j|jd|jdS)z=Returns headers that should be attached to each post request.zapplication/json)z x-api-keyz customer-id Content-TypezX-Source)r_r]rcrrs r)_get_post_headerszVectara._get_post_headerss*..44.   r(c l|j|j|d}|jjdt j |d|j |j}|jdk7r@tjd|d|jd|jd |jy y) z Delete a document from the Vectara corpus. Args: doc_id (str): ID of the document to delete. Returns: bool: True if deletion was successful, False otherwise. ) customer_id corpus_id document_idz$https://api.vectara.io/v1/delete-docT)dataverifyheaderstimeoutz#Delete request failed for doc_id = z with status code , reason z, text F) r]r^rfpostjsondumpsrvrj status_coder`errorreasontext)rNdoc_idbodyresponses r) _delete_doczVectara._delete_docs 4400!  ==%% 2D!**,,, &    3 & LL5fX=O''( (//1B'==/#  r(cni}|j|d<|j|d<||d<|rdnd}|jj|j |t j ||jd}|j}|j }d|vr|dd nd}|d k(s|r|d k(ry |r|d k(ryy)Nrxrydocumentz$https://api.vectara.io/v1/core/indexzhttps://api.vectara.io/v1/indexT)r}urlr{r~r|statuscodeALREADY_EXISTSE_ALREADY_EXISTS FORBIDDENE_NO_PERMISSIONS E_SUCCEEDED) r]r^rfrrvrrrjr) rNdoc use_core_apirequest api_endpointrrresult status_strs r) _index_doczVectara._index_doc s"$!%!:!: #66 !  32  ==%%**,G$,, & ** 19V1CVH%f- # ?O1O% Z;6% r(c d|r)|Dcgc]}|j|}}t|Sycc}w)zDelete by vector ID or other criteria. Args: ids: List of ids to delete. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. T)rall)rNidskwargsidsuccesss r)deletezVectara.delete(s7 69:t''+:G:w< ;s-c g}t|D]v\}}tjj|stj d|d@|r||ni}|t |dftj|d}|j} | jd|jjd|jd|jd|d | |j } | j d k(r4| jd d } tj#d|d| d| j dk(r)| jd d } |j%| Ntj#d|d| jy|S)ac Vectara provides a way to add documents directly via our API where pre-processing and chunking occurs internally in an optimal way This method provides a way to use that API in LangChain Args: files_list: Iterable of strings, each representing a local file path. Files could be text, HTML, PDF, markdown, doc/docx, ppt/pptx, etc. see API docs for full list metadatas: Optional list of metadatas associated with each file Returns: List of ids associated with each of the files indexed zFile z does not exist, skippingrb)file doc_metadataruz https://api.vectara.io/upload?c=z&o=z&d=TrueT)filesr|r}r~rr documentIdz# already exists on Vectara (doc_id=z ), skippingrzError indexing file z: ) enumeraterZpathexistsr`ropenrrrvpoprfrr]r^rjrinfoappend) rN files_list metadatasrdoc_idsinxrmdrr}rrs r) add_fileszVectara.add_files7s(":. NIC77>>$' uTF*CDE#,3"BtD$/0 $ 2E,,.G KK '}}))243L3L2MSQUQhQhPiipq00 *H##s*!4\B D6!DVHKX%%,!4\Bv& 24&8==?:KLM9 N<r(c >t}|D]!}|j|j#|j}||Dcgc]}i}}|rd|d<nddi}|j dd} | rdnd} d|dt j || t||D cgc]\} } | t j | d c} } i} |j| | }|d k(r%|j||j| |gS|d k(r td |gScc}wcc} } w)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. doc_metadata: optional metadata for the document This function indexes all the input text strings in the Vectara corpus as a single Vectara document, where each input text is considered a "section" and the metadata are associated with each section. if 'doc_metadata' is provided, it is associated with the Vectara document. Returns: document ID of the document added langchainrnrFpartssectionrz metadataJson)rr)rrrziNo permissions to add document to Vectara. Check your corpus ID, customer ID and API key) rupdateencode hexdigestr\rrziprrprint)rNtextsrrrdoc_hashtr_r section_keyrrr success_strs r) add_textszVectara.add_textsls:.5 (A OOAHHJ ' (##%  %*++I+ %0L "$k2Lzz.%8 !-g9 6 DJJ|4  #E9 5D"tzz"~> ooc oE , ,   V $ OOC x . . A x7,s  D"Dc Zt|jtrtdi|j|_t|jtrt di|j|_d|d|jj dvr|jjn |j|j|jd|j|jdgdgi}|jdkDrd|ji|ddddd <|jj d k(r)td |jjid |ddd <nj|jj dk(r't |jj"d|ddd <n*|jj dk(rdt$i|ddd <|jj&rb|jj(|jj*|jj,dg|ddd<|rd|d|ddddd<|S)zBuild the body for the API Args: query: Text to look up documents similar to. config: VectaraQueryConfig object Returns: A dictionary with the body of the query queryr)rIudfrerank_multilingual_v1)sentencesBeforesentencesAfter)corpusIdmetadataFilter)rstart numResults contextConfig corpusKeylambdarlexicalInterpolationConfigrI diversityBias) rerankerId mmrConfigrerankingConfigr)r userFunctionrr)maxSummarizedResults responseLangsummarizerPromptNamesummaryT)storeconversationIdchatr') isinstancerFdictr2rGrr4r5r<rCrDr^r?r>MMR_RERANKER_IDr6UDF_RERANKER_IDr8RERANKER_MULTILINGUAL_V1_IDrrrr )rNrconfigr chat_conv_idrrs r)_get_query_bodyzVectara._get_query_bodysP f**D 1#/#G&2F2F#GF f++T 2$1$JF4I4I$JF ! "#0099GH,,55 $XX,2+C+C*0*A*A& )-(?(?.4mm" 6   q &++NDM! [ )! ,-I J    ( (E 1--v/C/C/V/VW3DM! . / ! ! * *e 3- & 4 4 B B3DM! . / ! ! * *.F F93DM! . /  + +-3,A,A,M,M$*$9$9$G$G,2,A,A,M,M+DM! Y '!&2:W a +A.v6 r(c |j||fi|}|jj|jdt j ||j }|jdk7r@tjdd|jd|jd|jdgS|j }|jr+|d d d Dcgc]}|d |jkDr|}}n |d d d }|d d d } g} |D]g} | dD cic] } | d| d } } | d}| |dD cic] } | d| d }} d|vrd|d<| j|| j| it|| D cgc]\} } t!| d| | d f}} } |j"j$dvr|d|j&}|j(j*rF|d d dd d}|d d dd dd }|jt!|d|ddf|Scc}wcc} wcc} wcc} } w)a7Run a Vectara query Args: query: Text to look up documents similar to. config: VectaraQueryConfig object Returns: A list of k Documents matching the given query If summary is enabled, last document is the summary text with 'summary'=True zhttps://api.vectara.io/v1/query)r}rr{r~rQuery failed %s(code r , details ) responseSetrrscorermetadatanamevalue documentIndexrnvectarar page_contentrrIrNrfactualConsistencyT)rfcsr=)rrfrrvrrrjrr`rrrrArrrrrFr4r<rGr)rNrrrrrrr responses documentsrxmrdoc_numdoc_mdresrrs r) vectara_queryzVectara.vectara_querys$t##E6w5G 5STai7+TFTv%#,x IIf    R  !"Y 2  2 !"6'          ( (,M Mj/C  + +]+A.y9!c @tdi|}|j||}|S)aReturn Vectara documents most similar to query, along with scores. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 10. any other querying variable in VectaraQueryConfig like: - lambda_val: lexical match parameter for hybrid search. - filter: filter string - score_threshold: minimal score threshold for the result. - n_sentence_before: number of sentences before the matching segment - n_sentence_after: number of sentences after the matching segment - rerank_config: optional configuration for Reranking (see RerankConfig dataclass) - summary_config: optional configuration for summary (see SummaryConfig dataclass) Returns: List of Documents most similar to the query and score for each. r')r:r)rNrrrdocss r)similarity_search_with_scorez$Vectara.similarity_search_with_scoreGs(0$-f-!!%0 r(c ^|j|fi|}|Dcgc]\}}| c}}Scc}}w)aReturn Vectara documents most similar to query, along with scores. Args: query: Text to look up documents similar to. any other querying variable in VectaraQueryConfig Returns: List of Documents most similar to the query )r)rNrrdocs_and_scoresrrs r)similarity_searchzVectara.similarity_searchcs;<$;;   #22Q222s )c Ptd|d|z |d<|j|fi|S)aSReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 5. fetch_k: Number of Documents to fetch to pass to MMR algorithm. Defaults to 50 lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. kwargs: any other querying variable in VectaraQueryConfig Returns: List of Documents selected by maximal marginal relevance. rIrJrF)r2r )rNrfetch_k lambda_multrs r)max_marginal_relevance_searchz%Vectara.max_marginal_relevance_searchws80#/W[# &t%%e6v66r(c f|jdi}|di|}|j||fd|i||S)aConstruct Vectara wrapper from raw documents. This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Vectara vectara = Vectara.from_texts( texts, vectara_customer_id=customer_id, vectara_corpus_id=corpus_id, vectara_api_key=api_key, ) rr')rr)clsr embeddingrrrrs r) from_textszVectara.from_textss@6zz."5 --%PPPr(c :|di|}|j|||S)aConstruct Vectara wrapper from raw documents. This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Vectara vectara = Vectara.from_files( files_list, vectara_customer_id=customer_id, vectara_corpus_id=corpus_id, vectara_api_key=api_key, ) r')r)rrrrrrs r) from_fileszVectara.from_filess$.--%+r(ct||S)zReturn a Vectara RAG runnable. VectaraRAGrNrs r)as_ragzVectara.as_rags$''r(ct||dS)z'Return a Vectara RAG runnable for chat.T)rrrs r)as_chatzVectara.as_chats$T22r(c Lt||jdtS)zreturn a retriever object.r) vectorstorer)VectaraRetrieverr\r:)rNrs r) as_retrieverzVectara.as_retrievers#VZZ:L:N%O  r()NNNxr) rk Optional[str]rlr"rmr"rjrrnr)returnOptional[Embeddings])r#r)rrr#rF)rrrrr#rrq)rzOptional[List[str]]rrr#Optional[bool])r Iterable[str]rOptional[List[dict]]rrr# List[str])NN) rr'rr(rzOptional[dict]rrr#r))FN) rrrr:rr&rr"rrr#r)rrrr:rrr#List[Tuple[Document, float]])rrrrr#r*)rrrrr#List[Document])r,g?) rrr rrr/rrr#r+) r Type[Vectara]rr)rr$rr(rrr#rS) rr,rr)rr$rr(rrr#rS)rr:r#r)rrr#r)r"r#r$r%rQpropertyrsrvrrrrrrrrr r classmethodrrrrr r'r(r)rSrSs$.2+/)-#&! 7* 7) 7' 7 ! 7  7D <!< $+/3!3(3 3  3p+/'+ 77(7% 7  7  7z %&* QQ#Q Q $ Q  Q QfMM#M M & M^ & 8333  3. 777 7  7  7:+/*.  ((     >+/*.  ((     4(3 r(rSc\eZdZUdZded< ded< edZ d dZd d Zy ) rzVectara Retriever class.rSrr:rT)arbitrary_types_allowedc |jj||jfi|}|Dcgc]\}}| c}}Scc}}wrq)rrr)rNr run_managerrr rrs r)_get_relevant_documentsz(VectaraRetriever._get_relevant_documentss?9$**88 VvV"12Q222s >c <|jj|fi|S)zAdd documents to vectorstore.)r add_documents)rNrrs r)r5zVectaraRetriever.add_documentss -t--iB6BBr(N)rrr2rrrr#r+)rr+rrr#r)) r"r#r$r%r&r model_configr3r5r'r(r)rrsO"+ + $L33*H3TW3 3 Cr(rcZeZdZdZ d ddZ d d dZ d d dZy) rzVectara RAG runnable. Parameters: vectara: Vectara object config: VectaraQueryConfig object chat: bool, default False c<||_||_||_d|_yrq)rrrconv_id)rNrrrs r)rQzVectaraRAG.__init__s     r(Nc +K|jj||j|j|j}|jj j |jjdtj||jjd}|jdk7r?tjdd|jd|jd|jd y g}g}d |i|j!D]}|stj"|j%d } | d } | d} | C| j'dd } | Mt)| j'ddkDr:tjd| j'ddj'd| j'dd } | rR| j'dd r@| d}tj+d||dk(rd |_tjd | r| j'dd nd }|r||_| j'dd r*| j'dij'dd }d|iet-| d}d|i{|jj.r/| dDcgc] }|d|jj.kDr|"}}n| d}| d}g}|D]g}|d Dcic] }|d!|d" }}|d#}||d Dcic] }|d!|d" }}d$|vrd%|d$<|j1||j3|it5||Dcgc]\}}t7|d|&|df}}}|jj8j:d'vr|d |jj<}d(|iy cc}wcc}wcc}wcc}}ww))aGet streaming output from Vectara RAG. Args: input: The input query config: RunnableConfig object kwargs: Any additional arguments Returns: The output dictionary with question, answer and context z&https://api.vectara.io/v1/stream-queryT)r}rr{r~r!rrrrrrNquestionzutf-8rrrrrz&Summary generation failed with status statusDetailrzChat query failed with code RESOURCE_EXHAUSTEDz-Sorry, Vectara chat turns exceeds plan limit.rrrrranswerrrrrrrrnrrrcontext)rrrrr9rfrrvrrrjrr`rrr iter_linesloadsdecoder\lenrrrArrrrrFr4r<)rNinputrrrrrrliner{r response_setrrst_coder9rchunkrrrrrrrrs r)r!zVectaraRAG.streams ||++E4;; 4<<X<<((--LL2248D!LL44 .    3 & LL!--.i7H ==/$    5!!'')N +Dzz$++g"67h%m4 '$jjD9G 7;;x01A5 D&{{84Q7;;NKLN!#;;vt4D4 8"&x. &B7)$LM"&::+/DL"LL O%BFdhh'7>DG'. {{#7>%kk*>CGGQUV$cl*  0E#U++{{22&2*%=% ! zDKK,G,GG% % %1$< ,Z 8I "I& -=>z]Kai73KK"#O"4&/w%7 %C" !fIqz1""$61/8F8, &)!((, -&&)I%> "Ar %-.vY)+gJ  C {{0099>"/DKKMM2$c**]N +^ I%L" s>DOFO(%N> O(O8O O>OO 6AOc ddi}|j|D]V}d|vr |d|d<d|vr |d|d<d|vr|dxx|dz cc<2d|vr |d|d<?tjd|X|S)Nr>r7r?r;rzUnknown chunk type: )r!r`r)rNrDrrrrHs r)invokezVectaraRAG.invokes n[[' =EE!!&y!1Iu$"' "3JU"H x0 %"5\E  3E7;< = r(r%)rrSrr:rrrq)rDrrOptional[RunnableConfig]rrr#zIterator[dict])rDrrrKrrr#r)r"r#r$r%rQr!rJr'r(r)rrsJO(:BF,0vv)v v  vv,0)   r(r)/ __future__rrloggingrZrK dataclassesrrhashlibrtypingrrr r r r r rd langchain_core.callbacks.managerrlangchain_core.documentsrlangchain_core.embeddingsrlangchain_core.runnablesrrlangchain_core.vectorstoresrrpydanticr getLoggerr"r`rrrrr+r2r:rSrrr'r(r)rXs"  (GGG.0=I   8 $'  "     (  * O0O0 O0do ko dC+C0[[r(