h#).dZddlZddlmZddlmZmZmZddlm Z ddl m Z m Z m Z mZmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZmZdd l m!Z!ddl"m#Z#edZ$edeZ%dee$de e$ge%fdee$fdZ&GddeZ'y)zo Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion N) defaultdict)HashableIterableIterator)chain)AnyCallableOptionalTypeVarcast)#AsyncCallbackManagerForRetrieverRunCallbackManagerForRetrieverRun)Document) BaseRetriever RetrieverLike)RunnableConfig) ensure_config patch_config)ConfigurableFieldSpecget_unique_config_specs)model_validator)overrideTH)bounditerablekeyreturnc#rKt}|D]$}||x}|vs|j||&yw)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)setadd)rrseeneks [/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain/retrievers/ensemble.py unique_by_keyr&%s> 5D QKA $ HHQKGs77c eZdZUdZeeed<eeed<dZe ed<dZ e e ed<e deefd Zed ed ee efdefd Ze dde de ededeefdZe dde de ededeefdZde dedeefdZde dedeefdZddde dede edeefdZddde dede edeefdZdeeedeefdZ y)EnsembleRetrieveraeRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. Default is 60. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. retrieversweights<cNid_keyrc:td|jDS)z+List configurable fields for this runnable.c3BK|]}|jD]}|ywN) config_specs).0 retrieverspecs r% z1EnsembleRetriever.config_specs..Ns-' i>T>T' 6:D' ' s)rr))selfs r%r1zEnsembleRetriever.config_specsKs"'' "&//'   before)modevaluesc\|jdst|d}d|z g|z|d<|S)Nr*r))getlen)clsr: n_retrieverss r% set_weightszEnsembleRetriever.set_weightsRs;zz)$vl34L!"\!1 2\ AF9  r7inputconfigkwargsc ddlm}t|}|j|j dd|j dd|j dg|j |j di|j }|jd|fd |j d xs|ji|} |j||| }|j|fi||S#t$r}|j|d}~wwxYw) Nr)CallbackManager callbacksverboseFtagsmetadatarHinheritable_tags local_tagsinheritable_metadatalocal_metadatanamerun_name run_managerrC) langchain_core.callbacksrFr configurer=rIrJon_retriever_startget_name rank_fusionon_retriever_end Exceptionon_retriever_error) r6rBrCrDrFcallback_managerrSresultr#s r%invokezEnsembleRetriever.invokeZs =v&*44 JJ{ # JJy%0#ZZ3yy!'J!;==5 :&99   J':4==?   %%eV%TF )K ( (  M   * *1 -  s5C C>'C99C>c XKddlm}t|}|j|j dd|j dd|j dg|j |j di|j }|jd|fd |j d xs|ji|d{} |j||| d{}|j|fi|d{|S7>7#7 #t$r }|j|d{7d}~wwxYww) Nr)AsyncCallbackManagerrGrHFrIrJrKrPrQrR) rTr`rrUr=rIrJrVrW arank_fusionrYrZr[) r6rBrCrDr`r\rSr]r#s r%ainvokezEnsembleRetriever.ainvokesK Bv&/99 JJ{ # JJy%0#ZZ3yy!'J!;==: @,??   J':4==?     ,,'-F/+..  M)    003 3 3  sfB7D*9C8:D*?C>C:C>D*1C<2D*:C><D*> D'D"DD""D''D*queryrSc&|j||S)z Get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. )rXr6rcrSs r%_get_relevant_documentsz)EnsembleRetriever._get_relevant_documentss"{33r7cBK|j||d{S7w)z Asynchronously get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. N)rares r%_aget_relevant_documentsz*EnsembleRetriever._aget_relevant_documentss!"&&uk::::s )rCct|jDcgc]8\}}|j|t||j d|dz:}}}t t |D]B}||Dcgc].}t|trttt|n|0c}||<D|j|Scc}}wcc}w)z Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. retriever_r<tagrG page_content) enumerater)r^r get_childranger> isinstancestrrr weighted_reciprocal_rankr6rcrSrCir3retriever_docsdocs r%rXzEnsembleRetriever.rank_fusions6!*$// :  9   )33*QUG8L3M     s>*+ A*!,!:DC9Md3n5SVV!N1  ,,^<<'  !s =C73C cKtjt|jDcgc]8\}}|j |t ||j d|dz:c}}d{}tt|D]4}||Dcgc] }t|ts t|n|"c}||<6|j|Scc}}w7fcc}ww)z Asynchronously retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. rjr<rkrmNrn) asynciogatherrpr)rbrrqrrr>rsrrurvs r%razEnsembleRetriever.arank_fusions& '~~%.doo$> !Ay!! "-"7"7jQ*+ A*!,!3=S(2Kc*QTT!N1  ,,^<<)   !s.(C=C 'C/C0"C%C7 CC doc_listsct|tjk7r d}t|ttt |jD]f\}}t |dD]Q\}}j |jn|jjxx||jzz z cc<Shtj|}tt|fddfdS)a Perform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: list: The final aggregated list of items sorted by their weighted RRF scores in descending order. zz.Fs4{{*$$dkk2r7Tcpj|jS|jjSr0r)ry rrf_scorer6s r%rz.Ms5I$(KK$7  =@\\$++=Vr7)reverser)r>r* ValueErrorrfloatziprpr-rorJr,r from_iterablesortedr&) r6r}msgdoc_listweightrankryall_docsrs ` @r%ruz*EnsembleRetriever.weighted_reciprocal_rank s y>S. .PCS/ !'2%&8 #It|| < . Hf&xq9 . c ;;.(( \\$++6  tdff}- . . .&&y1    r7r0)!__name__ __module__ __qualname____doc__listr__annotations__rr,intr-r rtpropertyrr1r classmethoddictrrArrrr^rbrrfr rhrXrarur7r%r(r(6s ]## %[ AsK FHSM   d#89  (#c3hC$ ,0""(" " h ""H,0&&(& & h &&P444 4 h 4&;;9 ; h ;0,0 &=&=4&= ( &= h &=Z,0 (=(=9(= ( (= h (=T0 X'0  h0 r7r()(rr{ collectionsrcollections.abcrrr itertoolsrtypingrr r r r rTr rlangchain_core.documentsrlangchain_core.retrieversrrlangchain_core.runnablesrlangchain_core.runnables.configrrlangchain_core.runnables.utilsrrpydanticrtyping_extensionsrrrr&r(rr7r%rs #88.B3G%& CL Cx HQKhsAv.>8A;"Z Z r7