h*dZddlmZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddZddZ GddeeeZ!GddeeeZ"Gdde"Z#y)z-LLM Chains for evaluating question answering.) annotationsN)Sequence)AnyOptional) Callbacks)BaseLanguageModel)PromptTemplate) ConfigDict)override)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYctjd|jtj}|rF|j dj dk(ry|j dj dk(ry |jj djtjddtj}|j dk(ry|j dk(ry|jj d jtjddtj}|j dk(ry|j dk(ry y#t$rYywxYw) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translatestr maketransstring punctuation IndexError)textmatch first_word last_words `/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain/evaluation/qa/eval_chain.py _get_scorer+sC II5tzz|R]] SE ;;q>   !Y . ;;q>   ![ 0! JJL    # - -cmmBFDVDV.W X      *     ,! JJL UWR  Ys}}RV-?-?@ A  ??  ) ??  +! ,     s&=A&E5$E58A&E5E55 FFc^|j}t|}|d\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. )NN) reasoningvaluescore)rr+)r&r- parsed_scoresr.r/s r*_parse_string_eval_outputr13sD Iy)M! u$ u cFeZdZUdZdZded<edZeddZ e ddZ e dd Z e dd Z e d dd Z dd d  ddZddZed d d dd ddZed d d dd ddZy ) QAEvalChainz,LLM Chain for evaluating question answering.resultsr! output_keyignoreextracyNFclss r*is_lc_serializablezQAEvalChain.is_lc_serializableRr2cy)N correctnessr<selfs r*evaluation_namezQAEvalChain.evaluation_nameVsr2cyNTr<rCs r*requires_referencezQAEvalChain.requires_referenceZr2cyrGr<rCs r*requires_inputzQAEvalChain.requires_input^rIr2Nc |xst}hd}|t|jk7rd|d|j}t||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. >queryanswerresultInput variables should be , but got llmpromptr<)rsetinput_variables ValueError)r>rSrTkwargsexpected_input_varsmsgs r*from_llmzQAEvalChain.from_llmbsm,!6; #f&<&<"= =,-@,AB!1124 S/ !4s64V44r2 callbacksct|Dcgc]\}}|||||||d} }}|j| |Scc}}w)5Evaluate question answering examples and predictions.rMrNrOr\ enumerateapply) rDexamples predictions question_key answer_keyprediction_keyr]iexampleinputss r*evaluatezQAEvalChain.evaluatesd"(1  7 !.!*-%a.8   zz&Iz66 Acft||j}t|vr|t|t<|SNr1r6rrDrO parsed_results r*_prepare_outputzQAEvalChain._prepare_output11&2IJ f %+G_M' "r2F referenceinputr]include_run_infoc B||||d||}|j|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r`r]rxrsrD predictionrvrwr]rxrXrOs r*_evaluate_stringszQAEvalChain._evaluate_stringss72#$   - ##F++r2cpK|j|||d||d{}|j|S7w)Nr`rkr]rxacallrsr|s r*_aevaluate_stringszQAEvalChain._aevaluate_stringssHzz"i:N-"   ##F++  646returnboolrr!ro)rSrrTOptional[PromptTemplate]rXrrr4r`)rdSequence[dict]rerrfr!rgr!rhr!r]rr list[dict]rOdictrrr}r!rv Optional[str]rwrr]rrxrrXrrr)__name__ __module__ __qualname____doc__r6__annotations__r model_config classmethodr?propertyrErHrKr[rlrsr r~rr<r2r*r4r4Is6JL,05 5)5 5  55F$"& 7 $7 7$7 7  7  77 7,  $(##!&!,!,! !,  !,  !,!,!, !,!,F $(##!&,,! ,  ,  ,,, ,,r2r4cHeZdZdZeddZeddZeddZe dZ eddZ eddZ e d dd Z dd d  dd Zdd Zed d d dd ddZed d d dd ddZy )ContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextcyr;r<r=s r*r?z%ContextQAEvalChain.is_lc_serializabler@r2cy)z.Whether the chain requires a reference string.Tr<rCs r*rHz%ContextQAEvalChain.requires_referencer2cy)z+Whether the chain requires an input string.Tr<rCs r*rKz!ContextQAEvalChain.requires_inputrr2r7r8cvhd}|t|jk7rd|d|j}t|y)N>rMrOcontextrPrQ)rUrVrW)r>rTrYrZs r*_validate_input_varsz'ContextQAEvalChain._validate_input_varssQ< #f&<&<"= =,-@,AB!1124 S/ ! >r2cy)NzContextual Accuracyr<rCs r*rEz"ContextQAEvalChain.evaluation_names$r2Nc N|xst}|j||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rRr<)r rr>rSrTrXs r*r[zContextQAEvalChain.from_llms1,)>   (4s64V44r2r\ct|Dcgc]\}}|||||||d} }}|j| |Scc}}w)r_rMrrOr\ra) rDrdrerf context_keyrhr]rirjrks r*rlzContextQAEvalChain.evaluatesd"(1  7 !.";/%a.8   zz&Iz66 rmcft||j}t|vr|t|t<|Srorprqs r*rsz"ContextQAEvalChain._prepare_output(rtr2Fruc B||||d||}|j|S)Nrrzr{r|s r*r~z$ContextQAEvalChain._evaluate_strings.s7$$   - ##F++r2cpK|j|||d||d{}|j|S7w)Nrrrr|s r*rz%ContextQAEvalChain._aevaluate_stringsDsHzz"yJO-"   ##F++  rr)rTr rNonerro)rSrrTrrXrrrr)rdrrerrfr!rr!rhr!r]rrrrr)rrrrrr?rrHrKr rrrEr[rlrsr r~rr<r2r*rrs=L""%%,05 5)5 5  55:$$& 7 $77 7 7  7  77 7,  $(##!&,,! ,  ,  ,,, ,,* $(##!&,,! ,  ,  ,,, ,,r2rcXeZdZdZeddZeddZe d d dZy) CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.cyr;r<r=s r*r?z!CotQAEvalChain.is_lc_serializableZr@r2cy)NzCOT Contextual Accuracyr<rCs r*rEzCotQAEvalChain.evaluation_name^s(r2Nc N|xst}|j||d||d|S)zLoad QA Eval Chain from LLM.rRr<)rrrs r*r[zCotQAEvalChain.from_llmbs1%:   (4s64V44r2rrro)rSrrTrrXrrr) rrrrrr?rrEr[r<r2r*rrWsjG)),0 5  5) 5 5  5 5r2r)r&r!rzOptional[tuple[str, int]])r&r!rr)$r __future__rrr#collections.abcrtypingrrlangchain_core.callbacksrlangchain_core.language_modelsrlangchain_core.promptsr pydanticr typing_extensionsr langchain.chains.llmr #langchain.evaluation.qa.eval_promptr rrlangchain.evaluation.schemarrlangchain.schemarr+r1r4rrr<r2r*rsr3" $ .<1&)RRE$:,I,(O\I,X,?L,D5'5r2