hYHBdZddlmZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZdd lmZdd lmZej2eZGd dee ZGddeZGddZGddeeZ GddeeZ!GddeeZ"y)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)AnyOptionalUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)ChainceZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZy) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATIONY/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain/evaluation/schema.pyrrs& B F%JS'O!L7H1@#VH<)7'OPKIKN9=-M"?;#M.#M=-T5Ir?rc,eZdZdZeeddZy) LLMEvalChainz,A base class for evaluators that use an LLM.c y)z#Create a new evaluator from an LLM.Nr>)clsllmkwargss r@from_llmzLLMEvalChain.from_llmNr?N)rEr rFrreturnrB)r&r'r(r) classmethodrrGr>r?r@rBrBKs622r?rBcpeZdZdZeddZeddZed dZed dZ d d dZ y) _EvalArgsMixinz(Mixin for checking evaluation arguments.cyz2Whether this evaluator requires a reference label.Fr>selfs r@requires_referencez!_EvalArgsMixin.requires_referenceWr?cy)0Whether this evaluator requires an input string.Fr>rOs r@requires_inputz_EvalArgsMixin.requires_input\rRr?c6d|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r&rOs r@_skip_input_warningz"_EvalArgsMixin._skip_input_warningas$DNN$;$;#<^^,,--GHCS/ !  d&9&9 ))a 8  " "y'8^^,,--JKCS/ !  )@)@ --! <*A r?rIboolrIstr)NN)ra Optional[str]rbrirINone) r&r'r(r)propertyrQrUrZr\rdr>r?r@rLrLTs~2 VV  $( $= ==  =r?rLceZdZdZed dZed dZeddd d dZddd d dZ ddd d dZ ddd d d Z y)StringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.c.|jjS)zThe name of the evaluation.rXrOs r@evaluation_namezStringEvaluator.evaluation_names~~&&&r?cyrNr>rOs r@rQz"StringEvaluator.requires_referencerRr?Nrainputc y)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr>rP predictionrarrrFs r@_evaluate_stringsz!StringEvaluator._evaluate_stringsrHr?cTKtd|jf|||d|d{S7w)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrurarr)rrvrts r@_aevaluate_stringsz"StringEvaluator._aevaluate_stringssE,%   " " "       s (&(c T|j|||jd|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rarbrxr>)rdrvrts r@evaluate_stringsz StringEvaluator.evaluate_stringssD$ ##i#F%t%% !    r?cpK|j|||jd|||d|d{S7w)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r{rxNr>)rdryrts r@aevaluate_stringsz!StringEvaluator.aevaluate_stringssR$ ##i#F,T,, !      s -646rgre) ruzUnion[str, Any]raOptional[Union[str, Any]]rrrrFrrIdict) rurhrarirrrirFrrIr) r&r'r(r)rkrorQrrvryr|r~r>r?r@rmrms; '' 04+/  $ -  )       604+/  $ -  )      F$(#   !        <$(#   !        r?rmceZdZdZeddd ddZddd ddZddd ddZddd ddZy) PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nrqc y)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr>rPru prediction_brarrrFs r@_evaluate_string_pairsz.PairwiseStringEvaluator._evaluate_string_pairsrHr?cVKtd|jf||||d|d{S7w)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrurrarr)rrrs r@_aevaluate_string_pairsz/PairwiseStringEvaluator._aevaluate_string_pairssH(%   ' ' "%      )')c V|j|||jd||||d|S)rr{rr>)rdrrs r@evaluate_string_pairsz-PairwiseStringEvaluator.evaluate_string_pairs8sG( ##i#F*t** !%     r?crK|j|||jd||||d|d{S7w)rr{rNr>)rdrrs r@aevaluate_string_pairsz.PairwiseStringEvaluator.aevaluate_string_pairsUsU( ##i#F1T11 !%       .757) rurhrrhrarirrrirFrrIr) r&r'r(r)rrrrrr>r?r@rrs8N $(#     !        4$(#     !       F$(#     !       D$(#     !       r?rceZdZdZed dZedd d dZdd d dZdd d dZ dd d dZ y) AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.cy)rTTr>rOs r@rUz'AgentTrajectoryEvaluator.requires_inputvsr?N)rac y)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nr>rPruagent_trajectoryrrrarFs r@_evaluate_agent_trajectoryz3AgentTrajectoryEvaluator._evaluate_agent_trajectory{rHr?cVKtd|jf||||d|d{S7w)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)rurrarr)rrrs r@_aevaluate_agent_trajectoryz4AgentTrajectoryEvaluator._aevaluate_agent_trajectorysH*%   + + "-      rc V|j|||jd||||d|S)rr{rurrrrar>)rdrrs r@evaluate_agent_trajectoryz2AgentTrajectoryEvaluator.evaluate_agent_trajectorysG* ##i#F.t.. !-     r?crK|j|||jd||||d|d{S7w)rr{rNr>)rdrrs r@aevaluate_agent_trajectoryz3AgentTrajectoryEvaluator.aevaluate_agent_trajectorysU* ##i#F5T55 !-       rre) rurhrz!Sequence[tuple[AgentAction, str]]rrrhrarirFrrIr) r&r'r(r)rkrUrrrrrr>r?r@rrss@6 $(   <    !      8$(   <    !     J$(   <    !     H$(   <    !     r?r)#r) __future__rloggingabcrrcollections.abcrenumrtypingrr r warningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configrlangchain.chains.baser getLoggerr&loggerrhrrBrLrmrrr>r?r@rs9"#$''-<;'   8 $3JC3Jl2520=0=fw ncw to nco dx ~sx r?