hdZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZmZmZddlmZddlmZdd lmZ dd lmZmZmZmZddlZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&ejNe(Z)Gd deZ*GddedZ+GddeZ,GddedZ-GddZ.ee,e-e/fZ0GddeZ1ee1e/fZ2Gdde.Z3 d'dZ4dZ5d(dZ6Gd d!Z7 d)d"Z8 d*d#Z9 d+d$Z: d,d%Z;ee e ejLe ejJgee,e-ffe ey#e$rdd lmZmZmZmZY)wxYw).z?This module contains the evaluator classes for evaluating runs.) annotationsN)abstractmethod) AwaitableSequence)AnyCallableLiteralOptionalUnioncast) TypedDict run_helpers)schemas) BaseModelFieldValidationError validator)wraps) SCORE_TYPE VALUE_TYPEExampleRunc(eZdZUdZded< ded<y)Categoryz$A category for categorical feedback.Optional[Union[float, int]]valuestrlabelN__name__ __module__ __qualname____doc____annotations__\/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.pyrr/s. &&C J&r'rc@eZdZUdZded< ded< ded< ded<y ) FeedbackConfigziConfiguration to define a type of feedback. Applied on on the first creation of a feedback_key. z0Literal['continuous', 'categorical', 'freeform']typerminmaxz%Optional[list[Union[Category, dict]]] categoriesNr r&r'r(r*r*8s* ;: $$; $$A55r'r*F)totalceZdZUdZded< dZded< dZded< dZd ed < dZd ed < e e Z ded< dZ ded< dZ ded< dZded< dZd ed< GddZedddZy)EvaluationResultzEvaluation result.rkeyNrscorerrz Optional[str]commentzOptional[dict] correction)default_factorydictevaluator_infoz%Optional[Union[FeedbackConfig, dict]]feedback_configOptional[Union[uuid.UUID, str]] source_run_id target_run_idextraceZdZdZdZy)EvaluationResult.ConfigzPydantic model configuration.FN)r!r"r#r$ allow_extrar&r'r(Configr?bs + r'rAT)prectd|vs|d.t|ttfrtj d||S)z$Check that the value is not numeric.r3zJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluess r(check_value_non_numericz(EvaluationResult.check_value_non_numericgsE & F7O$;!c5\*C! r')r!r"r#r$r%r3rr4r5rr7r8r9r;r<r=rArrLr&r'r(r1r1Gs H@E:0E:8!G]!2!%J%: 6ND65=AO:A;59M29659M29!E> ) wD! " r'r1ceZdZUdZded<y)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsNr r&r'r(rNrNvs $#!r'rNcReZdZdZe d ddZ d ddZy) RunEvaluatorzEvaluator interface class.Ncy)zEvaluate an example.Nr&)selfrunexampleevaluator_run_ids r( evaluate_runzRunEvaluator.evaluate_runsr'cKtjfd}tjj d|d{S7w)z#Evaluate an example asynchronously.ctjdi5jcdddS#1swYyxYw)Nr&)rhtracing_contextrW)current_contextrVrUrTrSsr(_run_with_contextz5RunEvaluator.aevaluate_run.._run_with_contexts@##6o6 I((g7GH I I Is4=N)rZget_tracing_contextasyncioget_running_looprun_in_executor)rSrTrUrVr]r\s```` @r( aevaluate_runzRunEvaluator.aevaluate_runsH002 I I--/??FWXXXXsAA A ANNrTrrUOptional[Example]rVOptional[uuid.UUID]return*Union[EvaluationResult, EvaluationResults])r!r"r#r$rrWrbr&r'r(rQrQs}$&*04 # ###. # 4 ##&*04 Y  Y# Y. Y 4 Yr'rQcHeZdZUdZded< ded< dZded< dZd ed <y) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr2z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr:r;z6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r4)r!r"r#r$r%r;r4r&r'r(rjrjs8 H@ 33459M296FJG CJ:r'rjceZdZdZ d d dZ d ddZ ddZ ddZeddZ d ddZ d dfd Z d dd Z dd Z xZS)DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. ct|\}|rt|\}d fd }t||ddlm}|*|j |||_t |dd|_tj|r8| td|j |||_t |dd|_y|j tttttgt f|||_t |dd|_y) zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. cf|S|jd|jd\}}}|S)NrTrUgetinputs_ traced_inputsprepare_inputss r(process_inputsz4DynamicRunEvaluator.__init__..process_inputss>% $2 5!6::i#8% !Q=! r'rrNrwr!rmFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.rsr7rgr7)_normalize_evaluator_funcr langsmithrensure_traceableafuncgetattr_nameinspectiscoroutinefunction TypeErrorr rrr r_RUNNABLE_OUTPUTfuncrSrr~rwrrvs @r(__init__zDynamicRunEvaluator.__init__s(";4!@~ &?&F #UN ! d D)  $55n6DJ! 4IJDJ  & &t , 3 %55^6DJ!z3HIDJ#44XsHW$568HHI4P-5DI!z3HIDJr'c<ttrjs|_S stddvr|r|jd<t fddDrtdtdid|iS#t $r}td|d}~wwxYw) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r2c3&K|]}|v ywNr&).0kresults r( z@DynamicRunEvaluator._coerce_evaluation_result.. sJq1F?J)r3rr4zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r;z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r&)rDr1r; ValueErrorrallr)rSrr; allow_no_keyes ` r(_coerce_evaluation_resultz-DynamicRunEvaluator._coerce_evaluation_results f. /'''4$M  FFLXOF"| $ u J,IJJ OOUhX$Q &P&PQ Q 44:8=  sAA>> BBBcd|vrB|j}|dDcgc]}|j||c}|d<tdi|S|jtt||dScc}w)NrO)r;T)r;rr&)copyrrNr r7)rSrOr;cprs r(_coerce_evaluation_resultsz.DynamicRunEvaluator._coerce_evaluation_resultss  B!+..q .NByM%*r* *-- w }4.  sA)ct|tr|js||_|St|}|j ||Sr)rDr1r;_format_evaluator_resultr)rSrr;s r(_format_resultz"DynamicRunEvaluator._format_result)sB f. /'''4$M)&1..v}EEr'ct|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. r~hasattrrSs r(is_asynczDynamicRunEvaluator.is_async7tW%%r'ct|dsPtj}|jr t d|j |j ||S|tj}d|ji}t|ddrt|j|d<|j||||d}|j||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.Nr< session_id experimentrun_idmetadatalangsmith_extra)rr_get_event_loop is_running RuntimeErrorrun_until_completerbuuiduuid4idrrrrr)rSrTrUrV running_looprrs r(rWz DynamicRunEvaluator.evaluate_run@s"tV$"113L&&("R $66t7I7I#w7WXX  ##zz| $3SVV#< 3 d +%(%8H\ "  '7XN  ""6+;<rrs r(__repr__zDynamicRunEvaluator.__repr__s&tzzl!44r'r)rXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]r~zIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])F)rzUnion[EvaluationResult, dict]r; uuid.UUIDrboolrgr1)rOzUnion[dict, EvaluationResults]r;rrgrh)rzMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]r;rrgrhrgrrcrd)rTrrUrerVrf)rTrrUrergrhrgr)r!r"r#r$rrrrpropertyrrWrbrr __classcell__)rs@r(rmrms;, 8J 8J 8J|# -!   < / !  4  " F  F ! F 4 F&&&*04 $= $=#$=. $= 4 $=R&*04 = =#=. =D6://!2/ 3/"5r'rmct|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rmrs r( run_evaluatorrs t $$r'i'c^t|}t|tkDr|dtdz dz}|S)Nz...))reprlen_MAXSIZE)objss r( _maxsize_reprrs1 S A 1v n1  & Hr'ceZdZdZ d d dZed dZ d ddZ d ddZ d ddZ ddZ e dd Z dd Z y)DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.Nc t|\}|rt|\}d fd }t||ddlm}|*|j |||_t |dd|_tj|r8| td|j |||_t |dd|_y|j tttttt gt"f|||_t |dd|_y) zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. cf|S|jd|jd\}}}|S)NrunsrUrprrs r(rwz>DynamicComparisonRunEvaluator.__init__..process_inputss>% $2 6"FJJy$9% !Q=! r'rrNrxr!rmryrz)$_normalize_comparison_evaluator_funcrr|rr}r~rrrrrr rrrr r_COMPARISON_OUTPUTrrs @r(rz&DynamicComparisonRunEvaluator.__init__s(("Fd!K~ &J5&Q #UN ! d D)  $55n6DJ! 4IJDJ  & &t , 3 %55^6DJ!z3HIDJ#44!#(9:*,  .5 DI!z3HIDJr'ct|dSrrrs r(rz&DynamicComparisonRunEvaluator.is_asyncrr'cXt|dsPtj}|jr t d|j |j ||Stj}|j|}|j||||d}|j|||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rrrtagsr) rr_rrrr acompare_runsrr _get_tagsr_format_results)rSrrUrr;rrs r( compare_runsz*DynamicComparisonRunEvaluator.compare_runsstV$"113L&&("R $66&&tW5 ~~d#  '4dC  ##FM4@@r'cKt|ds|j||Stj}|j |}|j ||||dd{}|j |||S7w)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. r~rrN)rrrrrr~r)rSrrUr;rrs r(rz+DynamicComparisonRunEvaluator.acompare_runss tW%$$T73 3 ~~d#zz  '4dC"   ##FM4@@  sAA: A8!A:c&|j||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rSrrUs r(rz&DynamicComparisonRunEvaluator.__call__:s  w//r'c"d|jdS)rz>r'cg}|D]^}|jdt|jzt|dds8|jdt|jz`|S)zExtract tags from runs.zrun:rNz experiment:)appendrrrr)rrrTs r(rz'DynamicComparisonRunEvaluator._get_tagsOs` AC KKSVV, -sL$/ MC,??@ A r'ct|tr|js||_|St|tr9t ||Dcic]\}}|j |c}}|j |d}n4t|trd|vr |j |d<nd|}t| tdid|i|Scc}}w#t$r}td||d}~wwxYw)N)rkr2r;r2zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r;zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r&) rDrjr;listziprrr7rr)rSrr;rrTr3msgrs r(rz-DynamicComparisonRunEvaluator._format_resultsZs f8 9'''4$M  %;>tV;LMZS%3665=Mzz!.F  %F" $ u -%+I/ S/ ! -"M >J >J >J@&&AEA!A,=A #A@AEA!A,=A #A:AE0!0,=0 #0"?"="!" " $ "r'rct|S)z.Create a comaprison evaluator from a function.)rrs r(comparison_evaluatorrs ) ..r'c d tj jjDcgc]!\}}|j|j k7s |#}}} jjDcgc]-\}}|j tjjur|/c}} |r6t fd|Ds2t|Dcgc] }| vs| c}dk7rd d}t|t fd|Dr|ddgk(rdfStjrF d fd  dfd }td r td n |j|_|fS d fd dfd }td r td n |j|_|fScc}}wcc}}wcc}w)N)rTrUrsoutputsreference_outputs attachmentsc32K|]}|vxs|vywrr&rpnameargs_with_defaultssupported_argss r(rz,_normalize_evaluator_func..) GLE^ # Bu0B'B B UInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc32K|]}|vxs|vywrr&rs r(rz,_normalize_evaluator_func..)CH>5,>#>>rrTrUc|||r |jni|jxsi|r|jxsini|r|jxsinid}i}g}i}jj D]e\}}||vs |j |j |jfvr|j||n||||<|dvrt||n||||<g|||fSN)rTrUrsrrr)rTrU rsrr parametersitemskindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrr rTrUarg_mapkwargsargsru param_nameparamsigs r(_prepare_inputsz2_normalize_evaluator_func.._prepare_inputs&07gnnR"{{0b@G7#6#6#<"RBI)>Br  " ),)=)=)?%J!W, ::!77!11*!KK (;<181DF:. *-??*'**=>!(!4&j1V]22r'cJK||\}}}|i|d{S7wrr&rTrUr rrtr rs r(awrapperz+_normalize_evaluator_func..awrappers2%4C$A!vq!4262222 #!#r!c|||r |jni|jxsi|r|jxsini|r|jxsinid}i}g}i}jj D]e\}}||vs |j |j |jfvr|j||n||||<|dvrt||n||||<g|||fSrrrs r(r z2_normalize_evaluator_func.._prepare_inputsrr'c.||\}}}|i|Srr&rs r(wrapperz*_normalize_evaluator_func..wrappers&$3C$A!vqT,V,,r')rTrrUrergtuple[list, dict, dict])rTrrUrergrr signaturerrr VAR_KEYWORDdefault Parameteremptyrrrrrrr! rrpall_argsarrrr rr rs ` @@@@r(r{r{sN   D !C&)nn&:&:&<X(%!--@WXHX,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@AF G oLT   Tz  & &t , 3 3#4 3( 3> 3 3#4 3! 34,j)&&   o. . 3 3#4 3( 3> - 4,j)%%   _- -kYEs!F:F:2G G Gc d tj jjDcgc]!\}}|j|j k7s |#}}} jjDcgc]-\}}|j tjjur|/c}} |r6t fd|Ds2t|Dcgc] }| vs| c}dk7rd d}t|t fd|Dr|ddgk(rdfStjrF d fd  dfd }td r td n |j|_|fS d fd  dfd }td r td n |j|_|fScc}}wcc}}wcc}w)NrrUrsrrc32K|]}|vxs|vywrr&rs r(rz7_normalize_comparison_evaluator_func..%rrrrrc32K|]}|vxs|vywrr&rs r(rz7_normalize_comparison_evaluator_func..3rrrrUc|||r |jni|Dcgc]}|jxsic}|r|jxsinid}i}g}i} jjD]e\}}||vs |j|j |j fvr|j||n||||<|dvrt||n||||<g|||fScc}wNr")rrU rsrrrrrrrr rrUrTrrr rur r r s r(r z=_normalize_comparison_evaluator_func.._prepare_inputs=!&07gnnR=ABc 1r 1BBI)>Br  " ),)=)=)?%J!W, ::!77!11*!KK (;<181DF:. *-@@*'**=>!(!4&j1V]22+ CC cJK||\}}}|i|d{S7wrr&rrUr rrtr rs r(rz6_normalize_comparison_evaluator_func..awrapper[s2%4D'$B!vq!4262222rr!c|||r |jni|Dcgc]}|jxsic}|r|jxsinid}i}g}i} jjD]e\}}||vs |j|j |j fvr|j||n||||<|dvrt||n||||<g|||fScc}wr&r'r(s r(r z=_normalize_comparison_evaluator_func.._prepare_inputsjr)r*c.||\}}}|i|Srr&r,s r(rz5_normalize_comparison_evaluator_func..wrappers(%4D'$B!vqT,V,,r')rrrUrergr)rrrUrergrrrs ` @@@@r(rrs-SN   D !C&)nn&:&:&<X(%!--@WXHX,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@AF G oLT  Tz  & &t , 3# 3.? 3( 3< 3# 3.? 3# 34,j)&&   _, , 3# 3.? 3( 3< -# -.? -# -4,j)%%   O+ +kYEs!GG2G G  G c@t|tttfrd|i}|S|st d|t|t r't d|Dst d|dd|i}|St|trd|i}|St|tr |St d|) Nr3zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got c3<K|]}t|tywr)rDr7)rxs r(rz+_format_evaluator_result..s71:a&7sz8Expected a list of dicts or EvaluationResults. Received .rOrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rDrrFrErrrrr7)rs r(rrs&4,-6"* M);;A( D   FD !777J6(RST V$ M FC 6" M FD ! M &&,X /  r'c d tjjjDcgc]\}}| }}}jjDcgc]-\}}|jtj j ur|/c}}|r6t fd|Ds=t|Dcgc] }|vs| c}dk7rd d}|r |d|dz }t|t fd|Dr|dd gk(rS d fd }td rtd |_ |S|j|_ |Scc}}wcc}}wcc}w) Nrexamplesrsrrc32K|]}|vxs|vywrr&rs r(rz/_normalize_summary_evaluator..rrrrr2z Received arguments c3&K|]}|v ywrr&)rrrs r(rz/_normalize_summary_evaluator..s?U.(?rrr5c |||Dcgc]}|jc}|Dcgc]}|jxsic}|Dcgc]}|jxsic}d}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M |i|} t| tr| St| Scc}wcc}wcc}w)Nr4) rsrrrrrrrrDr1r) rr5rUrTrrr r r rrr s r(rz-_normalize_summary_evaluator..wrappers $9ABg7>>B9=>#CKK-2->KS%Tgoo&;&;%T GFD%(^^%9%9%; A! E(zz33--& GJ$78-4Z-@z* A4*6*F&"23 +F3 3)C>%TsC+C0 C5 r!)rzSequence[schemas.Run]r5zSequence[schemas.Example]rgrh) rrrrrrrrrrrrr!) rrrrr rrrr rs ` @@@r(_normalize_summary_evaluatorr9sSN   D !C&)nn&:&:&<=(%=H=,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@ C   )(15 5Co?h? ?8PD  4' 43L 4 7 4:*1z)BGD* % IPHXHX w>Es E %2E; EE)rr)rr)rrrgr)rrrgztuple[Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]], Optional[Callable[..., dict]]])rrrgztuple[Union[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]], Optional[Callable[..., dict]]])rz;Union[EvaluationResults, dict, str, int, bool, float, list]rgzUnion[EvaluationResults, dict])rrrgSUMMARY_EVALUATOR_T)?r$ __future__rr_rrabcrcollections.abcrrtypingrrr r r r typing_extensionsr r|rrZr pydantic.v1rrrr ImportErrorpydanticlogging functoolsrlangsmith.schemasrrrr getLoggerr!rGrr*r1rNrQr7rrjrrmrrrrrr{rrrr:r9r&r'r(rGsE" /(' BB   8 $'y' 6Ye 6,y,^" "YY8)+s E''E<;E<