h&dZddlZddlZddlmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZerddlmZddlmZdd lmZdd lmZmZmZej4eZGd d eZGd deZGddeZy)zQLoader that uses Playwright to load a page, then uses unstructured to parse html.N)ABCabstractmethod) TYPE_CHECKING AsyncIteratorDictIteratorListOptionalUnion)Document) BaseLoader)Browser)Page)Response)rrrc TeZdZdZedddddddefd Zedd dd dd defd Zy)PlaywrightEvaluatorzAbstract base class for all evaluators. Each evaluator should take a page, a browser instance, and a response object, process the page as necessary, and return the resulting text. pagerbrowserrresponserreturncy)aSynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nselfrrrs q/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/document_loaders/url_playwright.pyevaluatezPlaywrightEvaluator.evaluates AsyncPage AsyncBrowser AsyncResponsec Kyw)aAsynchronously process the page and return the resulting text. Args: page: The page to process. browser: The browser instance. response: The response from page.goto(). Returns: text: The text content of the page. Nrrs revaluate_asyncz"PlaywrightEvaluator.evaluate_async+s  sN)__name__ __module__ __qualname____doc__rstrrr"rrrrrsl   V  i  :  RU        *8  DS      rrcZeZdZdZddeeefdZddddd d d efd Zdd ddd dd efdZ y)UnstructuredHtmlEvaluatorz@Evaluate the page HTML content using the `unstructured` library.Nremove_selectorscL ddl}||_y#t$r tdwxYw)z%Initialize UnstructuredHtmlEvaluator.rNzQunstructured package not found, please install it with `pip install unstructured`) unstructured ImportErrorr*)rr*r,s r__init__z"UnstructuredHtmlEvaluator.__init__?s7  !1  -  s#rrrrrrrcTddlm}|jxsgD]J}|j|j }|D]$}|j s|j d&L|j}||}dj|D cgc] } t| c} Scc} w)z3Synchronously process the HTML content of the page.rpartition_htmlelement => element.remove()text unstructured.partition.htmlr1r*locatorall is_visiblercontentjoinr' rrrrr1selectorelementselement page_sourceels rrz"UnstructuredHtmlEvaluator.evaluateKs>--3 DH||H-113H# D%%'$$%BC D D lln !{3{{h7CG7887s B%rrr cKddlm}|jxsgD]b}|j|j d{}|D]4}|j d{s|j dd{6d|jd{}||}dj|D cgc] } t| c} S77m7U7;cc} ww)z4Asynchronously process the HTML content of the page.rr0Nr2r3r5r6r=s rr"z(UnstructuredHtmlEvaluator.evaluate_asyncYs ?--3 JH!\\(37799H# J ++---!**+HIII J J !LLN* !{3{{h7CG788:-I*7s]: "username": "username", "password": "password" } loader = PlaywrightURLLoader(urls, proxy=proxy) data = loader.load() Nurlscontinue_on_failureheadlessr* evaluatorproxybrowser_sessionc ddl}||_||_||_||_||_|r |r td|xs t||_ y#t$r tdwxYw)z%Load a list of URLs using Playwright.rNzMplaywright package not found, please install it with `pip install playwright`z:`remove_selectors` and `evaluator` cannot be both not None) playwrightr-rFrGrHrJrK ValueErrorr)rI) rrFrGrHr*rIrJrKrMs rr.zPlaywrightURLLoader.__init__s~   #6    .  L  #Q&?@P&Q% +  s A A"rc #Kddlm}|5}|jj|j|j }d}|j rhtjj|j r|j|j }n"tjd|j ||j}|jD]} |j}|j|}|t!d||j#d|j$j'|||}|j)d |i} t+|| |j)dddy#t,$r4} |j.rtj1d |d | n| Yd} ~ d} ~ wwxYw#1swYyxYww) zLoad the specified URLs using Playwright and create Document instances. Returns: A list of Document instances with loaded content. r)sync_playwrightrHrJN storage_stateSession file not found: "page.goto() returned None for url loadsource page_contentmetadataError fetching or processing , exception: )playwright.sync_apirPchromiumlaunchrHrJrKospathexists new_contextloggerwarningrFnew_pagegotorNwait_for_load_staterIrcloser ExceptionrGerror) rrPprcontexturlrrr4rZes r lazy_loadzPlaywrightURLLoader.lazy_loads 8  ! !jj''djj'QGG##77>>$"6"67%11@T@T1UGNN%=d>R>R=S#TU!--/yy  "++-D#yy~H'(+McU)STT,,V4>>224(KDJJL (#H"xHH ( MMOC! ! 4! // ;C5 aSQ   5! ! sHGC F7BE7F7. G7 F4*F/*F7/F44F77G<Gc`K|j2cgc3d{}| 76c}Scc}ww)Load the specified URLs with Playwright and create Documents asynchronously. Use this function when in a jupyter notebook environment. Returns: A list of Document instances with loaded content. N) alazy_load)rdocs raloadzPlaywrightURLLoader.aloads*&*__%677c7777s$.)%# %)%).c Kddlm}|4d{}|jj|j|j d{}d}|j rptjj|j r%|j|j d{}n"tjd|j ||jd{}|jD]} |jd{}|j|d{}|t!d||j#dd{|j$j'|||d{}|j)d{d |i} t+|| |j)d{dddd{y777.77777l7V#t,$r5} |j.rtj1d |d | n| Yd} ~ (d} ~ wwxYw7k7]#1d{7swYyxYww) rrr)async_playwrightNrQrRrTrUrVrWrXr[r\)playwright.async_apirwr^r_rHrJrKr`rarbrcrdrerFrfrgrNrhrIr"rir rjrGrk) rrwrlrrmrnrrr4rZros rrszPlaywrightURLLoader.alazy_loads  :#%# "# "JJ--t}}DJJ-WWGG##77>>$"6"67$+$7$7&*&:&:%8%GNN%=d>R>R=S#TU ' 3 3 55yy  !(!1!1!33D%)YYs^3H'(+McU)STT226:::!%!>!>tWh!WWD**,&& (#H"xHH (--/ ! !G# "# "# "W 643;W&! // ;C5 aSQ    "G# "# "# "# "s H?GH?5H* GAH*(G)%)0437*.BFR3iR"R R #49- R /0 RS#X'R"%R[[-=(=">?RB)8H-)V8T(^8,"-"9,"rrE) r&loggingr`abcrrtypingrrrrr r r langchain_core.documentsr )langchain_community.document_loaders.baser rxrrrrrr r] getLoggerr#rdrr)rErrrrsoW #VVV-@<6>;;   8 $# ## L+9 3+9\^"*^"r