h=ddlZddlmZmZmZmZmZmZmZddl m Z ddl m Z ddl mZddlmZddlmZGdd eZy) N)Any AsyncIteratorIteratorListOptionalSetUnion)urlparse) BeautifulSoup)Document) BaseLoader) WebBaseLoaderceZdZdZ d!ddddededeededed ed eed eeefd Zd edefdZ d"de ed ededefdZ de ee efde fdZdedefdZdede efdZ d#dedeedee de efdZ d#dededeedee de ef dZdeefdZdeefdZ d#dedeedeefdZdede efd Zy)$ GitbookLoadera Load `GitBook` data. 1. load from either a single page, or 2. load all (relative) paths in the sitemap, handling nested sitemap indexes. When `load_all_paths=True`, the loader parses XML sitemaps and requires the `lxml` package to be installed (`pip install lxml`). N) sitemap_urlallowed_domainsweb_pageload_all_pathsbase_urlcontent_selectorcontinue_on_failure show_progressrrc|xs||_|jjdr|jdd|_||_||_||_||_||_||_|jt|j} | r| h|_|r|xs|jd|_ n||_ |j|js%td|jd|jy)aTInitialize with web page and whether to load all paths. Args: web_page: The web page to load or the starting point from where relative paths are discovered. load_all_paths: If set to True, all relative paths in the navbar are loaded instead of only `web_page`. Requires `lxml` package. base_url: If `load_all_paths` is True, the relative paths are appended to this base url. Defaults to `web_page`. content_selector: The CSS selector for the content to load. Defaults to "main". continue_on_failure: whether to continue loading the sitemap if an error occurs loading a url, emitting a warning instead of raising an exception. Setting this to True makes the loader more robust, but also may result in missing data. Default: False show_progress: whether to show a progress bar while loading. Default: True sitemap_url: Custom sitemap URL to use when load_all_paths is True. Defaults to "{base_url}/sitemap.xml". allowed_domains: Optional set of allowed domains to fetch from. If None (default), the loader will restrict crawling to the domain of the `web_page` URL to prevent potential SSRF vulnerabilities. Provide an explicit set (e.g., {"example.com", "docs.example.com"}) to allow crawling across multiple domains. Use with caution in server environments where users might control the input URLs. /Nz /sitemap.xmlz Domain in z% is not in the allowed domains list: ) rendswithrrrrrrr netloc start_url_is_url_allowed ValueError) selfrrrrrrrrinitial_domains j/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/document_loaders/gitbook.py__init__zGitbookLoader.__init__sJ!,H == ! !# & MM#2.DM  , 0#6 *.    '%h/66N(6'7$ (Jt}}o\,JDN%DN##DNN3T^^,,Q''(* 4urlreturnc|jy t|}|jdvry|jsy|j|jvS#t$rYywxYw)z0Check if a URL has an allowed scheme and domain.F)httphttps)rr schemer Exception)r!r&parseds r#rzGitbookLoader._is_url_allowedYsf    ' c]F}}$55====D$8$88 8  sA AA AAurl_listurl_typec|j|r|j|ytjd|d|y)aBSafely add a URL to a list if it's from an allowed domain. Args: url_list: The list to add the URL to url: The URL to add url_type: Type of URL for warning message (e.g., "sitemap", "content") Returns: bool: True if URL was added, False if skipped TzSkipping disallowed z URL: F)rappendwarningswarn)r!r.r&r/s r# _safe_add_urlzGitbookLoader._safe_add_urlps>    $ OOC  MM0 &F Gr% url_or_urlscFt||j|jS)zCreate a new WebBaseLoader instance for the given URL(s). This ensures each operation gets its own isolated WebBaseLoader. )web_pathrr)rrr)r!r5s r#_create_web_loaderz GitbookLoader._create_web_loaders&  $ 8 8,,  r%soupc(|jdduS)z+Check if the soup contains a sitemap index. sitemapindexN)find)r!r9s r#_is_sitemap_indexzGitbookLoader._is_sitemap_indexsyy(44r%c|jd}g}|D]@}|jd}|s|js$|j||jdB|S)z*Extract sitemap URLs from a sitemap index.sitemaploc)find_allr<textr4)r!r9 sitemap_tagsurlsr?r@s r#_extract_sitemap_urlsz#GitbookLoader._extract_sitemap_urlssX}}Y/ # >G,,u%Csxx""49= > r%processed_urls web_loaderc6||j|j}|j|r|j|}g}|D]}||vrt j d| |j | |j}|g|_|jd}||_|j|||} |j| |S|j|S#t$r3} |jrt j d|d| nYd} ~ d} ~ wwxYw)aOProcess a sitemap, handling both direct content URLs and sitemap indexes. Args: soup: The BeautifulSoup object of the sitemap processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None Nz(Skipping already processed sitemap URL: lxml-xmlparserError processing sitemap : )r8rr=rEr2r3add web_pathsscrape_process_sitemapextendr,r _get_paths) r!r9rFrG sitemap_urlsall_content_urlsroriginal_web_paths sitemap_soup content_urlses r#rQzGitbookLoader._process_sitemaps@  00@J  ! !$ '55d;L! +  .0MMB;-P"";/)3)=)=&,7=J($.#4#4J#4#GL,>J($(#8#8$nj$L%++L91 >$ #??4( (!// (A+bQRPS&TUVs6AC D%)DDcK||j|j}|j|r|j|}g}|Dcgc] }||vs| }}|sgS|j} ||_|j |dd{} | |_t || D]D\} } |j|  |j| |||d{} |j| F|S|j|Scc}w7v71#t$r3}|jrtjd| d|nYd}~d}~wwxYww)a^Async version of _process_sitemap. Args: soup: The BeautifulSoup object of the sitemap base_url: The base URL for relative paths processed_urls: Set of already processed URLs to avoid cycles web_loader: WebBaseLoader instance to reuse for all requests, created if None NrIrJrLrM)r8rr=rErO ascrape_allziprN_aprocess_sitemaprRr,rr2r3rS)r!r9rrFrGrTrUr&new_urlsrVsoupsrrWrXrYs r#r]zGitbookLoader._aprocess_sitemapsm"  00@J  ! !$ '55d;L! (4Qs.7PQHQ ",!5!5 #+J %00*0MME$6J -05-A ) \"";/ )-)?)?$h *$L%++L9 $ #??4( (CRN$!// (A+bQRPS&TUVs`AD? C7C7/D?C<.D?5D C> D"D?>D D< )D72D?7D<<D?c#K|jsO|j|j}|j}|j ||j}|r|yy|j|j }|jd}t }|j||}|s.|jr"tjd|j g}|D]}|j||d|sy|j|} | j|} t| |D]\}}|j ||}|s| yw)zDFetch text from one single GitBook page or recursively from sitemap.rIrJ$No content URLs found in sitemap at contentN)rr8rrP _get_documentrsetrQrr2r3r4 scrape_allr\) r! temp_loaderr9doc soup_inforFrelative_pathsrDr&content_loader soup_infoss r# lazy_loadzGitbookLoader.lazy_loadsD""11$--@K%%'D$$T4==9C 11$..AK#****=I(+uN!229nMN!d&8&8 DT^^DTUV!D% 9""4i8 9"44T:N(2248J"%j$"7  3((C8I s DC 11$..AK%114>>2B:1VVEaI(+uN#'#9#94==.$N"d&8&8 DT^^DTUV!D% 9""4i8 9"44T:N .99$??J"%j$"7 $ 3 ..y#> (#O $ICW (@sPAF F A*F4F53F(F)A8F!F"*F FFFF custom_urlc|j|j}|sy|jdj}|jd}|r |jnd}|xs |j |d}t ||S)z,Fetch content from page and return Document.N ) separatorh1)sourcetitle) page_contentmetadata)r<rget_textstriprBrr )r!r9rppage_content_rawrbtitle_if_existsrwrys r#rczGitbookLoader._get_documentmsz 99T%:%:;"++d+;AAC*//5(7$$R(9DMMEJWx@@r%cg}|jdD]*}|js|j|j,|S)zFetch all URLs in the sitemap.r@)rArBr1)r!r9rDr@s r#rSzGitbookLoader._get_pathszs?==' &Cxx CHH%  &  r%)FNmainFT)URL)N)__name__ __module__ __qualname____doc__strboolrrr$rrr4r rr8r r=rErQr]rr rlrrorrcrSr%r#rr s %"& &$)"A&*.2AAA3- A  A " AAc]A"#c(+AF340>CS (+7: (  eCcN.C   5m55-DI/3 9)9)C9)]+ 9) c 9)@/3 ;););)C ;) ]+ ;) c ;)z(8H-(T,$-"9,$^6: A A%-c] A (  AstCyr%r)r2typingrrrrrrr urllib.parser bs4r langchain_core.documentsr )langchain_community.document_loaders.baser -langchain_community.document_loaders.web_baserrrr%r#rs.KKK!-@GvJvr%