hM< dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZer dd lmZdd lmZej8eZeGd dZe dddGddeZ y)zModule contains a PDF parser based on Document AI from Google Cloud. You need to install two libraries to use this parser: pip install google-cloud-documentai pip install google-cloud-documentai-toolbox N) dataclass) TYPE_CHECKINGIteratorListOptionalSequence) deprecated)Document) batch_iterate)BaseBlobParser)Blob)get_client_info) OperationDocumentProcessorServiceClientc&eZdZUdZeed<eed<y)DocAIParsingResultsz/Dataclass to store Document AI parsing results. source_path parsed_pathN)__name__ __module__ __qualname____doc__str__annotations__p/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/document_loaders/parsers/docai.pyrrs9rrz0.0.32z1.0z&langchain_google_community.DocAIParser)sinceremovalalternative_importceZdZdZddddddeddeedeedeefd Zd ed ee fd Z d"d ede deedee e d ee f dZ d#deedeede de d ee f dZde ed ee fdZde ed e dfdZde dd e fdZdddd dddeedeedeede de deed e dfd Zde dd e efd!Zy)$ DocAIParserz`Google Cloud Document AI` parser. For a detailed explanation of Document AI, refer to the product documentation. https://cloud.google.com/document-ai/docs/overview N)clientlocationgcs_output_pathprocessor_namer$rr%r&r'cht|t|k(r tdd}|r%tj||std|d||_||_|r||_y ddlm}ddl m }||d  } || td |_y #t$r}td|d }~wwxYw)aInitializes the parser. Args: client: a DocumentProcessorServiceClient to use location: a Google Cloud location where a Document AI processor is located gcs_output_path: a path on Google Cloud Storage to store parsing results processor_name: full resource name of a Document AI processor or processor version You should provide either a client or location (and then a client would be instantiated). zGYou must specify either a client or a location to instantiate a client.z?projects\/[0-9]+\/locations\/[a-z\-0-9]+\/processors\/[a-z0-9]+zProcessor name z has the wrong format. If your prediction endpoint looks like https://us-documentai.googleapis.com/v1/projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID:process, use only projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID part.r) ClientOptionsrZdocumentai package not found, please install it with `pip install google-cloud-documentai`Nz-documentai.googleapis.com) api_endpointz document-ai)module)client_options client_info) bool ValueErrorre fullmatch_gcs_output_path_processor_name_clientgoogle.api_core.client_optionsr)google.cloud.documentair ImportErrorr) selfr$r%r&r'patternr)rexcoptionss r__init__zDocAIParser.__init__2s* <4> )  U ",,w"G!.!12 !0- !DL HR $ (z)CDG:&+=ADL != s% B B1 B,,B1blobreturnc#\K|j|g|jEd{y7w)zParses a blob lazily. Args: blobs: a Blob to parse This is a long-running operation. A recommended way is to batch documents together and use the `batch_parse()` method. r&N) batch_parser3)r9r>s r lazy_parsezDocAIParser.lazy_parsels)##TFDresponses r z-DocAIParser.online_process..sT   .t{{HrDrErFrIrJrKrLr;rVrWrNrhs ` @@ronline_processzDocAIParser.online_processws<$  /    W) 0I J  9C " 4 !<<00  % %))'33 II"mm@/@4!/)-E!#'% &    !))//  I 9   A  : sQDC C)B(DDD C& C!!C&&D) D2 C>>DDblobs timeout_seccheck_in_interval_secc#K|xs |j}|s td|j||}|Dcgc]}|jj}}t j d|d} |j|rUtj|| |z } | |kDrtd|dt j d|j|rU|j|} |j| Ed {y cc}w7 w) aParses a list of blobs lazily. Args: blobs: a list of blobs to parse. gcs_output_path: a path on Google Cloud Storage to store parsing results. timeout_sec: a timeout to wait for Document AI to complete, in seconds. check_in_interval_sec: an interval to wait until next check whether parsing operations have been completed, in seconds This is a long-running operation. A recommended way is to decouple parsing from creating LangChain Documents: >>> operations = parser.docai_parse(blobs, gcs_path) >>> parser.is_running(operations) You can get operations names and save them: >>> names = [op.operation.name for op in operations] And when all operations are finished, you can use their results: >>> operations = parser.operations_from_names(operation_names) >>> results = parser.get_results(operations) >>> docs = parser.parse_from_results(results) :An output path on Google Cloud Storage should be provided.rAz9Started parsing with Document AI, submitted operations %srz#Timeout exceeded! Check operations z later!.) operationsN) r3r0 docai_parse operationrXloggerdebug is_runningtimesleep TimeoutError get_resultsparse_from_results) r9rrr&rsrt output_pathrxopoperation_names time_elapsedresultss rrBzDocAIParser.batch_parses 4&>)>)> L %%e[%I 7AB2<<,,BB G  ooj) JJ, - 1 1Lk)"9/9J'R LL ooj)""j"9**7333C 4s#5DDA?D&D:D; Drc# K ddlm}ddlm}ddlm|D]4 | j\}}|||} fd|DEd{6y#t $r}t d|d}~wwxYw7$w)Nr) split_gcs_uri) _get_shardsrMrOc3K|]T}|jD]C}t|j|j|jj dEVywr])rQr rbrdrer)rgshardr^rNresults rriz1DocAIParser.parse_from_results..sf !KK   !24;; !K&*&6&6&BTBTUsAA)7google.cloud.documentai_toolbox.utilities.gcs_utilitiesr1google.cloud.documentai_toolbox.wrappers.documentrrlrNr8r) r9rrrr;gcs_bucket_name gcs_prefixshardsrNrs @@rrzDocAIParser.parse_from_resultss   V W  F*78J8J*K 'OZ *=F $      A   s3A1A2A1 A/ A1 A, A''A,,A1rrc ddlm}|Dcgc]%}|jj ||'c}S#t$r}td|d}~wwxYwcc}w)z5Initializes Long-Running Operations from their names.r)GetOperationRequestzhlong running operations package not found, please install it with `pip install gapic-google-longrunning`N)rX)request)!google.longrunning.operations_pb2rr8r5 get_operation)r9rrr;rXs roperations_from_namesz!DocAIParser.operations_from_names sn  (  LL & &/B/M & N   :    s9*A A AArxc&td|DS)Nc3>K|]}|j yw)N)done)rgrs rriz)DocAIParser.is_running..s6Rrwwy=6s)any)r9rxs rr}zDocAIParser.is_runnings6:666ri)r&r' batch_sizerDrErc  ddlm}ddlm}m} |xs |j } | td|xs |j}| tdg} t||D]} |j|j| Dcgc]-}|j|j|jxsd /c}  }|j|jj!| | }|r| ||nd}| j#|j$j'|j)||||d| S#t $r} t d| d} ~ wwxYwcc}w)a3Runs Google Document AI PDF Batch Processing on a list of blobs. Args: blobs: a list of blobs to be parsed gcs_output_path: a path (folder) on GCS to store results processor_name: name of a Document AI processor. batch_size: amount of documents per batch enable_native_pdf_parsing: a config option for the parser field_mask: a comma-separated list of which fields to include in the Document AI response. suggested: "text,pages.pageNumber,pages.layout" Document AI has a 1000 file limit per batch, so batches larger than that need to be split into multiple requests. Batch processing is an async long-running operation and results are stored in a output GCS bucket. rrH)rKrLr*Nrvz0A Document AI processor name should be provided.)sizeiterablerRrS) documents) gcs_documents)rTrE)gcs_output_configrP)rVT)rXinput_documentsdocument_output_configrZr[)rjrIrkrKrLr8r3r0r4r BatchDocumentsInputConfig GcsDocumentsrorfrpDocumentOutputConfigGcsOutputConfigappendr5batch_process_documentsBatchProcessRequest)r9rrr&r'rrDrErIrKrLr;rrxbatchr> input_config output_configrZs rryzDocAIParser.docai_parses6  / R&>)>)>  L (?4+?+?  !OP P " UC& E%??(55 %*  ! #..$(II&*mm&H7H/6@ L';;","A"A"Q"Q'J#R#<M- (2K       4422+(4/<(7*. 3 9& No 9  &sD862E8 E E  Ec x ddlm}|Dcgc]}t|j|r|jj n.|j |jjj D]#}t|j|j%c}}S#t$r}td|d}~wwxYwcc}}w)Nr)BatchProcessMetadatar*)rr) google.cloud.documentai_v1rr8 isinstanceraindividual_process_statuses deserializevaluerinput_gcs_sourceoutput_gcs_destination)r9rxrr;rstatuss rrzDocAIParser.get_resultsus  G!  bkk+?@ 77)55KK%%--   "33"99       9    sBBB6 B3" B..B3)TNN)Ni<)rrrrrrr=r rr rCr/rintrqrrBrrrr}ryrrrrr#r#&s >B"&)-(, 89:83- 8 "# 8 ! 8t St S(: S+/$(*. F F $(F SM F T#Y' F (  F V*.%' /4~/4"#/4 /4 # /4 (  /4b/0 ( 4 T#Y 4 CT "7T+%6747*.(,*.$(U~U"# U ! U  U$(USMU k Un d;&7 DATrs !DD6-3DBB3F   8 $    ? ` .`  ` r