ohs:ddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZddlmZmZddlmZddlZddlZddlmZddlZej4ej6dd Zd Zd Zeej>d xsd Z e e_!e dZ"e dZ#e dZ$e$jKdej>ddZ&e'ej>ddZ(e)ej>ddZ*e dZ+dZ,dZ-GddeZ.de e/dee0fd Z1d!e0de efd"Z2d#e/de/fd$Z3d4d%e e/fd&Z4d4d'e0fd(Z5d)Z6ddlmZddl7Z7ddlZddl7Z7ddlZddlZddlm Z dd*l m Z dd+lmZm8Z8e'ej>d,d-Z9dd.lmZm8Z8m:Z:e9fd/e d0e'de e0fd1Z;de0fd2Z*1$)M8D>)r+rresultsreturnct}|s|S|D]]}t|ts|jdxs$|jdxs|jd}|sOt|tro|D]i}t|t st jj|}|jddjdd}|j|kt|t s|jdD]k}|j}|st jj|}|jddjdd}|j|m`|S)z Extract all unique figure references from the final results. Handles different formats of FigureRefs in the results. r figureRefs figure_refs?r#,) set isinstancedictgetlistr'ospathbasenamesplitaddstrip)r-referenced_imagesresultr1reffilenames r,&extract_figure_references_from_resultsrD;sJ    8&$' jj.g&**\2JgfjjYfNg +t,&8C!#s+#%77#3#3C#8#+>>##6q#9#?#?#DQ#G)--h7 8K-&,,S18C))+C#%77#3#3C#8#+>>##6q#9#?#?#DQ#G)--h7 8%82 r+textc tj|S#tj$rYnwxYwgd}|D]`}tj||tj }|D]4} tj|ccS#tj$rY2wxYwb |j d}|jddz}|dk\r||kDrtj|||Sy#tj$rYywxYw)zGExtract JSON from text, handling various formats including code blocks.)z```(?:json)?\s*([\s\S]*?)\s*```z```\s*([\s\S]*?)\s*```z `([^`]*)`{}rN)jsonloadsJSONDecodeErrorrefindall IGNORECASEfindrfind)rE json_patternspatternmatchesmatchstartends r,extract_json_from_textrX`s zz$      M !**WdBMM: E zz%((''     #jjo! A:#+::d5o. .       s1--$A==BBAC""C87C8mcq_datac|jdi}t|tr@i}gd}t|D]%\}}|t |kst ||||<'||d<|St|t rOi}gd}t|jD](\}\}}|t |kst ||||<*||d<|S)z@Normalize the Options field to be a dictionary with letter keys.r)ABCDEFGH)r8r6r9 enumeratelenr'r7items)rYoptions normalizedlettersioptionkeyvalues r,normalize_options_fieldrmsll9b)G'4  :"7+ 5IAv3w<),V 71:& 5) O GT " :(9 4OA|U3w<),U 71:& 4) Or+ final_resultscV tjjtrPt j ttj tdtjdt tjjtrAt|}tjdt|dt|t}gd }|D]}t!j tjj#t|D]R}tjj%|s#tjj'|}|j)|T||z }d } |D]x} tjj#t| } tjj|r2tj*|tjd | | d z } ztj,tD]} tjj#t| } tjj/| sG tj,| s.tj0| tjd| ntjd| tjdt|d| dyy#t$r*}tjdtd|Yd}~d}~wwxYw#t$r&}tjd | d|Yd}~d}~wwxYw#t$r&}tjd| d|Yd}~^d}~wwxYw#t$r)}tjdtd|Yd}~yd}~wwxYw)z Clean up directories while preserving images referenced in final results. - Always clean debug_outputs completely - Only clean crops_out images that are not referenced in final results TrzCleaned up debug directory: zFailed to clean up : NzFound z referenced images: )z*.jpgz*.jpegz*.pngz*.bmpz*.tiffz*.webprzRemoved unreferenced image: rIzFailed to remove image zRemoved empty subdirectory: z Keeping non-empty subdirectory: zFailed to remove subdirectory z"Cleaned crops_out directory. Kept z referenced images, removed z unreferenced images.)r:r;existsDEBUG_OUTPUTS_DIRshutilrmtreemakedirslogginginfo Exceptionwarning CROPS_OUT_DIRrDrdr9r5globjoinisfiler<r>removelistdirisdirrmdir) rner@ all_imagesimage_patternsrSimg_pathrCimages_to_delete deleted_countimg_nameitem item_paths r,cleanup_directoriesrs H 77>>+ , MM+ , KK)D 9 LL78I7JK L 1D 77>>- ( F} U  LL6#&7"8!99MdSdNeMfg hJVN) 1 $ "'',,}g*N O1Hww~~h/#%77#3#3H#="x01 1 *,== M, O77<< x@Oww~~h/ (+ 'CH:%NO%*  O =1 VGGLL= 77==+V!zz)4HHY/#LL+Gv)NO#LL+KD6)RS V LL=cBS>T=UUqrrAAVW X[ ) H-.?-@1#FGGHB!OOO&=hZr!$MNNO%V*HbQRPS(TUUV D-m_BqcBCCDsA3K6CM6?A$M6$AL5A!M6AM3'M6 L%L  L ML<6M6<MM6 M3 M.(M6.M33M66 N(?N##N(task_idc |r;tjtjjtd|dng}|j tjtjjtd|j tjtjjtd|D]O} tjj |r-tj|tjd|Qy#t$r%}tjd|d|Yd}~|d}~wwxYw#t$r"}tjd|Yd}~yd}~wwxYw) z< Clean up temporary files created during processing *z*_results.jsonz *_mcqs.jsonzRemoved temp file: zFailed to remove temp file rpNzFailed to clean temp files: ) r{r:r;r|TMP_DIRextendrqr~rvrwrxry)r temp_files temp_filers r,cleanup_temp_filesrs>),IIi(LL#6yk!BC  P  P"=i[1# NOO P <6qc:;;E99E>clg}|D],}t|tsd|vs|j|d.|S)z9Flatten the page-wise results into a single list of MCQs.mcqs)r6r7r) json_datamergedpages r, merge_pagesrs= F( dD !fn MM$v, '( Mr+)r)r ImageOpsMAX_IMAGE_BYTES30000)r r ImageChopsr; max_bytesc tj|jd}tj |j|jd}tj||}|j}|r|j|}|j\}}d} t||| kDrI| t||z } |jt|| zt|| zftj }|jd} t#j$} | j'| dd d t)| j+} | |d zkr| jd}d }d}t-dD]}t#j$}|j'|d|d d|j+}t)||kr+t/j0|j3d}d|cS|dkDrtdt|dz}|j\}}|jt||zt||zftj }y#t$r%}t j d|d|Yd}~yd}~wwxYw)a Compress image to a JPEG data URI under max_bytes. - Auto-crops whitespace - Converts to grayscale if it saves significant space - Aggressive resizing & quality tuning Returns data URI string or None if compression can't produce small enough image. RGBzFailed to open image rpN)rriLJPEG(r)rquality subsamplingg?2g333333?T)rrroptimizeasciizdata:image/jpeg;base64,g?)r openconvertrxrvrynewmodesizer differencegetbboxcropmaxresizeintLANCZOSioBytesIOsaverdgetvaluerangebase64 b64encodedecode)r;rimgrbgdiffbboxwhmax_dim_targetscaleimg_graybuf_gray gray_sizer scale_factor_bufrawb64s r,compress_image_to_data_urir s0jj&&u- 388SXX 7B  b )D <<>D hhtn 88DAqN 1ay>!Q*jj#a%i.#a%i.95==I{{3Hzz|H MM(621MEH%%'(I9s?"u%GL 1X\jjl VW!dSlln s8y ""3'..w7C,SE2 2 R<"c'C-01G88DAq**c!l"23S\9I5JKU]][C\ _ /vRs;<s$I I<I77I<cy)zKBuild the system prompt for the first pass (extraction and reconstruction).aYou are an expert at extracting multiple choice questions (MCQs) from text. Your task is to: 1. Extract all complete MCQs from the provided text 2. Reconstruct mathematical expressions using LaTeX when possible 3. Associate figures with questions when appropriate 4. Skip questions that are clearly incomplete or continue on another page Return a JSON array of objects with the following structure for each MCQ: { "Question": "The question text", "Options": {"A": "Option A", "B": "Option B", ...}, "Answer": "Correct answer letter (A, B, C, etc.) if available", "Explanation": "Explanation if available", "FigureRefs": ["List of figure names if applicable"], "QuestionLatex": "LaTeX version of the question if it contains math", "OptionsLatex": {"A": "LaTeX for option A", ...} if options contain math, "AmbiguousMath": true/false if mathematical notation is ambiguous } If no MCQs are found, return an empty array []. r*r*r+r,build_system_prompt_reconstructrKs r+cy)zDBuild the system prompt for the second pass (validation and repair).aYou are an expert at validating and repairing multiple choice questions (MCQs). Your task is to: 1. Validate the logical consistency of each MCQ 2. Repair any issues with formatting, clarity, or completeness 3. Return a valid JSON object for each MCQ (even if you make changes) 4. Only skip questions that are fundamentally flawed or incomplete Return a JSON object with the same structure as the input, or if the question should be skipped: {"skip": true, "reason": "Explanation for skipping"} If the question is valid, return it in the proper format: { "Question": "The question text", "Options": {"A": "Option A", "B": "Option B", ...}, "Answer": "Correct answer letter (A, B, C, etc.) if available", "Explanation": "Explanation if available", "FigureRefs": ["List of figure names if applicable"], "QuestionLatex": "LaTeX version of the question if it contains math", "OptionsLatex": {"A": "LaTeX for option A", ...} if options contain math, "AmbiguousMath": true/false if mathematical notation is ambiguous } r*r*r+r,build_system_prompt_validatercs r+)N)>r:rMrJtimerrvconcurrent.futures concurrentpathlibrtypingrrrrrpydanticr r PILr rsr{dotenvr openai basicConfigINFOPER_PAGE_TIMEOUT MAX_RETRIES RETRY_DELAYgetenvrapi_keyINPUT_PAGES_JSONOUTPUT_MCQ_JSON DEBUG_DIRmkdir MODEL_NAMErrfloatrrrzrrrr7r'rDrXrmrrrrrrrrrrr*r+r,rs 11/ ',,/VW  +,?)*'  !  RYY)8 4   "5v>?IBIImU34 | * * *) *#DJ#3s8#J!!#!Hdt0ADtDzADF<<* ibii 17;<++