Oh(8 rdZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z m Z mZmZddlmZmZddlmZddlddlmZddlZej2ej4d d Zd Zd Zeej<dxsdZee_ e dZ!e!jEdej<ddZ#e$ej<ddZ%e&ej<ddZ'GddeZ(de$de)de e)de e e*fdZ+d Z,d!e*de e*fd"Z-d#e*de)de.fd$Z/d%e*de e*fd&Z0y)'a- Modified pipeline implementing the user's plan: 1) First pass: extract MCQs and *try to reconstruct mathematical expressions / figure refs* (single prompt function) 2) Second pass: validate/repair each MCQ for logical validity and optionally rephrase while keeping key points 3) Skip any questions that appear to continue on another page (heuristics applied) Outputs structured, Pydantic-validated JSON using an extended MCQ model. Notes: - Keep your OPENAI_API_KEY in .env or env vars as before. - Tweak MODEL_NAME, timeouts and retry values to taste. N)Path)ListOptionalDictAny) BaseModelValidationError)Image)*) load_dotenvz%%(asctime)s %(levelname)s %(message)s)levelformatxg?OPENAI_API_KEY YOUR_KEY_HEREzpublic/debug_outputsT)exist_okOPENAI_VIS_MODELz gpt-4o-miniMAX_OUTPUT_TOKENS2000 TEMPERATUREz0.0ceZdZUeed<eeefed<dZeeed<dZeeed<dZ ee eed<dZ eeed<dZ eeeefed<dZ eeed <y) MCQQuestionOptionsNAnswer Explanation FigureRefs QuestionLatex OptionsLatex AmbiguousMath)__name__ __module__ __qualname__str__annotations__rrrrrrrr r!bool1/var/www/html/eduruby.in/utils/openai_pipeline.pyrr3swM #s(^ FHSM !%K#%&*Jc#*#'M8C='-1L(4S>*1$(M8D>(r)rpage_num page_text figure_pathsreturnc (t}|rdj|}nd}dd|d|d|dg}|D]y}t|}|jrA t |}|r|j dd |id nt jd |d |_t jd |d|{d} | tkrQ d|dd|dg} tjjjt| tt } t)| j*dj,dd} | r;t/d|d| d| ddd t0d|d| dz j3| d !n>t j4d |d"| d#| dz } t#j$t&| zt7| }|Kt9|t:r|gSt9|t<r|St jd |d$t?|gSt j4d |d% d&| d'}d|dd|dg}tjjjt|td(}t)|j*dj,dd}|rLt7|}|%t9|t:r|gSt9|t<r5|St j4d |d*nt j4d |d+| dz } t#j$t&| z| tkrQt j@d |d,gS#t$r)} t jd |d |d| Yd} ~ !d} ~ wwxYw#t$rJ} t jd |d| d| | dz } t#j$t&| zYd} ~ d} ~ wwxYw#t$rY wxYw#t$rG} t jd |d)| | dz } t#j$t&| zYd} ~ \d} ~ wwxYw)-a First pass: ask the model to extract MCQs and attempt to reconstruct math/figure refs. Skips obviously incomplete/continued questions (prompt tells model to skip). Properly sends images in multimodal format, falling back to listing paths if unavailable. z, NonetextzPAGE z TEXT: z IMAGE PATHS: )typer1 image_urlurl)r2r3[Page z(] Could not compress image under limit: z] Image compression failed for : Nz] Missing image file: rsystemrolecontentusermodelmessages max_tokens temperaturez] OpenAI API error on attempt r:zRaw response for page z on attempt ...page_extract_attemptz_raw.txtzutf-8)encodingz ] Empty model output on attempt z . Retrying...z] Unexpected JSON type: z,] Failed to parse JSON; running repair pass.a'The previous reply was not valid JSON. Re-format it into a valid JSON array of objects with keys: Question, Options, optional Answer, optional Explanation, optional FigureRefs, optional QuestionLatex, optional OptionsLatex, optional AmbiguousMath. Return ONLY the JSON array. PREVIOUS_OUTPUT: z z] Repair API error: z2] Repair attempt failed to produce parseable JSON.z ] Repair attempt returned empty.z(] All attempts exhausted; skipping page.)!build_system_prompt_reconstructjoinrexistscompress_image_to_data_uriappendloggingwarning Exception MAX_RETRIESopenaichat completionscreate MODEL_NAMErrtimesleep RETRY_DELAYgetattrchoicesmessageprint DEBUG_DIR write_textinfoextract_json_from_text isinstancedictlistr2error)r+r,r- system_prompt img_list_str user_parts fig_path_strfig_pathdata_urieattemptr>respraw_text parsed_json repair_promptrepair_messages repair_resp repair_textrepaireds r*extract_and_reconstructruAsf 45Myy.  5 *YKGXYeXf!ghJ % Q  % ??  c5h?%%{%QYIZ&[\OOfXJ6^_g^h$ij OOfXJ.DXJO P QG K  !m<J7H;;**11 !,' 2D4<<?22ItD  *8*L HUYVYNK[[^_ ` tH:-=gYhOO[[\dov[w LL6(+KG9Tab c qLG JJ{W, - ,X6  "+t,#}$K.""& 2J4P[K\J] ^_   vhZ'STU '(0j 6 "m<M:O!++1188 (, 9K "+"5"5a"8"@"@)TRK -k:H#h-$:%$/#O vhZ/abc LL6(+KL M1  ;()e K h MMF8*$LMN Iw c& 2QRZQ[[]^_]` abb c$  OOfXJ.LWIUWXYWZ[ \ qLG JJ{W, -     P  OOfXJ.B1#F G qLG JJ{W, -   s\?L&;AM=!N1A)O& M/MM N.$?N))N.1 N>=N> P

rmrkrnparseds r*validate_singlers 12M::hU;LG K  !m<L9H;;**11 ! 2D(- LLA B qLG JJ{W, - (1 > LLD E qLG JJ{W, -  fd #F q(8Zq SW=XAYF fd # 6(: LL6vzz(K7X6YZ [  LL@A G  OO>wir!M N qLG JJ{W, -   s;F G|6L%Mc!f%MH\ " //C 3  !121> &N  3  !12 3  OOfXJ.UVWUXXZ[]Z^_ `  s< E% F1E(FFFF F9F44F9)1__doc__osrer~rVbase64rMconcurrent.futures concurrentpathlibrtypingrrrrpydanticrr PILr utils.extractor_functionsdotenvr rQ basicConfigINFOPER_PAGE_TIMEOUTrPrXgetenvrapi_keyr]mkdirrUintrfloatrrr%rbrurwrr'rrr(r)r*rsv  ,,/'',,/VW  +,? ' (  RYY)= 9   "5v>?IBIImU34 )))wcwcwcwW_`dei`jWkwt8d8x~8xdst.)t)T )r)