hoddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z mZddlmZmZddlmZmZddlmZddl mZddlmZmZddlmZdd lmZddlZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(d d l)m*Z*m+Z+m,Z,m-Z-m.Z.d dl/m0Z0m1Z1ddl)m2Z2e0rddl3Z3ddlm4Z4m5Z5m6Z6m7Z7d dl8m9Z9m:Z:e$rddl;Z;e(rddle>rddl?Z?ddl@mAZAmBZBddlCmDZDddlEmFZFmGZGddlHmIZIddlJmKZKddlLmMZMddlNmOZOmPZPmQZQmRZRmSZSddlTmUZUddlVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcmdZdmeZeddlfmgZgddlhmiZimjZjmkZkGdd egd!"ZlGd#d$eUd!"ZmGd%d&eKd!"ZnejelZoejemZpejenZqhd'Zrhd(Zshd)Zte1jevZwd*d+d,d-iZxeyexjZ{Gd.d/ejZ}d0efd1Z~d2ed3d4d5d4fd6ZGd7d8ZGd9d:ZeGd;d<ZGd=d>e2Zevd?k(reZej yy)@N)ArgumentParser Namespace) GeneratorIterable) dataclassfield)BytesIO)Thread)OptionalUnion) model_info)HF_HUB_OFFLINE)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES)is_fastapi_availableis_librosa_availableis_openai_availableis_pydantic_availableis_uvicorn_availableis_vision_available) AutoConfigLogitsProcessorListPreTrainedTokenizerFastProcessorMixinTextIteratorStreamer)is_torch_availablelogging)BaseTransformersCLICommand) AutoProcessorBitsAndBytesConfigGenerationConfigPreTrainedModel)ContinuousBatchingManager RequestStatus)Image)FastAPI HTTPException)CORSMiddleware) JSONResponseStreamingResponse) Transcription)TranscriptionCreateParamsBase)ChatCompletionMessageParam)ChatCompletionChunkChoice ChoiceDeltaChoiceDeltaToolCallChoiceDeltaToolCallFunction)CompletionCreateParamsStreaming)ResponseResponseCompletedEventResponseContentPartAddedEventResponseContentPartDoneEventResponseCreatedEvent ResponseErrorResponseErrorEventResponseFailedEventResponseInProgressEventResponseOutputItemAddedEventResponseOutputItemDoneEventResponseOutputMessageResponseOutputTextResponseTextDeltaEventResponseTextDoneEvent)ResponseCreateParamsStreaming) BaseModel TypeAdapterValidationErrorceZdZUdZeed<y))TransformersResponseCreateParamsStreamingz OpenAI's ResponseCreateParamsStreaming with an additional field for the generation config (as a json string). generation_configN__name__ __module__ __qualname____doc__str__annotations__[/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/commands/serving.pyrJrJw rTrJF)totalceZdZUdZeed<y)+TransformersCompletionCreateParamsStreamingz OpenAI's CompletionCreateParamsStreaming with an additional field for the generation config (as a json string). rKNrLrSrTrUrYrY~rVrTrYc:eZdZUdZeed<eed<dZee ed<y)%TransformersTranscriptionCreateParamsz OpenAI's TranscriptionCreateParamsBase with an additional field for the generation config (as a json string). filerKFstreamN) rMrNrOrPbytesrRrQr]r boolrSrTrUr[r[s"  !&&rTr[> textuserstorepromptinclude reasoning background truncation tool_choice service_tier top_logprobsmax_tool_callsprevious_response_id>nstopraaudiorblogprobsmetadata functions modalities predictionrhrirj function_callstream_optionsresponse_formatpresence_penaltyreasoning_effortweb_search_optionsparallel_tool_callsmax_completion_tokens>rcrdlanguagerwchunking_strategytimestamp_granularitiesqwenz z )startendceZdZdZdZdZdZy)ModalityLLMVLMSTTTTSN)rMrNrOrrrrrSrTrUrrs C C C CrTrargsct|S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand ) ServeCommand)rs rUserve_command_factoryrs  rTreqmodel_generation_configr#returnc `|jd"td itj|d}nt j |}|j d i|}|jD]\}}| t||||jdt|d|_ |jdt|d|_ |jdt|d|_ |jd |d|_ |jd |d|_|jd+t|d|_t|ddk(rd |_|jd t|d |_|jd t%j&|d |S) a Creates a generation config from the parameters of the request. If a generation config is passed in the request, it will be used as a baseline for parameterization. Otherwise, we will use the model's default generation config. Other parameters in the request will be applied on top of the baseline. Args: req (`dict`): The request which may optionally contain generation parameters. model_generation_config (`GenerationConfig`): The model's default generation config. kwargs (`dict`): Additional parameters to set in the generation config. Returns: The prepared `GenerationConfig` object. rKmax_output_tokens max_tokensfrequency_penalty logit_biasrn temperaturegFtop_pseedrS)getr#jsonloadscopydeepcopyupdateitemssetattrintmax_new_tokensfloatrepetition_penalty sequence_bias stop_stringsr do_samplertorch manual_seed)rrkwargsrKnon_standard_kwargskvs rU!create_generation_config_from_reqrs. ww"#/,Ttzz#>Q:R/ST MM*AB2+22Reset the tool call state (assumes we're outside a tool call).FrN)inside_tool_callhas_tool_name_definedarg_nesting_levelbufferrs rUrzToolState.reset$s! %%*"!" rTN)rMrNrOrPrrrSrTrUrrsArTrc FeZdZdZ d dddedeedfdZd Zd Z d Z y) TimedModelz A class that holds a PreTrainedModel instance and its associated processor. Automatically deletes the instances after a specified timeout. Nmodelr$timeout_seconds processor)rrc||_t|j|_||_||_t j|j |j|_ |jjyr) rrQ name_or_path _name_or_pathrr threadingTimer _delete_model_timerr)rrrrs rUrzTimedModel.__init__2s[   !3!34".ood&:&:DDelete the wrapped model and processor and clean up resources.rNz was removed from memory after z seconds of inactivity) hasattrrrgccollectrcuda is_available empty_cacheloggerinforrrs rUrzTimedModel._delete_modelEs 4 !djj&< DJ!DN JJLzz&&( &&( KK%%&&EdFZFZE[[qr '= !rTc<t|d xs|jduS)z)Check if the instances have been deleted.rN)rrrs rU is_deletedzTimedModel.is_deletedVs 4))?TZZ4-??rTr) rMrNrOrPrr r rrrrrSrTrUrr,sJSW    E"MNO  "@rTrceZdZUdZedddiZeed<eddgd d Ze eed <edd gd d Z e eed <edddiZ e ed<edddiZ e eed<edddiZe ed<edddiZe ed<eddddgd Zeed<edddiZe ed<edddiZeed<ed dd!iZeed"<ed#dd$iZeed%<ed&dd'iZeed(<eddd)iZe eed*<eddd+iZe ed,<eddd-iZe ed.<eddd/iZe eed0<d1Zy)2ServeArgumentsz Arguments for the serve CLI. See the metadata arg for each argument's description -- the metadata will be printed with `transformers serve --help` autohelpzfDevice to use for inference; will default to `auto` andplace the model on an accelerator if available.)defaultrqdeviceNzA`torch_dtype` is deprecated! Please use `dtype` argument instead.)rbfloat16float16float32)rchoices torch_dtypezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.dtypeFz2Whether to trust remote code when loading a model.trust_remote_codezWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz$Interface the server will listen to.hosti@zPort the server will listen to.porti,z@Time in seconds after which a model will be removed from memory. model_timeoutrz8Logging level as a string. Example: 'info' or 'warning'. log_levelz1The default seed for torch, should be an integer. default_seedztWhether to enable CORS. Some apps that make requests from external domains (e.g. Cursor) require CORS to be enabled. enable_corsz+Whether to turn on strict input validation.input_validationzName of the model to be forced on all requests. This is useful for testing Apps that don't allow changing models in the request. force_modelc`|j"|jdk(r|j|_yyy)z(Only used for BC `torch_dtype` argument.Nr)rrrs rU __post_init__zServeArguments.__post_init__s1    'DJJ&,@))DJ-A 'rT)rMrNrOrPrrrQrRrr rrr_rrrrrrrrrrrrrrrrSrTrUrr[s4 > FC"'WA "K#!PA E8C=$)] ^t*/ r *#efL$efL$ %UFZhmotgu=vww!&uHm?n!o$okV=c4deD#edf6W-XYD#Y\]M3 &*d!eIs#(([\#L(3- & K# B d "' 2 "K#*rTrcDeZdZedefdZdefdZdedddd d e fd Z defd Z defd Z defdZ d1deedeedeedeedeedeeddefdZdddefdZdZej*ddeeeeffdZdedeeddffd Zedd!defd"Zed#efd$Zdedeeddffd%Zdedeeddffd&Zdedeeddffd'Zdede fd(Z!ededed)fd*Z"d+edefd,Z#d-efd.Z$d-ede%d!e&ffd/Z'd-ede%d!e(ffd0Z)y)2rparsercdtf}|jd|}|jty)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments serve)dataclass_types)funcN)r add_parser set_defaultsr)rr serve_parsers rUregister_subcommandz ServeCommand.register_subcommands3*+((/(R !!'<!=rTrcts td||_|jjdk(|_|jj |_|jj )tj|jj tjd}|jtj|jjjtjd}|jtj|jjji|_d|_d|_d|_d|_y)NzaMissing dependencies for the serving CLI. Please install with `pip install transformers[serving]` sdpa_paged transformersz+transformers.generation.continuous_batching)serve_dependencies_available ImportErrorrruse_continuous_batchingrrrrr get_loggersetLevel log_levelsrlower loaded_models#running_continuous_batching_manager last_messages last_kv_cache last_model)rrtransformers_logger cb_loggers rUrzServeCommand.__init__s+s   '+yy'D'D 'T$9900 99 ! ! -   dii44 5&00@$$W%7%7 8K8K8Q8Q8S%TU&&'TU 7--dii.A.A.G.G.IJK57X\0"!rTrequestschema_TypedDictMeta validatorrG unused_fieldsc0tjd|t|j}|j}||z }|r(tj d|t dd||jjrB |j|||z} | r(tj d| t dd| yy#t$rF}tj d|jt d|jd}~wwxYw)a Validates the request against the schema, and checks for unexpected keys. Args: request (`dict`): The request to validate. schema (`_TypedDictMeta`): The schema of the request to validate. It is a `TypedDict` definition. validator (`TypeAdapter`): The validator to use to validate the request. Built from `schema`. unused_fields (`set`): Fields accepted by `schema`, but not used in `transformers serve`. Raises: HTTPException: If the request is invalid or contains unexpected or unused fields. zValidating request: z Unexpected keys in the request: i) status_codedetailzValidation error: NzUnused fields in the request: ) rdebugsetkeys__mutable_keys__errorr)rrvalidate_pythonrHerrors) rrrrr input_keys possible_keysunexpected_keyseunused_fields_in_requests rU_validate_requestzServeCommand._validate_requests.  +G956( // $}4  LL;O;LM NC:Z[jZk8lm m 99 % % H))'2 (2M'A $' =>V=WXY# #.LMeLf,g( &# H 1!((*>?#AHHJGG HsC DADDcF|j|tttyN)rrrr)r$rJresponse_validatorUNUSED_RESPONSE_FIELDSrrs rUvalidate_response_requestz&ServeCommand.validate_response_request!s! <(0  rTcF|j|tttyr&)r$rYcompletion_validatorUNUSED_CHAT_COMPLETION_FIELDSr)s rU validate_chat_completion_requestz-ServeCommand.validate_chat_completion_request)s! >*7  rTcF|j|tttyr&)r$r[transcription_validatorUNUSED_TRANSCRIPTION_FIELDSr)s rUvalidate_transcription_requestz+ServeCommand.validate_transcription_request1s! 8-5  rTN request_idcontentrrole finish_reason tool_callsr3rc t|ttj|tt |||d|gdd}d|j d d S) a Builds a chunk of a streaming OpenAI Chat Completion response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: request_id (`str`): The request ID. content (`str`, *optional*): Content of the response from the model. model (`str`, *optional*): The model that generated the content. role (`str`, *optional*): The role of the next content, until a new role is defined. finish_reason (`str`, *optional*): The reason the generation by the model has finished. tool_calls (`list[ChoiceDeltaToolCall]`, *optional*): Data about the tool calls, when they are triggered. Returns: `str`: The built chunk, a string containing a JSON string with the payload. )r4r5r7r)deltaindexr6rzchat.completion.chunk)idcreatedrrsystem_fingerprintobjectdata: T exclude_none )r0rtimer1r2model_dump_json)rr3r4rr5r6r7chunks rUbuild_chat_completion_chunkz(ServeCommand.build_chat_completion_chunk9sq@$ $% '!#- "/  "*! $--4-@AFFrTresponserFc.d|jddS)a Builds a event of a streaming OpenAI Response response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: response (`BaseModel`): The response to build an event from. One of the multiple OpenAI Response output types Returns: `str`: The built chunk, a string containing a JSON string with the payload. r?Tr@rB)rD)rrGs rUbuild_response_eventz!ServeCommand.build_response_eventms"00d0CDDIIrTct}jr3|jtdgddgdgtj dntj d|j ddtffd }|j d dtffd }d d lm }|j d d|ffd }|jd|jdfd}tj|jjjj jj"y)N*T) allow_originsallow_credentials allow_methods allow_headerszUCORS allow origin is set to `*`. This is not recommended for production environments.zaSome apps may require CORS. Consider launching the server with `--enable-cors` if you see errors.z/v1/chat/completionsrcj|jrj|}nj|}t |dSNrtext/event-stream media_type)r.r#continuous_batching_chat_completiongenerate_chat_completionr,routputrs rUchat_completionz)ServeCommand.run..chat_completionsK  1 1' 1 B++AA'J66w?$V8KL LrTz /v1/responsescdj|j|}t|dSrQ)r*generate_responser,rXs rU responsesz#ServeCommand.run..responsess2  * *7 * ;++G4F$V8KL LrTr)Requestz/v1/audio/transcriptionsc K|j4d{}t|djd{|d}tj d|dj d|dj d|djdz dd dddd{j j|}t|d S7778#1d{7swYHxYww) Nr\r)r\rzReceived file: z ; MIME type: z; size: z.2fz KiBrRrSrT) formr[readrrfilename content_typesizer2generate_transcriptionr,)rraparsed_requestrYrs rUaudio_transcriptionsz.ServeCommand.run..audio_transcriptionss||~  !F#F|0022w-"  %d6l&;&;%data)r+get_gen_modelsrsrUget_all_modelsz(ServeCommand.run..get_all_modelss 64;N;N;P QR RrT)rrr)r(radd_middlewarer*r warning_oncepostdictfastapir^optionsruvicornrunrrrr)rapprZr]r^rhrms` rUruzServeCommand.run}s;i      "e"&"e"e     g    s  ( ) MT M * M / " Mt M # M $ , - M M . M" \ "   S  # S  Cdiinn499>>TYYM`M`arT)maxsizec gd}trQ|Dcgc]E}|dtjjj|j dddGc}S|Dcgc] }t |}}|Dcgc]5}|j d|jj|jd7c}Scc}wcc}wcc}w)a. This is by no means a limit to which models may be instantiated with `transformers serve`: any chat-based model working with generate can work. This is a limited list of models to ensure we have a discoverable /v1/models endpoint for third-party integrations. ) zMenlo/Jan-nanozMenlo/Jan-nano-128kzQwen/Qwen2.5-0.5B-InstructzQwen/Qwen2.5-3B-InstructzQwen/Qwen2.5-7B-InstructzQwen/Qwen2.5-14B-Instructz meta-llama/Llama-3.1-8B-Instructz meta-llama/Llama-3.2-1B-Instructz!meta-llama/Llama-3.3-70B-InstructzHuggingFaceTB/SmolVLM-Instructz!ibm-granite/granite-vision-3.2-2bzQwen/Qwen2.5-VL-7B-Instructr/r)r;r>r<owned_by) rdatetimenow timestampsplitr r; created_atauthor)rmodelsr model_infoss rUrlzServeCommand.get_gen_modelss   $  %'00446@@B % C 0 3  ;AA:e,AKA)  ((%$//99; %  BsA B5 B:8:B?rc  jd jk7} _|r0j$jjddd_j \}}t |dr |j n|}t|j|j|jddd dd d jK|jd _tj_ jj|jdddj!|j"} fd}||dS)a' Generates an OpenAI Chat Completion using continuous batching. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. rNTr)blocktimeout tokenizerFrr` fifo) r eos_token_id pad_token_id use_cache num_blocks block_sizermax_batch_tokens scheduler)rK streamingmessagespt)return_tensorsadd_generation_promptc3K jj|jdj}d} j |d jD]}|j |k7rjd"|s |j tjk(rDd}|j tjk(rdnd}|j tjk(r j ||y j ||j y#t$r9}tjt|d t|d Yd}~yd}~wwxYww) Nr3)r3rF assistantr5rTrnr6r)r3r4rdata: {"error": ""})r  add_requestrrrFr3statusr&FINISHED next_token ExceptionrrrQ) _inputsr3queue_is_flushedresultr6r"rKmodel_id_and_revisionrrs rUstream_chat_completionzPServeCommand.continuous_batching_chat_completion..stream_chat_completions_! 7!EEQQ (=N_NnNnR $) 66z [p6qq"FFF((J6 ww|,8AQ!==M,B,BB$/3,.4mm}?U?U.UF[_M}} (>(>>">>&mK`?">>'16;L;LTi?!( 7 SV$*3q6(#66 7s;E$C6D;E$<"DE$ E!(/EE$E!!E$r)process_model_namer r rnload_model_and_processorrrrrKrrinit_continuous_batchingrlogit_processorrapply_chat_templatetor) rrmust_discard_cacherrrinputsrrKrs `` @@rUrVz0ServeCommand.continuous_batching_chat_completionsc!% 7 7G E2dooE/ 77C88==DRS=T;?8889NOy+29k+JI''PY = $)$;$;"//"//    3 3 ;7<7U7U"3t8V8D 4 H[G\D 4 4 D  4 4 : : <..s:tko.pss LL " 7H&fQi00rTr$c|jj}|tjvrtj }|S|t jvrtj}|Std|)NzUnknown modality: ) __class__rMrvaluesrrrr ValueError)rmodel_classnamemodalitys rUget_model_modalityzServeCommand.get_model_modalityEsm//22 HOOQ Q||H   A H H J J||H1/1BCD DrTrc g}|D]}|dgd}|tjk(rmt|dtr|d}nMt|dtr:g}|dD]}|ddk(s |j |d!dj |}|d<n(|tjk(rt|dtr|dj d|ddn|dD]}|ddk(r|dj | |ddk(s)d |dd vrtjd d |dd }tjttj|}tj d d} | j"} |j%| j"n|dd } |dj d| d|j ||S)Nr5r5r4r4typer` )rr` image_urlbase64urlz^data:image/.+;base64,rz.pngF)suffixdeleteimage)rr)rr isinstancerQrjappendjoinrresubr'openr r b64decodetempfileNamedTemporaryFilenamesave) rrprocessor_inputsmessageparsed_messageparsed_contentr4 image_datarr\rs rU*get_processor_inputs_from_inbound_messagesz7ServeCommand.get_processor_inputs_from_inbound_messagesQs' 4G&-fo"EN8<<'gi0#6%,Y%7N 2D9%'N#*9#5C"6?f4*11'&/BC&)XXn%=N,:y)X\\)gi0#6"9-44fgV_N`5ab#*9#5\"6?f4*95<final<|message|>cLjdi|}|j_yNrSgeneraterr rgenerate_outputrrs rUgenerate_with_cachezbServeCommand.generate_chat_completion..stream_chat_completion..generate_with_cache$"0%..":6":%4%D%D"rTtargetrrrr <|return|>rrr7)r3r5r6rz\"name\": \"(.*?)\"r)rfunction _tool_call)rr:rr;z"arguments": {{}) arguments)rr:r)r3r5r7r)r4rrnrrr)config architecturesrr rrrFendswithlenstrip_TOOL_CALL_TOKENSrrrrrsearchgroupr3r4rcountrr~rrrrQ)r _request_id filter_cot cot_trace_endrthreadresults tool_stater tool_nametoolr"generation_kwargsrrr3rtool_model_familys rUrzEServeCommand.generate_chat_completion..stream_chat_completions_J M5<<55a8>>@@! =  E#6?PQFGh  &[ 66z [p6qq&WF5<<#=#=a#@#F#F#HH!??<8%+,@s->?P-QRY-ZZ:>J7$"<<>->?P-QRW-XX&,,."&"B"B+6%).:&; #C#%%66&--7-$.#C#C,.II6LjN_N_,` #,#4$,090BICG @':-Hi-X*+)3'2\'A ("$*R<$,$4:;L;L#L$,!+ < < S@Q Q < * < < S@Q Q <#-#?#?!#C-/WWV\\#5Fs5K-Ls-RF':-HSY-Z*+)3(" #'"B"B+6Ttf\q#C#%|">>'?T?kWp66{RX`u6vv   7 SV$*3q6(#66 7  sCAM I K*?K*M* L,3/L'"L/'L,,L//MM)rrrr rrr_MODELS_WITH_TOOL_SUPPORTrrrrrrrrrrKis_continuationr )rrrrrrrsupported_model_familiesrrgeneration_streamerrKr rrrrr3rs` @@@@@rUrWz%ServeCommand.generate_chat_completions 99 ,9900CL9c[`[r[rs    $-? ..M  +!2'+,  y y v&&9:FFrTc jd jk7} _j \ }tdtr'dvr dddgng}|j dddntdt r8dvr.ddddk7rdddgd}nYd}d|dd <nHd}nBtdtr$dvr dddgng}|j dn td |j|d d }|j j}jdd d }d jjdjvrd}t!||d }t# j$}d}j'r|s j(}|t+j,|||d |d  fd} | | S)a  Generates an OpenAI Response using `generate`. Args: req (`dict`): The request to generate an OpenAI Response for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Response events. rinput instructionssystemrrarr5r4z%inputs should be a list, dict, or strTr)rrrlrrFrrN)rattention_maskrrKrrc3L Kd}d}djjdjvrd}d}fd}t|}d}d}d} |j t j } t d|td | d jd d d diidggjdddjd } |dz }j| td|td | djd d d diidggjdddjd } |dz }j| td||tddddg} |dz }j| tdd|||tdd g!"} |dz }j| d }|D]}djjdjvr |jd#r|dt!d# }||z }|r ||vrd}d }\]t#d$d||||d d%d&g'}|dz }j|t%d(d||d|d d%d&g)}|dz }j|t'd*d|||td|j(g!"}|dz }|dz }j|t+d+||tddd,d|j,gg-}|dz }|dz }j|t/d.|td | d,jd d d dii|j0gdgjdddjd/ }|dz }j||j3|j3y#t4$r}t6j9d0t;|t=d1|t;|2}|dz }j|t?d3|td  d4jd d d diigdgddjdtAd5t;|67 }|dz }j|Yd}~d}~wwxYw#|j3wxYww)8NFrrTrcLjdi|}|j_yrrrs rUrzTServeCommand.generate_response..stream_response..generate_with_cacherrTrzresponse.createdresp_queuedr formatrr`rGr{rrq) r;rrrr r`r>rrYr{rhrq)rsequence_numberrGrzresponse.in_progress in_progresszresponse.output_item.addedmsg_rr)r;rrr5r4)rr output_indexitemzresponse.content_part.added output_textr)rr` annotations)ritem_idrr content_indexpartrzresponse.output_text.deltagX@)tokenlogprob)rrrrrr9rpzresponse.output_text.done)rrrrrr`rpzresponse.content_part.donezresponse.output_item.done completed)r;rrr5r4rzresponse.completed) r;rrrr r`rYr>rr{rhrqz"Exception in response generation: r)rrrzresponse.failedfailed server_error)coder) r;rrrr r`rYr>rr{rhrqr)!rrrr rrCr:r6rrIr>r?rAr8rBrrrCrDr9r`r@rr7rrrrrrQr<r=r;)rrrrrrrrrrresponse_createdresponse_in_progressresponse_output_item_addedresponse_content_part_addedrrresponse_output_text_deltaresponse_output_text_doneresponse_content_part_doneresponse_output_item_doneresponse_completedr" error_eventresponse_failedrrrrr3rs rUstream_responsez7ServeCommand.generate_response..stream_responsesJ M5<<55a8>>@@! =  E#6?PQFOLMN  !YY[ $8+$3%":,/#-'3%(WW^%<&(89) !,/GG4I5,Q$*!$!4 $ $ 1$//0@AA'>/$3%":,/#-,3%(WW^%<&(89) !,/GG4I5,Q$*!$!4 ($$ 1$//0DEE.J5$3!-.!*.Y}[fpr .* 1$//0JKK/L6":,/$3!-"/+RUWX /+ 1$//0KLL&PF5<<#=#=a#@#F#F#HH!??<8%+,@s >?) ,/GG4I5,Q$*!$!4 &"$ 1$//0BCC J I! A A#a&JK0 $3F  1$// <<"5*$3%":,/#-'3%(WW^%<&(89!) ,1$*!$!4+!/$'F#, 1$//@@C! AH s>A R$MN4#R$4 R =CRRR  RR!!R$)rr rrrQrrjrqrrrrrrrrrrrKrr r ones_like)rrrrrrrrKr r,rrrr3s`` @@@@rUr\zServeCommand.generate_responseGs1!% 7 7G E2dooE/889NOy c'lC (M[_bMbxC4GHIhjF MM6c'lC D G d +$w<?6*h6'/CF1Ii(W G d +M[_bMbxC4GHIhjF MM#g, 'DE E..vTbf.g5<<(WW3W= # u||11!4::< <"' 2  3  >c[`[r[rs    $-? ..M#oof5+!2'+,  a a F2J??rTc( ts td|j|d}|j|\ t j dd}t | j} jj}tj|d}tj||d\}} ||d j j d j j  d <||dd  fd } | S) a Generates an OpenAI Transcription using the audio file. Args: req (`dict`): The request containing the audio file and model information. Returns: `Generator[str, None, None]`: A generator that yields the transcription result. z]Missing librosa dependency for audio transcription. Please install with `pip install librosa`rTrrr\)srmonor) sampling_raterinput_features)rrKrc3Kjdi}j|jdd}t|}|j dyw)NT)rr)r`r@rS)r batch_decode sequencesr-rD) generated_idstranscription_text transcription audio_inputs audio_modelaudio_processorrs rU_generate_transcriptionzDServeCommand.generate_transcription.._generate_transcriptionsg0K00U<UCTUM!0!=!=m>U>Uko!=!pqr!s )/ABM"222EF GsAA)rrrload_audio_model_and_processorrrrrKfeature_extractorr1ior librosaloadrrr)rrrrrKmodel_sampling_rate audio_bytes audio_array_r<r9r:r;rs @@@@rUrfz#ServeCommand.generate_transcriptionks"$%o !% 7 7G E'+'J'JK`'a$ _2  % %4T > )F)F  .??MMjjV-  k6IPTU Q&{BUfjknn    *66F)G)J)J;K\K\)] %&,!2'+   H '((rTc>|jdxs|jd}d}|jd}n`t|jt|k\rd}n p5CZCZCiCilpCp ! "%=59E # # 2 m$9#:;<n$$rTc||jvs|j|jrG|j|\}}t||jj ||j|<||fS|j|j |j|j}|j|j}||fS)a\ Loads the text model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, PreTrainedTokenizerFast]`: The loaded text model and processor. rr r rrarrrrrr)rrrrs rUrz%ServeCommand.load_model_and_processor$s !(:(: :d>P>PQf>g>r>r>t#BBCXY E98B $ 7 7#9D  4 5i   4 5 A A C&&'<=CCE**+@AKKIirTc||jvs|j|jrG|j|\}}t||jj ||j|<||fS|j|j |j|j}|j|j}||fS)aU Loads the audio model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, ProcessorMixin]`: The loaded audio model and processor. rcrd)rrr:r;s rUr=z+ServeCommand.load_audio_model_and_processor?s !(:(: :d>P>PQf>g>r>r>t+/+N+NOd+e (K8B $ 7 7)9D  4 5O++   4 5 A A C,,-BCIIK"001FGQQOO++rT)rNNNNN)*rMrNrO staticmethodrrrrrqrr$r*r.r2r rQrjrFrIru functools lru_cacheanyrlrrVrrrrWr\rfr_rrOrratuplerrrr=rSrTrUrrs >N > >^@//!/! /  /b     d %'!%#"'+<@2GSM2G#2G} 2G sm 2G  } 2GT"7892G 2GhJ[JSJ ?bBY&+T#s(^ 4+'+ZW1tW1 #tUY/@ZW1r "3   + x+ + ZFGDFGYsD$5OFGPb@Tb@iT4.Hb@H .)$.)9S$_3M.)`+4+D+<#n#BV9W##8"3"3"";%C;%z %(  "99 : 6,C,ERcesRsLt,rTr__main__)rrr{enumrgrr?rrrrrCargparserrcollections.abcrr dataclassesrrr r typingr r huggingface_hubr huggingface_hub.constantsrr&transformers.models.auto.modeling_autorrtransformers.utils.import_utilsrrrrrrrrrrrrutilsrrr rr!r"r#r$generation.continuous_batchingr%r&r@PILr'rrtrrr(r)fastapi.middleware.corsr*fastapi.responsesr+r, openai.types.audio.transcriptionr-.openai.types.audio.transcription_create_paramsr.openai.types.chatr/'openai.types.chat.chat_completion_chunkr0r1r2r3r4*openai.types.chat.completion_create_paramsr5openai.types.responsesr6r7r8r9r:r;r<r=r>r?r@rArBrCrD-openai.types.responses.response_create_paramsrEpydanticrFrGrHrJrYr[r'r,r0r(r-r1rrMrrrjrrEnumrrrqrrrrrrrurSrTrUrs[    ./("&40(Zk 4 6k;O;QkViVk .6A>\<["\@@4QY^6U]b'0MUZ'%%NO&'RS)*OP %!.#   H %    !!2!7!7!9:tyy 8 8/8 8v  ,@,@^ d*d* d*NS,-S,l$ z NE IIKrT