h dZddlmZmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e'r ddl-m.Z.ddl/m0Z0e(jbe2Z3GddejhZ5 d3dejldejndejndejndeejnde8de8fdZ9Gd d!ejlZ:Gd"d#eZ;e%Gd$d%e Z<Gd&d'e<Z=e%Gd(d)e<Z>Gd*d+eL(>9A=CCEL'+A+B(BCLw|dkk9::r0)rN) __name__ __module__ __qualname____doc__intr,r7 LongTensorrr: __classcell__r.s@r/r$r$4s]FsF3F'(37 ;((;!$;u//0 ;;r0r$modulequerykeyvaluer1scalingdropoutc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)N)r6dtypeptrainingrr)) r7matmul transposer functionalsoftmaxfloat32torLrHrO contiguous) rCrDrErFr1rGrHkwargs attn_weights attn_outputs r/eager_attention_forwardrZQs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r0cDeZdZdZ ddedeeffd Zeddd dd e jdee e jd ee jd ee jd e d ee jde e jee jee ffdZxZS) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperconfig layer_idxc t|||_|j|_|j |_|j|_|j|_ ||_ |-tjd|jjd|j|j z|_d|_|j|j z|jk7r&t#d|jd|j d|jdz|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r+r,r] hidden_size embed_dimnum_attention_heads num_headsattention_dropoutrH enable_biasr^logger warning_oncer.r;head_dim is_causal ValueErrorrGrLineark_projv_projq_projout_proj)r-r]r^rWr.s r/r,zOPTAttention.__init__ks  ++33// !--"      !8!8 9:,,  $..8  MMDNN *t~~ =MdnnM]$T^^$4B8 }}d* iiTEUEUV iiTEUEUV iiTEUEUV  $..$..tGWGWX r0past_key_valuepast_key_values4.58new_nameversion hidden_statesr1layer_head_maskoutput_attentionscache_positionreturnc |j\}} } |j||jz} | j|d|j|j j dd} |j|} |j|} | j|d|j|j j dd} | j|d|j|j j dd} |#|j| | |jd|i\} } t}|jjdk7rN|jjdk(r|rtjdnt |jj}||| | | |f|j"sd n |j$d d |\}}|j'|| dj)}|j+|}|sd}||fS) z#Input shape: Batch x Time x ChannelrJrr)Nr{eagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.?)rHrG)sizerprGviewrerjrQrnroupdater^rZr]_attn_implementationrhrirrOrHreshaperVrq)r-rxrsr1ryrzr{rWbsztgt_len_ query_states key_states value_statesattention_interfacerYrXs r/r:zOPTAttention.forwards(,,.Wa{{=1DLL@ #((b$..$--PZZ[\^_` [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  &'6'='=L$..;K^:\( $J )@ ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7      % $}}C$,, %  % ! \"))#w;FFH mmK0  LL((r0N)NNNFN)r;r<r=r>r rr?r,rr7Tensortupleboolr r:rArBs@r/r\r\hsG $(!Y!YC=!YF%0A6R:>1526"'15<)||<)"% "56<)!. <) "%,,/ <)  <)!.<) u||Xell3Xe_D E<)S<)r0r\ceZdZddedeeffd Zeddd ddejd eejd eejdee ejd ee d ee d eejdeejde ede ejee ejejfffdZxZS)OPTDecoderLayerr]r^ct||j|_t |||_|j |_|j|_t|j|_ tj|j|j|_tj|j|j |j"|_tj|j |j|j"|_tj|j|j|_y)N)r]r^elementwise_affiner`)r+r,rbrcr\ self_attndo_layer_norm_beforerHr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrmffn_dimrgfc1fc2final_layer_norm)r-r]r^r.s r/r,zOPTDecoderLayer.__init__s ++%VyI$*$?$?!~~ #F$>$>?$&LL NNv/S/S% !99T^^V^^&BTBTU99V^^T^^&BTBTU " T^^PVPtPt ur0rrrsrtrurxr1ryrz use_cacher3r{rWr|c |} |jr|j|}|jd|||||||d| \}} tjj ||j |j }| |z}|js|j|}|j} |jd|jd}|} |jr|j|}|j|}|j|}|j|}tjj ||j |j }| |zj| }|js|j|}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`, *optional*): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence.. )rxrsr3r1ryrzr{rMrJ)rrrrrRrHrOshaperrrrrrr)r-rxr1ryrsrzrr3r{rWresidualself_attn_weightshidden_states_shapeoutputss r/r:zOPTDecoderLayer.forwards>!  $ $ 55mDM,:4>> , '+%)+/) ,  , ( ( --mt||VZVcVc-d  =0 (( 55mDM,11%--b-2D2DR2HI    $ $ 11-@M/ **=9 /  --mt||VZVcVc-d !M1778KL (( 11-@M "  )+ +Gr0r)NNNFFNN)r;r<r=r rr?r,rr7rrrr@rr FloatTensorr:rArBs@r/rrs3vyvXc]v"%0A6R26269=,1$)3715P||P!.P"%,,/ P "% "56 P $D> PD>Pu//0P!.P-.P u  (51B1BEDUDU1U+V"WW XPSPr0rc@eZdZUeed<dZdZdgZdZdZ dZ dZ dZ dZ y)OPTPreTrainedModelr]modelTrc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tjrJ|j j jd|jj jyy)Nr)meanstdr)r]init_std isinstancerrmweightdatanormal_razero_ Embedding padding_idxrfill_)r-rCrs r/ _init_weightsz OPTPreTrainedModel._init_weightsAskk"" fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ) KK   " " $.r0N)r;r<r=r __annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrrr0r/rr4s? &*#*+"&N! %r0rceZdZdZdeffd Z ddeejdfdejdejde d e f d Z e dejd e d e d ejdejde f dZe ddeej$deejdeejdee deej&dee d ee dee dee deej$deejdeedeeeffdZxZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig r]c 8t|||j|_|j|_|j|_|j |_|j|_tj|j|j|j |_ t|j |j|_|j|jk7r2tj |j|jd|_nd|_|j|jk7r2tj |j|jd|_nd|_|j&r=|j(s1tj*|j|j,|_nd|_tj0t3|j4Dcgc]}t7||c}|_d|_|j=ycc}w)NFr`r)r^)r+r,rH layerdrop pad_token_idrmax_position_embeddingsmax_target_positions vocab_sizerrword_embed_proj_dim embed_tokensr$rbembed_positionsrm project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing post_init)r-r]ir.s r/r,zOPTDecoder.__init__Xs  ~~ ))!..$*$B$B! ++LL):):FT]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr0rrrLrc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerLrr)diagonalrrJr)r6r7rrfullrtriuarangerexpandclonerrU masked_fill) r1rrrLr{rrWrr mask_length padding_masks r/rz@OPTDecoder._prepare_4d_causal_attention_mask_with_cache_positions>  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r0 input_ids head_maskrroutput_hidden_states return_dictr3rWr|c ||n|jj}||n|jj}||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||jd|jd}||j|}|r|t|j}||jnd} | 2tj | | |jdz|j" } |A| |jdz}tj$|jd||j" }|j'||| ||}| 8tj(|d } | |zdz j+} | dd| df} |j-|| | }|j.|j/|}||j1|j"z}|rd nd}|rd nd}t3|gd gD]j\}}| |j5dt7|j8k7s3t d|dt7|j8d|j5ddt;|j8D]g\}}|r||fz }|jr%tj<g}||j>kr?||f|| |||nd|||| d| }|d}|s_||dfz }i|j@|jA|}|jB|jC|}|r||fz }tE||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FrJ)r]rrrr5)r3rrzThe `z` should be specified for z layers, but it is for .)r1r3ryrsrzrr{last_hidden_statersrx attentions)#r]rzrruse_return_dictrlrrOrhrirrrr rr7rronesrr8r9rrrUziprlenr enumeraterandrrrr)r-rr1rrsrrrzrrr3r{rWr seq_lengthr pos_embedsrxall_hidden_statesall_self_attns attn_mask mask_nameidx decoder_layerdropout_probability layer_outputss r/r:zOPTDecoder.forwardsP2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?I   --i8M 0*$++>O?N?Z?99;`a  !"\\ "2]5H5H5K"KTaThThN  !)M,?,?,BBJ"ZZ (;(;A(> S`SgSghN.. M>?L]   <<A>L(>9A=CCEL'+;+<(<=L)).:JYe)f ?? & OOM:M% m6J6J(KK #7BD0d%( k]$C  Iy$>>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03 #,DKK"8 6 C#!m%55!}}&+jjn#&7) *)3<3H3d /"3#-  M*!,M =#3"553 66  , 11-@M    ' ,,];M -!1 1 &+++%   r0)F NNNNNNNNNNN)r;r<r=r>r r,rr7rr rr staticmethodr?rLrrrr@rrrrrr:rArBs@r/rrPs#y#X#( BellK78BllB B  B  BH4 444{{ 4  4  44l1515,0+/59$(,0/3&*3715u E,,-u !.u ELL) u "% u   1 12 u D>u $D>u 'tnu d^u u//0u !.u -.u  u-- .u u r0rceZdZdeffd ZdZdZee dde e jde e jde e jde e ee jefd e e jd e ed e ed e ed e ede e jde e jdeede eeffdZxZS)OPTModelr]cdt||t||_|j yr)r+r,rdecoderrr-r]r.s r/r,zOPTModel.__init__s&  !&)  r0c.|jjSrr rr-s r/get_input_embeddingszOPTModel.get_input_embeddingss||(((r0c&||j_yrr#r-rFs r/set_input_embeddingszOPTModel.set_input_embeddingss$) !r0rr1rrsrrrzrrr3r{rWr|c  |||n|jj}||n|jj}||n|jj}| | n|jj} |j d||| ||||||d| d | } t | j| j| j| jS)NT rr1r3rrsrrrzrrr{rr) r]rzrrr r rr rsrxr )r-rr1rrsrrrzrrr3r{rWdecoder_outputss r/r:zOPTModel.forwards"2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK->(?(F"GH +   1 12 + D>+ $D>+ 'tn+ d^+ u//0+ !.+ -.+  u-- .+ + r0rc!eZdZdgZfdZdZdZdZdZe e dde e jde e jd e e jd e eee j"efd e e j"d e e jd e ede ede ede ede e jde e jdeedeeeffdZxZS)OPTForCausalLMzlm_head.weightct||t||_t j |j |jd|_|jyNFr`) r+r,rrrrmrrlm_headrr!s r/r,zOPTForCausalLM.__init__sK  f% yy!;!;V=N=NUZ[  r0cB|jjjSrrr rr$s r/r%z#OPTForCausalLM.get_input_embeddingszz!!...r0c:||jj_yrr3r's r/r(z#OPTForCausalLM.set_input_embeddings*/ 'r0c&||j_yrrr )r-r s r/ set_decoderzOPTForCausalLM.set_decoders$ r0c.|jjSrr8r$s r/ get_decoderzOPTForCausalLM.get_decoderszz!!!r0rr1rrsrlabelsrrzrrr3r{rWr|c  ||n|jj}| | n|jj} | | n|jj} |jj d||| |||||| d| d | }|j |dj}d}|E|j|j}|j||fd|jji| }t|||j|j|jS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```NTr*rrlosslogitsrsrxr r)r]rzrr rr r1rVrUr loss_functionrrrsrxr )r-rr1rrsrr<rrzrrr3r{rWrr@r?s r/r:zOPTForCausalLM.forward s;R2C1N-TXT_T_TqTq$8$D $++JjJj &1%->(?(F"GH P   1 12 P ))*P D>P $D>P 'tnP d^P u//0P !.P +,P  u,, -P P r0r.a The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcreZdZdeffd Ze ddeejdeejdeejdee e eje fdeejdeejd ee d ee d ee d ee d eejde eeffdZdZdZxZS)OPTForSequenceClassificationr]ct|||j|_t||_t j |j|jd|_|jyr0) r+r, num_labelsrrrrmrscorerr!s r/r,z%OPTForSequenceClassification.__init__nsT   ++f% YYv994??QVW  r0rr1rrsrr<rrzrrr3r|c | | n|jj} |j|||| ||||| |  } | d} |j| }||jdd\}}n|jdd\}}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d |tj||j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:| j<S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N rsr1r3rrrrzrrrr)rz=Cannot handle batch sizes > 1 if no padding token is defined.rJ)rrLz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr>)r]r rrIrrrlrUrr7int32rargmaxrhrir.r; problem_typerHrLr9r?r squeezerrrrrsrxr )r-rr1rrsrr<rrzrrr3transformer_outputsrxr@rrlast_non_pad_token non_pad_mask token_indices pooled_logitsr?loss_fctoutputs r/r:z$OPTForSequenceClassification.forwardws*&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r0cB|jjjSrr3r$s r/r%z1OPTForSequenceClassification.get_input_embeddingsr4r0c:||jj_yrr3r's r/r(z1OPTForSequenceClassification.set_input_embeddingsr6r0r)r;r<r=r r,rrr7r@rrr,r rrrr:r%r(rArBs@r/rFrF_sLy156:15KO59-1$(,0/3&*37\ E,,-\ !!2!23\ E--. \ "%U->->(?(F"GH \   1 12 \ ))*\ D>\ $D>\ 'tn\ d^\ u//0\  u66 7\ \ |/0r0rFceZdZdeffd Ze ddeejdeejdeejdee e eje fdeejdeejd eejd ee d ee d ee d ee deejde eeffdZdZdZxZS)OPTForQuestionAnsweringr]ct||t||_t j |j d|_|jyr() r+r,rrrrmr qa_outputsrr!s r/r,z OPTForQuestionAnswering.__init__s@  f% ))F$>$>B r0rr1rrsrstart_positions end_positionsrrzrrr3r|c | | n|jj} |j|||| |||| | |  } | d}|j|}|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|j|j}|jd|j|j}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```NrKrrrJr5) ignore_indexr))r? start_logits end_logitsrxr )r]r rr_splitrRrVrrclamprUrrrrxr )r-rr1rrsrr`rarrzrrr3rSrxr@rdre total_loss ignored_indexrX start_lossend_lossrYs r/r:zOPTForQuestionAnswering.forwards`&1%->(?(F"GH _   1 12 _ "%"2"23_   0 01_ D>_ $D>_ 'tn_ d^_ u//0_  u22 3_ _ B/0r0r])r.rrrFr])r)Cr>typingrrrr7torch.utils.checkpointrtorch.nnrrr activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_utilsrrprocessing_utilsrutilsrrrrrutils.deprecationrconfiguration_optr !torch.nn.attention.flex_attentionr!integrations.flex_attentionr" get_loggerr;rhrr$ModulerfloatrZr\rrrrr.rFr]__all__rr0r/rs,, AA!.)>B9 G&pp0( !;J   H %;BLL;H% II% <<% % << % U\\* %  %%.c)299c)Lc0cL%%%6` #` F : !: : zk 'k \  m0#5m0 m0`o00o0o0d r0