h}dZddlmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZd dlmZej>e Z!dZ"dZ#d#dZ$GddejJZ&dZ'GddejJZ(eGddeZ)eGdde)Z*edGdde)eZ+edGd d!e)Z,gd"Z-y)$zPyTorch CTRL model.)OptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss) DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcPdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess d/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr (s/eiiQ!V 'DEEK  cttj|tjj |j dtj|tjj |j d|}tj |dddddf}tj|dddddf}tj||gd}|S)Ndtyperrrdim) r rarangeint64to unsqueezesincoscat)positionrr$ angle_radssinescosines pos_encodings rpositional_encodingr4-s XU[[144U;EEaH \588?II!LJ IIjADqD) *Eii 1add7+,G99eW-26L r!c tj||jdddd}|jd}|t j |z }|6|j d|j d} } ||| | z | d| fdzz }|||z}tj|d} || |z} tj| |} | | fS) Nrrr rr%gr&)rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask head_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputs rscaled_dot_product_attentionrK<s Q !Q1 56I B'"''"+5 (--b13J3O3OPR3SB4R" crc(9#:T#AA!"9N"J &=2F- 9 \\+Q /F $ $$r!c@eZdZdfd ZdZdZ ddZxZS)MultiHeadAttentionc|t|||_||_||_t ||jz |_tj|||_ tj|||_ tj|||_ tj|||_ t|_yN)super__init__ num_headsr layer_idxintdepthrLinearWqWkWvdenseset pruned_heads)selfrrRrS __class__s rrQzMultiHeadAttention.__init__Ws "("67 ))L,7))L,7))L,7YY|\: Er!c|j|jz}t|dk(ryt||j||j\}}t |j ||_t |j||_t |j||_t |j|d|_ |jt|z |_||jz|_|jj||_y)Nrrr&) rrRlenrr\rrWrXrYrZunion)r]headsattention_head_sizeindexs r prune_headszMultiHeadAttention.prune_headsfs"//4>>A u:? 7t~~Obdhduduv u%TWWe4$TWWe4$TWWe4' EqA #e*4/$..@ --33E:r!cx|j|d|j|j}|jgdS)Nr%rrrr )reshaperRrUr8)r]x batch_sizes rsplit_into_headsz#MultiHeadAttention.split_into_headsws- IIj"dnndjj Ayy&&r!c |jd} |j|}|j|}|j|}|j || }|j || }|j || }|#|j |||j d| i\}}t||||||} | djgd} | d}| j| d|j}|j|} | |fS)Nrcache_positionrgrr%) r9rWrXrYrkupdaterSrKr8rhrrZ)r]r@r?r>rA layer_pastrBrC use_cacheoutput_attentionsrmrjrJscaled_attentionattnoriginal_size_attentions rforwardzMultiHeadAttention.forward{sWWQZ GGAJ GGAJ GGAJ  ! !!Z 0  ! !!Z 0  ! !!Z 0  !$$Q4>>eps) rPrQrMmulti_head_attentionrffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)r]rrRrraterSr^s rrQzEncoderLayer.__init__sr $6|YZc$d!2<E,,|>,,|> 4(  4( r!c |j|} |j| | | ||||||| } | d} |j| } || z} |j| } |j | }|j |}| |z} | f| ddz}|S)NrorBrCrprqrmrr)rrrrrr)r]rirArorBrCrprqrmnormed attn_outputs attn_outputout1out2 ffn_outputoutputss rruzEncoderLayer.forwards#00    !)/)1  #1o mmK0 ;t$XXd^ ]]:. j 'L,,r!)g?Nrv)rwrxryrQrurzr{s@rrrs! ) "r!rc"eZdZUeed<dZdZy)CTRLPreTrainedModelconfig transformerct|tjtfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyy)zInitialize the weights.g)meanstdN?) isinstancerrVrweightdatanormal_rinitializer_rangebiaszero_ Embedding padding_idxrfill_)r]modules r _init_weightsz!CTRLPreTrainedModel._init_weightss  fryy&1 2 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r!N)rwrxryr__annotations__base_model_prefixrr!rrrs %*r!rceZdZfdZdZdZdZe ddee jdee e e jdee jdee jd ee jd ee jd ee jd ee d ee dee dee dee jdee e jeffdZxZS) CTRLModelc t|||j|_|j|_t |j|jtj|_ tj|j|j|_tj|j |_tj$t'|jDcgc]:}t)|j|j*|j,|j.|<c}|_tj2|j|j4|_|j9ycc}w)Nrr)rPrQn_embdrn_layer num_layersr4 n_positionsrfloatr3rr vocab_sizewr embd_pdropdropout ModuleListrangern_headr resid_pdrophrlayer_norm_epsilon layernorm post_init)r]rrr^s rrQzCTRLModel.__init__s  "MM ../0B0BDDUDUW\WbWbcf//?zz&"3"34 v~~. V]]FMM6::vGYGYefg   fmm9R9RS  s ?E,c|jSrOr)r]s rget_input_embeddingszCTRLModel.get_input_embeddingss vv r!c||_yrOr)r]new_embeddingss rset_input_embeddingszCTRLModel.set_input_embeddingss r!c|jD]-\}}|j|jj|/y)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrre)r]heads_to_prunelayerrbs r _prune_headszCTRLModel._prune_headss>+002 BLE5 FF5M . . : :5 A Br! input_idspast_key_valuesrBtoken_type_ids position_idsrC inputs_embedsrprqoutput_hidden_states return_dictrmreturnc "| | n|jj} ||n|jj}| | n|jj} | | n|jj} | | t d|G|j |||j}|jd|d}|jd}n0|#|jdd}|jd}n t d| |jn |j}|r|t|j}|r:t|tr*tjdtj |}||j#nd}|>t%j&||d|zt$j(|}|j+d}||dkr t d |j|d}|j+d j+d }|j-|j. }d |z t%j0|j.j2z}|j5||jj6}|I|jd|d}|j9|}|t;j<|j>z}nd}||j9|}|d}t%j@t%jB||z||zd j-|}|t;j<|j>z}|jDj-||_"|jD|ddf}||z|z}|jG|}| rdnd}| rdnd}tI|jJD]1\}}| r||fz}||||||||| | }|d}| s)||d fz }3|jM|}| r||fz}| std||||fDStO||||S)aE input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer%rz5You have to specify either input_ids or inputs_embeds)rzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.)r$devicez$batch_size has to be defined and > 0rrr#rrrc3&K|] }|| ywrOr).0r@s r z$CTRLModel.forward..sbcbos)last_hidden_stater hidden_states attentions)(rrqrpruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr<viewr9rr rtuplelogger warning_oncefrom_legacy_cacheget_seq_lengthrr(longr+r*r$finfomin get_head_maskrrr:r;rtriuonesr3r enumeraterrr )r]rrrBrrrCrrprqrrrmkwargs input_shaperjr past_lengthtoken_type_embedsseq_lenrA pos_embedsrall_hidden_statesall_attentionsrrrs rruzCTRLModel.forwards6b2C1N-TXT_T_TqTq!*!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T 0*$++>O OU;   U  +<<_MO:I:Uo446[\   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,55a8BB1EN,..TZZ.@N!N2ekk$**6M6Q6QQN&&y$++2E2EF  %+00[_EN $~ 6  ):):!; ;  !    FF9-Mb/zz%**W{%:Gkd $D>d 'tnd d^d !.d  uU\\"$;; <d d r!rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc eZdZdgZfdZe ddeejdee e ejdeejdeejdeejdeejd eejd eejd ee d ee d ee dee deejde e ejeffdZddZxZS)CTRLLMHeadModelzlm_head.weightct||t||_t j |j |jd|_|jy)NTr) rPrQrrrrVrrlm_headrr]rr^s rrQzCTRLLMHeadModel.__init__sG  $V,yy0A0AM  r!rrrBrrrCrlabelsrprqrrrmrc ~| | n|jj} |j|||||||| | | | |  }|d}|j|}d}|*|j||fd|jj i|}| s|f|ddz}||f|zS|St |||j|j|jS)a  input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N) rrBrrrCrrprqrrrmrrr)losslogitsrrr) rrrr loss_functionrr rrr)r]rrrBrrrCrrrprqrrrmrtransformer_outputsr lm_logitsrrJs rruzCTRLLMHeadModel.forwardsx&1%c $D>c 'tnc d^c !.c uU\\"$:: ;!c c Jdr!ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZfdZe ddeejdeeeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d ee de eeje ffdZxZS)CTRLForSequenceClassificationct|||j|_t||_t j |j|jd|_|jy)NFr) rPrQ num_labelsrrrrVr classifierrrs rrQz&CTRLForSequenceClassification.__init__LsR   ++$V,))FMM4??O r!rrrBrrrCrrrprqrrrc | | n|jj} |j|||||||| | | |  } | d}|j|}||jdd\}}n|jdd\}}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d |tj||j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:S)a2 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```N) rrBrrrCrrprqrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r%)rr$z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrr)rrrrr9 pad_token_idrr*rrint32r(argmaxrrr^rw problem_typerr$rrTrsqueezerrrrrr)r]rrrBrrrCrrrprqrrrrrrjsequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrJs rruz%CTRLForSequenceClassification.forwardUsP&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E' -;;*55   r!r)rwrxryrQrrrrrrrrrrrurzr{s@rrr@sW15EI6:59371559-1$(,0/3&*p E,,-p "%e.?.?(@"ABp !!2!23 p !!1!12 p u//0 p E--.p   1 12p ))*p D>p $D>p 'tnp d^p  uU\\"$<< =p p r!r)rrrrr ).__doc__typingrrnumpyr:rrtorch.nnrrr cache_utilsr generationr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_ctrlr get_loggerrwrr r4rKModulerMrrrrrr__all__rr!rr1s$ " AA')ii-YY+   H % %4DDN`/299/d*/***I #I I X Ad)?Ad  AdH { $7{  { | cr!