hhBddlZddlmZmZddlZddlmZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZdd lmZmZdd lmZmZmZmZdd lmZmZdd lmZmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.e(j^e0Z1GddejdZ3dZ4d9dZ5dejlde7dejlfdZ8dZ9GddejdZ:Gd d!e:Z;Gd"d#e:Z<ed$Gd%d&ejdZ=e:e;eGd(d)eZ?e&Gd*d+e!Z@Gd,d-ejdZAe&Gd.d/e@ZBe&Gd0d1e@e ZCGd2d3ee@ZDGd4d5ee@ZEGd6d7ee@ZFgd8ZGy):N)OptionalUnion)nn)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)_flash_attention_forward!flash_attn_supports_top_left_mask)GenericForQuestionAnswering GenericForSequenceClassificationGenericForTokenClassificationGradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg)check_model_inputs)DiffLlamaConfigc$eZdZfdZdZxZS) DiffLlamaMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFbias) super__init__config hidden_sizeintermediate_sizerLinear gate_projup_proj down_projr hidden_actact_fnselfr* __class__s n/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/diffllama/modeling_diffllama.pyr)zDiffLlamaMLP.__init__8s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ c|j|j|j||j|z}|SN)r0r2r.r/)r4xr0s r6forwardzDiffLlamaMLP.forwardBs6NN4;;t~~a/@#ADLLQRO#ST r7)__name__ __module__ __qualname__r)r; __classcell__r5s@r6r#r#7s 0r7r#c|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)r:x1x2s r6 rotate_halfrKGsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r7c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezerK)qkcossin position_ids unsqueeze_dimq_embedk_embeds r6apply_rotary_pos_embrVNsY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r7 hidden_statesn_repreturnc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r N)rFexpandreshape)rWrXbatchnum_key_value_headsslenhead_dims r6 repeat_kvraiso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr7c>ddtjd|zzz S)Ng?g333333?g333333ӿ)mathexp) layer_idxs r6lambda_init_fnrfus txxy 011 11r7cxeZdZdZddedeeffd Zeddd dd e jd e e je jfd ee jd ee jdee d edee jde e jee jee e jffdZxZS)DiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr*rect|||_||_|-tj d|j jd|j|_|j|_ |j|_ t|d|j|jz|_ |j|_|j|jz|_|j |_|j"|_d|_t'j(|j|j|jz|j*|_t'j(|j|j|jz|j*|_t'j(|j|j|jz|j*|_t'j(|j|jz|j|j*|_t5||_t'j8t;j<d|j>|jf|_ t'j8t;j<d|j>|jf|_!t'j8t;j<d|j>|jf|_"t'j8t;j<d|j>|jf|_#t'jHd|jz|jJd |_&y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.r`Tr&r)sizerCF)epselementwise_affine)'r(r)r*relogger warning_oncer5r<attention_dropoutr+num_attention_heads num_headsgetattrr`r^num_key_value_groupsmax_position_embeddings rope_theta is_causalrr-attention_biasq_projk_projv_projo_projrf lambda_init ParameterrGnormallambda_std_dev lambda_q1 lambda_k1 lambda_q2 lambda_k2RMSNorm rms_norm_eps groupnormr4r*rer5s r6r)zDiffLlamaAttention.__init__|s~  "      !8!8 9:,,  "(!9!9!--33 D4D4D4VW #)#=#= $(NNd6N6N$N!'-'E'E$ ++ii 0 0$..4==2PW]WlWlm ii 0 0$2J2JT]]2Zagavavw ii 0 0$2J2JT]]2Zagavavw ii >@P@PW]WlWlm ))4ell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdA $56;N;Nchir7past_key_valuepast_key_values4.58new_nameversionrWposition_embeddingsattention_maskrR use_cachecache_positionrYc |j\} } } | } |j|} |j|}|j|}| j | | |j |j jdd} |j | | |j|j jdd}|j | | |j|j jdd}|\}}t| |||\} }|'|||d}|j|||j|\}}t||j}t||j}tjtj |ddd}|j#dddd}tj$| |jddt'j(|j z }|#|ddddddd|j*df}||z}t,j.j1|dtj2j5| j6}t,j.j9||j:|j< }tj>tj@|jB|jDzdtj2j5| j6}tj>tj@|jF|jHzdtj2j5| j6}||z |jJz}tj$||}tj |dd\}}|||zz }d|jJz |jM|z}|jddjO}|jQ| | d}|jS|}||fS) Nr rCrQrPrrDrBrrEdtype)ptraining)*rjrxryrzviewrqr` transposer^rVupdatererarsrGrHchunkrepeatmatmulrcsqrtrFr functionalsoftmaxfloat32tordropoutrorrdsumrrrrr|r contiguousr\r{)r4rWrrrRrrrkwargsbsz target_len_q_len query_states key_states value_statesrPrQ cache_kwargs attn_weights causal_masklambda_1lambda_2 lambda_full attn_output attn_output1 attn_output2s r6r;zDiffLlamaAttention.forwards|+//1Z{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J z4+D+DE  t/H/HI yy\1!!D"M #**1aA6 ||L*2F2Fq!2LMPTPYPYZ^ZgZgPhh  %(Aq2HJ4D4DR4H2H)HIK'+5L}},,\r,WZZ[g[m[mn }},,\T=S=S^b^k^k,l 99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< ll<> %*[[aQ%G" l"[<%?? 4+++t~~k/JJ !++Aq1<<> !))#ub9 kk+. L((r7r9NNNFN)r<r=r>__doc__r!rintr)rrGTensortuple LongTensorrboolr;r?r@s@r6rhrhysG j j8C= jD%0A6R 2637+/59<)||<)#5<<#=><)!. <) u//0 <) "% <)<)!!1!12<) u||Xell3XeELL>Q5RR S<)S<)r7rhc*eZdZdZfdZeddd ddejd eejejfd e ejd e ejde e d e d e ejdeejdffdZ xZS)DiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cBt||i|t|_yr9)r(r)r_flash_attn_uses_top_left_mask)r4argsrr5s r6r)z!DiffLlamaFlashAttention2.__init__s# $)&) /P.Q+r7rrrrNrWrrrRrrrYc@ t|tr td|j\}} } |j |} |j |} |j |} | j|| |j|jjdd} | j|| |j|jjdd} | j|| |j|jjdd} |+tjd|j| |\}}n|\}}t| | ||\} } |'|||d}|j!| | |j"|\} } | jdd} | jdd} | jdd} |j$r |j&nd}| j(}| j*j,dk7r| j*j,nd}|t.j0k(rt/j2r:t5t.d rt/j6|nt/j8}nMt5|j:d r|j:j<}n |jj>j(}tjd |d | jA|} | jA|} | jA|} t/jB| dd \}}|jEdddd}|jEdddd}tG| | ||| ||tI|dd|jJ|jL }tG| | ||| ||tI|dd|jJ|jL }t/jN||gd }t/jB|dd \}}t/jPt/jR|jT|jVzdt.j0jA| j(}t/jPt/jR|jX|jZzdt.j0jA| j(}||z |j\z}|||zz }d|j\z |j_|z}|ja|| djc}|je|}|dfS)Nz`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersr rCaYThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be removed and `position_embeddings` will be mandatory.rmpscpuget_autocast_dtype_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rDsliding_window)rRrruse_top_left_maskrvrBr)3 isinstancer ValueErrorrjrxryrzrrqr`rr^rmrn rotary_embrVrrerrordevicetyperGris_autocast_enabledhasattrrget_autocast_gpu_dtyper*rweightrrrrrrrrvrHrdrrrrrr|rr\rr{)r4rWrrrRrrrrrrrrrrPrQr dropout_rate input_dtype device_type target_dtype value_states1 value_states2rrrrrrs r6r;z DiffLlamaFlashAttention2.forwards o{ 3}  &**, UA{{=1 [[/ {{=1 $((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm  &   G  |\BHC*HC#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J $--a3 ))!Q/ #--a3 15t--C #(( 2>2E2E2J2Je2Sl))..Y^ %-- '((*u&:;,,[9557 &?@#{{BB #{{1177    >$  (??<8L#|4J'??<8L',{{<'J$ }%,,Q1a8 %,,Q1a8 /     % "4)94@"AAnn  0     % "4)94@"AAnn  ii| <"E %*[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !))#ub9DDF kk+. D  r7r)r<r=r>rr)rrGrrrrrrr;r?r@s@r6rrs R%0A6R 6:37+/59B!||B!#5<<#=>B!!!1!12 B! u//0 B! "% B!B!!!1!12B! u||T! "B!SB!r7rcTeZdZdZeddd ddej deej ej fd eej d eejdee d e d eejd eej eej eeej ffdZ y)DiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. rrrrNrWrrrRrrrYc |j\} } } |j|} |j|} |j|}| j | | |j |j jdd} | j | | |j|j jdd} |j | | |j|j jdd}|\}}t| | ||\} } |'|||d}|j| ||j|\} }t| |j} t||j}tjtj |ddd}|j#dddd}|}||ddddddd| j$df}| j&j(dk(r2|0| j+} | j+} |j+}|duxr| dkD}tj,j.j1| | |||j2r |j4nd| }tj |dd\}}tj6tj8|j:|j<zdtj> jA| jB}tj6tj8|jD|jFzdtj> jA| jB}||z |jHz}|||zz }d|jHz |jK|z}|jddj+}|j | | d}|jM|}|dfS) Nr rCrrDrBrcudar) attn_mask dropout_prvr)'rjrxryrzrrqr`rr^rVrrerarsrGrHrrrFrrrrrscaled_dot_product_attentionrrordrrrrrrrrr|rr{)r4rWrrrRrrrrrrrrrrrPrQrrrvrrrrrrs r6r;zDiffLlamaSdpaAttention.forward{sm&**, UA{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J z4+D+DE  t/H/HI yy\1!!D"M #**1aA6 $  %%aA/E1A1A"1E/E&EFK    # #v -+2I'224L#..0J'224L 4'5EAI hh))FF   !04 d,,3 G &+[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !++Aq1<<> !&&sE26 kk+. D  r7r) r<r=r>rrrGrrrrrrr;r7r6rrss%0A6R 2637+/59I!||I!#5<<#=>I!!. I! u//0 I! "% I!I!!!1!12I! u||Xell3XeELL>Q5RR SI!SI!r7rrc,eZdZdfd ZdZdZxZS)DiffLlamaRMSNormct|tjt j ||_||_y)z? DiffLlamaRMSNorm is equivalent to T5LayerNorm N)r(r)rr}rGonesrvariance_epsilon)r4r+rkr5s r6r)zDiffLlamaRMSNorm.__init__s1 ll5::k#:; #r7c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrCrBT)keepdim) rrrGrpowmeanrsqrtrr)r4rWrvariances r6r;zDiffLlamaRMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r7c^t|jjd|jS)Nz, eps=)rrrFr)r4s r6 extra_reprzDiffLlamaRMSNorm.extra_reprs* ))*+6$2G2G1HIIr7)gư>)r<r=r>r)r;rr?r@s@r6rrs$;Jr7r)eagerflash_attention_2sdpac>eZdZdedeffd Zeddd ddejd e ejd e ejde e d e e d e ejd e e ejejfdeedejfdZxZS)DiffLlamaDecoderLayerr*rec:t||j|_t|j|||_t ||_t|j|j|_ t|j|j|_ y)N)r*rerk) r(r)r+DIFFLLAMA_ATTENTION_CLASSES_attn_implementation self_attnr#mlprrinput_layernormpost_attention_layernormrs r6r)zDiffLlamaDecoderLayer.__init__sz !--4V5P5PQY_ktu'/0B0BH[H[\(89K9KQWQdQd(e%r7rrrrrWrrRrrrrrYc |} |j|}|jd|||||||d|\}} | |z}|} |j|}|j|}| |z}|S)N)rWrrRrrrrr)rrrr) r4rWrrRrrrrrresidualrs r6r;zDiffLlamaDecoderLayer.forwards!,,]; )4>>  ')%+) 3     q!=0 !55mD /  =0 r7)NNNFNN)r<r=r>r!rr)rrGrrrrrrrrr;r?r@s@r6rrsff3f%0A6R2637+/$)59KO||!.u//0  "%  D> !!1!12&eELL%,,,F&GH+, Sr7rc\eZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZfdZxZS) DiffLlamaPreTrainedModelr*modelTrrF)rW attentionsct||t|tr|jj j d|jj|jj j d|jj|jj j d|jj|jj j d|jjyy)Nr) r( _init_weightsrrhrdatanormal_r*rrrr)r4moduler5s r6rz&DiffLlamaPreTrainedModel._init_weights$s f% f0 1    ! ! ) )!T[[-G-G H    ! ! ) )!T[[-G-G H    ! ! ) )!T[[-G-G H    ! ! ) )!T[[-G-G H 2r7)r<r=r>r!__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrrh_can_record_outputsrr?r@s@r6rrs^ &*#01#4"5N!"'.( IIr7rc~eZdZUejed<ddeffd Zeje dZ xZ S)DiffLlamaRotaryEmbeddinginv_freqr*ct|t|drUt|jt r;|jj d|jj d|_nd|_|j|_ |j|_ ||_ t|j|_ |j|j|\}|_|jd|d|j |_y)N rope_scaling rope_typerdefaultrF) persistent)r(r)rrrdictgetrrtmax_seq_len_cachedoriginal_max_seq_lenr*r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)r4r*rrr5s r6r)z!DiffLlamaRotaryEmbedding.__init__0s  6> *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r9) r<r=r>rGrr r!r)no_gradrr;r?r@s@r6rr-s=ll//"U]]_ < rrRrr?rrrpast_seen_tokensrrWr decoder_layers r6r;zDiffLlamaModel.forwardcs[  -t"; <YZ Z  *.*;*;I*FM 0*$++>O  !CRC^==?de +0<< "2]5H5H5K"KTaThTh,N  )33A6L(;;&))+%  & "oom\J![[)H4;;+H+HI M)*) /-$7 M  -0 &++  r7)NNNNNNN)r<r=r>r!r)rrrrGrrr FloatTensorrrrrr;r?r@s@r6r/r/Qs 151537+/5959$(8 E,,-8 !.8 u//0 8 "% 8   1 12 8 !!1!128 D>8 +,8  !8 8 r7r/cdeZdZdgZddiZddgdgfiZfdZee dde e jde e jd e e jd e e d e e jd e e jd e ede e jdeee jfdeedefdZxZS)DiffLlamaForCausalLMzlm_head.weightlm_head colwise_reprWlogitsct||t||_|j|_t j |j|jd|_|jyr%) r(r)r/rr4rr-r+rJr=r3s r6r)zDiffLlamaForCausalLM.__init__sU  #F+  ++yy!3!3V5F5FUS  r7r>rrRrr?labelsrrlogits_to_keeprrYc z|jd|||||||d| } | j} t| tr t | dn| } |j | dd| ddf}d}|*|j d|||jjd| }t||| j| j| jS)a Example: ```python >>> from transformers import AutoTokenizer, DiffLlamaForCausalLM >>> model = DiffLlamaForCausalLM.from_pretrained("google/diffllama-7b") >>> tokenizer = AutoTokenizer.from_pretrained("google/diffllama-7b") >>> prompt = "What is your favorite condiment?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is your favorite condiment?" ```)r>rrRrr?rrN)rLrNr4)lossrLrrWrr) rrBrrslicerJ loss_functionr*r4rrrWr)r4r>rrRrr?rNrrrOroutputsrW slice_indicesrLrQs r6r;zDiffLlamaForCausalLM.forwards@,64:: , )%+') ,  ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r7) NNNNNNNNr)r<r=r>_tied_weights_keys_tp_plan_pp_planr)rrrrGrrrrGrrrrrrr;r?r@s@r6rIrIs0*+=)H_-z:;H151537+/59-1$(59348 E,,-8 !.8 u//0 8 "% 8   1 12 8 ))*8 D>8 !!1!128 c5<</08 +,8  8 8 r7rIc eZdZy)"DiffLlamaForSequenceClassificationNr<r=r>rr7r6rZrZr7rZceZdZdZy)DiffLlamaForQuestionAnswering transformerN)r<r=r>r rr7r6r^r^s%r7r^c eZdZy)DiffLlamaForTokenClassificationNr[rr7r6rarar\r7ra)rr/rIrZr^ra)Nr )HrctypingrrrGr activationsr cache_utilsrr r generationr integrationsr masking_utilsr modeling_flash_attention_utilsrrmodeling_layersrrrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrprocessing_utilsrutilsrrrrutils.deprecationr utils.genericrconfiguration_diffllamar! get_loggerr<rmModuler#rKrVrrrarfrhrrrrrrrr/rIrZr^ra__all__rr7r6rus0 " !;;)7/i PK-&RR0/4   H % 299  (6 UU\\ U# U%,, U2b)b)JR!1R!jR!/R!jY'JryyJ(J* 1 "+6+\III4!