hIfddlmZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1e(jde3Z4ee&dGdde$Z5edGddejlZ7GddejlZ8Gd d!ejlZ9d"Z:dGd#Z;d$ejxd%e=d&ejxfd'Z> dHd(ejld)ejxd*ejxd+ejxd,eejxd-e?d.e?d/e"e%fd0Z@Gd1d2ejlZAGd3d4eZBe&d5e&Gd6d7e ZCe&Gd8d9eCZDGd:d;ejlZEe&d<Gd=d>eCeZFGd?d@ejlZGe&GdAdBeCZHe&dCGdDdEeCe1ZIgdFZJy)I) dataclass)CallableOptionalUnionN)check_model_inputs)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee ejdfed<dZ ee ejdfed<dZeejed <dZejed <dZee e ejed <dZee ejdfed <dZee ejdfed <dZeejed<y)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r&rtorch FloatTensor__annotations__r'r(tupler)r*r+r,r-r.r/r0b/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/csm/modeling_csm.pyr%r%2s'<)-D(5$$ %, $FE  $AEOXeE%*;*;$<=>E=AM8E%"3"3S"89:A:>Ju00#567>6:!2!23:.2%++2OS!8E%8I8I2J,K#LSKO%0A0A30F*G!HOHLhuU->->-C'DEL15M8E--.5r:r%RMSNormc,eZdZdfd ZdZdZxZS) CsmRMSNormct|tjt j ||_||_y)z9 CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr5onesweightvariance_epsilon)self hidden_sizeeps __class__s r;rAzCsmRMSNorm.__init__fs1 ll5::k#:; #r:c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NrT)keepdim) dtypetor5float32powmeanrsqrtrFrE)rGr) input_dtypevariances r;forwardzCsmRMSNorm.forwardnsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r:c^t|jjd|jS)Nz, eps=)r8rEshaperFrGs r; extra_reprzCsmRMSNorm.extra_reprus* ))*+6$2G2G1HIIr:)gư>)r1r2r3rArVrZ __classcell__rJs@r;r>r>ds$;Jr:r>c~eZdZUejed<ddeffd Zeje dZ xZ S)CsmRotaryEmbeddinginv_freqconfigct|t|drUt|jt r;|jj d|jj d|_nd|_|j|_ |j|_ ||_ t|j|_ |j|j|\}|_|jd|d|j |_y)N rope_scaling rope_typetypedefaultr_F persistent)r@rAhasattr isinstancerbdictgetrcmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr`r rope_init_fnattention_scalingregister_bufferr_original_inv_freq)rGr`devicer_rJs r;rAzCsmRotaryEmbedding.__init__|s  6> *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.N) r1r2r3r5Tensorr7r rAno_gradrrVr[r\s@r;r^r^ys=ll/y/"U]]_ <