hcBbdZddlZddlmZmZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ddl-m.Z.m/Z/e*j`e1Z2dZ3dejhde5de5fdZ6 d~dejhde5deejhfdZ7 dde8e5e5fde9de5deejtde5d ejvf d!Z<Gd"d#eZ=Gd$d%eZ>Gd&d'eZ?Gd(d)e jZAGd*d+e jZBGd,d-e jZCGd.d/ejjZDGd0d1e jZEGd2d3e jZFGd4d5e jZGGd6d7e jZHGd8d9e jZIGd:d;e jZJGd<d=e jZKGd>d?e je&ZLGd@dAe je&ZMGdBdCe je&ZNGdDdEe jZOGdFdGe jZPGdHdIeZQGdJdKeZRe)GdLdMe'ZSGdNdOeSZTGdPdQeSZUGdRdSeSZVGdTdUeSZWGdVdWeSZXGdXdYeSZYGdZd[eSZZGd\d]eSZ[Gd^d_e jZ\Gd`dae jZ]e)dbcGdddeeSZ^e)dfcGdgdheSeZ_ ddieSdejdjeejdeejtdke9dle9dme9dnee jdoeadpead eeje8ejejfffdqZbe)drcGdsdteSZce)ducGdvdweSZdGdxdye jZee)dzcGd{d|e'Zfgd}Zgy)zPyTorch SpeechT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossL1Loss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSpectrogramOutput)EmbeddingAccessMixinPreTrainedModel)auto_docstringlogging)deprecate_kwarg)SpeechT5ConfigSpeechT5HifiGanConfig input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r!r"r#shifted_input_idss l/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/speecht5/modeling_speecht5.pyshift_tokens_rightr-7s}"++IOO<(CRC0668ae4adLMM""#4#AB%%&:f&DcJ  //r.r' mask_prob mask_length min_masksreturnc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr8r7r9sequence_lengths r,compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :-=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r.r!past_key_values_lengthc|j\}}|j||j|j|j}|jdz|z}||j jdkDr4|j ||jz|j|j|j jd|jdj||djS)Nrrr%) size"create_position_ids_from_input_idsrrrrrrr index_selectrrL)r{r!rbszseq_len position_idsmax_poss r,rz-SpeechT5SinusoidalPositionalEmbedding.forwardDs ~~' W>>y$JZJZ\rsvv    ""Q&0 T\\&&q) )   g 3T5G5GIYIY Z||((L,=,=b,ABGGWVXY``bbr.c|j|j}tj|dj ||z|z}|j |zS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner>rcumsumtype_aslong)r{r!rrmaskincremental_indicess r,rzHSpeechT5SinusoidalPositionalEmbedding.create_position_ids_from_input_idsSsW||K(,,.$||Da8@@FI__cgg"'')K77r.rr)rrr__doc__r>rror staticmethodrrno_gradTensorrrrrs@r,rr sNScS#SHUXMSG3GsGQYZ]Q^G1c1#1HUXM11"U]]_ c cs c cbc88478QYZ]Q^8r.rc$eZdZfdZdZxZS)SpeechT5PositionalConvEmbeddingct|tj|j|j|j |j dz|j |_tjj}ttjjdr$tjjj}tr(ddl }|jj|jj d5||jdd|_dddt|jdrU|jjj j"}|jjj j$}n,|jj&}|jj(}|jj+|||jj+||n||jdd|_t-|j |_t0|j2|_y#1swY'xYw) Nr)rkpaddinggroups weight_normr) modifier_rankweight)namerparametrizations)rnrorrs hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsrwutilsrrrr deepspeedzeroGatheredParametersr original0 original1weight_gweight_vregister_external_parameterSpeechT5SamePadLayerrr rxry)r{r|rrrrr~s r,roz(SpeechT5PositionalConvEmbedding.__init__fs II      6622a777   hh** 288,,m <((33??K % ' 224993C3CST2U I' aH  Ityy"459955<<FF9955<<FF99--99-- NN 6 6tX F NN 6 6tX F#DIIH!DDI+F,J,JK  !?!?@ I Is IIc|jdd}|j|}|j|}|j|}|jdd}|SNrr)rrwrryrs r,rz'SpeechT5PositionalConvEmbedding.forwardsV%//15  -0  ]3  6 %//15 r.rrs@r,rresABr.rc*eZdZdZdfd ZdZxZS) SpeechT5ScaledPositionalEncodingu[ Scaled positional encoding, see §3.2 in https://huggingface.co/papers/1809.08895 ctj||}tjd|jd}tjtjd|dtj j tjd|z z}tj|j |z|dddddf<tj|j |z|dddddf<|jd}t|1|jd|dtj| |_||_tj$tj&d |_y) NrrrrEg@peFrp?)rrPrSrrrrrrrrrnrorrDropoutdropoutr Parametertensoralpha)r{rrmax_lenrpositiondiv_termr~s r,roz)SpeechT5ScaledPositionalEncoding.__init__s  [[# &<<7+55a899U\\!S!5;;GMMOTXT\T\]dTehkTkRllmii 08 ;<1add7 ii 08 ;<1add7 \\!_  T2%8zzG, \\%,,s"34 r.c||j|jddd|jdfzz}|j|}|S)Nr)rrrr)r{rs r,rz(SpeechT5ScaledPositionalEncoding.forwardsBDJJMchhqkM)9!:::ll3 r.)i)rrrrrorrrs@r,rrs 5r.rc&eZdZdfd ZdZxZS)"SpeechT5RelativePositionalEncodingct|||_||_tj j d|z||_yr)rnror max_lengthrr Embeddingpe_k)r{rrr~s r,roz+SpeechT5RelativePositionalEncoding.__init__s8 $HH&&q:~s; r.c~|jd}tjd|j|jtj }|dddf|dddfz }|j |||j k<|j dz |||j k\<||j z}|j|S)NrrrrF)r'rrSrrrrr)r{rrpos_seqs r,rz*SpeechT5RelativePositionalEncoding.forwards%%a(,,q'*--]5I5IQVQ[Q[-\!T'"WT1W%55/3.>4??**+.2oo.A4??*+DOO+yy!!r.)irrs@r,rrs < "r.rc$eZdZfdZdZxZS)rcPt||dzdk(rd|_yd|_y)Nrrr)rnronum_pad_remove)r{rr~s r,rozSpeechT5SamePadLayer.__init__s) #:Q#>!#Car.cV|jdkDr|ddddd|j f}|SNr)r rs r,rzSpeechT5SamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr.rrs@r,rrsKr.rc.eZdZdZfdZdZdZxZS)SpeechT5FeatureEncoderz.Construct the features from raw audio waveformc t||jdk(rDt|dgt |j dz Dcgc]}t ||dzc}z}nV|jdk(r.t |j Dcgc]}t||}}ntd|jdtj||_ d|_ d |_ ycc}wcc}w) Ngroupr)r}rlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rnrofeat_extract_normrrOnum_feat_extract_layersrhrr)r ModuleList conv_layersgradient_checkpointing_requires_grad)r{r|irr~s r,rozSpeechT5FeatureEncoder.__init__s   # #w .5fqIJNSTZTrTruvTvNwNIJ,Va!eDNK % % 0HMfNlNlHmCD*6A>K01I1I0JJst ==5&+#"Ns C" C'cJ|jD] }d|_ d|_y)NF) parameters requires_gradr)r{params r,_freeze_parametersz)SpeechT5FeatureEncoder._freeze_parameterss(__& (E"'E  (#r.c|dddf}|jr|jrd|_|jD] }||} |SNT)rtrainingrr)r{r/r conv_layers r,rzSpeechT5FeatureEncoder.forwardsP$QW-    4==*.M '** 6J&}5M 6r.)rrrrrorrrrs@r,rrs8#&$ r.rc$eZdZfdZdZxZS)SpeechT5FeatureProjectionc4t|tj|jd|j |_tj|jd|j|_ tj|j|_ y)Nr%eps) rnrorrrplayer_norm_epsrLinearr projectionrfeat_proj_dropoutrr{r|r~s r,roz"SpeechT5FeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r.cp|j|}|j|}|j|}||fSr)rr*r)r{rnorm_hidden_statess r,rz!SpeechT5FeatureProjection.forwards:!__];(:;  ]3 000r.rrs@r,r$r$s < 1r.r$c6eZdZfdZdZ d dej deejdeejfdZ de dejfdZ d e eje ffd Z d d ejdeejdeejfd ZxZS)SpeechT5SpeechEncoderPrenetct|||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_t!||_t%|j&|j(zdz|j|j(|_y)Nr4r)rnror|rfeature_encoderr$feature_projectionmask_time_probmask_feature_probrrrrruniform_masked_spec_embedrpos_conv_embedrmax_speech_positionsr"pos_sinusoidal_embedr,s r,roz$SpeechT5SpeechEncoderPrenet.__init__s  5f=";F"C  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "=fE$I  ' '&*=*= = A      % !r.c8|jjyr)r2rr{s r,freeze_feature_encoderz2SpeechT5SpeechEncoderPrenet.freeze_feature_encoders //1r.r/r1mask_time_indicesc|j|}|jdd}||j|jd|}|j |\}}|j |||}|j |}||z}| |jdj}n=tj|jddtj|j}|j|}||z}||fS)Nrr)r>r1r) r2r"_get_feature_vector_attention_maskr'r3_mask_hidden_statesr8rrrrPrr:) r{r/r1r>extract_featuresrpositional_conv_embedding padding_mask positional_sinusoidal_embeddingss r,rz#SpeechT5SpeechEncoderPrenet.forwards  // =+55a;  %!DD &&q)N +/*A*ABR*S' '00 ->~1 %)$7$7 $F!%(AA  %),,Q/446L ;;}':':2A'>ejjYfYmYmnL+/+D+D\+R(%(HH n,,r.feature_vector_lengthc|jddddf}|j|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)Nr%rrrrr) r _get_feat_extract_output_lengthsrrrr'rPrFrrSfliprQ)r{rFr1non_padded_lengthsoutput_lengthsr]s r,r@z>SpeechT5SpeechEncoderPrenet._get_feature_vector_attention_mask:s,22r2:1b5A>>?QRUUV[V`V`a#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr.r_cd}t|jj|jjD]\}}||||}|S)zH Computes the output length of the convolutional layers c>tj||z |ddzS)Nfloor) rounding_moder)rdiv)r@rkrls r,_conv_out_lengthzVSpeechT5SpeechEncoderPrenet._get_feat_extract_output_lengths.._conv_out_lengthOs"99\K7wWZ[[ [r.)zipr|rtru)r{r_rRrkrls r,rIz'> @W@W#X Q K,]KPM Qr.rct|jdds|S|j\}}}|)|jj |j ||<n|jj dkDr|jrt||f|jj |jj||jj}tj||jtj}|jj |j ||<|jjdkDr|jrt||f|jj|jj |jj"}tj||jtj}|dddfj%d|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r7r8r1r9r)r7r8r9r%)getattrr|rr7rrFr4r!rfmask_time_lengthmask_time_min_masksrrrrQr5mask_feature_lengthmask_feature_min_masksexpand)r{rr>r1r]rCrmask_feature_indicess r,rAz/SpeechT5SpeechEncoderPrenet._mask_hidden_statesZst{{$8$? 4A3E3E3G0 O[  (/3/E/E/H/HI\I\/]M+ , [[ ' '! + 5_-++44 KK88-++99 ! !& ->}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r.NN)rrrror=rrr LongTensor FloatTensorrr>r@rrIrArrs@r,r0r0s "2 6:9= -ll -!!1!12 -$E$5$56 -F  ]b]m]m  eEDTDTVYDY>Z &:>59 ,((,$E$5$56,!!1!12 ,r.r0cfeZdZfdZdZ ddej deej fdZxZ S)SpeechT5SpeechDecoderPrenetc Xt|||_tjt |j Dcgc]=}tj|dk(r |jn |j|j?c}|_ tj|j|j|_ t|j|j|j|_tj|j"|jz|j|_ycc}wr )rnror|rrrOspeech_decoder_prenet_layersr) num_mel_binsspeech_decoder_prenet_unitslayersr final_layerrpositional_dropoutr9encode_positionsspeaker_embedding_dimspeaker_embeds_layerr{r|rr~s r,roz$SpeechT5SpeechDecoderPrenet.__init__s  mm vBBC     +,6F''v7Y7Y66   99V%G%GI[I[\ @  % %     ' '!  %'IIf.J.JVM_M_._agasas$t! sAD'ctj|d|}|jdj|j ddd}tj |dk(|ddzd|z z S)Nrrr)r bernoullirrepeatrwhere)r{ inputs_embedsrr all_maskss r,_consistent_dropoutz/SpeechT5SpeechDecoderPrenet._consistent_dropoutsd}Q/15NN1%,,]-?-?-BAqI {{9>=!>LII  <<99A=!C   ..6 f::Q> > ggiDO"DOzz&"G"GH r.c|j|}|j|}|j|j|}|j|}|Sr)rwrryrrs r,rz"SpeechT5BatchNormConvLayer.forwardsJ -0  6 ?? & OOM:M ]3 r.rrrs@r,r{r{sI<r.r{c^eZdZfdZdej fdZdej fdZxZS)SpeechT5SpeechDecoderPostnetc t|||_tj|j |j |jz|_tj|j |j|_ tjt|jDcgc]}t||c}|_ycc}wr)rnror|rr)rrdr0feat_outprob_outrrOr~r{rfrls r,roz%SpeechT5SpeechDecoderPostnet.__init__s   &"4"4f6I6IFLcLc6cd  &"4"4f6M6MN mmeZdZfdZdej fdZxZS)SpeechT5TextEncoderPrenetct|||_tj|j |j |j|_t|j|j |j|_ yr) rnror|rr vocab_sizerr" embed_tokensrrhmax_text_positionsrir,s r,roz"SpeechT5TextEncoderPrenet.__init__se  LL):):F}|jA|}||fS)z#Input shape: Batch x Time x ChannelNr%rrrTz$Attention weights should be of size z , but is rrz!Attention mask should be of size rz/Head mask for a single layer should be of size )rr!z `attn_output` should be of size )!rrrrr is_updatedgetrcross_attention_cacheself_attention_cacherfkeysvaluesrrrrrrupdater[rbmmr) contiguousmatmulrrvsoftmaxrr!rr)r{rrrr1rrrris_cross_attentionrtgt_lenr^ query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weights reshape_q rel_pos_biasattn_weights_reshaped attn_probs attn_outputs r,rzSpeechT5Attention.forwardps .T9',,.Wa{{=1DLL@  &/+>?,77;;DNNK %*9*O*O'*9*N*N'&5#-?)] /"=*,33DNNCHHJ.55dnnELLL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*7It+>+E+E dnn?OQ_>`,( L&AEO..t~~>DNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   $$//166sT^^7KRQUQ^Q^_iijkmnoI << =3J3J2r3RSL'11!Q7<<dnn$m&8&8&;]=O=OPQ=RL L (L  %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<rrrQrorrrr tuplerrrs@r,rrMsX$'%*#$(CCC% C TN C tn CD>C:%0A6R48+/152604"'15~2||~2#5<<0~2"% ~2 !. ~2 "%,,/ ~2  -~2 ~2!.~2 u||Xell3Xe_D E~2S~2r.rc$eZdZfdZdZxZS)SpeechT5FeedForwardct|tj|j|_tj |j||_t|jtrt|j|_ n|j|_ tj ||j|_tj|j|_yr)rnrorractivation_dropoutintermediate_dropoutr)rintermediate_denser hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropout)r{r|intermediate_sizer~s r,rozSpeechT5FeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@Q"R f'' -'-f.?.?'@D $'-'8'8D $II&79K9KL jj)>)>?r.c|j|}|j|}|j|}|j|}|j |}|Sr)rrrrrrs r,rzSpeechT5FeedForward.forwardsX// > 00? 11-@ ))-8 ++M: r.rrs@r,rrs @r.rc eZdZdeffd Z d dej deej deej deej def dZ xZ S) SpeechT5EncoderLayerr|ct|t|j|j|j d|_tj|j|_ tj|j|j|_ t||j|_tj|j|j|_y)NF)rrrrr&)rnrorrencoder_attention_headsattention_dropout attentionrrrrrr(rrencoder_ffn_dim feed_forwardfinal_layer_normr,s r,rozSpeechT5EncoderLayer.__init__ s *((44,,   zz&"7"78 ,,v'9'9v?T?TU/8N8NO " V-?-?VEZEZ [r.rr1rrrc|}|j|||||\}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|S)as Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(config.encoder_attention_heads,)`. position_bias (`torch.FloatTensor`): relative position embeddings of size `(seq_len, seq_len, hidden_size // encoder_attention_heads)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rr1rrr)rrrrr) r{rr1rrrresidualroutputss r,rzSpeechT5EncoderLayer.forwards.!&*nn')+'/ '5' # | ]3  =0  6 %(9(9-(HH --m<  "    &Gr.)NNNF) rrrrrorrrrQrrrs@r,rr ss \~ \ 262604"' ,||,!.,"%,,/ ,   - ,  ,r.rcHeZdZddeffd Zeddd ddejdeejd eejd eejd eejd eejdee d ee dee deejfdZ xZ S)SpeechT5DecoderLayerr|ct|t|j|j|j d||_tj|j|_ tj|j|j|_ t|j|j|j d||_tj|j|j|_t!||j"|_tj|j|j|_y)NT)rrrrrr&)rrr)rnrorrdecoder_attention_headsr self_attnrrrrrr(self_attn_layer_norm encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrr)r{r|rr~s r,rozSpeechT5DecoderLayer.__init__Hs *((44,,  zz&"7"78 $&LL1C1CI^I^$_!-     * *,,  (*||F4F4FFLaLa'b$/8N8NO " V-?-?VEZEZ [r.rrrrrr1encoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskr use_cacherc ~|} |j|||||| \}} |j|}| |z}|j|}d} |D|} |j||||||| \}} |j|}| |z}|j |}||j |z}|j |}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_values (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrr1rrrN)rrr1rrrr)rrrrrrr)r{rr1rr rr rrr rrself_attn_weightscross_attn_weightsrs r,rzSpeechT5DecoderLayer.forward`s@!,0>>'+)+/) ,:, ( ( ]3  =0 11-@ " ,$H040A0A+!65 : /"3-1B1 -M-!LL7M$}4M 88GM&(9(9-(HH --m<  "  )+=> >Gr.r) NNNNNNFTN) rrrrrorrrrr rQrrrs@r,rrGs\~\0%0A6R268<9=26=A+/,1$(15I||I!.I ( 5 I !) 6 I "%,,/ I%-U\\$:I"%I$D>ID>I!.ISIr.rcDeZdZUeed<dZdZdZdejfdZ y)SpeechT5PreTrainedModelr|speecht5r/Tmodulec F|jj}t|trtj j |jjddtjd|jjd|jjzz ztj j|jjdnt|tr'|j j"j%dnt|t&rtjd|j(j*z }tj j-|j(j| |tj j-|j(j| |n/t|tj.rZ|jj"j d||j|jj"j1nt|tj2tj4tj6frK|jj"j1|jj"j%dn7t|tj8rtj j;|j|jtj|j<|j|jdzz }tj j-|j| |n~t|tj>rd|jj"j d||j@1|jj"|j@j1tC|d r*tj j-|jDyy) zInitialize the weightsrrrmeanstdr)abr4Nr7)#r|initializer_rangerrrinitnormal_rwrrrrk in_channels constant_rmrrdatafill_r$r* in_featuresr6r)zero_rrrrskaiming_normal_rrrrr7)r{rrks r, _init_weightsz%SpeechT5PreTrainedModel._init_weightsskk++ f= > GGOO "" !v{{'>'>q'AFKKD[D['["\]]   GG  fkk.. 2  @ A LL   # #C (  9 : !f//;;; 6. / GG  V55 6 0r.N) rrrr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointingrModuler$r.r,rrs) "$O&*#"7BII"7r.rceZdZdZdeffd Z d dejdeejdeejdee dee d ee d e e e ffd ZxZS) SpeechT5Encoderzu Transformer encoder consisting of *config.encoder_layers* layers. Each layer is a [`SpeechT5EncoderLayer`]. r|ct||tj|j|j |_tj|j|_ |j|_ tjt|jDcgc] }t|c}|_t#|j|j$z|j&|_d|_|j-ycc}w)Nr&F)rnrorrrr(rrrrencoder_layerdrop layerdroprrOencoder_layersrrfrrencoder_max_relative_positionrr post_init)r{r|r^r~s r,rozSpeechT5Encoder.__init__s  ,,v'9'9v?T?TUzz&"7"78 11mm5QWQfQfKg$ha%9&%A$hi A   &"@"@ @&BfBf ',# %isDrr1 head_maskroutput_hidden_states return_dictr:c||n|jj}||n|jj}||n|jj}|t ||j }|j |}|j|}|j|}txs t|}|rdnd} |rdnd} |_|jdt|jk7r6tdt|jd|jddt|jD]l\} } |r| |fz} d} |j r$t#j$g}||j&k} | r|r| |||||| nd|}|d}| rd }|sd| d fz} n|r| |fz} |st)d || | fDSt+|| | S) a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the encoder prenet. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr*rz&The head_mask should be specified for  layers, but it is for .F)r1rrrr]rc3&K|] }|| ywrr*.0vs r, z*SpeechT5Encoder.forward..Osmq_`_lmslast_hidden_stater attentions)r|rr4use_return_dictrrFrrrrrrrTrfr) enumerater!rrJr/rr)r{rr1r3rr4r5r synced_gpusall_hidden_statesall_self_attentionsidx encoder_layerskip_the_layerdropout_probability layer_outputss r,rzSpeechT5Encoder.forwardsH2C1N-TXT_T_TqTq$8$D $++JjJj &1% f 'tn f d^f  uo% &f r.r,ceZdZdZdeffd Z d dejdeejdeejdee dee d ee d e e e ffd ZxZS) SpeechT5EncoderWithSpeechPrenetz Wrapper around SpeechT5Encoder that applies SpeechT5SpeechEncoderPrenet to convert the audio waveform data to hidden features. r|ct||t||_t ||_|j yr)rnror0prenetr,wrapped_encoderr2r,s r,roz(SpeechT5EncoderWithSpeechPrenet.__init__^5  1&9 .v6 r.r/r1r3rr4r5r:c^|j||\}}|j||||||}|SN)rr1r3rr4r5rPrQ r{r/r1r3rr4r5rrs r,rz'SpeechT5EncoderWithSpeechPrenet.forwardfsG)- L.(Q% ~&&')/!5# ' r.rKrLrs@r,rNrNXs ~26,0,0/3&*''!.ELL)  $D>  'tn d^ uo% &r.rNceZdZdZdeffd ZdZdZ ddejde ejde ejd e e d e e d e e d e eeffd ZxZS)SpeechT5EncoderWithTextPrenetz| Wrapper around SpeechT5Encoder that applies SpeechT5TextEncoderPrenet to convert the input_ids to hidden features. r|ct||t||_t ||_|j yr)rnrorrPr,rQr2r,s r,roz&SpeechT5EncoderWithTextPrenet.__init__5  /7 .v6 r.c6|jjSrrPget_input_embeddingsr<s r,r]z2SpeechT5EncoderWithTextPrenet.get_input_embeddings{{//11r.c:|jj|yrrPset_input_embeddingsr{values r,raz2SpeechT5EncoderWithTextPrenet.set_input_embeddings ((/r.r/r1r3rr4r5r:cV|j|}|j||||||}|SrTrUrVs r,rz%SpeechT5EncoderWithTextPrenet.forwards@ L1 &&')/!5# ' r.rK)rrrrrror]rarr_rrrQrrrrrrs@r,rXrX}s~20 26,0,0/3&*''!.ELL)  $D>  'tn d^ uo% &r.rXceZdZdZdeffd Z d dejdeejdeejdee dee d ee d e e e ffd ZxZS) SpeechT5EncoderWithoutPrenet This wrapper class is a helper class to correctly load pretrained checkpoints when used in combination with [`SpeechT5Model`]. r|cdt||t||_|j yr)rnror,rQr2r,s r,roz%SpeechT5EncoderWithoutPrenet.__init__)  .v6 r.r/r1r3rr4r5r:c0|j||||||SrT)rQ)r{r/r1r3rr4r5s r,rz$SpeechT5EncoderWithoutPrenet.forwards.##&)/!5# $  r.rKrLrs@r,rgrgs ~26,0,0/3&* '' !. ELL)  $D>  'tn  d^  uo% & r.rgcveZdZdZdeffd Z ddeejdeejdeejdeejdeejd eejd ee ejd ee d ee d ee dee deejde eeffdZxZS)SpeechT5Decoderzt Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`SpeechT5DecoderLayer`] r|c  t|||j|_t j t |jDcgc]}t||c}|_ d|_ |jycc}w)N)rF) rnrodecoder_layerdropr/rrrOdecoder_layersrrfrr2rls r,rozSpeechT5Decoder.__init__sh  11mmX]^d^s^sXt$uST%9&A%N$uv &+#  %vsBrr1rr r3cross_attn_head_maskrr rr4r5rr:c | | n|jj} | | n|jj} ||n|jj}| | n|jj} |j dd} |j r%|jr|rtjdd}|r6|4tt|jt|j}|r:t|tr*tjdtj|}||jnd}t!|| ||}||t#||j$| d}t'xs t)|}| rd nd}| rd nd}| r|d nd}t+||gd d gD]j\}}| |j dt-|j.k7s3t1d |d t-|j.d|j ddt3|j.D]\}}| r||fz}d}|jr$t5j6g}||j8k}|r|sE||||||||nd|||nd|| ||  }|d}| sm||dfz}|y||dfz}| r||fz}| std|||||fDSt;|||||S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the decoder prenet. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r|zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)rr*r3rqzThe `z` should be specified for r7r8)r rr rrr rrrc3$K|]}|| ywrr*r:s r,r=z*SpeechT5Decoder.forward..{s=s)r?rrr@cross_attentions)r|rr4r rArrr!logger warning_oncer r rrfrom_legacy_cacherrrrFrrrSrTrfr)rBrrJr/r)r{rr1rr r3rqrr rr4r5rrrrCrDrEall_cross_attentions attn_mask mask_namerF decoder_layerrHrIrJs r,rzSpeechT5Decoder.forwardscT2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03 #,DKK"8 V C#$58H$H!#N}}&+jjn#!4t~~!Ek)%'=3<3H3dI]Ii,@,Eos /"3#- M*!,M &9]1=M 1]4D D ':KM`bvw  9+++*1   r. NNNNNNNNNNNNrrrrrrorrr_r^rlistrQrrrrrrs@r,rmrms? ~ 6:59=A=A,07;=A$(,0/3&*15p  1 12p !!1!12p  ((9(9: p !))9)9 : p ELL) p 'u||4p "$u'8'8"9:p D>p $D>p 'tnp d^p !.p  u?? @p r.rmceZdZdZdeffd Z ddeejdeejdeejdeejdeejd eejd eejd ee ejd ee d ee dee dee deejde eeffdZxZS)SpeechT5DecoderWithSpeechPrenetz Wrapper around SpeechT5Decoder that applies SpeechT5SpeechDecoderPrenet to convert log-mel filterbanks to hidden features. r|ct||t||_t ||_|j yr)rnrorarPrmwrapped_decoderr2r,s r,roz(SpeechT5DecoderWithSpeechPrenet.__init__rRr.r/r1rr rtr3rqrr rr4r5rr:cd|j||}|j|||||||| | | | |  }|SN) rr1rr r3rqrr rr4r5rrPr)r{r/r1rr rtr3rqrr rr4r5rdecoder_hidden_statesrs r,rz'SpeechT5DecoderWithSpeechPrenet.forwardsV !% L:L M&&/)"7#9!5+/!5#)'  r.) NNNNNNNNNNNNNr}rs@r,rrsG ~5959=A=A59,07;=A$(,0/3&*15!u001!!!1!12! ((9(9: ! !))9)9 : ! %U\\2 !ELL)!'u||4!"$u'8'8"9:!D>!$D>!'tn!d^!!.! u?? @!r.rceZdZdZdeffd ZdZdZ ddee jdee jdee jd ee jd ee jd ee jd ee e jd eedeedeedeedee jdeeeffdZxZS)SpeechT5DecoderWithTextPrenetz{ Wrapper around SpeechT5Decoder that applies SpeechT5TextDecoderPrenet to convert input tokens to hidden features. r|ct||t||_t ||_|j yr)rnrorrPrmrr2r,s r,roz&SpeechT5DecoderWithTextPrenet.__init__rZr.c6|jjSrr\r<s r,r]z2SpeechT5DecoderWithTextPrenet.get_input_embeddingsr^r.c:|jj|yrr`rbs r,raz2SpeechT5DecoderWithTextPrenet.set_input_embeddingsrdr.r/r1rr r3rqrr rr4r5rr:c l|j|||\} }|j| |||||||| | | |  }|Srr)r{r/r1rr r3rqrr rr4r5rrrs r,rz%SpeechT5DecoderWithTextPrenet.forwards]15 L.Zi0j-~&&/)"7#9!5+/!5#)'  r.r|)rrrrrror]rarrr_r^rr~rQrrrrrrs@r,rrs;~20 5959=A=A,07;=A$(,0/3&*15 u001 !!1!12  ((9(9:  !))9)9 :  ELL) 'u||4 "$u'8'8"9: D> $D> 'tn d^ !.  u?? @ r.rcveZdZdZdeffd Z ddeejdeejdeejdeejdeejd eejd ee ejd ee d ee d ee dee deejde eeffdZxZS)SpeechT5DecoderWithoutPrenetrhr|cdt||t||_|j yr)rnrormrr2r,s r,roz%SpeechT5DecoderWithoutPrenet.__init__rjr.r/r1rr r3rqrr rr4r5rr:c @|j||||||||| | | |  } | Sr)r)r{r/r1rr r3rqrr rr4r5rrs r,rz$SpeechT5DecoderWithoutPrenet.forwardsD&&&)"7#9!5+/!5#)'  r.r|r}rs@r,rrs1 ~5959=A=A,07;=A$(,0/3&*15u001!!1!12 ((9(9:  !))9)9 :  ELL) 'u||4"$u'8'8"9:D>$D>'tnd^!. u?? @r.rceZdZdZdeffd ZdejdejdejdejfdZ d Z e d Z xZS) $SpeechT5GuidedMultiheadAttentionLossz Guided attention loss from the paper [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://huggingface.co/papers/1710.08969), adapted for multi-head attention. r|cft||j|_|j|_yr)rnroguided_attention_loss_sigmasigmaguided_attention_loss_scalescaler,s r,roz-SpeechT5GuidedMultiheadAttentionLoss.__init__%s( 77 77 r.r@ input_masks output_masksr:cF|j|||j}|jd|jdz}|j|jjd}||z}t j |j |}|j|zS)aY Compute the attention loss. Args: attentions (`torch.FloatTensor` of shape `(batch_size, layers * heads, output_sequence_length, input_sequence_length)`): Batch of multi-head attention weights input_masks (`torch.BoolTensor` of shape `(batch_size, input_sequence_length)`): Input attention mask as booleans. output_masks (`torch.BoolTensor` of shape `(batch_size, output_sequence_length)`): Target attention mask as booleans. Returns: `torch.Tensor` with the loss value r%rr)_make_guided_attention_masksrrrrr masked_selectr)r{r@rrguided_attn_masksmaskslosseslosss r,rz,SpeechT5GuidedMultiheadAttentionLoss.forward*s"!==k