h9dZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,e&jZe.Z/ee$dGdde#Z0Gdde jbZ2Gdde jbZ3Gdde jbZ4de3iZ5Gdde jbZ6Gd d!e jbZ7Gd"d#e jbZ8Gd$d%eZ9Gd&d'e jbZ:e$Gd(d)eZ;Gd*d+e jbZ<Gd,d-e jbZ= dJd.e jbd/ej|d0ej|d1ej|d2eej|d3e?d4e?fd5Z@Gd6d7e jbZAGd8d9eZBGd:d;e jbZCGd<d=e jbZDe$d>Gd?d@e;ZEGdAdBe jbZFe$dCGdDdEe;ZGe$dFGdGdHe;eZHgdIZIy)KzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int)deprecate_kwarg) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r$rtorch FloatTensor__annotations__r%r&tupler'b/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/git/modeling_git.pyr#r#5sr 15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r1r#c eZdZdZfdZ d deejdeejdeejde dejf dZ xZ S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.cBt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_t|dd|_|j#dt%j&|jj)ddy) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr9register_bufferr,arangeexpandselfconfig __class__s r2rAzGitEmbeddings.__init__Ks !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  r1 input_idsr; inputs_embedspast_key_values_lengthreturncJ||j}n|jdd}|d}||jdd|||zf}||j|}n|}|jdk(r|j |}||z }|j |}|j |}|S)Nr=rr:)sizer;rFr9rHrIrM) rSrVr;rWrX input_shape seq_length embeddingsrHs r2forwardzGitEmbeddings.forwardZs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  --i8J&J  ' ': 5"&":":<"H  - -J^^J/ \\*- r1)NNNr) r(r)r*r+rArr, LongTensorr-intTensorr_ __classcell__rUs@r2r4r4HssE  "153759&' E,,-u//0  1 12  !$   r1r4ceZdZd fd Zeddd ddej deejdeejdee d ee d ee d e ej fd Z xZ S)GitSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd||_|-tjd|jjd|j|_t|j|jz |_ |j|jz|_ t|jj|jj z dzd z|_|j$|xj"|j$zc_t'j(|j|j|_t'j(|j|j|_t'j(|j|j|_t'j0|j2|_|xs t7|d d |_|j8d k(s|j8d k(rG|j:|_t'j<d|j:zd z |j|_yy)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr9r: relative_keyrelative_key_query) r@rArDnum_attention_headshasattr ValueError layer_idxlogger warning_oncerUr(raattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerKattention_probs_dropout_probrMrNr9rGrBdistance_embeddingrSrTr9rprUs r2rAzGitSelfAttention.__init__ys(     : : :a ?PVXhHi#F$6$6#78 445Q8 #      !8!8 9:,,  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr1past_key_valuepast_key_values4.58new_nameversionr&attention_mask head_maskoutput_attentionspixel_values_presentrYc d|j\}}} |j|j|d|j|jj dd} |r |j nd} |j|j|d|j|jj dd} |j|j|d|j|jj dd} ||j| dddd| dddf| dddd| dddf|j\}}tj| ddddd| ddf|gd} tj| ddddd| ddf|gd} tj| | j dd}|jdk(s|jdk(r| jd| jd}}|Dtj|dz tj |j" jdd}n@tj$|tj |j" jdd}tj$|tj |j" jdd}||z }|j'||j(zdz }|j+| j, }|jdk(rtj.d | |}||z}nE|jdk(r6tj.d | |}tj.d | |}||z|z}|t1j2|jz }|||z}t4j6j9|d}|j;|}|||z}tj|| }|j=dddd j?}|jAdd|jBfz}|j|}||fS)Nr=rrjrdimrkrldtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"shaper{viewrmrs transposerxr|r}updaterpr,catmatmulr9tensorlongrrPrrGtoreinsummathsqrtr functionalsoftmaxrMpermute contiguousr[rt)rSr&rrrrr batch_sizer]_ query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_pastattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r2r_zGitSelfAttention.forwards%2$7$7! J JJ} % T*b$":":D/E/E!Q*+[Avw9I-JDNN0 ,N, 9Q7F7A-=#>"OUVWI))[Aww1A%BDT$U[\]K!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L*!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC o--r1NNNNNFF)r(r)r*rArr,rbrr-r boolr/r_rcrds@r2rfrfxs uD%0A6R7;15+/,1/4R.||R.!!2!23R.E--. R. "% R. $D> R.'tnR. u|| R.SR.r1rfcneZdZfdZdej dej dej fdZxZS) GitSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr7) r@rArrzrDdenserIrJrKrLrMrRs r2rAzGitSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r1r& input_tensorrYcr|j|}|j|}|j||z}|SNrrMrIrSr&rs r2r_zGitSelfOutput.forward7 =1  ]3 }|'CD r1r(r)r*rAr,rbr_rcrds@r2rr1> U\\RWR^R^r1reagerceZdZdfd ZdZeddd ddejdeejd eejdee d ee d ee d e ejfd Z xZS) GitAttentionct|t|j||||_t ||_t|_y)N)r9rp) r@rAGIT_SELF_ATTENTION_CLASSES_attn_implementationrSroutputset pruned_headsrs r2rAzGitAttention.__init__sE .v/J/JK ,Cy  $F+ Er1c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrSrmrsrrr{r|r}rrrtunion)rSheadsindexs r2 prune_headszGitAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r1rrrrr&rrrrrYc`|j||||||\}}|j||} | |fSr)rSr) rSr&rrrrr attn_outputself_attn_weightsattention_outputs r2r_zGitAttention.forward!sI*.      * & & ;;{MB!222r1rr)r(r)r*rArrr,rbrr-r rr/r_rcrds@r2rrs";$%0A6R7;15+/,1/43||3!!2!233E--. 3 "% 3 $D> 3'tn3 u|| 3S3r1rcVeZdZfdZdej dej fdZxZS)GitIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r@rArrzrDintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrRs r2rAzGitIntermediate.__init__9s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r1r&rYcJ|j|}|j|}|Sr)rrrSr&s r2r_zGitIntermediate.forwardAs& =1 00? r1rrds@r2rr8s#9U\\ellr1rcneZdZfdZdej dej dej fdZxZS) GitOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r@rArrzrrDrrIrJrKrLrMrRs r2rAzGitOutput.__init__Is` YYv779K9KL f&8&8f>S>STzz&"<"<= r1r&rrYcr|j|}|j|}|j||z}|Srrrs r2r_zGitOutput.forwardOrr1rrds@r2rrHrr1rceZdZdfd Zeddd ddej deejdeejdee d ee d ee d e ej fd Z d Z xZS)GitLayerct||j|_d|_t |||_t ||_t||_ y)Nr)rp) r@rAchunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rSrTrprUs r2rAzGitLayer.__init__WsK '-'E'E$%f B+F3' r1rrrrr&rrrrrYc|j||||||\}}t|j|j|j|} | |fS)N)rrr)rrfeed_forward_chunkrr) rSr&rrrrrrself_attention_weights layer_outputs r2r_zGitLayer.forward_sg48>>   /+!5 4B4 001  # #T%A%A4CSCSUe 333r1cL|j|}|j||}|Sr)rr)rSrintermediate_outputrs r2rzGitLayer.feed_forward_chunkxs,"//0@A{{#68HI r1rr)r(r)r*rArr,rbrr-r rr/r_rrcrds@r2rrVs(%0A6R7;15+/,1/44||4!!2!234E--. 4 "% 4 $D> 4'tn4 u|| 4S40r1rceZdZfdZ d dej deejdeejdeee e e ejfdee dee dee d ee d ee d ee ej e ffd Z xZS) GitEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wNF) r@rArTr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rSrTirUs r2rAzGitEncoder.__init__sP  ]]vG_G_A`#aAHVQ$7#ab &+#$bsA$r&rrr use_cacheroutput_hidden_statesr return_dictrYc |jr%|jr|rtjdd}t |t dt fs td|r|t|j}|rdnd} |rdnd} t|jD]4\} } |r| |fz} ||| nd}| ||||||}|d}|s,| |dfz} 6|r| |fz} | std||| | fDSt||| | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzBThe `past_key_values` should be either a `Cache` object or `None`.)rTr0rrc3$K|]}|| ywrr0).0vs r2 z%GitEncoder.forward..s  = sr%rr&r')rtrainingrqrrrtyper ror rT enumeraterr/r)rSr&rrrrrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r2r_zGitEncoder.forwardsQ  & &4==##p" /DJ+>?ab b 0*$++>O"6BD$5b4(4 POA|#$58H$H!.7.CilO(!$ M*!,M &9]1=M B $D>B 'tnB 'tnB d^B  uU\\"$;; <B r1rc&eZdZUeed<dZdZdZy)GitPreTrainedModelrTgitTct|trtjj |j d|j jtjj |jj|j jtjj |jj|j jt|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j 2|jj|j jyyt|tj"rJ|jjj|jjj%dyy)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitnormal_class_embeddingrTinitializer_rangepatch_embeddingweightposition_embeddingrzdatabiaszero_rBr6rIfill_)rSmodules r2 _init_weightsz GitPreTrainedModel._init_weightss f1 2 GGOOF22$++B_B_O ` GGOOF2299t{{?\?\O ] GGOOF55<<$++B_B_O ` fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r1N)r(r)r*rr.base_model_prefixsupports_gradient_checkpointingr%r0r1r2rrs &*#*r1rceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) rrTct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestrider!rjrr;r<r>)r@rArTrD embed_dimrvrwr Parameterr,randnrConv2d num_channelsr num_patches num_positionsrBrrOrPrQrRs r2rAzGitVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr1r^heightwidthrYc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr=g?rrjbicubicF)r[mode align_cornersr)rrr unsqueezer,jit is_tracingr;rwrreshaperrr interpolaterr) rSr^r5r6r3rr4class_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r2interpolate_pos_encodingz,GitVisionEmbeddings.interpolate_pos_encodingsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr1 pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model ().rrjrr=r)rrvrorrrrflattenrrrQr,rrErr;) rSrFrErrr5r6 target_dtype patch_embeds class_embedsr^s r2r_zGitVisionEmbeddings.forward&s6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr1F) r(r)r*r rAr,rbrarEr-r_rcrds@r2rrsdqq,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr1rcVeZdZfdZdej dej fdZxZS) GitVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) r@rArTr r activation_fnrrzrDrfc1fc2rRs r2rAzGitVisionMLP.__init__:sd  #F$5$5699V//1I1IJ99V55v7I7IJr1r&rYcl|j|}|j|}|j|}|Sr)rSrRrTrs r2r_zGitVisionMLP.forwardAs4/ **=9 / r1rrds@r2rPrP9s$KU\\ellr1rPr$r{r|r}rscalingrMc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nr=r)rr)pr rrj) r,rrrrrfloat32rrrMr r) r$r{r|r}rrVrMkwargs attn_weightsrs r2eager_attention_forwardr\Is<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r1ceZdZdZfdZ d dej deej deej deede ej eej ff dZ xZ S) GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rIgF)r@rArTrDr.rm num_headshead_dimroscaleattention_dropoutrM is_causalrrzk_projv_projq_projout_projrRs r2rAzGitVisionAttention.__init__cs  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r1r&rcausal_attention_maskrrYc |j\}}}|j|}|j|} |j|} |j |||j |j jdd}| j |||j |j jdd} | j |||j |j jdd} |jjdk7r||||z}n| |}n |du|_ t} |jjdk7rN|jjdk(r|rtjdnt|jj} | ||| | ||j|j|j sdn |j" \} } | j%|||j'} |j)| } |sd} | | fS) z#Input shape: Batch x Time x Channelrrjflash_attention_2Nrsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r)rdrVrM)rrgrerfrr`rarrTrrdr\rqrrrrbr rMr>rrh)rSr&rrirrr]r.querieskeysvaluesattention_interfacerr[s r2r_zGitVisionAttention.forwardws-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc ;; + +/B B).C.O!/2G!G&2!62$>DN(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0  LL((r1)NNF) r(r)r*r+rAr,rbrrr/r_rcrds@r2r^r^`s}GB.268<,1 5)||5)!.5) ( 5 5) $D> 5) u||Xell33 4 5)r1r^c eZdZdeffd Z d dej dej dej deede ejf dZ xZ S) GitVisionEncoderLayerrTcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yr) r@rArDr.r^ self_attnrrIrJ layer_norm1rPmlp layer_norm2rRs r2rAzGitVisionEncoderLayer.__init__sm +++F3<<F & u  ! &r1rrceZdZdZdeffd Ze d deejdeejdee dee dee d e e e ff d ZxZS) GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rTct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) r@rArTrrrrrrlayersr)rSrTrrUs r2rAzGitVisionEncoder.__init__sP  mmERXRjRjLk$lq%:6%B$lm &+#%msA#rrirrrrYcj||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|} t |j D]*\} } |r|| fz}| | |||} | d} |s"|| dfz},|r|| fz}t | ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr0)rrrr%r&r')rTrruse_return_dictr r~r) rSrWrrirrrencoder_statesall_attentionsr&idx encoder_layerrs r2r_zGitVisionEncoder.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj &1%>N+>Vd  r1)NNNNN)r(r)r*r+r rArrr,rbrrr/rr_rcrds@r2r|r|s,, 268<,0/3&*D !.D  ( 5 D $D> D 'tn D d^D  uo% &D D r1r|ceZdZdeffd Ze d deejdee dee dee dee de e e ff d Z xZS) GitVisionTransformerrTc t|||_|j}t ||_t j||j|_ t||_ t j||j|_ yr) r@rArTrDrr^rrIrJ pre_layrnormr|encoderpost_layernorm)rSrTr.rUs r2rAzGitVisionTransformer.__init__<sj  && -f5LL8M8MN'/  ll9&:O:OPr1rFrrrErrYc||n|jj}||n|jj}||n|jj}| t d|j ||}|j |}|j||||}|d}|j|}|s |f|ddzSt||j|jS)Nz You have to specify pixel_valuesrE)rWrrrrrr) rTrrrror^rrrrr&r') rSrFrrrErr&encoder_outputsr%s r2r_zGitVisionTransformer.forwardFs2C1N-TXT_T_TqTq$8$D $++JjJj &1%& 'tn & #+4. & d^ &  uo% && & r1rzY The vision model from CLIP, used in GIT, without any head or projection on top. ceZdZUeed<dZdeffd ZdejfdZ e d de e jde ede eded e edeeeff d ZxZS) GitVisionModelrTrFcdt||t||_|j yr)r@rAr vision_model post_initrRs r2rAzGitVisionModel.__init__zs'  08 r1rYcB|jjjSr)rr^rrSs r2get_input_embeddingsz#GitVisionModel.get_input_embeddingss  ++;;;r1rrrErcb||n|jj}|j|||||S)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```)rFrrrEr)rTrr)rSrFrrrErs r2r_zGitVisionModel.forwardsA8&1%# 'tn # #' # d^ #  uo% &# # r1rc\eZdZdeffd Zdej dej fdZxZS) GitProjectionrTc0t|||_tjtj |j j|jtj|j|j j|_ yr) r@rArTr SequentialrzrurDrIrJvisual_projectionrRs r2rAzGitProjection.__init__sf  !# IIf**668J8J K LL++1E1E1T1T U" r1r^rYc$|j|Sr)r)rSr^s r2r_zGitProjection.forwards%%j11r1) r(r)r*rrAr,rbr_rcrds@r2rrs* y 2%,,25<<2r1rzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states ceZdZfdZdZdZdZdedejdejdejfd Z dd Z e dd eejd eejd eejdeejdeejdeejdeeeeej&fdeedeedeededeedeeejeffdZxZS)GitModelclt||_t|_t j |_t|_ t|_ j6tjfdtjD|_|j#y)Nc3K|]B}tjtjddjj Dyw)rN)rr/r,zerosrurD)rrrTs r2rz$GitModel.__init__..s;; U[[Av/C/C/O/OPQ;sAA )r@rArTr4r^rru image_encoderrrrrryr ParameterListrimg_temperal_embeddingrrRs `r2rAzGitModel.__init__s   '/+F,@,@A!&) !.v!6  * * 6*,*:*:;v>>?;+D ' r1c.|jjSrr^rFrs r2rzGitModel.get_input_embeddingss...r1c&||j_yrr)rSr}s r2set_input_embeddingszGitModel.set_input_embeddingss*/'r1c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rSheads_to_prunerrs r2 _prune_headszGitModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr1r[rrrYctjtj||||d}|j|dk(t d}|S)Nrrr)diagonal-inf)r,triuones masked_fillfloat)rSr[rrmasks r2_generate_future_maskzGitModel._generate_future_masksAzz%**T4eLWXY 5=9 r1c|jd}|jd}|j}|j} tj||f|| } tj |||zft d|j| } tj||f| |j} |dkDrAtj|jd|jd|zf| |j}tj| | fd} tj| |j| fd}tj| |fddddf}|4tj |jd|jdfd|}|jtjk7r td tj||j }t d||<|j|jd||z||z|zf}|j}|ddddd|f}|dddddf}||z|ddddd|f<|dddddddf}|S) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr,rfullrrrrro zeros_likerQclone)rStgtmemorytgt_maskrXmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrs r2create_attention_maskzGitModel.create_attention_masksF))A,\\!_  ;; J7eTJJ #99 : &M::  kk j !?? "A %{{"HNN1$58N$NOH yy(K0a8 9hkk%&89qA#iiu 1=dAgF " *&+jj&,,q/6<>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer=z5You have to specify either input_ids or inputs_embedsrrrrz#pixel_values must be of rank 4 or 5)rVr;rWrXrjr)rrrrX)tgt_len)rrrrrrrrr)%rTrrrrro%warn_if_padding_and_no_attention_maskr[rr get_seq_length get_head_maskrndimrr%rrrappendr,rrr^rrrrepeatrrr rrrrr&r')rSrVrr;rFrrWrrrrrErr\r]rXprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr&rcombined_attention_maskexpanded_attn_maskrsequence_outputs r2r_zGitModel.forwards7J2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K  &',,.s3KTU U ^ "#  &"/59 ..0$335 #&&y$++2O2OP $(!  #  A%"&"4"4 ;S#5### ""a'"$!&|'9'9!',>$Q 1a%78Sk-?-''**T-H-H-SS)#**+@A B#())O"C!!FGG(,(>(>(O %??%'#9 +  % ,(- !''*A/?/E/Ea/HI&,,'..) %%>$D$D  ! !! $(A(F(Fq(I I1a% !  #<>N"OUVW --j:J:P:PRbRiRij#'"<"< ,#9 #=#   %"< 0 6 6 B"b!(() &)%71?U>U>VXY8Y%Z"'1{1~o.?+a.AR(RSWiiS,, 2+/!5#!-T!9'  *!,#%(;; ;&-+;;)77&11   r1r) NNNNNNNNNNFN)r(r)r*rArrrrar,rrrbrrrrrr listr-rr/rr_rcrds@r2rrs &/0C#ekk5<<\a\h\h 0#d-115/3/3,004KO$(,0/3).&*c ELL)c !.c u||, c u||, c ELL) c   -c "%tE4E4E/F(F"GHc D>c $D>c 'tnc #'c d^c  uU\\"$>> ?c c r1rz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c eZdZdgZfdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee jd ee e e e jfd eedeedeededeede ee jeffdZ ddZxZS)GitForCausalLMz output.weightct||t||_t j |j |j|_|jyr) r@rArrrrzrDrCrrrRs r2rAzGitForCausalLM.__init__sF  F#ii 2 2F4E4EF  r1c|jSrrrs r2get_output_embeddingsz$GitForCausalLM.get_output_embeddingss {{r1c||_yrr)rSnew_embeddingss r2set_output_embeddingsz$GitForCausalLM.set_output_embeddingss $ r1rVrr;rFrrWlabelsrrrrrErrYc | | n|jj} |d} |j|||||||| | | | |  }|d}|j|}d}||jjj dj jj}|dd|dddfj}|ddddfj}|j|jd|jj|jdfd|jji|}| s|f|ddz}||f|zS|St|||j|j|j S) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr;rFrrWrrrrrErrr=rrC)losslogitsrr&r')rTrrrrrrrSrxr loss_functionrrCrrr&r')rSrVrr;rFrrWrrrrrrErrZrzrrrnum_image_tokensshifted_logitsrs r2r_zGitForCausalLM.forwardsj&1%IIKNAqrE]--/F%4%%##B (>(>? B ;;11 DY,F)-)9TGf$ Ev E%#33!//))   r1c |B|j}|jd|kDr|}n|jddz }|dd|df}|j}||j|}|||jd||dS)NrrF)rVrrFrr)rrnew_onesget) rSrVrrrrZ past_lengthremove_prefix_lengthr\s r2prepare_inputs_for_generationz,GitForCausalLM.prepare_inputs_for_generations  &)88:Kq!K/'2$(1q'9A'=$!!%9%:":;I oo  !&// A $D>A 'tnA #'A d^A uU\\"$:: ;!A A HOS r1r)rrrr)r)Jr+r dataclassesrtypingrrrr,torch.utils.checkpointr activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrrutils.deprecationrconfiguration_gitrr get_loggerr(rqr#rr4rfrrrrrrrrrrPrbrr\r^rrr|rrrrr__all__r0r1r2r s  !,, !.)B9 Gll19   H %  ?; ?   ?-BII-`v.ryyv.t BII   /3299/3f bii   %)%PI I X***6P"))Pf 299 .% II% <<% % << % U\\* %  %%.L)L)`/6/fT ryyT n3 2993 l 2 '2  2 j 2BII 2 ~ !~  ~ B s 's  s l Qr1