hbddlZddlmZddlmZmZmZddlZddl Z ddl m Z ddl m cm Zddl mZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZdd lmZdd lm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,ee!dGdde Z-ee!dGdde Z.Gdde j^Z0Gdde j^Z1Gdde j^Z2 dRd e j^d!e jd"e jd#e jd$ee jd%e3d&e3fd'Z4dSd(e jd)ee5d*e jfd+Z6Gd,d-e j^Z7Gd.d/e j^Z8d0Z9d1Z:Gd2d3eZ;ee!d4Gd5d6e Ze!d;Gd<d=e=Z?Gd>d?e j^Z@Gd@dAe j^ZAGdBdCe j^ZBGdDdEe j^ZCGdFdGe j^ZDGdHdIe j^ZEGdJdKe jZGGdLdMe j^ZHe!dNGdOdPe=ZIgdQZJy)TN) dataclass)CallableOptionalUnion)Tensor)OutputRecorder)ACT2FN)GradientCheckpointingLayer)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)#compile_compatible_method_lru_cache) ModelOutputauto_docstring)TransformersKwargscheck_model_inputs) AutoModel) Sam2ConfigSam2HieraDetConfigSam2MaskDecoderConfigSam2PromptEncoderConfigSam2VisionConfigz,Base class for the vision encoder's outputs.) custom_introceZdZUdZdZej ed<dZe ej ed<dZ e ej ed<dZ e e ej dfed<dZ e e ej dfed<y) Sam2VisionEncoderOutputa last_hidden_state (`torch.FloatTensor` of shape `(batch_size, height, width, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. fpn_hidden_states (`tuple(torch.FloatTensor)`): Tuple of `torch.FloatTensor` (one for each feature level, from high to low resolution) of shape `(batch_size, hidden_size, height, width)`. Feature maps from the Feature Pyramid Network neck. fpn_position_encoding (`tuple(torch.FloatTensor)`): Tuple of `torch.FloatTensor` (one for each feature level, from high to low resolution) of shape `(batch_size, hidden_size, height, width)`. Positional encodings corresponding to the `fpn_hidden_states`. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, height, width, hidden_size)`. Hidden-states of the model at the output of each stage. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlast_hidden_statefpn_hidden_statesfpn_position_encoding. hidden_states attentions)__name__ __module__ __qualname____doc__r torch FloatTensor__annotations__r!rr"r#tupler$d/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/sam2/modeling_sam2.pyrr7s&,0u((/59x 1 1299=8E$5$56==AM8E%"3"3S"89:A:>Ju00#567>r.rz'Base class for the Sam2 model's output.c:eZdZUdZdZej ed<dZej ed<dZ ej ed<dZ e ej dfed<dZ e e ej dfed<dZe e ej dfed <dZe e ej dfed <y) Sam2ImageSegmentationOutputa iou_scores (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_masks)`): The Intersection over Union (IoU) scores of the predicted masks. pred_masks (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_masks, height, width)`): The predicted low-resolution masks. This is an alias for `low_res_masks`. These masks need to be post-processed by the processor to be brought to the original image size. object_score_logits (`torch.FloatTensor` of shape `(batch_size, point_batch_size, 1)`): Logits for the object score, indicating if an object is present. image_embeddings (`tuple(torch.FloatTensor)`): The features from the FPN, which are used by the mask decoder. This is a tuple of `torch.FloatTensor` where each tensor has shape `(batch_size, channels, height, width)`. vision_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of each stage) of shape `(batch_size, height, width, hidden_size)`. Hidden-states of the vision model at the output of each stage. vision_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the vision model. mask_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the mask decoder. N iou_scores pred_masksobject_score_logits.image_embeddingsvision_hidden_statesvision_attentionsmask_decoder_attentions)r%r&r'r(r2r)r*r+r3r4r5r,r6rr7r8r-r.r/r1r1Ts,%)J!!($(J!!(-1**16:eE--s23:DH(5):):C)?#@AHAExe&7&7&< =>EGKXeE,=,=s,B&CDKr.r1c.eZdZdZdeffd ZdZxZS)Sam2PatchEmbeddingsa Turns pixel values into patch embeddings for transformer consumption. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`Sam2ImageProcessorFast.__call__`] for details. Returns: embeddings (`torch.FloatTensor`): Patch embeddings depend on image_size, patch_kernel_size, patch_stride and patch_padding configct||j}|j}t j |||j |j|j|_ y)N) kernel_sizestridepadding) super__init__ num_channels hidden_sizennConv2dpatch_kernel_size patch_stride patch_padding projection)selfr;rBrC __class__s r/rAzSam2PatchEmbeddings.__init__sU ** (( ))  00&&((  r.cn|j\}}}}|j|jdddd}|S)Nrrr r)shaperIpermute)rJ pixel_values_rBheightwidth embeddingss r/forwardzSam2PatchEmbeddings.forwards;)5););&<__\2::1aAF r.)r%r&r'r(rrArT __classcell__rKs@r/r:r:vs   1  r.r:c eZdZdZ ddedededeeffd Ze d dd e jd e e jefd e jd eed ef dZxZS)Sam2SinePositionEmbeddingz This is a more standard version of the position embedding, very similar to the one used by the Attention is all you need paper, generalized to work on images. num_pos_feats temperature normalizescalect|||dur td||_||_||_|dt jz|_y||_y)NFz+normalize should be True if scale is passedr) r@rA ValueErrorrYrZr[mathpir\)rJrYrZr[r\rKs r/rAz"Sam2SinePositionEmbedding.__init__sY   e!3JK K*&"$)MQ[ u r.r)maxsizerMdevicedtypemaskreturnc |2tj|d|d|df|tj}|j|}|j d}|j d}|j rDd}||ddddddf|zz |j z}||ddddddf|zz |j z}tj|jtj|j|} |jdtj| dd z|jz z} |dddddddf| z } |dddddddf| z } tj| dddddddddfj| dddddddddfjfd jd} tj| dddddddddfj| dddddddddfjfd jd} tj | | fd j#dddd} | S) Nrrr rbrcrư>rcrbfloor) rounding_modedim)r)zerosbooltocumsumr[r\arangerYint64rZdivstacksincosflattencatrN) rJrMrbrcrdnot_masky_embedx_embedepsdim_tpos_xpos_yposs r/rTz!Sam2SinePositionEmbedding.forwards <;;a%(E!H=fTYT^T^_DE::e$//!$//!$ >>CBC!3c!9:TZZGGArs!3c!9:TZZGG T//u{{6RUUV[\  Q5!7)S%SVZVhVh%hi1a &.1a &. U1aADqD=1557q!Q1}9M9Q9Q9STZ[\ddefg U1aADqD=1557q!Q1}9M9Q9Q9STZ[\ddefgiiA.66q!QB r.)@i'FNN)r%r&r'r(intrqrfloatrArr)SizerrbstrrcrrTrUrVs@r/rXrXs tx =  =47 =LP =aijoap =)3 "& zzellC'({{  v   4r.rXceZdZdeffd Zdej deeej dfeej dfffdZxZ S)Sam2VisionNeckr;c t|||_t|jdzd|_t j|_|jD]]}|jjt j||j|j|j|j_|j|_y)NrT)rYr[) in_channels out_channelsr=r>r?)r@rAr;rXfpn_hidden_sizeposition_encodingrD ModuleListconvsbackbone_channel_listappendrEfpn_kernel_size fpn_stride fpn_paddingfpn_top_down_levels)rJr;rrKs r/rAzSam2VisionNeck.__init__s  !:I_I_cdIdpt!u]]_ !77 K JJ   +!'!7!7 & 6 6!,,"..   $*#=#= r.r#re.cHd}d}t|jdz }t|ddD]}||jdddd}|j||z |}||jvs||k(r|}nVt j jtjdd dd j|j}||z}|j|j|j|jj|j} ||fz }|| fz }||fS) Nr-rrirr r)rcg@nearestF) scale_factormode align_corners antialias)lenrrangerNrF interpolaterrr)float32rcrrMrb) rJr#r!r"nilateral_features prev_featurestop_down_featuresprev_position_encodings r/rTzSam2VisionNeck.forwardsE "  Oa q"b! ?A,Q/771aC 0tzz!a%01AB 000AF 0 $%MM!$$5==$9!$""&# % "%++, "!13D D %)%;%;##]%9%9=;N;N&b$$% # -!1 1  !&<%> > !) ?,!"777r.) r%r&r'rrAr)rr,rTrUrVs@r/rrsS>/>$8U\\8eE%,,PSBS ceil_mode)rNrDr max_pool2d)rrs r/do_poolrsX !Q1A    \]b cA !Q1A Hr.ceZdZ d dededededeeeeff fd Zdejdejfd Z xZ S) Sam2MultiScaleAttentionr;rodim_outnum_attention_headsrct|||_||_||_||_||_||z}|dz|_tj||dz|_ tj|||_ d|_ y)Nr F) r@rAr;rorrrr\rDLinearqkvproj is_causal)rJr;rorrrhead_dimrKs r/rAz Sam2MultiScaleAttention.__init__sz   (#6 11t^ 99S'A+.IIgw/ r.r#rec |j\}}}}|j|j|||zd|jd}t j |d\}} } ||j z| jddz} tjjj| tjdj|j} |jr[t|j|||d|j}|jdd\}}|j|||z|jd}|jdd}| jdd} | jdd} t } |j"j$dk7rt&|j"j$} | ||| | fd|j(|j d|\} }| j|||d} |j+| } | S) Nr rir)rcroreager)rrr)rMrreshaperr)unbindr\rrDrrrrrrcrrrr;_attn_implementationr rr)rJr#r batch_sizerQrRrPrrrrrattention_interfacers r/rTzSam2MultiScaleAttention.forward2s'4':':$ FE1hh}%--j&5.!TMeMegij!LLa0sE *cmmB.CC xx**22V>VXZ[E1%mmAq!1%(? ;; + +w 6"9$++:Z:Z"[ ,       nnJJ     Q"))*feRH ii , r.r) r%r&r'rrrr,rAr)rrTrUrVs@r/rrse37 "  !  uS#X/ 0&U\\& &r.rc DeZdZ d dedededededef fd ZdZxZS) Sam2FeedForward input_dim hidden_dim output_dim num_layers activationsigmoid_outputc `t|||_t||_t j |||_t j |||_t jt|dz Dcgc]}t j ||c}|_ ||_ ycc}w)Nr) r@rArr rrDrproj_inproj_outrrlayersr) rJrrrrrrrPrKs r/rAzSam2FeedForward.__init__\s $ ,yyJ7  *j9 mmPUV`cdVdPe$f1RYYz:%F$fg ,%gs:B+c|j|}|j|}|jD]}|j||}|j|}|jrt j |}|Sr)rrrrrrsigmoid)rJr#layers r/rTzSam2FeedForward.forwardmss ]3  6 [[ BE OOE-,@AM B m4   IIm4Mr.)reluF) r%r&r'rrrqrArTrUrVs@r/rr[sO!$--- -  -  --" r.rc `|j\}}}}|||zz |z}|||zz |z}tjj|ddd|d|f}||z||z} }|j |||z|| |z||}|j ddddddj j d|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrr rrmri)rMrDrpadviewrNr) hidden_state window_sizerrQrRrB pad_height pad_width padded_height padded_widthwindowss r/window_partitionrys/;.@.@+J| 44 CJu{22kAI==$$\Aq!Y:3VWL"(:"5uy7H+>q+AqA$4]K$P !M6 dii '  $   **d.?.?.BBK>>!A&DAq 1{?2kAE 1{?2kAE%iU+F   a .}k6TUWXSYZM =0 ++M:%1A(BB r.) r%r&r'rrrAr)rrrr*rTrUrVs@r/rrs`-:"-:-: -:  -:^*||*+,*    *r.rzW Hiera model's outputs that also contains a pooling of the last hidden states. cleZdZUdZdZeejed<dZ ee ejdfed<y)Sam2HieraDetModelOutputat last_hidden_state (`torch.FloatTensor` of shape `(batch_size, height, width, hidden_size)`): hidden-states at the output of the last layer of the model. intermediate_hidden_states (`tuple[torch.FloatTensor]` of shape `(batch_size, height, width, hidden_size)`): Sequence of hidden-states at the output of the intermediate layers of the model. Nr .intermediate_hidden_states) r%r&r'r(r rr)r*r+rr,r-r.r/rrs?6:x 1 129JNu/@/@#/E)F GNr.rc*eZdZeZdZdZdZdZdZ dZ y)Sam2PreTrainedModelsam2rOTc|jj}t|tjtj tj frY|jjjd||j |jjjnt|tjre|jjjd||j|jj|jjnit|tjtfrI|jjj!d|jjjt|t"r`|j$$|j$jj|j&$|j&jjt|t(r2|j*%|j*jjyyy)N)meanstd?)r;initializer_range isinstancerDrrEConvTranspose2dweightdatanormal_biaszero_ Embedding padding_idxr Sam2LayerNormfill_Sam2HieraDetModel pos_embedpos_embed_window Sam2Modelno_memory_embedding)rJrrs r/ _init_weightsz!Sam2PreTrainedModel._init_weights0skk++ fryy"))R5G5GH I MM   & &CS & 9{{&   &&(  - MM   & &CS & 9!!- ""6#5#56<<> } = > MM   $ $S ) KK   " " $ f/ 0+  %%++-&&2'',,224 fi ())5**//5576 )r.N) r%r&r'r config_classbase_model_prefixmain_input_name_supports_sdpa_supports_flash_attn_2_supports_attention_backendr-r-r.r/rr's(L$ON!"&8r.rc eZdZeZdZeedZdeffd Z dZ de e e fde jfdZe d dee j$d eedee effd ZxZS) r(rOr#r$r;c t||t||_t j t jd|jg|j|_ t j t jd|j|jd|jd|_ tj|jdz j!|_t j$|_d}t)|jD]D\}}t+|D]1}t-||||}|j&j/||dz }3Fy)Nrr)r;rrr)r@rAr: patch_embedrD Parameterr)rprC+window_positional_embedding_background_sizer)rr*nprsblocks_per_stagetolist stage_endsrblocks enumeraterrr)rJr;rrr;rblockrKs r/rAzSam2HieraDetModel.__init__Os.  .v6 KK6-- c0b0b c !# KK6--v/K/KA/NPVPlPlmnPo p! 99V%<%<=AIIKmmo +4V5L5L+M % 'I'"#34 % +!Y)]l ""5)1$  % %r.c|jSr)r7rJs r/get_input_embeddingsz&Sam2HieraDetModel.get_input_embeddingsesr.hwrec 4|\}}|j}tj|j||fd}||j t |j |j Dcgc] \}}||z c}}z}|jdddd}|Scc}}w)Nbicubic)sizerrrr r)r*rrr)tileziprMrN)rJrDhw window_embedr)rys r/_get_pos_embedz Sam2HieraDetModel._get_pos_embedhs1,, MM$..1vIN  1 1c)//[g[m[mFn2oda162o pp %%aAq1 3ps%B rc | td|j|}||j|jddz}d}t |j D]#\}}||fi|}||j vs||fz}%t||S)N You have to specify pixel_valuesrr r-)r r)r^r7rNrMr?r>r=r)rJrOrr#rr block_modules r/rTzSam2HieraDetModel.forwardps  ?@ @((6 %(;(;M%>%D%DE !D$=$== "%%i+- yy%))K0%))K2HIrRRr.rr%r&r'rrArTrUrVs@r/rarasK6K Sr.rac*eZdZdeffd ZdZxZS)Sam2MaskEmbeddingr;c2t||jdz|_t|j|_t jd|jdd|_t j|j|jdd|_ t j|j|jd|_ t|j|jd|_t|jdz|jd|_y)Nrmrrr=r>)r=channels_firstr data_format)r@rAmask_input_channelsr rrrDrEconv1conv2rCconv3r&rrrr\s r/rAzSam2MaskEmbedding.__init__s #)#=#=#B  !2!23YYq$":":RST YYt779S9Sabklm YYv996;M;M[\] (  $ $&*?*?M] )  $ $q (f.C.CQa r.c|j|}|j|}|j|}|j|}|j |}|j|}|j |}|Sr)rtrrrurrv)rJmasksr#dense_embeddingss r/rTzSam2MaskEmbedding.forwardso 5) ((7  6  =1 ((7  6 ::m4r.rkrVs@r/rmrms  6    r.rmc eZdZdeffd Zdej dej dedej fdZdej dej fd Z d e e ej ej fd e ej d e ej d e ej de ej ej ff dZ xZ S)Sam2PromptEncoderr;ct|t||_t ||_t jd|j|_ |j|jz|j|jzf|_ d|jz|jzd|jz|jzf|_ |j|_t j|j|j|_|j|_t jd|j|_y)Nrrm)r@rArashared_embeddingrm mask_embedrDr$rC no_mask_embed image_size patch_sizeimage_embedding_sizemask_input_sizeinput_image_sizenum_point_embeddings point_embednot_a_point_embedr\s r/rAzSam2PromptEncoder.__init__s  7 ?+F3\\!V-?-?@%+%6%6&:K:K%KVM^M^bhbsbsMs$t! !F$5$5 59J9J JAPVPaPaLaekevevLvw & 1 1<<(C(CVEWEWX!--!#a1C1C!Dr.pointslabelsrrecP|dz}|rZtjjj|ddd}tjjj|ddd}|j|jf}|j ||}tj |ddk(|jj|}tj |dd k7|tj|}||j|jd |dk\jdzz}|S) zEmbeds point prompts.?)rrrrconstantr)rr)rrri).Ni)min) r)rDrrrr}whererr zeros_likerclamp unsqueeze)rJrrrripoint_embeddings r/ _embed_pointszSam2PromptEncoder._embed_pointss# XX((,,V\ Z[,\FXX((,,VV*TV,WF,,d.C.CD // D ++fY&72&=t?U?U?\?\^mn ++ 9  $    _ - *D,<,|jd}|j|}||}nt j ||gd}||j |} || fS|jjjddddj|d|jd|jd} || fS) au Embeds different types of prompts, returning both sparse and dense embeddings. Args: points (`torch.Tensor`, *optional*): point coordinates and labels to embed. boxes (`torch.Tensor`, *optional*): boxes to embed masks (`torch.Tensor`, *optional*): masks to embed Nrrz5If points are provided, labels must also be provided.)rrrnri) rMr^rrr)r{r~rrrexpandr) rJrrrrsparse_embeddingsrpoint_embeddingsbox_embeddingsrys r/rTzSam2PromptEncoder.forward%s'$!  #%++A.J# !XYY#11, S^bfSf1h  0   "$**1-J!..{;N ($2!$)II/@..QWX$Y!  "#{;  !"222 $1188@@B1MTTB 9 9! W>WXY>Z  !"222r.)r%r&r'rrAr)rrqrrrr,rTrUrVs@r/r{r{s E6 EELL%,,TV[VbVb2  %,,  5<<  (3uU\\5<<%?@A(3u||,(3ell+ (3 ell+ (3 u||U\\) * (3r.r{ceZdZdZd fd Z d dej dej dej deej dee de ej ej ff d Z xZ S) Sam2Attentionz SAM2's attention layer that allows for downscaling the size of the embedding after projection to queries, keys, and values. ct|| |jn|}||_|j|_|j|z|_|j |_|j |j z|_|jdz|_d|_ tj|j|j |_ tj|j|j |_ tj|j|j |_tj|j |j|_y)NrF)r@rAattention_downsample_rater;rC internal_dimrrrrrDrq_projk_projv_projo_proj)rJr;downsample_raterKs r/rAzSam2Attention.__init__Vs >M>U&::[j !--"../A#)#=#= ))V-G-GG }}d* ii 0 0$2C2CD ii 0 0$2C2CD ii 0 0$2C2CD ii 1 143C3CD r.rrrattention_similarityrrec |jdd\}}||zd|j|jf}|j|j|j dd}|j |j|j dd}|j|j|j dd}t} |jjdk7rt|jj} | ||||f|d|j|jd|\} } | j||d|j|jzj} |j!| } | | fS)Nrrirrr)rrrr)rMrrrrrrrrr;rr rrrrr) rJrrrrrrpoint_batch_size new_shaperrrs r/rTzSam2Attention.forwardfso(-{{2A$ $"22B8P8PRVR_R_` ' E"''3==aC#dkk###Y/99!Q?' E"''3==aC(? ;; + +w 6"9$++:Z:Z"[ $7     % 0LLnn %  % ! \")) ("d.F.F.V *, kk+. L((r.r) r%r&r'r(rAr)rrrrr,rTrUrVs@r/rrPs E*8< %)||%)\\%)|| %) 'u||4 %) +, %) u||U\\) *%)r.rc PeZdZd dedeffd Zdededededed eef d Z xZ S) Sam2TwoWayAttentionBlockr;skip_first_layer_pec,t|t|d|_t j |j |_t||_t j |j |_ t|j |j|j |j|_ t j |j |_t j |j |_t||_||_y)a A transformer block with four layers: (1) self-attention of sparse inputs (2) cross attention of sparse inputs -> dense inputs (3) mlp block on sparse inputs (4) cross attention of dense inputs -> sparse inputs Arguments: config (`Sam2MaskDecoderConfig`): The configuration file used to instantiate the block attention_downsample_rate (*optionalk*, int, defaults to 2): The downsample ratio of the block used to reduce the inner dim of the attention. skip_first_layer_pe (*optional*, bool, defaults to `False`): Whether or not to skip the addition of the query_point_embedding on the first layer. r)r)rN)r@rAr self_attnrDrrCrcross_attn_token_to_imagerrmlp_dimnum_hidden_layersr layer_norm3 layer_norm4cross_attn_image_to_tokenr)rJr;rrKs r/rAz!Sam2TwoWayAttentionBlock.__init__s &vqA<<(:(:;)6v)>&<<(:(:;"   0B0BvOgOg <<(:(:;<<(:(:;)6v)> r.querieskeysquery_point_embeddingkey_point_embeddingrrc |jr|j|||\}}n!||z}|j|||\} }|| z}|j|}||z}||z} |j|| ||\} }|| z}|j |}|j |} || z}|j |}||z}||z} |j| ||\} }|| z}|j|}||| fS)Nrrr)rrrr) rrrrrrrrr) rJrrrrrrrPrattn_outrmlp_outs r/rTz Sam2TwoWayAttentionBlock.forwards=  # #g7'RJGQ33E..u%w.OKHa(G""7+//((44SCW5 !H$""7+((7#G#""7+//((443EQX4Y !h%h&&r.)F) r%r&r'rrqrArrrrTrUrVs@r/rrs\74747>*'*'*' & *' $ *' % *'+,*'r.rcXeZdZdeffd Z d dededededeedee e ff d Z xZ S) Sam2TwoWayTransformerr;crt|||_|j|_t j |_t|jD]+}|j jt||dk(-t||_ t j|j|_y)Nr)r)r@rAr;rrDrrrrrrfinal_attn_token_to_imagerrClayer_norm_final_attn)rJr;rrKs r/rAzSam2TwoWayTransformer.__init__s  !'!9!9mmo t--. _A KK  7UVZ[U[] ^ _*7v)>&%'\\&2D2D%E"r.rr5image_positional_embeddingsrrrec | td|jdjdddjd}|jdjdddjd}|}|}|jD]} |||z }| d|||||d|\}}} ||z} ||z} |j | | |\} } || z}|j |}||fS)Nz&You have to specify an image_embeddingrrr)rrrrrrr-)r^rzrNrrrr)rJrr5rrtarget_embeddingrrrrrPrrrs r/rTzSam2TwoWayTransformer.forwards*  #EF F+33A6>>q!QGQQRST&A&I&I!&L&T&TUVXY[\&]&g&ghi&j##[[ E+++$ &6$?%9      GT1 **00445cQU4V !H$,,W5}r.r) r%r&r'rrArrrrr,r rTrUrVs@r/rrsf F4 F& ( (!(&, ( % (+,( uo% &(r.rcfeZdZdZdddfd Zdej dej ffd ZxZS) r&aALayerNorm that supports two data formats: channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch_size, height, width, channels) while channels_first corresponds to inputs with shape (batch_size, channels, height, width). rh channels_lastrqc \t||fd|i||dvrtd|||_y)Nr)rrpzUnsupported data format: )r@rANotImplementedErrorrr)rJnormalized_shaperrrrrKs r/rAzSam2LayerNorm.__init__s? )=s=f= A A%(A+&OP P&r.featuresrec|jdk(r9|jdddd}t| |}|jdddd}|St| |}|S)z Args: features: Tensor of shape (batch_size, channels, height, width) OR (batch_size, height, width, channels) rprrr r)rrrNr@rT)rJrrKs r/rTzSam2LayerNorm.forward sj   / /''1a3Hwx0H''1a3Hwx0Hr.) r%r&r'r(rAr)rrTrUrVs@r/r&r&s4 15/'    r.r&cheZdZdeffd Z ddej dej dej dej dedeej d e ej d e ej d e e d e ej ej ej ej ffd Z dZdZxZS)Sam2MaskDecoderr;ct|||_|j|_|j|_|jdz|_t jd|j|_t j|j |j|_ t||_ t j|j|jdzdd|_ t j|jdz|jdzdd|_t|jdzd|_t j"|_g}t'|j D]5}|t)|j|j|jdzdgz }7t j*||_t)|j|j.|j |j0d |_t j4|j|jdzdd|_t j4|j|jdzdd|_t jd|j|_t)|j|jdd|_|j>|_|j@|_ |jB|_!y) Nrrmrrorp)rrr T)r)"r@rAr;rCnum_multimask_outputsnum_mask_tokensrDr$ iou_token mask_tokensr transformerr upscale_conv1 upscale_conv2r&upscale_layer_normGELUrrrroutput_hypernetworks_mlpsiou_head_hidden_dimiou_head_depthiou_prediction_headrEconv_s0conv_s1obj_score_tokenpred_obj_score_headdynamic_multimask_via_stability!dynamic_multimask_stability_delta"dynamic_multimask_stability_thresh)rJr; mlps_listrPrKs r/rAzSam2MaskDecoder.__init__/s`  !--%+%A%A"%;;a?a)9)9:<<(<(N>NO08 //0@0@$BRBRVWBWefopq//0@0@A0EtGWGW[\G\jktuv"/0@0@A0ESc"d'') t++, iA /$*:*:DNTU=V&A&S&STdfg&h#-=T-=-=. --(C!5- .  . **)Aq!4 *1aa$:N:N6N1OQR+RS,55a;@@ ) )< 4++,:-aABh7 <<(<"E$4$>$>r$B$L$LR$P$Z$Z[]$^!$=$D$D A',,R02B2G2G2K% !!& -=qB[ \$)LL1EqJZJdJdegJh$i!,Aq!A#q!O< .q!QqSy 9556GH$(O(OO  ++ o & 0 01B C  !      5 6 ! %  ..r.)NN)r%r&r'rrAr)rrqlistrrrr,rTr rrUrVs@r/rr.s'\4'\b8<37mD,,mD&+\\mD#(,, mD "' mD  mD#'u||"4mD'u||4mD#5<<0mD+,mD u||U\\5<<E FmD^  #/r.rz Segment Anything Model 2 (SAM 2) for generating segmentation masks, given an input image and input points and labels, boxes, or masks. ceZdZdgZdgZdeediZgdZde ffd Z dZ d Z d e jfd Ze j d e j"d eed ee jfdZe j ddee j"dee j.dee j"dee j.fdZee dd ee j"dee j"dee j.dee j"dee j.dee j"dedee j"dee j"d eed efdZd e j"d eed eee jee jeee j"dfeee j"dfffdZxZ S)r+z4prompt_encoder.shared_embedding.positional_embeddingr8r)index)z ^memory_.*z^mask_downsample.*z^object_pointer_proj.*z0^temporal_positional_encoding_projection_layer.*no_memory_positional_encodingno_object_pointer%occlusion_spatial_embedding_parameterr;ct||t|j|_t j |j|_t|j|_ |j|j_ t|j|_|jj|_|jj |_|jj"|_t&j(j+t'j,dd|j$|_|j1y)Nr)r@rAraprompt_encoder_configshared_image_embeddingrrV vision_configvision_encoderr{prompt_encoderrmask_decoder_configr mask_decoderrZbackbone_feature_sizesrrr)rDr8rpr,r[r\s r/rAzSam2Model.__init__s  &=f>Z>Z&[#'33F4H4HI/0L0LM:@:U:U""7+F,F,FG"("6"6"I"I&,&:&:&Q&Q# ..>>#(88#5#5ekk!Q6X#Y  r.c|jjj|jjj_yr)r#rcr r&r}rBs r/ _tie_weightszSam2Model._tie_weightss1  ' ' < < A A ,,AAFr.c6|jjSr)r%rCrBs r/rCzSam2Model.get_input_embeddings$s""7799r.rec|jj}|jjj}|jjj }t j|||}|jddz }|jddz }||dz }||dz }|jt j||gd}|jdddjdS)Nrgrrnrrrir) r&rr#rcrbrcr)onesrsrwrNr)rJrG target_device target_dtypegridr}r~rcs r/$get_image_wide_positional_embeddingsz.Sam2Model.get_image_wide_positional_embeddings's""7733HHOO 22GGMM zz$}LI++!+$s*++!+$s*DG#DG##::5;;QXGY_a;bc#++Aq!4>>qAAr.rOrc |jd}|j|fi|\}}}}|d|jz|d<t||jDcgc]*\}}|j dddj |dg|,}}}|Scc}}w)z Returns the image embeddings by passing the pixel values through the vision encoder. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Input pixel values rrirr)rMget_image_featuresr,rIr)rNr) rJrOrr feature_mapsrPfeat feat_sizer5s r/get_image_embeddingszSam2Model.get_image_embeddings4s"''* 7 7 7 O O aA(+d.F.FF R $'|T5P5P#Q i 'DLLAq ! & &z2 B B     s/B rrrrc0|j||||}|S)a Returns the prompt embeddings by passing the input points, labels, boxes and masks through the prompt encoder. Args: input_points (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_points_per_image, 2)`): Optional input points for the prompt encoder. The padding of the point is automatically done by the processor. `point_batch_size` refers to the number of masks that we want the model to predict per point. The model will output `point_batch_size` times 3 masks in total. input_labels (`torch.LongTensor` of shape `(batch_size, point_batch_size, num_points_per_image)`): Optional input labels for the prompt encoder. The padding of the labels is automatically done by the processor, or can be fed by the user. input_boxes (`torch.FloatTensor` of shape `(batch_size, num_boxes_per_image, 4)`): Optional input boxes for the prompt encoder. The padding of the boxes is automatically done by the processor. users can also pass manually the input boxes. input_masks (`torch.LongTensor` of shape `(batch_size, image_size, image_size)`): Optional input masks for the prompt encoder. rrrr)r&)rJrrrr prompt_outputs r/get_prompt_embeddingszSam2Model.get_prompt_embeddingsOs-2++%%## , r.r5rrrc h|du|duz s td|V|T|jd|jdk7r5tdj|jd|jd|j} ||jdn|djd} | j | ddd} d} d}|x|j |fi| \}}}} |d|j z|d<t||jDcgc]*\}}|jdddj| dg|,}}}|?|=tj|dddddddftj|j}|m|ktj| ddd|dj |dj}tj"| ddtj$|dj }|{|jd d|j&j(k7rUt+j,|j/|j&j(d d d j1|j }|j'||||\}}|j2d|d| ||||dd|| d| \}}}}t5|||||| Scc}}w)a input_points (`torch.FloatTensor` of shape `(batch_size, num_points, 2)`): Input 2D spatial points, this is used by the prompt encoder to encode the prompt. Generally yields to much better results. The points can be obtained by passing a list of list of list to the processor that will create corresponding `torch` tensors of dimension 4. The first dimension is the image batch size, the second dimension is the point batch size (i.e. how many segmentation masks do we want the model to predict per input point), the third dimension is the number of points per segmentation mask (it is possible to pass multiple points for a single mask), and the last dimension is the x (vertical) and y (horizontal) coordinates of the point. If a different number of points is passed either for each image, or for each mask, the processor will create "PAD" points that will correspond to the (0, 0) coordinate, and the computation of the embedding will be skipped for these points using the labels. input_labels (`torch.LongTensor` of shape `(batch_size, point_batch_size, num_points)`): Input labels for the points, this is used by the prompt encoder to encode the prompt. According to the official implementation, there are 3 types of labels - `1`: the point is a point that contains the object of interest - `0`: the point is a point that does not contain the object of interest - `-1`: the point corresponds to the background We added the label: - `-10`: the point is a padding point, thus should be ignored by the prompt encoder The padding labels should be automatically done by the processor. input_boxes (`torch.FloatTensor` of shape `(batch_size, num_boxes, 4)`): Input boxes for the points, this is used by the prompt encoder to encode the prompt. Generally yields to much better generated masks. The boxes can be obtained by passing a list of list of list to the processor, that will generate a `torch` tensor, with each dimension corresponding respectively to the image batch size, the number of boxes per image and the coordinates of the top left and bottom right point of the box. In the order (`x1`, `y1`, `x2`, `y2`): - `x1`: the x coordinate of the top left point of the input box - `y1`: the y coordinate of the top left point of the input box - `x2`: the x coordinate of the bottom right point of the input box - `y2`: the y coordinate of the bottom right point of the input box input_masks (`torch.FloatTensor` of shape `(batch_size, image_size, image_size)`): SAM model also accepts segmentation masks as input. The mask will be embedded by the prompt encoder to generate a corresponding embedding, that will be fed later on to the mask decoder. These masks needs to be manually fed by the user, and they need to be of shape (`batch_size`, `image_size`, `image_size`). image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_channels, window_size, window_size)`): Image embeddings, this is used by the mask decoder to generate masks and iou scores. For more memory efficient computation, users can first retrieve the image embeddings using the `get_image_embeddings` method, and then feed them to the `forward` method instead of feeding the `pixel_values`. multimask_output (`bool`, *optional*): In the original implementation and paper, the model always outputs 3 masks per image (or per point / per bounding box if relevant). However, it is possible to just output a single mask, that corresponds to the "best" mask, by specifying `multimask_output=False`. attention_similarity (`torch.FloatTensor`, *optional*): Attention similarity tensor, to be provided to the mask decoder for target-guided attention in case the model is used for personalization as introduced in [PerSAM](https://huggingface.co/papers/2305.03048). target_embedding (`torch.FloatTensor`, *optional*): Embedding of the target concept, to be provided to the mask decoder for target-semantic prompting in case the model is used for personalization as introduced in [PerSAM](https://huggingface.co/papers/2305.03048). Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoModel, AutoProcessor >>> model = AutoModel.from_pretrained("danelcsb/sam2.1_hiera_tiny") >>> processor = AutoProcessor.from_pretrained("danelcsb/sam2.1_hiera_tiny") >>> img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/sam-car.png" >>> raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB") >>> input_points = [[[400, 650]]] # 2D location of a window on the car >>> inputs = processor(images=raw_image, input_points=input_points, return_tensors="pt") >>> # Get segmentation mask >>> outputs = model(**inputs) >>> # Postprocess masks >>> masks = processor.post_process_masks( ... outputs.pred_masks, inputs["original_sizes"], inputs["reshaped_input_sizes"] ... ) ``` NzAExactly one of pixel_values or image_embeddings must be provided.rzQYou should provide as many bounding boxes as input points per box. Got {} and {}.rrirrjrFbilinearT)rGrrrr:)r5rrrrrrr)r2r3r4r5r6r7r-)r^rMformatr2rr4r,rIr)rNrr) ones_likerrbrprcr.int32r&rrrrrrr(r1)rJrOrrrrr5rrrrrrr7r6r5rPr6r7rrylow_res_multimasksr2r4s r/rTzSam2Model.forwardps=z%*:d*BC`a a  # (?!!!$ (9(9!(<< gnn$**1-{/@/@/C '+&O&O&Q#.:.F\''*L\]_L`LfLfghLi &A&H&HUVXY[\&]# #  #G^tG^G^HH DL!13D ,B/$2J2JJL  (+<9T9T'U #D)+ Q1%**:rFIF     # (< ??<1a +C599]i]p]pqL  K$7 ;;Aq!+;B+?+E+EN^_aNbNiNiL"JJz1au{{ScdfSgSnSnooL  "  %)<)<)L)LLmm%%',,<<"'#"  "[&&' /3.A.A%%## /B/ ++ BSARAR B -b1(C%6$4-%5cr%:!5- B  B >J+>+!) 3-!5/   [ s/J..c |j|fi|}|j}|j}t|}|jj |d|d<|jj |d|d<|Dcgc]$}|jdjddd&}}|Dcgc]$}|jdjddd&}}|||j|jfScc}wcc}w)a Extract and preprocess image features using the vision encoder. Args: pixel_values (`torch.FloatTensor`): Input pixel values of shape `(batch_size, num_channels, height, width)`. Returns: `tuple`: A tuple containing: - feature_maps (`list[torch.Tensor]`): List of feature maps from different levels. - feature_maps_position_embeddings (`list[torch.Tensor]`): List of positional embeddings for each feature level. - vision_hidden_states (`tuple[torch.FloatTensor]`, *optional*): Hidden states from the vision encoder. - vision_attentions (`tuple[torch.FloatTensor]`, *optional*): Attention weights from the vision encoder. rrr) r%r!r"rr(rrrzrNr#r$)rJrOrvision_outputsr5 feature_maps_position_embeddings feature_mapfeature_map_position_embeddings r/r4zSam2Model.get_image_featuress03F$2E2E 3 3  &77 +9+O+O(L) ++33LOD Q++33LOD QT``K ++A.66q!Q?` `3S, . + 2 21 5 = =aA F, (, =~?[?[]k]v]vvv a, s =)C1,)C6)NNNN) NNNNNNTNN)!r%r&r'_tied_weights_keys_keys_to_ignore_on_load_missingrrrR"_keys_to_ignore_on_load_unexpectedrrAr+rCr)rr2no_gradr*rrrr8r LongTensorr<rrrqr1rTr,r4rUrVs@r/r+r+sQQ']&^#4nE]ef6gh*&z" : Bell BU]]_ '' +,  ell    4U]]_59373726 u001u//0e//0  e../ @59483737268r_s, !,, 5!9/F&@D KL?k?M ?6 FGL+LH L@"))D+ +\/8RYY/8r% II% <<% % << % U\\* %  %%, u|| 8C= ELL ?bii?Dbii<2@>Z4Zz  Ok O  O8/88>? +? D - )-  - `SbiiS2   6[3 [3|;)BII;)|J'ryyJ'Z6BII6rBLL4H/biiH/V Mw#Mw  Mw` Wr.