hddlZddlZddlmZddlmZmZddlZ ddl Z ddl m cm Zddl mZm Z ddlmZddlmZmZmZddlmZdd lmZmZdd lmZdd lmZmZm Z dd l!m"Z"d dl#m$Z$erddl%m&Z&e r ddl'm(Z(ddl)m*Z*eedGddeZ+ dNde jde jde jfdZ,dededefdZ-de jde jde jfdZ.Gdde j^Z0deded e1defd!Z2de jde jd e1de jfd"Z3Gd#d$e j^Z4Gd%d&e j^Z5Gd'd(e j^Z6 dOd)e j^d*e jd+e jd,e jd-ee jd.e7d/e7fd0Z8Gd1d2e j^Z9Gd3d4e j^Z:dPd5e jd6e7d7e;de jfd8Z<Gd9d:e j^Z=Gd;dGd=d>e j^Z?Gd?d@eZ@GdAdBe jZBGdCdDe j^ZCGdEdFe j^ZDGdGdHe j^ZEeGdIdJeZFedKGdLdMeFZGdJdMgZHy)QN) dataclass)CallableOptional)Tensornn)ACT2FN) ModelOutputis_scipy_availablerequires_backends)GradientCheckpointingLayer)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringis_accelerate_available)check_model_inputs) EomtConfig)linear_sum_assignment) PartialState)reducea Class for outputs of [`EomtForUniversalSegmentationOutput`]. This output can be directly passed to [`~EomtImageProcessor.post_process_semantic_segmentation`] or [`~EomtImageProcessor.post_process_instance_segmentation`] or [`~EomtImageProcessor.post_process_panoptic_segmentation`] to compute final segmentation maps. Please, see [`~EomtImageProcessor] for details regarding usage. ) custom_introc<eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeeej"ed <y) "EomtForUniversalSegmentationOutputa+ loss (`torch.Tensor`, *optional*): The computed loss, returned when labels are present. class_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, num_labels + 1)` representing the proposed classes for each query. Note the `+ 1` is needed because we incorporate the null class. masks_queries_logits (`torch.FloatTensor`): A tensor of shape `(batch_size, num_queries, height, width)` representing the proposed masks for each query. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Last hidden states (final feature map) of the last layer. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states all layers of the model. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tuple(torch.FloatTensor)` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Self and Cross Attentions weights from transformer decoder. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segementation. Nlossclass_queries_logitsmasks_queries_logitslast_hidden_state hidden_states attentions patch_offsets)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrr r!tupler"r#listrd/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/eomt/modeling_eomt.pyrr2s*)-D(5$$ %,8<(5#4#45<8<(5#4#45<59x 1 1298]st(!#-77:XX((44^SK\E\_bEbmflmN'//2 r.inputslabelsc|jjd}dtj||jz}|j ddddf|j ddddfz}d|dz|dzz z }|S)a A pair wise version of the dice loss, see `dice_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: `torch.Tensor`: The computed loss between each pairs. rr4N)sigmoidflattenr(matmulTsum)r?r@ numerator denominatorrs r/pair_wise_dice_lossrJ}s|^^  % %a (FELL22I**R.D)FJJrN47,CCK A +/2 2D Kr.c\|jd}tjd}||tj|}||tj |}tj ||z |j}tj ||z d|z j}||z}|S)a A pair wise version of the cross entropy loss, see `sigmoid_cross_entropy_loss` for usage. Args: inputs (`torch.Tensor`): A tensor representing a mask. labels (`torch.Tensor`): A tensor with the same shape as inputs. Stores the binary classification labels for each element in inputs (0 for the negative class and 1 for the positive class). Returns: loss (`torch.Tensor`): The computed loss between each pairs. rnone reduction)shaperBCEWithLogitsLossr( ones_like zeros_likerErF) r?r@height_and_width criterioncross_entropy_loss_poscross_entropy_loss_negloss_posloss_negrs r/$pair_wise_sigmoid_cross_entropy_lossrYs||A$$v6I&vuv/FG&vu/?/?/GH||25EEvxxPH||25EEF ~~VH h D Kr.c eZdZdZ d dedededeffd Zejdejdejd ejd ejd e e e f d Z xZ S)EomtHungarianMatcheraqThis class computes an assignment between the labels and the predictions of the network. For efficiency reasons, the labels don't include the no_object. Because of this, in general, there are more predictions than labels. In this case, we do a 1-to-1 matching of the best predictions, while the others are un-matched (and thus treated as non-objects). cost_class cost_mask cost_dice num_pointsct||dk(r|dk(r|dk(r td||_||_||_||_y)aHCreates the matcher Params: cost_class (`float`, *optional*, defaults to 1.0): Relative weight of the classification error in the matching cost. cost_mask (`float`, *optional*, defaults to 1.0): This is the relative weight of the focal loss of the binary mask in the matching cost. cost_dice (`float`, *optional*, defaults to 1.0): This is the relative weight of the dice loss of the binary mask in the matching cost. num_points (`int`, *optional*, defaults to 12544): No. of points to sample on which the mask loss will be calculated. The same set of K points are uniformly sampled for all prediction and ground truth masks to construct the cost matrix for bipartite matching. rzAll costs can't be 0N)super__init__ ValueErrorr_r\r]r^)selfr\r]r^r_ __class__s r/rbzEomtHungarianMatcher.__init__sK"  ?yA~)q.34 4$$""r.rr mask_labels class_labelsr2c g}|jd}t|D]}||jd}||} |dd||f } ||j| } | dddf} | dddf} t j d|j d| j} | j| jddd} t| | djd} | j| jddd}t| |djd} t| | }t| | }|j|z|j| zz|j|zz}t j |t j"d }t j$|t j"d }t j&|d}t)|j+}|j-||Dcgc]O\}}t j.|tj0 t j.|tj0 fQ}}}|Scc}}w) ao Params: masks_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, num_labels` with the classification logits. class_queries_logits (`torch.Tensor`): A tensor of dim `batch_size, num_queries, height, width` with the predicted masks. class_labels (`torch.Tensor`): A tensor of dim `num_target_boxes` (where num_target_boxes is the number of ground-truth objects in the target) containing the class labels. mask_labels (`torch.Tensor`): A tensor of dim `num_target_boxes, height, width` containing the target masks. Returns: matched_indices (`list[tuple[Tensor]]`): A list of size batch_size, containing tuples of (index_i, index_j) where: - index_i is the indices of the selected predictions (in order) - index_j is the indices of the corresponding selected labels (in order) For each batch element, it holds: len(index_i) = len(index_j) = min(num_queries, num_target_boxes). rrBNrr4deviceF align_cornersg _Bg _©dtype)rOrangesoftmaxtor(randr_rjrepeatr>r:rYrJr]r\r^minimumtensormaximum nan_to_numrcpuappend as_tensorint64)rdrrrfrgindices batch_sizei pred_probs pred_maskr\ target_maskr1target_coordinatespred_coordinatesr]r^ cost_matrixassigned_indicesjmatched_indicess r/forwardzEomtHungarianMatcher.forwards/8*,*//2 z" -A-a088ejjY_` 36wP]3^ G /FM< +FLLO+->v||A->@Q&,,q/@QQ R X XY_ `AFL*6<<?*,=fll1o,== > G},,&PsD+rrgr|c |}|j\}}}tj|j}|j |} t j t||D  cgc] \} \}} | | c} }} } t j||f|jt j|j} | | | <|jdd}||| }d|i}|Scc} }} w)aCompute the losses related to the labels using cross entropy. Args: class_queries_logits (`torch.Tensor`): A tensor of shape `batch_size, num_queries, num_labels` class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. Returns: `dict[str, Tensor]`: A dict of `torch.Tensor` containing the following key: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. )weight) fill_valuernrjrr4loss_cross_entropy) rOrCrossEntropyLossr$_get_predictions_permutation_indicesr(catrfullrr{rj transpose)rdrrgr| pred_logitsr} num_queriesrrTidxtargetrtarget_classes_otarget_classespred_logits_transposedloss_celossess r/ loss_labelszEomtLoss.loss_labelss"+ %0%6%6" K''t/@/@A 77@ 99-0w-G H H>66AqVAY H   %$//]h]o]o /s!,!6!6q!!<2NC&0  Is#C! rrfrcj|}j|}||}j|\}} ||}|dddf}|dddf}tj5j |fdj jj} t|| djd} dddt| djd} t|  |t| | |d} ~~| S#1swYExYw)aCompute the losses related to the masks using sigmoid_cross_entropy_loss and dice loss. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. indices (`tuple[np.array])`: The indices computed by the Hungarian matcher. num_masks (`int)`: The number of masks, used for normalization. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing two keys: - **loss_mask** -- The loss computed using sigmoid cross entropy loss on the predicted and ground truth. masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth, masks. Nc&j|SN)calculate_uncertainty)logitsrds r/z%EomtLoss.loss_masks..st99&Ar.Frkr) loss_mask loss_dice) r _get_targets_permutation_indicesrr(rsample_points_using_uncertaintyr_rrr>r:rr)rdrrfr|rsrc_idxtgt_idx pred_masks target_masksrr1 point_labels point_logitsrs` r/ loss_maskszEomtLoss.loss_maskss,4;;GD77@)'2 ::;G a#G,  4( #AtG, ]]_ i $ D DA%%,, ! ( 6GW\]eefghL i$J0AQVW__`ab 4L,PYZ"<yI    ) i is (ADD c tjt|Dcgc]\}\}}tj|| c}}}}tj|Dcgc]\}}| c}}}||fScc}}}wcc}}wrr(rr full_like)rdr|r~srcr batch_indicespredictions_indicess r/rz-EomtLoss._get_predictions_permutation_indicessj iX_N`"a"a{q(35??3#:"ab #iiW(E#q(EF111#b(E #A7  A> c tjt|Dcgc]\}\}}tj|| c}}}}tj|Dcgc]\}}| c}}}||fScc}}}wcc}}wrr)rdr|r~rtgtrtarget_indicess r/rz)EomtLoss._get_targets_permutation_indicessh iX_N`"a"a{q(1c5??3#:"ab #@HQC#@An,,#b#@rrc2tj| }|S)a In Eomt paper, uncertainty is estimated as L1 distance between 0.0 and the logit prediction in 'logits' for the foreground class in `classes`. Args: logits (`torch.Tensor`): A tensor of shape (R, 1, ...) for class-specific or class-agnostic, where R is the total number of predicted masks in all images and C is: the number of foreground classes. The values are logits. Returns: scores (`torch.Tensor`): A tensor of shape (R, 1, ...) that contains uncertainty scores with the most uncertain locations having the highest uncertainty score. )r(abs)rdruncertainty_scoress r/rzEomtLoss.calculate_uncertaintys %yy01!!r.r_rrc |jd}t||z}tj||d|j}t ||d} || } t||z} || z } tj | dddddf| dd} |tj|tj|j z}| |dddfz } |jd d| jd ddfj|| d}| dkDr:tj|tj|| d|jgd }|S) a This function is meant for sampling points in [0, 1] * [0, 1] coordinate space based on their uncertainty. The uncertainty is calculated for each point using the passed `uncertainty function` that takes points logit prediction as input. Args: logits (`float`): Logit predictions for P points. uncertainty_function: A function that takes logit predictions for P points and returns their uncertainties. num_points (`int`): The number of points P to sample. oversample_ratio (`int`): Oversampling parameter. importance_sample_ratio (`float`): Ratio of points that are sampled via importance sampling. Returns: point_coordinates (`torch.Tensor`): Coordinates for P sampled points. rr4riFrkNr)kr6rrBr6) rOrr(rrrjr>topkarangelongviewr)rdruncertainty_functionr_rr num_boxesnum_points_sampledr1rpoint_uncertaintiesnum_uncertain_pointsnum_random_pointsrshifts r/rz(EomtLoss.sample_points_using_uncertaintysI<LLO  .>!>?"JJy2DaPVP]P]^#F,=US 2<@"#:Z#GH&)==jj,Q1W59MSTUVWX"U\\)5::V\VcVc%dd uQW~-222q9#((2,/JOOPY[oqrs q % "EJJy:KQW]WdWd$ef! ! r.auxiliary_predictionsc|j||||}|j||dj}i|j|||||j |||}|jt |D]\\} } | d}| d}|j ||||} | jD cic] \} } | d| | } } } |j| ^|Scc} } w)a This performs the loss computation. Args: masks_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, height, width)`. class_queries_logits (`torch.Tensor`): A tensor of shape `(batch_size, num_queries, num_labels)`. mask_labels (`torch.Tensor`): List of mask labels of shape `(labels, height, width)`. class_labels (`list[torch.Tensor]`): List of class labels of shape `(labels)`. auxiliary_predictions (`dict[str, torch.Tensor]`, *optional*): if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], then it contains the logits from the inner layers of the EomtMaskedAttentionDecoder. Returns: losses (`dict[str, Tensor]`): A dict of `torch.Tensor` containing three keys: - **loss_cross_entropy** -- The loss computed using cross entropy on the predicted and ground truth labels. - **loss_mask** -- The loss computed using sigmoid cross_entropy loss on the predicted and ground truth masks. - **loss_dice** -- The loss computed using dice loss on the predicted on the predicted and ground truth masks. if `use_auxiliary_loss` was set to `true` in [`EomtConfig`], the dictionary contains additional losses for each auxiliary predictions. rrirrr) r get_num_masksrjrrrritemsupdate)rdrrrfrgrr|rrr aux_outputs loss_dictkeyvalues r/rzEomtLoss.forward<s H,,35I;Xde&&|LO CsB#r)r$r%r&rdictstrrrbr,rrrr+rnparrayrr(rrrrrrrrjrrrs@r/rrKs! z! S%Z8H! F$tCy/d3i-4<-E&RX.DY-" $* :>v, QVWYW_W_Q`  c6k  D<#ll<%,,'<rxx <  < c5<< <|2 - "ELL"U\\""5! 5! 5!  5! "' 5! 5!zDH 5#ll5$ll5%,,' 5 5<<( 5 (S%,,->(?@ 5 c5<< 5n %,,  QVQ]Q] r.rcZeZdZdZfdZdej dej fdZxZS)EomtPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr kernel_sizestride)rarb image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesrConv2d projection)rdrrrrr r%res r/rbzEomtPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir. pixel_valuesr2c|jd}||jk7rtd|jd|d|j|j dj dd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r4)rOrrcr'rDr)rdr(r embeddingss r/rzEomtPatchEmbeddings.forwardsz#))!, 4,, ,!../yaI __\2::1=GG1M r.) r$r%r&r'rbr(rrrrs@r/rrs) jELLU\\r.rcdeZdZdZdeddffd ZdejdejfdZxZ S)EomtEmbeddingszM Construct the CLS token, mask token, position and patch embeddings. rr2Nct|||_|j|_t j t jdd|j|_ t j t jd|j|j|_ t||_|jj}t j |j"|_d|jz|_t j(||j|_|j-dt j.|j1ddy)Nr position_ids)rrBF) persistent)rarbrrr Parameterr(randnr  cls_tokenrnum_register_tokensregister_tokensrpatch_embeddingsr%Dropouthidden_dropout_probdropoutnum_prefix_tokens Embeddingposition_embeddingsrrexpand)rdrr%res r/rbzEomtEmbeddings.__init__s   ++ekk!Q8J8J&KL!||EKK6;U;UW]WiWi,jk 3F ;++77 zz&"<"<= !"V%?%?!?#%<< V=O=O#P  ^U\\+-F-M-Mg-Vchir.r(c|j\}}}}|jjjj}|j|j |}|j j|dd}|jj|dd}||j|jz}tj|||gd}|j|}|S)NrmrBrr)rOr6r'rrnrqr3r=r5r<r/r(rr9)rdr(r}r target_dtyper+ cls_tokensr5s r/rzEomtEmbeddings.forwards*00 Aq!,,77>>DD ** ..55j"bI$":":4;L;L"MM YY OZHaP \\*- r.) r$r%r&r'rrbr(rrrrs@r/r-r-s9jzjdj ELL U\\ r.r-modulequeryr r attention_maskscalingr9c tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)NrB)r6rn)ptrainingrr4) r(rErrr8rpfloat32rqrnr9rH contiguous) rArBr r rCrDr9r< attn_weights attn_outputs r/eager_attention_forwardrMs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r.c eZdZdZfdZ ddej deej deej eej ffdZ xZ S) EomtAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rarbrr  embed_dimnum_attention_heads num_headshead_dimrcscaleattention_dropoutr9 is_causalrLineark_projv_projq_projout_projrdrres r/rbzEomtAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r.r!rCr2c :|j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rt|jj} | |||| ||j|j|jsdn |j\} } | j!|||j#} |j%| } | | fS)z#Input shape: Batch x Time x Channelrr4eager)rWrDr9)rOr[rYrZrrSrTrrMr_attn_implementationrrWrUrHr9reshaperJr\) rdr!rCr<r} seq_lengthrQquerieskeysvaluesattention_interfacerLrKs r/rzEomtAttention.forwardsa-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0 L((r.r) r$r%r&r'rbr(rrr+rrrs@r/rOrOsVGB.26$)||$)!.$) u||Xell33 4 $)r.rOcXeZdZdfd Zdej dej fdZxZS)EomtLayerScaler2ct|tj|jt j |jz|_yr) rarbrr1layerscale_valuer(rr lambda1r]s r/rbzEomtLayerScale.__init__s8 ||F$;$;ejjI[I[>\$\] r. hidden_statec ||jzSr)rlrdrms r/rzEomtLayerScale.forward#sdll**r.r2Nr$r%r&rbr(rrrrs@r/riris$^+ELL+U\\+r.riinput drop_probrHc|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r`rr)rr)rOndimr(rrrnrjfloor_div)rrrsrH keep_probrO random_tensoroutputs r/ drop_pathr{'sCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr.cxeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) EomtDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrsr2c0t|||_yr)rarbrs)rdrsres r/rbzEomtDropPath.__init__>s "r.r!cDt||j|jSr)r{rsrHrdr!s r/rzEomtDropPath.forwardBs FFr.c d|jS)Nzp=)rsrds r/ extra_reprzEomtDropPath.extra_reprEsDNN#$$r.r) r$r%r&r'rrrbr(rrrrrrs@r/r}r};sGb#(5/#T#GU\\GellG%C%r.r}cXeZdZdfd Zdej dej fdZxZS)EomtMLPr2c~t||jx}}t|j|jz}t j ||d|_t|jtrt|j|_ n|j|_ t j ||d|_ y)NTbias)rarbr r mlp_ratiorrXfc1r! hidden_actrr activationfc2rdr in_features out_featureshidden_featuresres r/rbzEomtMLP.__init__Js %+%7%77 lf0063C3CCD99[/E f'' -$V%6%67DO$//DO99_lFr.rmcl|j|}|j|}|j|}|Sr)rrrros r/rzEomtMLP.forwardUs2xx - |4 xx - r.rprqrs@r/rrIs$ GELLU\\r.rcXeZdZdfd Zdej dej fdZxZS) EomtSwiGLUFFNr2c0t||jx}}t|j|jz}t|dzdz dzdzdz}t j |d|zd|_t j ||d|_y)Nr4rTr) rarbr rrrrX weights_in weights_outrs r/rbzEomtSwiGLUFFN.__init__]s %+%7%77 lf0063C3CCD2Q67!;AAE))K_1D4P99_lNr.rmc|j|}|jdd\}}tjj ||z}|j |S)Nr4rBr)rchunkrr8silur)rdrmx1x2hiddens r/rzEomtSwiGLUFFN.forwardfsS|4 ##A2#.B##B'",''r.rprqrs@r/rr\s$O(ELL(U\\(r.rceZdZdZdeddffd Z d dejdeejdejfdZ xZ S) EomtLayerzCThis corresponds to the Block class in the original implementation.rr2Nct|tj|j|j |_t||_t||_ |jdkDrt|jntj|_tj|j|j |_|j rt#||_nt'||_t||_y)Nepsr`)rarbr LayerNormr layer_norm_epsnorm1rO attentionri layer_scale1drop_path_rater}Identityr{norm2use_swiglu_ffnrmlpr layer_scale2r]s r/rbzEomtLayer.__init__ps \\&"4"4&:O:OP &v.*62@F@U@UX[@[f&;&;>2Di#P q $ 1 12G H'<= M zz-0 xx - ((6 ~~l3mC r.r) r$r%r&r'rrbr(rrrrrs@r/rrmsPM3z3d3&-1||ELL)  r.rcXeZdZdfd Zdej dej fdZxZS)EomtLayerNorm2dc*t||||y)N)relementwise_affine)rarb)rdrraffineres r/rbzEomtLayerNorm2d.__init__s 36Jr.rmr2c|jdddd}tj||j|j|j |j }|jdddd}|S)Nrr4rr)permuteF layer_normnormalized_shaperrrros r/rzEomtLayerNorm2d.forwardsb#++Aq!Q7 ||L$2G2GVZV_V_aeaiaij #++Aq!Q7 r.)gư>Trqrs@r/rrs$KELLU\\r.rc\eZdZdeffd Zdej dej fdZxZS)EomtScaleLayerrct||j}tj||dd|_t |j|_tj||dd|d|_ t||_ y)Nr4rrrF)rpaddinggroupsr) rarbr rConvTranspose2dconv1r rrr&conv2r layernorm2drdrr res r/rbzEomtScaleLayer.__init__su (( '' [aXYZ  !2!23YY     +;7r.r!r2c|j|}|j|}|j|}|j|}|Sr)rrrrrs r/rzEomtScaleLayer.forwardsB =1  6  =1 ((7 r. r$r%r&rrbr(rrrrs@r/rrs*8z8 U\\ellr.rc\eZdZdeffd Zdej dej fdZxZS)EomtScaleBlockrct||j|_t j t |jDcgc] }t|c}|_ycc}wr) rarbnum_upscale_blocks num_blocksr ModuleListrorblockrdrrres r/rbzEomtScaleBlock.__init__sG  33]]E$//DZ#[qN6$:#[\ #[sA&r!r2c8|jD] }||} |Sr)r)rdr!rs r/rzEomtScaleBlock.forwards%ZZ 1E!-0M 1r.rrs@r/rrs,]z] U\\ellr.rc\eZdZdeffd Zdej dej fdZxZS) EomtMaskHeadrc t||j}tj|||_tj|||_tj|||_t|j|_ yr) rarbr rrXrrfc3r rrrs r/rbzEomtMaskHead.__init__sa (( 99[+699[+699[+6 !2!23r.r!r2c|j|j|}|j|j|}|j|}|Sr)rrrrrs r/rzEomtMaskHead.forwardsD(?@ (?@ / r.rrs@r/rrs*4z4U\\ellr.rcdeZdZUdZeed<dZdZdZdgZ dZ dZ e e dZd ej d d fd Zy ) EomtPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. reomtr(FrT)r!r"rAr2Nc|jj}t|tjtj tj frtjj|jtjd|jxtjj|j\}}|dkDrdtj|z nd}tjj|j| |yyt|tjrJ|jj j#d|jj j%yt|tj&rf|jj j)dd|j*2|jj |j*j%yyt|t,rGt/|dr:|j0j j#|jj2yyt|t4rtjj7|j8j j;t<j>d|j;|j8j@|j8_|jBj j%yy) N)arrr5r`)rstdrl)"rinitializer_ranger!rrXr&rinitkaiming_uniform_rmathsqrtr_calculate_fan_in_and_fan_outuniform_rdatafill_zero_r;normal_ padding_idxrihasattrrlrkr- trunc_normal_r3rqr(rIrnr5)rdrArfan_inrbounds r/ _init_weightsz!EomtPreTrainedModel._init_weightsskk++ fryy"))R5G5GH I GG $ $V]]diil $ C{{&GGAA&--P 17!DIIf--  ufe<' - MM   $ $S ) KK   " " $  - MM   & &CQ & 7!!- ""6#5#56<<>.  /vy)##))$++*F*FG*  /$&GG$9$9  %%((7cs%:%b!!''(    !  " " ' ' - - / 0r.)r$r%r&r'rr*base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attnrrO_can_record_outputsrModulerr-r.r/rrsX $O&+#$ N"# 0BII0$0r.rzV The EoMT Model with head on top for instance/semantic/panoptic segmentation. ceZdZdZdeffd Zdededededeeefd eeeff d Z d eeefd efd Z e e ddede eede eed e eedeed ef dZdZdej fdZedZxZS)EomtForUniversalSegmentationr(rct||||_|j|_t ||_t j|j|j|_ t j|j|j|_ t jt|jDcgc] }t!|c}|_t%||_t)||_t j,|j|j.dz|_|j2|j4z|j2|j4zf|_|j8|j:|j<d|_tA||j>|_!|jEdtGjH|jJ|jMycc}w)Nrr)rrr)rrattn_mask_probs)'rarbrnum_hidden_layersr-r+rrr r layernormr;rrBrrorlayersr upscale_blockr mask_headrXrclass_predictorrr grid_sizerrrrrrTrr(rr post_initrs r/rbz%EomtForUniversalSegmentation.__init__ sp   !'!9!9(0f&8&8f>S>ST\\&"4"4f6H6HI mmfF^F^@_$`1Yv%6$`a +F3%f-!yy););V=N=NQR=RS ++v/@/@@&BSBSW]WhWhBhi"("5"5++++.  "T=M=MN . 6;L;L0MN %%as>G*rrrfrgrr2c|j|||||}|jjD]'\}}|jD]\} } || vs | |z} )|S)Nrrrfrgr)rTrr) rdrrrfrgrr r rloss_keyrs r/ get_loss_dictz*EomtForUniversalSegmentation.get_loss_dict(s|(,~~!5!5#%"7 (6(  ++113 #KC"+//"3 #$(?FND # # r.r c4t|jSr)rGrf)rdr s r/get_lossz%EomtForUniversalSegmentation.get_loss@s9##%&&r.r#r<c (d\}}d}| td|j|} t|jD]\} } | |j|j j z k(rp|jjdddddfj| jdddj| j} tj| | fd} | |j|j j z k\r|js7|j | |jz |j j zdkDr|j#| } |j%| \}}||fz }||fz }tj&| jd| jd| jd| jtj(}t+j,||j.d }|j1|j3d|j3dd}|j j4}||jj6z}|dkD|ddd||df<|j9||j | |jz |j j z|||j }|dddd fjd|j j:dd}|j=j?|d }| | |} |j#| }|j%|\}}||fz }||fz }d}|B|@d}tA||D]/\}}|jC||||d}||jE|z }1tG|||||S)ah mask_labels (`list[torch.Tensor]`, *optional*): list of mask labels of shape `(num_labels, height, width)` to be fed to a model class_labels (`list[torch.LongTensor]`, *optional*): list of target class labels of shape `(num_labels, height, width)` to be fed to a model. They identify the labels of `mask_labels`, e.g. the label of `mask_labels[i][j]` if `class_labels[i][j]`. patch_offsets (`list[torch.Tensor]`, *optional*): list of tuples indicating the image index and start and end positions of patches for semantic segementation. )r-r-Nz You have to specify pixel_valuesrrBrr)rjrnbilinear)sizemode)probnum_query_tokensencoder_start_tokensrj.ger`r)rrrr r#)$rcr+rrrrrrBrr=rOrqrjr(rrHrrpredictrrr interpolaterrr rr:_disable_attention_maskrRr masked_fillrrr r)rdr(rfrgr#r<masks_queries_logits_per_layerclass_queries_logits_per_layerrCr!r layer_modulerBnorm_hidden_statesrrinterpolated_logitsrrsequence_outputrr s r/rz$EomtForUniversalSegmentation.forwardCs(JPF&(F  ?@ @ 5 !*4;;!7. H Cd,,t{{/E/EEE ))$1*5<<]=P=PQR=SUWY[\__`m`t`tu % 5-*@a H d,,t{{/E/EEE !5!5cD&>',,Q/1D1I1I!1Lb'#$(;;#:#: '7$//:[:['[$ObdeNeq"3#3"35I5JJK"&!=!="--cD4J4J.JT[[McMc.cd%5)=)00 ">""04 !=!D!DRIhIhjlnp!q!/!5!5!7!C!C^OUY!Z(GM]. H`..759\\/5R22&+?*AA&&+?*AA&  "|'?D>A.0N? 1:$&:!..)=)= +!-*. /  i00 12!5!5-'   r.c.|jjSr)r+r6rs r/get_input_embeddingsz1EomtForUniversalSegmentation.get_input_embeddingss///r.rc|ddd|jjddf}|j|}|dd|jj|jjzdddf}|j dd}|j |jddg|j}|j|}|j|}tjd||}||fS)Nrr4rrBzbqc, bchw -> bqhw) rrrr+r:rrbrOrrrr(einsum)rdr query_tokens class_logits prefix_tokens mask_logitss r/rz$EomtForUniversalSegmentation.predictsa!:4;;#:#:!:A=> ++L9 q$++"9"9DOO<]<]"]"_abbc %//15 - --m.A.A!.DbZ4>>Z ~~l3 **=9 ll#6 mT L((r.c|dkr9tj|jd|||kD}d|ddd||df|<|S)Nrrri)r(rrrO) attn_maskrrrrjrandom_queriess r/rz4EomtForUniversalSegmentation._disable_attention_masksW !8"ZZ (: Rr.)NNN)r$r%r&rrrbrrrrr rrrr,rrrrrr(r staticmethodrrrs@r/rrs7 %Oz8$%    $CK0  c6k 0'$sF{"3''/3/304 e e d6l+e tF|, e  V - e +, e  ,e e N0)ell) r.r)F)r`)r`F)Icollections.abcr"r dataclassesrtypingrrnumpyrr(torch.nn.functionalrr8rr activationsr file_utilsr r r modeling_layersr modeling_utilsrrprocessing_utilsrutilsrrr utils.genericrconfiguration_eomtrscipy.optimizer accelerateraccelerate.utilsrrr>rJrYrr[rrrrrr-rrMrOrirr{r}rrrrrrrrrr__all__r-r.r/r9s, !% !LL9F&PP/*4''  77  7BLQLL5:\\ \\@6,u||X]XdXd8g299gTff<u||U\\VY^c^j^j(uryyup "))B"RYY"X% II% <<% % << % U\\* %  %%.;)BII;)|+RYY+U\\eTV[VbVb( %299 %bii&(BII("'*'TbllRYY2 RYY 299"(0/(0(0V #6  D !"@ Ar.