h3JdZddlZddlmZmZddlZddlmZddlmZddl m Z ddl m Z m Z mZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZmZddlmZddl m!Z!ejDe#Z$GddejJZ&GddejJZ'GddejPZ)GddejPZ*GddeZ+eGddeZ,Gdde,Z-ed !Gd"d#e,Z.ed$!Gd%d&e,eZ/d&dgZ0y)'z/PyTorch TrOCR decoder model (based on RoBERTa).N)OptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) TrOCRConfigcneZdZdZdedeffd Zd dejdedejffd ZxZ S) TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|y)N)offsetsuper__init__)selfrr __class__s f/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/models/trocr/modeling_trocr.pyr z(TrOCRLearnedPositionalEmbedding.__init__0s$  $++5}E input_idspast_key_values_length position_idsc$|a|jdd\}}tj|||ztj|jj j |d}n|jd}t|%||jzS)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr*expand unsqueezerforwardr)r!r%r&r'bszseq_lenr"s r#r3z'TrOCRLearnedPositionalEmbedding.forward6s  $??2A.LC <<&(>(HPUPZPZcgcncncucufS"o (11!4Lw|dkk9::r$)rN) __name__ __module__ __qualname____doc__intr r-Tensorr3 __classcell__r"s@r#rr+sHFsF3F ; ;s ;^c^j^j ; ;r$rc `eZdZdZd dedededeeffd Zdejffd Z xZ S) TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rr padding_idx embed_scalec6t||||||_yN)rr rA)r!rrr@rAr"s r#r z!TrOCRScaledWordEmbedding.__init__Js  D&r$r%c<t|||jzSrC)rr3rA)r!r%r"s r#r3z TrOCRScaledWordEmbedding.forwardNswy)D,<,<<=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r$r%r&cP|j\}}|j||j|j|j}|jdz|z}|j ||j jdkDr,|j ||j|j|_|j j|j|_|j jd|jdj||dj}|S)Nrrr+) size"create_position_ids_from_input_idsr@r\r*rMrLrrK index_selectrZdetach)r!r%r&r4r5r'max_posxs r#r3z*TrOCRSinusoidalPositionalEmbedding.forwardps ~~' W>>y$JZJZ\rsvv    ""Q&0 << 7T\\->->q-A#A--gt7I7I4K[K[\DL||t'9'9: LL % %a):):2)> ? D DS'SU V ] ] _r$c|j|j}tj|dj ||z|z}|j |zS)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrQ)ner:r-cumsumtype_asr/)r!r%r@r&maskincremental_indicess r#rbzETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_idssW||K(,,.$||Da8@@FI__cgg"'')K77r$rC)r)r6r7r8r9r:rr staticmethodrLr-no_gradr;r3rbr<r=s@r#rHrHRsNDcD#DHUXMD1c1#1HUXM11$U]]_s&bc 8 847 8QYZ]Q^ 8r$rHceZdZdZ ddededeedeedeedeedeed eed eeffd Ze d d d dde jdee jd ee dee jdee jdeedee jde e jee jee e jffdZxZS)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper. embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc Pt |||_||n||_||n||_||_||_||z|_|j|z|jk(std|jd|d|jdz|_ ||_ | |_ tj|j|||_tj|j|||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rw)rr rqrsrtrrruhead_dim ValueErrorscalingrvryrLineark_projv_projq_projout_proj) r!configrqrrrsrtrurvrwrxryr"s r#r zTrOCRAttention.__init__s " ,D)  ,D) " !Y.  )T^^;MdnnM]^;b" }}d* $"ii 94@ ii 94@ ii 94@  )YTB r$past_key_valuepast_key_values4.58new_nameversion hidden_stateskey_value_statesattention_masklayer_head_maskoutput_attentionscache_positionreturnc |du}|j\} } } |j||jz} |St|trA|j j |j} |r |j}n|j}n|}|r|n|}|rK|I rGj|jj}|j|jj}n|j|}|j|}|j| d|j |j"j%dd}|j| d|j |j"j%dd}|D|s|nd}j'|||jd|i\}}|rd|j |j<| |j zd|j"f}| j| | |j |j"j%dd} | j(|} |j(|}|j(|}|jd}t+j,| |j%dd}|j| |j z| |fk7r/t/d| |j z| |fd|j|{|j| d| |fk7r#t/d | d| |fd|j|j| |j | ||z}|j| |j z| |}t0j2j5|d }||j|j fk7r*t/d |j fd|j|jdddd|j| |j | |z}|j| |j z| |}|r?|j| |j | |}|j| |j z| |}nd}t0j2j7||j6|j8 }t+j,||}|j| |j z| |j"fk7r7t/d | |j | |j"fd|j|j| |j | |j"}|j%dd}|j)| | | }|j;|}||fS)z#Input shape: Batch x Time x ChannelNr+rrrTz$Attention weights should be of size z , but is z!Attention mask should be of size rQz/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )rarr~ isinstancer is_updatedgetrycross_attention_cacheself_attention_cachelayerskeysvaluesrrrZrrr| transposeupdatereshaper-bmmr}r functionalsoftmaxrurr)r!rrrrrrrrxr4tgt_lenrq query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r#r3zTrOCRAttention.forwards7.T9"/"4"4"6Wi{{=1DLL@  &/+>?,77;;DNNK %*9*O*O'*9*N*N'&5#-?)] /"=*,33DNNCHHJ.55dnnELLL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*7It+>+E+E dnn?OQ_>`,( L&AEO..t~~>DNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<!CF%0A6R48+/1526,115p2||p2#5<<0p2"% p2 !. p2 "%,,/ p2$D>p2!.p2 u||Xell3XeELL>Q5RR Sp2Sp2r$rpcHeZdZddeffd Zeddd ddejdeejd eejd eejd eejd eejdee d ee dee deejfdZ xZ S)TrOCRDecoderLayerrc bt||j|_t ||j|j |j d||_|j|_t|j|_ |j|_ tj|j|_|j rot ||j|j |j"|j"|j dd| |_tj|j|_tj(|j|j*|_tj(|j*|j|_tj|j|_y)NT)rqrrrurvry)rqrrrsrtrurvrxry)rr hidden_sizerqrpdecoder_attention_headsattention_dropout self_attnruractivation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normrvcross_attention_hidden_size encoder_attnencoder_attn_layer_normrdecoder_ffn_dimfc1fc2final_layer_norm)r!rryr"s r#r zTrOCRDecoderLayer.__init__+s= ++' nn44,,  ~~ #F$>$>?"(";";$&LL$@!    ... 88777700#'# !D ,.<<+GD (99T^^V-C-CD99V33T^^D " T^^ <r$rrrrrrencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskr use_cacherc 2|} |j|||||| \}} tjj||j|j}| |z}|j |}d} |i|} |j ||||||| \}} tjj||j|j}| |z}|j|}|} |j|j|}tjj||j|j}|j|}tjj||j|j}| |z}|j|}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size *(decoder_attention_heads,)*. past_key_values (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrrrrN)rrrrrrr) rrrrurrrrrrrrr)r!rrrrrrrrrrresidualself_attn_weightscross_attn_weightsoutputss r#r3zTrOCRDecoderLayer.forwardOs@!,0>>'+)+/) ,:, ( ( --mt||VZVcVc-d  =0 11-@ " ,$H040A0A+!65 : /"3-1B1 -M-MM11-4< >Gr$rC) NNNNNNFTN) r6r7r8rr rr-r;rr rr3r<r=s@r#rr*s"={"=H%0A6R268<9=26=A+/,1$(15Q||Q!.Q ( 5 Q !) 6 Q "%,,/ Q%-U\\$:Q"%Q$D>QD>Q!.QSQr$rc,eZdZUeed<dZdZdgZdZy)TrOCRPreTrainedModelrmodelTrc6|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjrf|j jjd||j2|j j|jjyyy)Nr)meanstd) rinit_stdrrrConv1dr0datanormal_rwzero_ Embeddingr@)r!modulers r# _init_weightsz"TrOCRPreTrainedModel._init_weightsskk"" fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r$N) r6r7r8r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr$r#rrs" &*#,- ?r$rcJeZdZdZdeffd Z ddZxZS) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rc `t|||j|_|j|_|j |_|jrtj|jnd}t|j|j|j ||_ |jr&t|j |j|_n@t%|j |j zdz|j|j |_|j&r%t)j*|j|_nd|_t)j,t/|j0Dcgc]}t3||c}|_d|_|j9ycc}w)NrE)rAr)ryF)rr rudecoder_layerdrop layerdrop pad_token_idr@scale_embeddingrSsqrtrr? vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrHlayernorm_embeddingrr ModuleListrangedecoder_layersrrgradient_checkpointing post_init)r!rrAir"s r#r zTrOCRDecoder.__init__sP  ~~ 11!..7=7M7Mdii 2 23SV 4   v1143C3CQ\   1 1#B6CaCacicucu#vD #E..1A1AAAE""  $D  % %')||F4F4F'GD $'+D $mmUZ[a[p[pUq$rPQ%6v%K$rs &+#  %ss3F+c | | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|"|}|j d|jd}n-| |jdd}|dddddf}n t d|jr%|jr| rtjdd} | rN|L|4tt|jt|jnt|j}| r:t|t r*tjdtj"|}||j%nd }||j'|}|jj(r|j+|| }n|j+|| }||z}|j,|j-|}t.j0j3||j2|j }|j}t5||||}||t7||j8|d }| rd nd}| rd nd}| r|d nd}t;||gddgD]j\}}| |jd t=|j>k7s3t d|dt=|j>d|jd dtA|j>D]{\}}| r||fz }|jr%tCjDg}||jFkr?||||||||nd|||nd|| | |  }|d }| sg||dfz }|s||dfz }}| r||fz }| st!d|||||fDStI|||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer+zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...F)rzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)r&r)rr head_maskcross_attn_head_maskzThe `z` should be specified for z layers, but it is for .)rrrrrrrrrc3$K|]}|| ywrCr).0vs r# z'TrOCRDecoder.forward..s=s)last_hidden_staterr attentionscross_attentions)%rroutput_hidden_statesruse_return_dictr}rZr,rarrlogger warning_oncer r rrfrom_legacy_cacheget_seq_lengthrrrrrrrurr r)ziplenr enumerater-randrr)r!r%rrrrrr inputs_embedsrrr return_dictrinput input_shaper& embed_posrall_hidden_statesall_self_attnsall_cross_attentions attn_mask mask_nameidx decoder_layerdropout_probability layer_outputss r#r3zTrOCRDecoder.forwardsXb2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E!r5;;r?;I  &',,.s3K!!Q(+Ede e  & &4==##t" 0)4$L $DlZ^ZeZeFfg!5  OU;   \  2CCOTOETE`!?!?!Afg   --i8M ;; 6 6,,UKa,bI,,YOe,fI% 1  # # / 44]CM --mt||VZVcVc-d kk : K8N  ! ,1G1S%?& (;(;[QS_& " #7BD0d&7>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03 #,DKK"8 @ C#!m%55!}}&+jjn#&7)%'=3<3H3dI]Ii,@,Eos /"3#- M*!,M =#3"55(4(]1-=,??(7 @< -!1 1 ':K^]qr  9+++%1   r$) NNNNNNNNNNNNN)r6r7r8r9rr r3r<r=s@r#rrsD{B"#!!Q r$ra The TrOCR Model with a language modeling head. Can be used for summarization. This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ) custom_introc$eZdZfdZdZxZS)TrOCRDecoderWrappercDt||t||_yrC)rr rdecoderr!rr"s r#r zTrOCRDecoderWrapper.__init__s  #F+ r$c&|j|i|SrC)r)r!argskwargss r#r3zTrOCRDecoderWrapper.forwardst||T,V,,r$)r6r7r8r r3r<r=s@r#rrs ,-r$rzy The TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and c"eZdZdgZfdZdZdZdZdZdZ dZ e dd e e jd e e jd e e j d e e jd e e jde e jde eee j de e j de e jde ede ede ede ede e jdeeeffdZxZS)TrOCRForCausalLMzoutput_projection.weightcd|_d|_t| |t ||_t j|j|jd|_ |jy)NTFr{) rvis_encoder_decoderrr rrrrrroutput_projectionrrs r#r zTrOCRForCausalLM.__init__sZ $)!  (0 !#6+=+=v?P?PW\!] r$cB|jjjSrCrrrr!s r#get_input_embeddingsz%TrOCRForCausalLM.get_input_embeddingsszz!!...r$c:||jj_yrCr()r!values r#set_input_embeddingsz%TrOCRForCausalLM.set_input_embeddingss*/ 'r$c|jSrCr&r)s r#get_output_embeddingsz&TrOCRForCausalLM.get_output_embeddingss%%%r$c||_yrCr/)r!new_embeddingss r#set_output_embeddingsz&TrOCRForCausalLM.set_output_embeddingss !/r$c&||j_yrCrr)r!rs r# set_decoderzTrOCRForCausalLM.set_decoders$ r$c.|jjSrCr5r)s r# get_decoderzTrOCRForCausalLM.get_decoderszz!!!r$r%rrrrrrr labelsrrrr rrcF| | n|jj} | | n|jj} | | n|jj} |jj ||||||||| | | | | }|j |d}d}| Ft}||jd|jj| jd}| s|f|ddz}||f|zS|St|||j|j|j|jS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import requests >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r%rrrrrrr rrrr rrr+r)losslogitsrrrr)rrrrrrr&rrZrrrrrr)r!r%rrrrrrr r9rrrr rrr<r;loss_fctoutputs r#r3zTrOCRForCausalLM.forwardsKZ2C1N-TXT_T_TqTq$8$D $++JjJj &1%u $D>u 'tnu d^u !.u u77 8!u u r$r#)1r9rStypingrrr-rtorch.nnr activationsr cache_utilsr r r generationr modeling_attn_mask_utilsr rmodeling_layersrmodeling_outputsrrmodeling_utilsrutilsrrutils.deprecationrconfiguration_trocrr get_loggerr6rrrr?ModulerHrprrrrr#__all__rr$r#rPs06 " %!CC):l-,0,   H %;bll;4 =r|| =;8;8|W2RYYW2tw2wt????$x 'x v-.-- V +_V  V r 5 6r$