hZu(4ddlZddlZddlmZddlmZmZddlZddlm cm Z ddl m Z mZmZmZmZej$eZdZdZdadadadadadd d Zd eefd Zd Zd eefdZ dZ!d:dZ"dZ#dejHde%ejHejHe&ffdZ'dejHdejHdejHdejHde&f dZ(dZ)dZ*dZ+ d:dejHdejHdejHd eejXfd!Z-Gd"d#ed$%Z. d;de&d&e&d'e/d(e0d)ee0d*ee&d+e/d,ee0d-ee/d.eejHd/ee1ee/ffd0Z2 dTY_hnFBB  ! ]j 0 0^5KPV T T&n6GNO%,^=UW[%\ "%-1H `ao`pqst 2I{ JJrctj|j}tjtj}i}|D]}tj ||}||v||<!t t|S)a Depending on the version and kernel some features are not supported. Due to limitations in `torch.compile`, we opt to statically type which (optional) kwarg parameters are supported within `_process_flash_attention_kwargs`. NOTE: While all supported kwargs are marked as `True`, everything else is marked as `False`. This might be confusing for kwargs that we use in any case, e.g. `is_causal`. )supports_mapping)inspect signature parameters_process_flash_attention_kwargs_hf_api_to_flash_mappinggetr)flash_functionflash_parametersprocess_parametersr0paramfa_params r_lazy_define_process_functionr<js~((8CC **+JKVV#B+//u=%-1A%A"B 2EU VVrctdttttfDrt |\aaaat ttattttft fS)a Lazily import flash attention and return the respective functions + flags. NOTE: For fullgraph, this needs to be called before compile, while no fullgraph can work without preloading. See `load_and_register_kernel` in `integrations.hub_kernels`. c3$K|]}|du ywrr).0ks r z.lazy_import_flash_attention..s P19 P)any _flash_fn_flash_varlen_fn_pad_fn _unpad_fnr._process_flash_kwargs_fnr<)rs rlazy_import_flash_attentionrIsZ Py*:GYO PP:G:W7 #Wi '#@AQ#R ') <>V VVrcL|jdg|jdd}||S)a A local implementation of the PyTorch indexing operation `tensor[indices]` on the first axis, after flattening the first two dimensions of the tensor. This is functionally equivalent to FA2's `index_first_axis` and replaces the need to import it. N)reshapeshape)tensorindicesreshaped_tensors r_index_first_axisrRs/%fnnR;&,,qr*:;O 7 ##rc|||zn|}|jdtj}|jdtj}tj|j dj }|j j }tjtj|dtjd}t||||||fS)a4 unpad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (batch, seqlen, ...) attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid. unused_mask: (batch, seqlen), bool / int, 1 means the element is allocated but unused. Return: hidden_states: (total_nnz, ...), where total_nnz = number of tokens selected in attention_mask + unused_mask. indices: (total_nnz), the indices of masked tokens from the flattened input sequence. cu_seqlens: (batch + 1), the cumulative sequence lengths, used to index into hidden_states. max_seqlen_in_batch: int seqused: (batch), returns the number of tokens selected in attention_mask + unused_mask. rKdimdtypeFas_tuplerrr) sumtorchint32nonzeroflattenmaxitemFpadcumsumrR) hidden_statesattention_mask unused_mask all_masksseqlens_in_batchused_seqlens_in_batchrPmax_seqlen_in_batch cu_seqlenss rr&r&s 3>2I+-~I }}5;;}?*..2U[[.ImmI--/%@HHJG*..0557u||$4!5;;OQWXJ -1  rc|jdd}tj||zg||j|jd}|||<|j ||g|S)a pad_input function for flash attention variants that do not have them within their pkg themselves, e.g. fa3. Arguments: hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask. indices: (total_nnz), the indices that represent the non-masked tokens of the original padded input sequence. batch: int, batch size for the padded sequence. seqlen: int, maximum sequence length for the padded sequence. Return: hidden_states: (batch, seqlen, ...) rN)devicerV)rNr[zerosrmrVview)rdrPbatchseqlenrUoutputs rr%r%sb   ab !C [[%&. hC h 8L8LTaTgTg hF#F7O 6;;uf +s ++rrereturncd|jdtj}tj|j dj }|j j }tjtj|dtjd}|||fS)aq Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. rKrTFrWrrY) rZr[r\r]r^r_r`rarbrc)rerhrPrjrks r_get_unpad_datarus &))b )DmmN224uEMMOG+..0557u||$4!5;;OQWXJ r query_layer key_layer value_layer query_lengthct|\}}}|jd|jdx} kDr"|ddd| ddddf|ddd| ddddf}}|j\} } } } t||}t||}|| k(rt||}|}|}|}nk|dk(rLd}tj| dztj |j }|dd}|jd}n|dd| df}|||^}}}}}||||||f||ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. unpad_input_func: The function to use for unpadding the input tensors. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rrKNrVrm)rurNrRr[aranger\rmsqueeze)rvrwrxreryunpad_input_func indices_k cu_seqlens_kmax_seqlen_in_batch_kseq_len batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_q_s r _upad_inputrs`R6E^5T2I|2q(<((9:JZ[fhvJwG Y .Ca  |$  56  rctj|jd}|jd}|dk(j jd}tj |j di|tj|jfi|f}|}|jj}|j}|}||f||ffS)a This function returns all the necessary kwargs to call `flash_attn_varlen_func` extracted from position_ids. Arguments: position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). r{rKrr) r[r\rmror]cattorOsizediffr_r`) position_ids tensor_kwargsr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_ks r#prepare_fa_kwargs_from_position_idsr<s $kk\5H5HIM$$R(L"++-2226III ILL )= ) LL**, > > M "M !%%'++-L  $$&LL = )L,+G GGrc|jjd|jd|jd}|jjd|jd|jd}|jjd|jd|jd}t|\\}}\}}|||||f||ffS)a This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cumulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). (cu_seqlens_q, cu_seqlens_k) (`tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rK) contiguousrorr)querykeyvaluerrrrrs r_prepare_from_posidsrhs:     # #B 2 2 GE ..   CHHRL#((2, ?C     # #B 2 2 GECfgsCt@"]M$@\< 3 }= l?[ \\rc|ytj|jd|j|j z}|dk(xr/||z j j jS)a Check the position ids whether packed sequences are indicated or not 1. Position ids exist 2. Flattened sequences only are supported 3. Compile-friendly `not (torch.diff(position_ids, dim=-1) >= 0).all()`, i.e. we have multiple increasing sequences Fr)rm)r[r|rNrmminabsrZbool)rrincreasing_position_sequencess r_is_packed_sequencerss  \''*<3F3FG,JZJZJ\\" ? ` = LQQSWWY^^``rqr@v target_dtypec|ri|jtjk(rLtj d|d|j ||j ||j |}}}|||fS)aa PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms! zCasting fp32 inputs back to z for flash-attn compatibility.)rVr[float32logger warning_oncer)rr@rrs rfa_peft_integration_checkrsb5==0:<.Hfgh$$|$add<&8!$$|:La1 a7NrczeZdZUdZeej ed<eej ed<eeed<eeed<y)FlashAttentionKwargsa Keyword arguments for Flash Attention with Compile. Attributes: cu_seq_lens_q (`torch.LongTensor`, *optional*) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`, *optional*) Gets cumulative sequence length for key state. max_length_q (`int`, *optional*): Maximum sequence length for query state. max_length_k (`int`, *optional*): Maximum sequence length for key state. rrrrN) __name__ __module__ __qualname____doc__rr[ LongTensor__annotations__intrrrrrs? E,,--E,,--3-3-rrF)total key_length is_causalr softmax_scaleruse_top_left_masksoftcap deterministics_auxr0c  |xr |xr|dk( |d} | dr|| d<| dr|||kDr |dz |dz f| d<| dr ||ntjdddk(| d<| d r||| d <| d r| | | d <| S) aZ Returns a set of kwargs that are passed down to the according flash attention function based on requested features and whether it is supported - depends on the version and kernel implementation which is dynamically configured at `lazy_import_flash_attention`. The (un)supported features can be inspected in `supports_mapping`, see `_lazy_define_process_function` for more details. Args: query_length (`int`): Length of the query states key_length (`int`): Length of the key states is_causal (`bool`): Whether we perform causal (decoder) attention or full attention. dropout (`float`): Attention dropout. softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to `1 / sqrt(head_dim)`. sliding_window (`int`, *optional*): The size of the sliding window, i.e. we look at a max of `sliding_window` tokens back. use_top_left_mask (`bool`): Deprecated behavior of older versions of flash attention requiring different masking. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. s_aux (`torch.Tensor`, *optional*): Attention sink auxiliary that adds a `bias` to the attention calculation via an additional head. Return: flash_kwargs (`dict`): A dict of kwargs that are requested and supported. r)causalrrrrFLASH_ATTENTION_DETERMINISTIC01rr)osgetenv) ryrrrrrrrrrr0kwargs flash_kwargss rr4r4s\M%6%L<1;L M&L  $$+ [! &>+E*WeJe (6'9>A;M&N ]#(*6MBIIFegj