h-:ddlZddlmZddlZddlmZmZddlmZddlm Z eje Z GddejZGd d ejZGd d ejZGd dejZGddejZGddejZGddejZGddejZGddejZGddejZGddejZGddeZGdd ejZid!ed"ed#d$d%fd&ed'ed(ed)d*ifd+ed,ed-ed.ej6d/ed0ed1ed2ej8d3ed4ej:d5ej<d6ej>ej>ej@ejBed7Z"ee"Z#d8Z$e$d(Z%e$d'Z&e$d!Z'e$d&Z(e$d1Z)e$d6Z*e$d0Z+e$d/Z,y)9N) OrderedDict)Tensornn)logging)is_torchdynamo_compilingc eZdZdZdedefdZy)PytorchGELUTanha& A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. inputreturncDtjj|dS)Ntanh) approximate)r functionalgeluselfr s V/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/transformers/activations.pyforwardzPytorchGELUTanh.forward%s}}!!%V!<<N__name__ __module__ __qualname____doc__rrrrr r s=V==rr c eZdZdZdedefdZy)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 r r c d|zdtjtjdtjz |dtj |dzzzzzS)N??@Hm?g@)torchrmathsqrtpipowrs rrzNewGELUActivation.forward/sPU{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrNrrrrrr)s wVwwrrcJeZdZdZddeffd ZdedefdZdedefdZxZ S) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 use_gelu_pythonct||r|j|_ytj j |_yN)super__init__ _gelu_pythonactrrr)rr+ __class__s rr/zGELUActivation.__init__;s/  ((DH}}))DHrr r cj|dzdtj|tjdz zzS)Nr r!r")r$erfr%r&rs rr0zGELUActivation._gelu_pythonBs,s{cEIIediin.D$EEFFrc$|j|Sr-r1rs rrzGELUActivation.forwardExxr)F) rrrrboolr/rr0r __classcell__r2s@rr*r*3s=**G&GVGVrr*c eZdZdZdedefdZy)FastGELUActivationz} Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs r r c\d|zdtj|dzdd|z|zzzzzS)Nr r!g3E?r#)r$rrs rrzFastGELUActivation.forwardNs:U{cEJJu|/CsXX]M]`eMeGe/f$gghhrNrrrrr<r<IsiViirr<c eZdZdZdedefdZy)QuickGELUActivationzr Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs r r c8|tjd|zzS)NgZd;?)r$sigmoidrs rrzQuickGELUActivation.forwardWsu}}UU]333rNrrrrr?r?Rs4V44rr?c<eZdZdZdedeffd ZdedefdZxZS)ClippedGELUActivationa Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 minmaxcl||kDrtd|d|dt| ||_||_y)Nzmin should be < max (got min: z, max: )) ValueErrorr.r/rDrE)rrDrEr2s rr/zClippedGELUActivation.__init__hs> 9=cU'#aPQ Q rxr cjtjt||j|jSr-)r$cliprrDrE)rrIs rrzClippedGELUActivation.forwardps!zz$q'488TXX66r) rrrrfloatr/rrr9r:s@rrCrC[s. E77F7rrCc2eZdZdZfdZdedefdZxZS)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) cxt|tjdtjz |_y)N)r.r/r%r&r'precomputed_constantrr2s rr/zAccurateGELUActivation.__init__|s' $(IIa$''k$:!rr r c d|zdtj|j|dtj|dzzzzzS)Nr rr#)r$rrQr(rs rrzAccurateGELUActivation.forwardsEU{a%**T-F-F%RZ]b]f]fglno]pRpJp-q"rrssr)rrrrr/rrr9r:s@rrNrNts#;tVttrrNcBeZdZdZfdZdedefdZdedefdZxZS)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish c`t|tjj|_yr-)r.r/rrmishr1rRs rr/zMishActivation.__init__s ==%%rr r cl|tjtjj |zSr-)r$rrrsoftplusrs r _mish_pythonzMishActivation._mish_pythons%uzz"--"8"8"?@@@rc$|j|Sr-r6rs rrzMishActivation.forwardr7r) rrrrr/rr[rr9r:s@rrVrVs6 &A&AVAVrrVc eZdZdZdedefdZy)LinearActivationz[ Applies the linear activation function, i.e. forwarding input directly to output. r r c|Sr-rrs rrzLinearActivation.forwards rNrrrrr^r^sVrr^ceZdZdZddZy)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability c||z j|tjdz}ddtj|zzS)Nr"r r!)divr%r&r$r4)rr musigmas rrzLaplaceActivation.forwards<  3!78cEIIe,,--rN)g۞?g ^/ ?rrrrrrrrraras .rraceZdZdZdZy)ReLUSquaredActivationz` Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668v2 cntjj|}tj|}|Sr-)rrrelur$square)rr relu_appliedsquareds rrzReLUSquaredActivation.forwards)}}))%0 ,,|,rNrfrrrrhrhs rrhceZdZfdZxZS)ClassInstantiercdt||}t|tr|n|if\}}|di|S)Nr)r. __getitem__ isinstancetuple)rkeycontentclskwargsr2s rrqzClassInstantier.__getitem__s6'%c*!+GU!;g'2 V}V}r)rrrrqr9r:s@rroros rrocteZdZdZddddej dffd Zdedefd Zdedefd Z d edefd Z xZ S) XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?r gưFc ft |tjt j t j t j||dz jd|_ tjt j t j t j||z |dz jd|_ |jdt j|||jdt j||||_ t|jjj!jj#|_t|j&jj!jj#|_d|_ ddl}tj.j0j3|_d} ddlm} | |j8|_|dz }t>jA|y#t<$r$} |d | d z }|j8|_Yd} ~ >d} ~ wwxYw#t<$r)} t>jAd tC| Yd} ~ yd} ~ wwxYw) N)dtyperrbetaepszUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.uCUDA-fused xIELU not available (%s) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)"r.r/r Parameterr$logexptensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsrLr|detachcpuitem _beta_scalarr} _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELU torch._dynamor~ _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_oncestr) r alpha_p_init alpha_n_initr|r}r{rrmsgr~errr2s rr/zXIELUActivation.__init__s ||EIIeii \Y^8_.`cd.d$e$o$opq$rs || IIeii \D-@ NORSS T ^ ^_` a   VU\\$e%DE UELLE$BC!2!$))"2"2"4"8"8":"@"@"B"G"G"IJ !2!6!6!8!>!>!@!E!E!GH#  #(==#6#6#<#<#>D 2C 78&4T5E5E&F#??    $ 7DSEIstt&*&6&6## 7    jC   sB"3I>"I8I> I;I61I>6I;;I>> J0J++J0rIr c tjj|j}|jtjj|j z}t j|dkD||z|z|j|zzt jt j||j|z |z|j|zzS)Nr) rrrZrr|rr$whereexpm1rDr})rrIrrs r _xielu_pythonzXIELUActivation._xielu_pythons--((6))bmm44T\\BB{{ E aK!Odii!m + [[1dhh/ 01 4 ?$))a- O  rc|j}|jdkr%|jd}|jdkr%|jdkDr"|jdd|j d}||jk7r!t j d||j|jj||j|j|j|j|j}|j|S)zDFirewall function to prevent torch.compile from seeing .item() callsrTrrz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimrviewsizerrrrrrrrr)rrIoriginal_shaperesults rrzXIELUActivation._xielu_cudaseegk AAeegk 557Q;r1affRj)A QWW $   q  %%-- LL LL        " " {{>**rr c|j<|jr0ts|j|Stj d|j |S)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudarrrrrrs rrzXIELUActivation.forwardsK    + +-**511##$`a!!%((r) rrrrr$bfloat16r/rrrrr9r:s@rryrys_  nn)V v & +V++2)V))rryrgelu_10i )rDrE gelu_fastgelu_new gelu_pythonr+Tgelu_pytorch_tanh gelu_accuratelaplace leaky_relulinearrX quick_gelurjrelu2relu6rAsilu)swishrprelurc ||tvr t|Std|dttj)Nz function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings rget_activationr5sBF"'((#4"55RSWX^XcXcXeSfRghiir)-r% collectionsrr$rrutilsrutils.import_utilsr get_loggerrrModuler rr*r<r?rCrNrVr^rarhrory LeakyReLUReLUReLU6SigmoidSiLUTanhPReLUACT2CLSrrrrrrrrrX linear_actrrrrs # 8   H % =bii =w wRYY,ii4"))47BII72 tRYY t RYY"ryy . .BIIk[)bii[)|  N  %s2'>? # !   N%6$=>     +   ",,    N %  BGG  "  RXX  rzz! " BGG# $WW GG XX + .  !j]+ * %f ; ' L ) ff H % r