h"vddlZddlmZmZmZmZddlmZddlm Z ddl m Z ddl m Z dZdZd ZGd d e Zy) N)AnyListMappingOptional)CallbackManagerForLLMRun)LLM) ConfigDict)enforce_stop_tokenszgoogle/flan-t5-largetext2text-generation)r text-generation summarizationcPeZdZUdZdZeed<eZe ed< dZ e e ed< dZ e e ed< edZe dde d e d e ed e e de e de e d e ed e ede ededefdZedee effdZede fdZ dde de ee de edede f dZy)WeightOnlyQuantPipelineaWeight only quantized model. To use, you should have the `intel-extension-for-transformers` packabge and `transformers` package installed. intel-extension-for-transformers: https://github.com/intel/intel-extension-for-transformers Example using from_model_id: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) config = WeightOnlyQuantConfig hf = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation" pipeline_kwargs={"max_new_tokens": 10}, quantization_config=config, ) Example passing pipeline in directly: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( AutoModelForSeq2SeqLM ) from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) from transformers import AutoTokenizer, pipeline model_id = "google/flan-t5-large" tokenizer = AutoTokenizer.from_pretrained(model_id) config = WeightOnlyQuantConfig model = AutoModelForSeq2SeqLM.from_pretrained( model_id, quantization_config=config, ) pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10, ) hf = WeightOnlyQuantPipeline(pipeline=pipe) Npipelinemodel_id model_kwargspipeline_kwargsallow)extrataskdevice device_map load_in_4bit load_in_8bitquantization_configkwargsreturnc | t|tr|dkDr tdtjj d td ddlm} m} ddl m } dd l m }dd l m }t|tr&|dk\r!| s td d t|z}nt|tr|dkrd}||d}|xsi}|j |fi|} |dk(r| j |f||| d|d|}n4|dvr| j |f||| d|d|}ntd|dt"dd|vr)|j%Dcic]\}}|dk7s ||}}}|xsi}|d|||||d|}|j&t"vr td|j&dt"d|d||||d| S#t$r td wxYw#t$r}td|d|d}~wwxYwcc}}w)z5Construct the pipeline object from model_id and task.Nz7`Device` and `device_map` cannot be set simultaneously!torchz;Weight only quantization pipeline only support PyTorch now!r)AutoModelForCausalLMAutoModelForSeq2SeqLM)is_ipex_available) AutoTokenizer)rzCould not import transformers python package. Please install it with `pip install transformers` and `pip install intel-extension-for-transformers`.z)Don't find out Intel GPU on this machine!zxpu:cpur F)rrruse_llm_runtimer)r r Got invalid task , currently only  are supportedzCould not load the z# model due to missing dependencies.trust_remote_code)rmodel tokenizerrr)rrrr) isinstanceint ValueError importlibutil find_spec-intel_extension_for_transformers.transformersr!r",intel_extension_for_transformers.utils.utilsr# transformersr$r ImportErrorstrfrom_pretrained VALID_TASKSitemsr)clsrrrrrrrrrrr!r"r#r$ hf_pipeline _model_kwargsr,r+ekv_pipeline_kwargsrs o/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/llms/weight_only_quantization.py from_model_idz%WeightOnlyQuantPipeline.from_model_idOs  !z&#'>6B;VW W >> # #G , 4M    W 2 < fc "v{$& !LMM#f+-J  $!F >!" $* 1M11(LmL  ((<,<<!-!-(;$)) $BB=-==!-!-(;$)) $!'v.&&1].B - /!.!4!4!6A!?R:R1M+0b &     == +#HMM?3""-n>  &,     K F  \ %dV+NO   s1F(AG? G! G!(F= G GGcJ|j|j|jdS)zGet the identifying parameters.rrrrFselfs rC_identifying_paramsz+WeightOnlyQuantPipeline._identifying_paramss'  --#33  cy)zReturn type of llm.weight_only_quantizationr-rGs rC _llm_typez!WeightOnlyQuantPipeline._llm_types*rJpromptstop run_managerc ||j|}|jjdk(r|ddt|d}nn|jjdk(r |dd}nL|jjdk(r |dd}n*td|jjd td |r t ||}|S) abCall the HuggingFace model and return the output. Args: prompt: The prompt to use for generation. stop: A list of strings to stop generation when encountered. Returns: The generated text. Example: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline llm = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation", ) llm.invoke("This is a prompt.") r rgenerated_textNr r summary_textr'r(r))rrlenr0r:r )rHrNrOrPrresponsetexts rC_callzWeightOnlyQuantPipeline._calls4==( ==  !2 2A;/0V?D ]]  #9 9A;/0D ]]  ? 2A;~.D#DMM$6$6#78""-n>  'tT2D rJ)rNNNFFN)NN)__name__ __module__ __qualname____doc__rr__annotations__DEFAULT_MODEL_IDrr8rrdictrr model_config classmethodr/boolrrDpropertyrrIrMrrrWr-rJrCrrs/bHc$Hc$*#'L(4.'1&*OXd^*4L !#$('+*.',',-1h h h  h SM h tn h "$h tnh tnh &c]h h  h h T WS#X%6  *3**%):> ++tCy!+67 +  + +rJr)r1typingrrrr langchain_core.callbacks.managerr#langchain_core.language_models.llmsrpydanticr langchain_community.llms.utilsr r] DEFAULT_TASKr:rr-rJrCris8//E3>)% J dcdrJ