hddlZddlmZmZmZmZmZmZddlm Z ddl m Z m Z ejeZGdde Zy)N)AnyCallableDictIteratorListOptional)Document) BaseModelmodel_validatorceZdZUdZdZeed<dZeed<dZe ed<dZ e e e gefed <eed <ed ed e defdZdeefdZdeefdZy)TensorflowDatasetsavAccess to the TensorFlow Datasets. The Current implementation can work only with datasets that fit in a memory. `TensorFlow Datasets` is a collection of datasets ready to use, with TensorFlow or other Python ML frameworks, such as Jax. All datasets are exposed as `tf.data.Datasets`. To get started see the Guide: https://www.tensorflow.org/datasets/overview and the list of datasets: https://www.tensorflow.org/datasets/catalog/ overview#all_datasets You have to provide the sample_to_document_function: a function that a sample from the dataset-specific format to the Document. Attributes: dataset_name: the name of the dataset to load split_name: the name of the split to load. Defaults to "train". load_max_docs: a limit to the number of loaded documents. Defaults to 100. sample_to_document_function: a function that converts a dataset sample to a Document Example: .. code-block:: python from langchain_community.utilities import TensorflowDatasets def mlqaen_example_to_document(example: dict) -> Document: return Document( page_content=decode_to_str(example["context"]), metadata={ "id": decode_to_str(example["id"]), "title": decode_to_str(example["title"]), "question": decode_to_str(example["question"]), "answer": decode_to_str(example["answers"]["text"][0]), }, ) tsds_client = TensorflowDatasets( dataset_name="mlqa/en", split_name="train", load_max_docs=MAX_DOCS, sample_to_document_function=mlqaen_example_to_document, ) dataset_nametrain split_named load_max_docsNsample_to_document_functiondatasetbefore)modevaluesreturnc ddl} ddl}|d td|j |d|d |d <|S#t$r tdwxYw#t$r tdwxYw) z7Validate that the python package exists in environment.rNz\Could not import tensorflow python package. Please install it with `pip install tensorflow`.znCould not import tensorflow_datasets python package. Please install it with `pip install tensorflow-datasets`.rzmsample_to_document_function is None. Please provide a function that converts a dataset sample to a Document.rr)splitr) tensorflow ImportErrortensorflow_datasets ValueErrorload)clsrrrs o/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/utilities/tensorflow_datasets.pyvalidate_environmentz'TensorflowDatasets.validate_environment?s     & / 0 8   044 > "&*>5 y - C   L  s9AAA&c`fdjjjDS)zYDownload a selected dataset lazily. Returns: an iterator of Documents. c3XK|]!}jj|#yw)N)r).0sselfs r" z/TensorflowDatasets.lazy_load..cs0 //;  , ,Q / s'*)rtakerr(s`r" lazy_loadzTensorflowDatasets.lazy_load]s*  \\&&t'9'9:  c4t|jS)zMDownload a selected dataset. Returns: a list of Documents. )listr,r+s r"r zTensorflowDatasets.loadis DNN$%%r-)__name__ __module__ __qualname____doc__rstr__annotations__rrintrrrrr rr classmethodr#rr,rr r-r"r r s,\L#JM3HL(D683C*D!EL L(#$3$8  8H-  &d8n&r-r )loggingtypingrrrrrrlangchain_core.documentsr pydanticr r getLoggerr0loggerr r8r-r"r?s4@@-/   8 $e&e&r-