hj"rddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZGddeZy) N)PathLike)Path)AnyCallableDictIteratorOptionalUnion)Document) BaseLoaderceZdZdZ ddeeefdedeedeedee e e ge fded efd Z d e e fd Zd eded e e fdZded efdZde eefded e eeffdZded dfdZy) JSONLoadera Load a `JSON` file using a `jq` schema. Setup: .. code-block:: bash pip install -U jq Instantiate: .. code-block:: python from langchain_community.document_loaders import JSONLoader import json from pathlib import Path file_path='./sample_quiz.json' data = json.loads(Path(file_path).read_text()) loader = JSONLoader( file_path=file_path, jq_schema='.quiz', text_content=False) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quizg .json', 'seq_num': 1} Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python {"sport": {"q1": {"question": "Which one is correct team name in NBA?", "options": ["New York Bulls" {'source': '/sample_quiz .json', 'seq_num': 1} N file_path jq_schema content_keyis_content_key_jq_parsable metadata_func text_content json_linesc ddl}||_t|j|_|j ||_||_||_||_ ||_ ||_ y#t$r tdwxYw)a~Initialize the JSONLoader. Args: file_path (Union[str, PathLike]): The path to the JSON or JSON Lines file. jq_schema (str): The jq schema to use to extract the data or text from the JSON. content_key (str): The key to use to extract the content from the JSON if the jq_schema results to a list of objects (dict). If is_content_key_jq_parsable is True, this has to be a jq compatible schema. If is_content_key_jq_parsable is False, this should be a simple string key. is_content_key_jq_parsable (bool): A flag to determine if content_key is parsable by jq or not. If True, content_key is treated as a jq schema and compiled accordingly. If False or if content_key is None, content_key is used as a simple string. Default is False. metadata_func (Callable[Dict, Dict]): A function that takes in the JSON object extracted by the jq_schema and the default metadata and returns a dict of the updated metadata. text_content (bool): Boolean flag to indicate whether the content is in string format, default to True. json_lines (bool): Boolean flag to indicate whether the input is in JSON Lines format. rNz=jq package not found, please install it with `pip install jq`) jq ImportErrorrresolvercompile _jq_schema_is_content_key_jq_parsable _content_key_metadata_func _text_content _json_lines) selfrrrrrrrrs n/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_community/document_loaders/json_loader.py__init__zJSONLoader.__init__TsD  DG i002**Y/+E('+)% O  s A%%A:returnc#vKd}|jra|jjd5}|D]5}|j}|s|j ||D] }||dz } 7 dddy|j |jj d|D] }||dz } y#1swYyxYww)z-Load and return documents from the JSON file.rz utf-8-sig)encodingN)r ropenstrip_parse read_text)r!indexflinedocs r" lazy_loadzJSONLoader.lazy_loads   $$k$: 'a'D::      ' 's#+B9B-"B-)AB9-B62B9contentr,c#`K|jjtj|}|j|j |t ||dzD]M\}}|j|}|j|t|j|}t||Oyw)z#Convert given content to documents.Nr')sample)r3sourceseq_num) page_contentmetadata) rinputjsonloadsr_validate_content_key enumerate _get_text _get_metadatastrrr )r!r1r,datair3textr7s r"r*zJSONLoader._parses$$TZZ%89    (  & &t ,"43 AIAv>>>0D))c$..&91*Hx@ @  AsB,B.r3c|ja|jrE|jj|j}|j |j }n||j}n|}|j r*t|ts|tdt|dt|tr|St|ttfr|rtj|SdS| t|SdS)zConvert sample to string formatz%Expected page_content is string, got z instead. Set `text_content=False` if the desired input for `page_content` is not a string)rrrrr8firstr isinstancer? ValueErrortypedictlistr9dumps)r!r3compiled_content_keyr1s r"r=zJSONLoader._get_texts    (//'+wwt7H7H'I$.44V<BBD !2!23G   j#&>7CV7W G34  %N $ .*14::g& 9r 9#*#63w< >B >additional_fieldsc |j<|j||}t|tstdt |d|S|S)z Return a metadata dictionary base on the existence of metadata_func :param sample: single data payload :param additional_fields: key-word arguments to be added as metadata values :return: zUExpected the metadata_func to return a dict but got ``)rrFrIrGrH)r!r3rNresults r"r>zJSONLoader._get_metadatas\    *((1BCFfd+ ""&v,q2M$ $rMr@c|j}t|tstdt |d|j s4|j |jtd|jd|j r\|jj|jj|jtd|jdyy)zCheck if a content key is validztExpected the jq schema to result in a list of objects (dict), so sample must be a dict but got `rPNz_Expected the jq schema to result in a list of objects (dict) with the key `z ` which should be parsable by jq) rErFrIrGrHrgetrrrr8rB)r!r@r3s r"r;z JSONLoader._validate_content_keys&$'77;F|nAG  00 4,,-5##'#4#4"5Q8   , , 1 1288@EEGO##'#4#4"55UW P -rM)NFNTF)__name__ __module__ __qualname____doc__r r?rr boolrrr#rr r0intr*rr=r>r;rMr"rr sFX&*5:@D! 1&h'1&1&c] 1& %-TN 1&  $t); <= 1&1&1&f8H-$AcA#A(82DA"???4%38n%;>% c3h%(#$rMr)r9osrpathlibrtypingrrrrr r langchain_core.documentsr )langchain_community.document_loaders.baser rrZrMr"r`s( AA-@ffrM