Recent chats
ASKGEEZ Q&A GUIDETechnology & devices

How can I combine small JSON arrays from several files?

Quick answer

List the input files you intend to combine, load each JSON document, and check that it contains an array. Then validate each entry against the fields and types your application expects before adding it to a combined list. Save that list to a different output file. Valid JSON syntax alone does not show that records from different files have compatible fields or meanings.

LET’S WORK THROUGH IT

The question, the details, the next step.

Question

I have a few small JSON files, each supposed to contain a list of records. What is a cautious way to combine just the files I choose?

Geez

Use an explicit list of paths rather than automatically collecting every file in a folder. Load each file, check that its top-level value is a list, and validate the entries before extending your combined list. In Python, the standard JSON decoder turns arrays into lists and objects into dictionaries; json.load reads a document from an open file. Those conversions do not determine whether records from different files mean the same thing or have compatible fields. [S1]

Here is a small example. The field names and types are illustrative: change id and label to match the records your own application expects.

import json
from pathlib import Path

input_paths = [
    Path("part_a.json"),
    Path("part_b.json"),
]
output_path = Path("combined.json")

combined = []

for path in input_paths:
    with path.open(encoding="utf-8") as file:
        data = json.load(file)

    if not isinstance(data, list):
        raise ValueError(f"{path} must contain a JSON array")

    for index, item in enumerate(data):
        if not isinstance(item, dict):
            raise ValueError(f"{path}: entry {index} must be an object")
        if not isinstance(item.get("id"), str):
            raise ValueError(f"{path}: entry {index} needs a string id")
        if not isinstance(item.get("label"), str):
            raise ValueError(f"{path}: entry {index} needs a string label")

    combined.extend(data)

with output_path.open("w", encoding="utf-8") as file:
    json.dump(combined, file, indent=2)
    file.write("\n")

The output path is separate from the inputs, so this example does not overwrite them. json.dump serializes a Python value to a writable file; write the combined list once as one JSON document. [S1]

Follow-up question

Why check each record if the files all parse as JSON?

Geez

Parsing checks whether a file can be read as JSON; it does not check whether its records suit your application. For example, one file might contain objects with id and label, while another contains objects with id and description. Both could be valid JSON arrays, yet downstream code expecting label may not handle the second set as intended. The JSON parser does not supply application-specific schema rules. [S1]

Decide what counts as a valid entry before combining. Check required fields and their types, and add any other rules your task needs. If a field is optional, make that explicit in your checks rather than treating a missing value as if it were valid by default. The sample checks only two illustrative fields; it does not establish that those fields are right for your data.

Follow-up question

What if one file has the right top-level array but a bad entry partway through?

Geez

The example raises an error and stops instead of quietly including an entry that fails its checks. Its message identifies the file and entry index, which gives you a starting point for reviewing the input. Decide whether you want the whole combine operation to stop on the first problem, as shown, or to collect errors for a later review. If you choose the latter, keep invalid entries out of the output until you have decided how to handle them.

For a first run, you can also print or inspect the explicit input list and output path before running the script. This helps catch a mistaken filename or an output path that is not actually separate from the inputs.

Follow-up question

How do I make sure I haven't missed a practical step?

Geez

Use this short review before combining:

  1. Write down the exact input paths; do not include files just because they are nearby.
  2. Choose the expected entry shape, including required fields and types.
  3. Check every file is an array and every entry meets that shape.
  4. Choose a distinct output path and confirm it is not one of the inputs.
  5. Review the combined file as one JSON document and check the records make sense for your task.

The script checks syntax and the specific rules you add. It does not establish that two fields with the same name have the same meaning, detect every application-specific inconsistency, or decide whether duplicate records should be kept. Those are decisions for your data and use case.

A practical checklist

  • List only the input paths you intend to combine.

  • Define required fields and expected types for each entry.

  • Check that every file contains an array and every entry passes your checks.

  • Write the combined list to a distinct output file.

  • Review the result for application-specific meaning and duplicate handling.

A few more useful answers

Does checking the fields prove the records are compatible?

No. It checks only the rules you wrote, such as required field names and types. It does not prove that fields have the same meaning across files.

Can I write each input array to the output file one after another?

Do not treat repeated JSON dumps as one combined document: repeated dumps to the same file do not form a valid sequence of JSON documents. Combine the values into a list and serialize that list once. [S1]

What should I do about duplicate records?

Choose a rule that fits your task, such as keeping all entries or identifying duplicates by a specific field. The example does not remove duplicates.

Sources

References used for this guide.

  1. [S1] Python 3.15 json reference: JSON-to-Python conversion and file serialization
    docs.python.org · Checked

YOUR QUESTION. YOUR DETAILS.

Let’s work through your situation.

Get a tailored answer or help with your next task. Starts at $1, with a quote before you pay.

Ask your own question
← Back to Geez at Work