How to Merge Multiple JSON Files Using Python

We are given multiple JSON files and our task is to merge those multiple JSON files into a single JSON file with the help of different approaches in Python. In this article, we will see how we can merge multiple JSON files in Python.

JSON Files

Below are the two JSON files that we will use in our article to merge them into a single JSON file.

f.json

{"key1":"value1"}

s.json

{"key2": "value2"}

Merging Multiple JSON Files in Python

Below are some of the ways by which we can merge multiple JSON files in Python:

1. Using json Module
2. Using List Comprehension
3. Using os Module with json Module
4. Using glob Module with json Module
5. Using Pandas Library

Merge Multiple JSON Files Using json Module

In this example, a Python function merge_json_files is defined to combine data from multiple JSON files specified by file_paths into a list called merged_data. The merged data is then written to a new JSON file named "merged.json," and a confirmation message is printed.

import json


def merge_json_files(file_paths, output_file):
    merged_data = []
    for path in file_paths:
        with open(path, 'r') as file:
            data = json.load(file)
            merged_data.append(data)
    with open(output_file, 'w') as outfile:
        json.dump(merged_data, outfile)


file_paths = ["f.json", "s.json"]

output_file = "merged.json"

merge_json_files(file_paths, output_file)

print(f"Merged data written to '{output_file}'")

Output:

merged.json

[{"key1": "value1"}, {"key2": "value2"}]

Merge Multiple JSON Files Using List Comprehension

In this example, the merge_json_files function reads JSON data from multiple files specified by file_paths using a list comprehension. The data is then stored in a list named merged_data. Subsequently, the combined data is written to a new JSON file, "merged.json," using the json.dump() method. Finally, the merged data list is printed for confirmation.

import json
def merge_json_files(file_paths):
    merged_data = [json.load(open(path, 'r')) for path in file_paths]
    return merged_data


file_paths = ["f.json", "s.json","e.json","t.json"]
output_file = "merged.json"
merged_data = merge_json_files(file_paths)
with open(output_file, 'w') as outfile:
    json.dump(merged_data, outfile)
print(merged_data)

Output:

merged.json

[{"key1": "value1"}, {"key2": "value2"}, {"key2": "value2"}, {"key2": "value2"}]

Merge Multiple JSON Files Using os Module with json Module

In this example, the merge_json_files function reads and merges JSON files from the specified directory ("./files"). The combined data is then written to a new JSON file named "merged.json," and the merged data is printed for confirmation.

import json
import os


def merge_json_files(directory_path):
    merged_data = []
    for filename in os.listdir(directory_path):
        if filename.endswith('.json'):
            with open(os.path.join(directory_path, filename), 'r') as file:
                data = json.load(file)
                merged_data.append(data)
    return merged_data


directory_path = "./files"
output_file = "merged.json"
merged_data = merge_json_files(directory_path)
with open(output_file, 'w') as outfile:
    json.dump(merged_data, outfile)
print(merged_data)

Output:

merged.json

[{"key1": "value1"}, {"key2": "value2"}]

Merge Multiple JSON Files Using glob Module with json Module

In this example, the merge_json_files function utilizes the glob module to obtain a list of JSON file paths from the specified directory ("./files"). It then reads and merges the JSON data from these files into a list named merged_data. The combined data is subsequently written to a new JSON file, "merged.json," and the merged data is printed for confirmation.

import json
import glob


def merge_json_files(directory_path):
    merged_data = []
    file_paths = glob.glob(directory_path + '/*.json')
    for path in file_paths:
        with open(path, 'r') as file:
            data = json.load(file)
            merged_data.append(data)
    return merged_data


directory_path = "./files"
output_file = "merged.json"
merged_data = merge_json_files(directory_path)
with open(output_file, 'w') as outfile:
    json.dump(merged_data, outfile)
print(merged_data)

Output:

merged.json

[{"key1": "value1"}, {"key2": "value2"}]

Merge Multiple JSON Files Using Pandas Library

In this example, the merge_json_files function reads JSON data from multiple files specified by file_paths and merges them into a Pandas DataFrame named merged_data. The combined data is then written to a new JSON file, "merged.json," using the to_json method with the 'records' orientation. Finally, the merged DataFrame is printed for confirmation.

import pandas as pd
import json


def merge_json_files(file_paths):
    merged_data = pd.DataFrame()
    for path in file_paths:
        with open(path, 'r') as file:
            data = json.load(file)
            row = pd.DataFrame([data])
            merged_data = pd.concat([merged_data, row], ignore_index=True)
    return merged_data


file_paths = ["f.json", "s.json", "e.json", "t.json"]
output_file = "merged.json"

merged_data = merge_json_files(file_paths)

merged_data.to_json(output_file, orient='records')

print(merged_data)

Output:

merged.json

[{"key1": "value1", "key2": null}, {"key1": null, "key2": "value2"}, {"key1": null, "key2": "value2"}, {"key1": null, "key2": "value2"}]