Convert CSV to JSON in Node.js

Data conversion between different formats is a common requirement in modern web development and data processing pipelines. Among the most frequently needed conversions is transforming CSV (Comma Separated Values) files to JSON (JavaScript Object Notation), especially when working with Node.js applications.This complete tutorial describes various proven methods for CSV to JSON conversion in Node.js with instructions on processing header rows while handling internal field commas and optimizing performance for large file sizes.

Why Convert CSV to JSON?

The following section provides an understanding of the reasons behind converting CSV files to JSON documents.

• The majority of current-day API systems rely on JSON as their native data configuration.
• JavaScript applications achieve direct compatibility with JSON data through their frontend operation.
• Data Manipulation: JSON's hierarchical structure makes complex data easier to manipulate
• Database Import: Many NoSQL databases like MongoDB store data in JSON-like format

Setting Up Your Node.js Project

Let's start by creating a new Node.js project and installing the necessary dependencies

// Initialize a new Node.js project
mkdir csv-to-json-converter

cd csv-to-json-converter
npm init -y

// Install required packages
npm install csvtojson fs-extra

Basic CSV to JSON Conversion

The CSV file conversion into JSON can be achieved through the popular csvtojson package as the simplest method. Here's a basic example

FileName: Converter.js

Example

const csv = require('csvtojson');
const fs = require('fs');

const csvFilePath = './data.csv';
const jsonOutputPath = './output.json';

// Basic conversion
async function convertCsvToJson() {
    try {
        // Convert CSV to JSON
        const jsonArray = await csv().fromFile(csvFilePath);
        
        // Write JSON to file
        fs.writeFileSync(jsonOutputPath, JSON.stringify(jsonArray, null, 2));
        
        console.log('Conversion completed successfully!');
        console.log(`JSON data saved to ${jsonOutputPath}`);
    } catch (error) {
        console.error('Error during conversion:', error.message);
    }
}

convertCsvToJson();

Output

Conversion completed successfully!
JSON data saved to ./output.json
 [
  {
    id: '1',
    name: 'John Doe',
    skills: 'JavaScript, Node.js, React',
    location: 'New York, NY'
  },
  {
    id: '2',
    name: 'Jane Smith',
    skills: 'Python, Data Analysis',
    location: 'London'
  },
  {
    id: '3',
    name: 'Bob Johnson',
    skills: 'Java, Spring, Hibernate',
    location: 'San Francisco, CA'
  }
]  

Handling Comma Separated Values Within Fields

The CSV files conversion into JSON can be achieved through the popular csvtojson package as the simplest method. Here's a basic example

Let's take a sample CSV file with comma-separated value inside some fields:

Example data.csv:

id,name,skills,location
1,"John Doe","JavaScript, Node.js, React","New York, NY"
2,"Jane Smith","Python, Data Analysis",London
3,"Bob Johnson","Java, Spring, Hibernate","San Francisco, CA"

To properly handle this case, we need to configure the CSV parser correctly:

FileName: advanced-converter.js

Example

const csv = require('csvtojson');
const fs = require('fs-extra');

const csvFilePath = './complex-data.csv';
const jsonOutputPath = './complex-output.json';

async function convertComplexCsv() {
    try {
        // Configure parser to handle quoted fields with commas
        const jsonArray = await csv({
            checkType: true,
            quote: '"', // Specify quote character
            escape: '"', // Specify escape character for quotes
            trim: true    // Trim whitespace around values
        }).fromFile(csvFilePath);
        
        // Process each row to handle arrays in skill field
        const processedData = jsonArray.map(row => {
            // If skills contains commas, split it into an array
            if (row.skills && row.skills.includes(',')) {
                row.skills = row.skills.split(',').map(skill => skill.trim());
            }
            return row;
        });
        
        // Write processed JSON to file
        await fs.writeJson(jsonOutputPath, processedData, { spaces: 2 });
        
        console.log('Complex CSV conversion completed successfully!');
    } catch (error) {
        console.error('Error during conversion:', error.message);
    }
}

convertComplexCsv();

Output

Complex CSV conversion completed successfully!
 [
  {
    id: 1,
    name: 'John Doe',
    skills: [ 'JavaScript', 'Node.js', 'React' ],
    location: 'New York, NY'
  },
  {
    id: 2,
    name: 'Jane Smith',
    skills: [ 'Python', 'Data Analysis' ],
    location: 'London'
  },
  {
    id: 3,
    name: 'Bob Johnson',
    skills: [ 'Java', 'Spring', 'Hibernate' ],
    location: 'San Francisco, CA'
  }
]   
 

Streaming Large CSV Files

CSV files that exceed a certain size require special attention regarding memory consumption. A streaming approach is more efficient.

FileName: stream-converter.js

Example:

const csv = require('csvtojson');
const fs = require('fs');

const csvFilePath = './large-data.csv';
const jsonOutputPath = './large-output.json';

async function streamConversion() {
    try {
        // Create write stream
        const writeStream = fs.createWriteStream(jsonOutputPath);
        
        // Write opening bracket for JSON array
        writeStream.write('[');
        
        let isFirstLine = true;
        
        // Process CSV as stream
        await new Promise((resolve, reject) => {
            csv()
                .fromFile(csvFilePath)
                .subscribe(
                    (json) => {
                        // Add comma before each item except the first
                        const prefix = isFirstLine ? '' : ',';
                        isFirstLine = false;
                        
                        // Write JSON object to file
                        writeStream.write(`${prefix}  ${JSON.stringify(json)}`);
                    },
                    reject,
                    resolve
                );
        });
        
        // Write closing bracket for JSON array
        writeStream.write(']');
        writeStream.end();
        
        console.log('Streaming conversion completed successfully!');
    } catch (error) {
        console.error('Error during streaming conversion:', error.message);
    }
}

streamConversion();

Handling CSV Files Without Headers

Sometimes you might encounter CSV files without headers. Here's how to handle them:

FileName: no-header.js

Example

const csv = require('csvtojson');
const fs = require('fs');

const csvFilePath = './no-header-data.csv';
const jsonOutputPath = './no-header-output.json';

async function convertNoHeaderCsv() {
    try {
        // Define custom headers for CSV without headers
        const jsonArray = await csv({
            noheader: true,
            headers: ['column1', 'column2', 'column3', 'column4']
        }).fromFile(csvFilePath);
        
        fs.writeFileSync(jsonOutputPath, JSON.stringify(jsonArray, null, 2));
        console.log('No-header CSV conversion completed!');
    } catch (error) {
        console.error('Error:', error.message);
    }
}

convertNoHeaderCsv();

Input Data :

1,"John Doe","JavaScript, Node.js, React","New York, NY"
2,"Jane Smith","Python, Data Analysis",London
3,"Bob Johnson","Java, Spring, Hibernate","San Francisco, CA"

Output

No-header CSV conversion completed!
 [
  {
    column1: '1',
    column2: 'John Doe',
    column3: 'JavaScript, Node.js, React',
    column4: 'New York, NY'
  },
  {
    column1: '2',
    column2: 'Jane Smith',
    column3: 'Python, Data Analysis',
    column4: 'London'
  },
  {
    column1: '3',
    column2: 'Bob Johnson',
    column3: 'Java, Spring, Hibernate',
    column4: 'San Francisco, CA'
  }
]  

Creating Nested JSON Structures from CSV

Sometimes you might want to transform your flat CSV structure into a nested JSON structure.

FileName: nested-structure.js

Example

const csv = require('csvtojson');
const fs = require('fs');

const csvFilePath = './user-data.csv';
const jsonOutputPath = './nested-output.json';

async function createNestedJson() {
    try {
        const jsonArray = await csv().fromFile(csvFilePath);
        
        // Transform flat structure to nested structure
        const nestedJson = jsonArray.map(item => {
            return {
                id: item.id,
                personalInfo: {
                    name: item.name,
                    email: item.email,
                    age: parseInt(item.age, 10)
                },
                location: {
                    city: item.city,
                    country: item.country,
                    postalCode: item.postalCode
                },
                skills: item.skills ? item.skills.split(',').map(skill => skill.trim()) : []
            };
        });
        
        fs.writeFileSync(jsonOutputPath, JSON.stringify(nestedJson, null, 2));
        console.log('Nested JSON structure created successfully!');
    } catch (error) {
        console.error('Error:', error.message);
    }
}

createNestedJson();

Input Data:

id,name,email,age,city,country,postalCode,skills
1,John Doe,john@example.com,32,New York,USA,10001,JavaScript,HTML,CSS
2,Jane Smith,jane@example.com,28,London,UK,EC1A 1BB,Python,SQL,Data Analysis
3,Alex Johnson,alex@example.com,35,Toronto,Canada,M5V 2A8,Java,C++,Docker
4,Maria Garcia,maria@example.com,30,Madrid,Spain,28001,React,Node.js,MongoDB

Output

Nested JSON structure created successfully!
[
   {
      "id":"1",
      "personalInfo":{
         "name":"John Doe",
         "email":"john@example.com",
         "age":32
      },
      "location":{
         "city":"New York",
         "country":"USA",
         "postalCode":"10001"
      },
      "skills":[
         "JavaScript"
      ]
   },
   {
      "id":"2",
      "personalInfo":{
         "name":"Jane Smith",
         "email":"jane@example.com",
         "age":28
      },
      "location":{
         "city":"London",
         "country":"UK",
         "postalCode":"EC1A 1BB"
      },
      "skills":[
         "Python"
      ]
   },
   {
      "id":"3",
      "personalInfo":{
         "name":"Alex Johnson",
         "email":"alex@example.com",
         "age":35
      },
      "location":{
         "city":"Toronto",
         "country":"Canada",
         "postalCode":"M5V 2A8"
      },
      "skills":[
         "Java"
      ]
   },
   {
      "id":"4",
      "personalInfo":{
         "name":"Maria Garcia",
         "email":"maria@example.com",
         "age":30
      },
      "location":{
         "city":"Madrid",
         "country":"Spain",
         "postalCode":"28001"
      },
      "skills":[
         "React"
      ]
   }
]

Conclusion

Node.js developers have diverse options to convert their CSV files into JSON according to different project specifications. Most CSV-to-JSON conversion needs can be resolved effectively using the csvtojson package regardless of file complexity or special requirements.

These techniques enable you to transform your CSV data into JSON format which provides versatility and efficiency during the development of data pipelines, API creation and data processing for analysis. During conversions focus on checking memory capacity and validating data together with handling special cases involving comma-separated values contained in fields to achieve reliable and accurate results.

The educational site TutorialsPoint contains detailed tutorials for Node.js, JSON and CSV processing methods which can be accessed for further exploration.