How to do Piping/Streaming of JavaScript objects in Node.js ?
Last Updated :
15 Jul, 2024
Node.js is well-known for its non-blocking I/O and stream-based data processing capabilities. While streams are typically used for handling raw binary data or strings, Node.js also provides mechanisms to stream JavaScript objects. This can be particularly useful for handling large datasets, processing data on the fly, or building efficient pipelines for data transformation.
What is Buffers ?
These are the temporary storage spot for a chunk of data that is being transferred from one place to another, the buffer is filled with data and then passed along. Buffer transfers small chunks of data at a time. Buffers are very useful as we don't have to wait for the complete transfer process because we fill up the buffer which is the temporary storage spot for the data which can then be passed altogether.
What is Streams ?
They are the data handling method that is used for the transfer of the data. It increases the performance as the data is processed even before the complete transfer of the data. It collects the small chunks of buffer together and when the buffer is filled then it passes the data down the stream to be processed and sent to the client. They are a data-handling method and are used to read or write input into output following a proper sequence. Buffer and streams together While playing an online video you don't need to wait for the complete video to be processed rather is small sections transferred with the help of buffers and streams bit by bit.
What is Readable Stream ?
Readable streams are used to extract information from any source they are used to read the data that is present on any source, and then we can use that data to transfer to any source or to store in our system. Readable streams emit an event in the form of chunks (received from the buffers) that is a small part of the data that needs to be processed.
Steps to Setup Project for Streaming
Step 1: Make a folder structure for the project.
mkdir myapp
Step 2: Navigate to the project directory
cd myapp
Step 3: Initialize the NodeJs project inside the myapp folder.
npm init -y
Project Structure:

Example: Create a read.txt file in the current project directory with some sample text, in this case, we have the following text.
THIS IS READ.TXT FILE
Example: Implementation to show the example for reading a file.
JavaScript
// app.js
// Requiring file system module
const fs = require('fs');
// Creating a read stream
const myReadStream = fs.createReadStream('read.txt', 'utf8');
myReadStream.on('data', function (chunk) {
console.log("new chunk received");
// Printing the chunk
console.log(chunk);
});
Step to Run Application: Run the application using the following command from the root directory of the project
node app.js
Output:
new chunk received
THIS IS READ.TXT FILE
Writable Streams
Writable stream lets you send the data to any destination source that can rather be any file or stream online. These streams also emit the events to ensure proper functioning at regular intervals of time.
Example: Implementation to show the example using writable stream in nodeJS.
JavaScript
// app.js
// Requiring file system module
const fs = require('fs');
// Creating a write stream
// It will create a txt file in provided location
const myWriteStream =
fs.createWriteStream(__dirname + '/write.txt');
// Writing on stream
myWriteStream.write("Greetings from GeeksforGeeks");
Step to Run Application: Run the application using the following command from the root directory of the project
node app.js
Output: A new file named write.txt will be generated in the current directory which will contain the following data.
Greetings from GeeksforGeeks
Piping using pipeline()
Piping is a method of transferring data from one stream to other directly with the help of buffers so that the data transfer process can start as soon as it fills up a buffer. In other words, piping is used to process streamed data in various steps. This is a module method to pipe between streams forwarding errors and properly clean up and provide a callback when the pipeline is complete.
Example: Create a read.txt file in the current project directory with some sample text, in this case, we have the following text.
THIS IS READ.TXT FILE
JavaScript
// app.js
const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream');
// Constructing promisify
const { promisify } = require('util');
const pipelineAsync = promisify(pipeline);
// Creating a read stream
const myReadStream = fs.createReadStream(
__dirname + '/read.txt', 'utf8');
// Creating a write stream
const myWriteStream = fs.createWriteStream(
__dirname + '/write.txt');
// Creating transform stream
const transform = zlib.createGzip();
(async function run() {
try {
// Pipelining three streams
await pipelineAsync(
// Reading from read stream
myReadStream,
// Transforming the file
transform,
// Writing on write stream
myWriteStream
);
console.log("Pipeline accomplished");
}
catch (err) {
console.error('pipeline failed with error:', err);
}
})();
Step to Run Application: Run the application using the following command from the root directory of the project
node app.js
Output: We will see the following output and the write.txt file will be generated in the current directory.
Pipeline accomplished
Note: The pipeline should be used instead of a pipe, as the pipe is unsafe, but implementation on the pipe is given below:
JavaScript
// app.js
// Requiring file system module
const fs = require('fs');
// Creating a read stream
const myReadStream = fs.createReadStream(
__dirname + '/read.txt', 'utf8');
// Creating a write stream
const myWriteStream = fs.createWriteStream(
__dirname + '/write.txt');
// Piping using pipe
myReadStream.pipe(myWriteStream);
console.log("Piping accomplished");
Step to Run Application: Run the application using the following command from the root directory of the project
node app.js
Output: We will see the following output and the write.txt file will be generated in the current directory.
Piping accomplished
Similar Reads
Node.js Tutorial Node.js is a powerful, open-source, and cross-platform JavaScript runtime environment built on Chrome's V8 engine. It allows you to run JavaScript code outside the browser, making it ideal for building scalable server-side and networking applications.JavaScript was mainly used for frontend developme
4 min read
Introduction & Installation
NodeJS IntroductionNodeJS is a runtime environment for executing JavaScript outside the browser, built on the V8 JavaScript engine. It enables server-side development, supports asynchronous, event-driven programming, and efficiently handles scalable network applications. NodeJS is single-threaded, utilizing an event l
5 min read
Node.js Roadmap: A Complete GuideNode.js has become one of the most popular technologies for building modern web applications. It allows developers to use JavaScript on the server side, making it easy to create fast, scalable, and efficient applications. Whether you want to build APIs, real-time applications, or full-stack web apps
6 min read
How to Install Node.js on LinuxInstalling Node.js on a Linux-based operating system can vary slightly depending on your distribution. This guide will walk you through various methods to install Node.js and npm (Node Package Manager) on Linux, whether using Ubuntu, Debian, or other distributions.PrerequisitesA Linux System: such a
6 min read
How to Install Node.js on WindowsInstalling Node.js on Windows is a straightforward process, but it's essential to follow the right steps to ensure smooth setup and proper functioning of Node Package Manager (NPM), which is crucial for managing dependencies and packages. This guide will walk you through the official site, NVM, Wind
6 min read
How to Install NodeJS on MacOSNode.js is a popular JavaScript runtime used for building server-side applications. Itâs cross-platform and works seamlessly on macOS, Windows, and Linux systems. In this article, we'll guide you through the process of installing Node.js on your macOS system.What is Node.jsNode.js is an open-source,
6 min read
Node.js vs Browser - Top Differences That Every Developer Should KnowNode.js and Web browsers are two different but interrelated technologies in web development. JavaScript is executed in both the environment, node.js, and browser but for different use cases. Since JavaScript is the common Programming language in both, it is a huge advantage for developers to code bo
6 min read
NodeJS REPL (READ, EVAL, PRINT, LOOP)NodeJS REPL (Read-Eval-Print Loop) is an interactive shell that allows you to execute JavaScript code line-by-line and see immediate results. This tool is extremely useful for quick testing, debugging, and learning, providing a sandbox where you can experiment with JavaScript code in a NodeJS enviro
5 min read
Explain V8 engine in Node.jsThe V8 engine is one of the core components of Node.js, and understanding its role and how it works can significantly improve your understanding of how Node.js executes JavaScript code. In this article, we will discuss the V8 engineâs importance and its working in the context of Node.js.What is a V8
7 min read
Node.js Web Application ArchitectureNode.js is a JavaScript-based platform mainly used to create I/O-intensive web applications such as chat apps, multimedia streaming sites, etc. It is built on Google Chromeâs V8 JavaScript engine. Web ApplicationsA web application is software that runs on a server and is rendered by a client browser
3 min read
NodeJS Event LoopThe event loop in Node.js is a mechanism that allows asynchronous tasks to be handled efficiently without blocking the execution of other operations. It:Executes JavaScript synchronously first and then processes asynchronous operations.Delegates heavy tasks like I/O operations, timers, and network r
5 min read
Node.js Modules , Buffer & Streams
NodeJS ModulesIn NodeJS, modules play an important role in organizing, structuring, and reusing code efficiently. A module is a self-contained block of code that can be exported and imported into different parts of an application. This modular approach helps developers manage large projects, making them more scal
6 min read
What are Buffers in Node.js ?Buffers are an essential concept in Node.js, especially when working with binary data streams such as files, network protocols, or image processing. Unlike JavaScript, which is typically used to handle text-based data, Node.js provides buffers to manage raw binary data. This article delves into what
4 min read
Node.js StreamsNode.js streams are a key part of handling I/O operations efficiently. They provide a way to read or write data continuously, allowing for efficient data processing, manipulation, and transfer.\Node.js StreamsThe stream module in Node.js provides an abstraction for working with streaming data. Strea
4 min read
Node.js Asynchronous Programming
Node.js NPM
NodeJS NPMNPM (Node Package Manager) is a package manager for NodeJS modules. It helps developers manage project dependencies, scripts, and third-party libraries. By installing NodeJS on your system, NPM is automatically installed, and ready to use.It is primarily used to manage packages or modulesâthese are
6 min read
Steps to Create and Publish NPM packagesIn this article, we will learn how to develop and publish your own npm package (also called an NPM module). There are many benefits of NPM packages, some of them are listed below: Reusable codeManaging code (using versioning)Sharing code The life-cycle of an npm package takes place like below: Modu
7 min read
Introduction to NPM scriptsNPM is a Node Package Manager. It is the world's largest Software Registry. This registry contains over 800,000 code packages. Many Open-source developers use npm to share software. Many organizations also use npm to manage private development. "npm scripts" are the entries in the scripts field of t
2 min read
Node.js package.jsonThe package.json file is the heart of Node.js system. It is the manifest file of any Node.js project and contains the metadata of the project. The package.json file is the essential part to understand, learn and work with the Node.js. It is the first step to learn about development in Node.js.What d
4 min read
What is package-lock.json ?package-lock.json is a file that is generated when we try to install the node. It is generated by the Node Package Manager(npm). package-lock.json will ensure that the same versions of packages are installed. It contains the name, dependencies, and locked version of the project. It will check that s
3 min read
Node.js Deployments & Communication
Node DebuggingDebugging is an essential part of software development that helps developers identify and fix errors. This ensures that the application runs smoothly without causing errors. NodeJS is the JavaScript runtime environment that provides various debugging tools for troubleshooting the application.What is
3 min read
How to Perform Testing in Node.js ?Testing is a method to check whether the functionality of an application is the same as expected or not. It helps to ensure that the output is the same as the required output. How Testing can be done in Node.js? There are various methods by which tasting can be done in Node.js, but one of the simple
2 min read
Unit Testing of Node.js ApplicationNode.js is a widely used javascript library based on Chrome's V8 JavaScript engine for developing server-side applications in web development. Unit Testing is a software testing method where individual units/components are tested in isolation. A unit can be described as the smallest testable part of
5 min read
NODE_ENV Variables and How to Use Them ?Introduction: NODE_ENV variables are environment variables that are made popularized by the express framework. The value of this type of variable can be set dynamically depending on the environment(i.e., development/production) the program is running on. The NODE_ENV works like a flag which indicate
2 min read
Difference Between Development and Production in Node.jsIn this article, we will explore the key differences between development and production environments in Node.js. Understanding these differences is crucial for deploying and managing Node.js applications effectively. IntroductionNode.js applications can behave differently depending on whether they a
3 min read
Best Security Practices in Node.jsThe security of an application is extremely important when we build a highly scalable and big project. So in this article, we are going to discuss some of the best practices that we need to follow in Node.js projects so that there are no security issues at a later point of time. In this article, we
4 min read
Deploying Node.js ApplicationsDeploying a NodeJS application can be a smooth process with the right tools and strategies. This article will guide you through the basics of deploying NodeJS applications.To show how to deploy a NodeJS app, we are first going to create a sample application for a better understanding of the process.
5 min read
How to Build a Microservices Architecture with NodeJSMicroservices architecture allows us to break down complex applications into smaller, independently deployable services. Node.js, with its non-blocking I/O and event-driven nature, is an excellent choice for building microservices. How to Build a Microservices Architecture with NodeJS?Microservices
3 min read
Node.js with WebAssemblyWebAssembly, often abbreviated as Wasm, is a cutting-edge technology that offers a high-performance assembly-like language capable of being compiled from various programming languages such as C/C++, Rust, and AssemblyScript. This technology is widely supported by major browsers including Chrome, Fir
3 min read
Resources & Tools
Node.js Web ServerA NodeJS web server is a server built using NodeJS to handle HTTP requests and responses. Unlike traditional web servers like Apache or Nginx, which are primarily designed to give static content, NodeJS web servers can handle both static and dynamic content while supporting real-time communication.
6 min read
Node Exercises, Practice Questions and SolutionsNode Exercise: Explore interactive quizzes, track progress, and enhance coding skills with our engaging portal. Ideal for beginners and experienced developers, Level up your Node proficiency at your own pace. Start coding now! #content-iframe { width: 100%; height: 500px;} @media (max-width: 768px)
4 min read
Node.js ProjectsNode.js is one of the most popular JavaScript runtime environments widely used in the software industry for projects in different domains like web applications, real-time chat applications, RESTful APIs, microservices, and more due to its high performance, scalability, non-blocking I/O, and many oth
9 min read
NodeJS Interview Questions and AnswersNodeJS is one of the most popular runtime environments, known for its efficiency, scalability, and ability to handle asynchronous operations. It is built on Chromeâs V8 JavaScript engine for executing JavaScript code outside of a browser. It is extensively used by top companies such as LinkedIn, Net
15+ min read