Node.js Streams and Backpressure Handling

Streams are one of the most powerful and most misunderstood features in Node.js. They let you process data piece by piece instead of loading everything into memory at once. But if you ignore backpressure, your application will buffer unbounded data in memory until it crashes. This guide covers how streams actually work and how to handle backpressure properly.

The four stream types

Node.js has four fundamental stream types:

import { Readable, Writable, Transform, Duplex } from 'node:stream';

• Readable: produces data (file reads, HTTP requests, database cursors)
• Writable: consumes data (file writes, HTTP responses, database inserts)
• Transform: modifies data passing through (compression, encryption, parsing)
• Duplex: both readable and writable (TCP sockets, WebSockets)

Every stream extends EventEmitter and communicates through events like data, end, drain, and error.

What is backpressure?

Backpressure occurs when a writable stream cannot consume data as fast as a readable stream produces it. Without handling it, the internal buffer grows without limit.

import { createReadStream, createWriteStream } from 'node:fs';

const source = createReadStream('huge-file.csv'); // Reads at 500 MB/s
const dest = createWriteStream('output.csv');     // Writes at 100 MB/s

// WRONG — ignores backpressure
source.on('data', (chunk) => {
  dest.write(chunk); // Returns false when buffer is full, but we ignore it
});

In this example, dest.write() returns false when its internal buffer exceeds the highWaterMark (default 16 KB for object mode, 16 KB for buffer mode). Ignoring this return value means data keeps piling up in memory.

The correct way: using pipe

The pipe method handles backpressure automatically. When the writable stream’s buffer is full, it pauses the readable stream. When the buffer drains, it resumes reading.

import { createReadStream, createWriteStream } from 'node:fs';

const source = createReadStream('huge-file.csv');
const dest = createWriteStream('output.csv');

source.pipe(dest);

dest.on('finish', () => {
  console.log('Done writing');
});

Under the hood, pipe does this:

1. Listens for data events on the readable stream
2. Calls write() on the writable stream for each chunk
3. If write() returns false, calls pause() on the readable stream
4. Listens for the drain event on the writable stream
5. When drain fires, calls resume() on the readable stream

Manual backpressure handling

Sometimes you need more control than pipe provides. Here is the manual equivalent:

import { createReadStream, createWriteStream } from 'node:fs';

const source = createReadStream('huge-file.csv');
const dest = createWriteStream('output.csv');

source.on('data', (chunk) => {
  const canContinue = dest.write(chunk);

  if (!canContinue) {
    // Buffer is full — pause reading
    source.pause();

    // Resume when the buffer drains
    dest.once('drain', () => {
      source.resume();
    });
  }
});

source.on('end', () => {
  dest.end();
});

source.on('error', (err) => {
  console.error('Read error:', err);
  dest.destroy(err);
});

dest.on('error', (err) => {
  console.error('Write error:', err);
  source.destroy(err);
});

Notice the error handling on both streams. If either stream fails, you must destroy the other to prevent resource leaks.

pipeline: the modern approach

The stream.pipeline function is the recommended way to chain streams. It handles backpressure, error propagation, and cleanup automatically.

import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';

await pipeline(
  createReadStream('input.log'),
  createGzip(),
  createWriteStream('input.log.gz')
);

console.log('Compression complete');

Unlike pipe, pipeline destroys all streams if any stream in the chain errors. With pipe, you have to handle cleanup yourself or risk dangling file descriptors.

Building custom Transform streams

Transform streams are the workhorse of stream processing. Here is a CSV line parser:

import { Transform } from 'node:stream';

class CSVParser extends Transform {
  #remainder = '';

  constructor() {
    super({ objectMode: true });
  }

  _transform(chunk, encoding, callback) {
    const data = this.#remainder + chunk.toString();
    const lines = data.split('\n');

    // Last line might be incomplete
    this.#remainder = lines.pop();

    for (const line of lines) {
      if (line.trim()) {
        const fields = line.split(',').map((f) => f.trim());
        this.push(fields); // push to the readable side
      }
    }

    callback(); // Signal that we're done processing this chunk
  }

  _flush(callback) {
    // Handle any remaining data
    if (this.#remainder.trim()) {
      const fields = this.#remainder.split(',').map((f) => f.trim());
      this.push(fields);
    }
    callback();
  }
}

Usage with pipeline:

import { pipeline } from 'node:stream/promises';
import { createReadStream } from 'node:fs';

const parser = new CSVParser();

let rowCount = 0;

parser.on('data', (row) => {
  rowCount++;
  // Process each row
});

await pipeline(createReadStream('data.csv'), parser);

console.log(`Processed ${rowCount} rows`);

Async generators as readable streams

Node.js lets you use async generators as readable stream sources with Readable.from:

import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';
import { createWriteStream } from 'node:fs';

async function* generateData() {
  for (let i = 0; i < 1_000_000; i++) {
    yield `Line ${i}: ${Date.now()}\n`;
  }
}

await pipeline(
  Readable.from(generateData()),
  createWriteStream('output.txt')
);

Backpressure works naturally here. When the writable stream’s buffer is full, the generator pauses at the next yield.

highWaterMark tuning

The highWaterMark option controls the buffer size threshold that triggers backpressure:

import { createReadStream, createWriteStream } from 'node:fs';

// Larger buffer for high-throughput scenarios
const source = createReadStream('large-file.bin', {
  highWaterMark: 64 * 1024, // 64 KB instead of default 16 KB
});

const dest = createWriteStream('output.bin', {
  highWaterMark: 64 * 1024,
});

source.pipe(dest);

A larger highWaterMark reduces the number of drain events (better throughput) but uses more memory. A smaller value reduces memory usage but increases the overhead of pausing and resuming. The default of 16 KB is a good starting point for most applications.

Common mistakes

Mistake 1: Forgetting to handle errors on piped streams.

// BAD — unhandled error crashes the process
source.pipe(transform).pipe(dest);

// GOOD — use pipeline
await pipeline(source, transform, dest);

Mistake 2: Mixing pipe with async/await incorrectly.

// BAD — pipe does not return a promise
await source.pipe(dest); // This resolves immediately!

// GOOD
await pipeline(source, dest);

Mistake 3: Not ending the writable stream.

// BAD — dest never emits 'finish'
source.on('end', () => {
  console.log('Done'); // But dest might still be flushing
});

// GOOD
source.on('end', () => {
  dest.end(); // Flush remaining data and emit 'finish'
});

Key takeaways

Always use pipeline from node:stream/promises for chaining streams in production code. It handles backpressure, error propagation, and resource cleanup. If you must handle streams manually, always check the return value of write() and listen for drain. Set highWaterMark based on your throughput and memory requirements. And never forget error handling on both ends of a stream chain.