Node fs/promises and Streams Tutorial

What and Why

Reading a one-line config file and reading a ten-gigabyte log file are very different problems. The first fits in memory and is best handled with a single async call. The second has to be chunked or your process runs out of heap. Node provides two complementary APIs: promise-based file helpers for small files and streams for large ones. Choosing the right tool saves both memory and your weekend.

Mental Model

fs/promises treats a file as a value. You ask for it and receive the whole thing. Streams treat a file as a sequence of chunks that flow through a pipeline of transforms. The pipeline applies backpressure automatically, so a slow consumer pauses the producer instead of overflowing buffers.

readFile         readable stream
[ entire file ]   [ chunk ][ chunk ][ chunk ]
                 -> transform -> writable
single Promise    backpressure aware

Hands-on Example

Reading and writing a small JSON config.

import { readFile, writeFile } from 'node:fs/promises';

const raw = await readFile('config.json', 'utf8');
const config = JSON.parse(raw);
config.lastRun = new Date().toISOString();
await writeFile('config.json', JSON.stringify(config, null, 2));

For a large CSV that does not fit in memory, switch to streams.

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';
import { Transform } from 'node:stream';

const upper = new Transform({
  transform(chunk, _enc, cb) {
    cb(null, chunk.toString().toUpperCase());
  },
});

await pipeline(
  createReadStream('input.csv'),
  upper,
  createWriteStream('output.csv'),
);

pipeline wires the three streams together, propagates errors, and closes every resource on failure. It is the safest way to compose streams.

You can also use streams to compute a hash without loading the file.

import { createReadStream } from 'node:fs';
import { createHash } from 'node:crypto';
import { pipeline } from 'node:stream/promises';

const hash = createHash('sha256');
await pipeline(createReadStream('movie.mp4'), hash);
console.log(hash.digest('hex'));

The file streams through the hasher chunk by chunk and the heap stays flat regardless of file size.

Common Pitfalls

The classic mistake is using readFile on a multi-gigabyte file. Node has to allocate one big buffer and your process either crashes or pauses for garbage collection. If a file might grow beyond a few megabytes, default to a stream.

Streams without pipeline are leak-prone. People wire source.pipe(dest) directly, then forget to handle error on the source. A failure mid-stream leaves descriptors open. Always use pipeline or the promise version of it.

Encoding bugs hide in transform streams. The default chunk type is a Buffer. If you call .toString() without specifying utf8, multibyte characters that span chunk boundaries get corrupted. Use setEncoding('utf8') on the readable side or decode with TextDecoderStream to handle boundaries correctly.

Practical Tips

The high water mark controls chunk size. The default is 64 kilobytes for file streams. For network-bound work you can lower it to ship data faster, and for CPU-heavy transforms a larger value reduces per-chunk overhead. Pass it via { highWaterMark: 256 * 1024 }.

Use async iterators for readability. Modern streams are async iterables, so you can write straightforward loops.

import { createReadStream } from 'node:fs';
import readline from 'node:readline';

const rl = readline.createInterface({
  input: createReadStream('access.log'),
  crlfDelay: Infinity,
});

let errors = 0;
for await (const line of rl) {
  if (line.includes(' 500 ')) errors++;
}
console.log(`Found ${errors} server errors`);

This pattern keeps memory low and reads like synchronous code. Pair it with pipeline for transforms when you need backpressure.

When writing to disk, prefer writeFile for atomic semantics. If the process is killed mid-write, a streamed file may be left half written. For critical config, write to a temp file and rename, since rename is atomic on the same filesystem.

Wrap-up

Use fs/promises for small files where ergonomics matter most. Reach for streams once the data outgrows memory or when you want pipelines that compose. Wrap stream chains in pipeline, mind your encodings, and pick a sensible high water mark. With those habits, Node handles megabytes and gigabytes with the same code shape.