Streams and buffers — processing large data — Node.js — Runtime Internals, Express, APIs and SQL Server

5. Streams and buffers — processing large data

Intermediate★ Must knowLesson 5 of 11

Streams process data piece by piece — constant memory regardless of file size.

Type	Example
Readable	`fs.createReadStream`, HTTP request, DB cursor
Writable	`fs.createWriteStream`, HTTP response
Duplex	TCP socket
Transform	`zlib.createGzip()`, CSV parser

import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

await pipeline(createReadStream('orders.csv'), createGzip(), createWriteStream('orders.csv.gz'));   // handles errors + backpressure

// Stream a large file to an HTTP client
app.get('/export', (req, res) => {
  res.setHeader('Content-Type', 'text/csv');
  createReadStream('export.csv').pipe(res);
});

Backpressure: when the writer is slower than the reader, write() returns false; pipe/pipeline pause and resume the source automatically so memory doesn't explode.