Unit 1: Getting Started with Node.js & Handling Data I/O in Node.js — Comprehensive Study Notes

1. Node.js Architecture, Philosophy, and Runtime Internals

1.1 What is Node.js?

Node.js is an open-source, cross-platform JavaScript runtime environment that executes JavaScript code outside a web browser. Authored by Ryan Dahl in 2009, Node.js was engineered to overcome the scalability bottlenecks inherent in traditional multi-threaded web servers (such as Apache HTTP Server) by utilizing an event-driven, non-blocking I/O model.

CODECopy Code
+-----------------------------------------------------------------------------------+
|                           NODE.JS RUNTIME ARCHITECTURE                            |
|                                                                                   |
|  +-----------------------------------------------------------------------------+  |
|  |                           NODE.JS STANDARD LIBRARY                          |  |
|  |     fs, http, net, stream, events, buffer, crypto, path, zlib, cluster     |  |
|  +-----------------------------------------------------------------------------+  |
|  |                 NODE.JS BINDINGS (C++ WRAPPERS / NODE-API)                  |  |
|  |                 Bridges JavaScript API calls to C/C++ libraries             |  |
|  +--------------------------------------+--------------------------------------+  |
|                                         │                                         |
|                    ┌────────────────────┴────────────────────┐                    |
|                    ▼                                         ▼                    |
|  +-----------------------------------+     +-----------------------------------+  |
|  |       V8 JAVASCRIPT ENGINE        |     |               LIBUV               |  |
|  |  - Written in C++ (Google)        |     |  - Multi-platform C Library       |  |
|  |  - Compiles JS to machine code    |     |  - Event Loop implementation      |  |
|  |  - Call Stack & Heap Management   |     |  - Thread Pool (worker threads)   |  |
|  |  - Garbage Collection (Orinoco)   |     |  - Non-blocking Network I/O (epoll)| |
|  +-----------------------------------+     +-----------------------------------+  |
|  | OPENSSL (Crypto/TLS) | C-ARES (DNS) | ZLIB (Deflate) | HTTP-PARSER / LLHTTP |  |
|  +-----------------------------------------------------------------------------+  |
|  |                      OPERATING SYSTEM KERNEL & HARDWARE                     |  |
+-----------------------------------------------------------------------------------+

1.2 The Two Pillars: Google V8 Engine and Libuv

Node.js is not an interpreter or a standalone programming language; it is a composition of runtime libraries centered around two foundational engines:

Pillar 1: The Google V8 Engine

• Source Language: Written in C++.
• Role: Compiles and executes JavaScript source code directly into optimized native machine code at runtime.
• Components:
  • Ignition: The fast bytecode interpreter that converts parsed AST into efficient bytecode for immediate execution.
  • TurboFan: The optimizing compiler that analyzes running bytecode, identifies "hot" functions, and compiles them into optimized machine code based on type feedback.
  • Garbage Collector: Implements generational garbage collection (Scavenger for young generation allocations, Mark-Sweep-Compact for old generation heap objects).

Pillar 2: Libuv (The Asynchronous Engine)

• Source Language: Written in C.
• Role: Provides the cross-platform abstraction layer for asynchronous, event-driven I/O operations across Linux, macOS, and Windows.
• Kernel Demultiplexing: Uses OS-level non-blocking event notification facilities:
  • epoll on Linux.
  • kqueue on macOS/BSD.
  • IOCP (Input/Output Completion Ports) on Windows.
• The Worker Thread Pool: Operating systems do not provide non-blocking asynchronous APIs for all system operations (notably file system access and certain DNS lookups). Libuv manages a background Worker Thread Pool (defaulting to 4 threads) to execute blocking tasks without stalling the main JavaScript execution thread.

1.3 Single-Threaded Event Loop vs. Multi-Threaded Server Models

2. The Node.js Event Loop Deep Dive

The Node.js Event Loop begins execution when Node.js initializes. It executes the provided input script, processes microtasks, and continues cycling through dedicated phases as long as asynchronous event handles, timers, or I/O listeners remain active.

CODECopy Code
+-----------------------------------------------------------------------------------+
|                        THE SIX PHASES OF THE LIBUV EVENT LOOP                     |
|                                                                                   |
|     ┌────────────────────────► +--------------------+                             |
|     │                          |     1. TIMERS      | (setTimeout, setInterval)   |
|     │                          +---------+----------+                             |
|     │                                    │ * microtasks check (nextTick, Promise) |
|     │                                    ▼                                        |
|     │                          +--------------------+                             |
|     │                          | 2. PENDING CALLBACKS (I/O errors, system events) |
|     │                          +---------+----------+                             |
|     │                                    │ * microtasks check (nextTick, Promise) |
|     │                                    ▼                                        |
|     │                          +--------------------+                             |
|     │                          |  3. IDLE, PREPARE  | (Internal libuv operations) |
|     │                          +---------+----------+                             |
|     │                                    │ * microtasks check (nextTick, Promise) |
|     │                                    ▼                                        |
|     │                          +--------------------+                             |
|     │                          |      4. POLL       | (Retrieve I/O events, read  |
|     │                          +---------+----------+  incoming sockets & files)  |
|     │                                    │ * microtasks check (nextTick, Promise) |
|     │                                    ▼                                        |
|     │                          +--------------------+                             |
|     │                          |      5. CHECK      | (setImmediate callbacks)    |
|     │                          +---------+----------+                             |
|     │                                    │ * microtasks check (nextTick, Promise) |
|     │                                    ▼                                        |
|     │                          +--------------------+                             |
|     │                          | 6. CLOSE CALLBACKS | (socket.on('close'))        |
|     └──────────────────────────+--------------------+                             |
+-----------------------------------------------------------------------------------+

2.1 The 6 Discrete Phases of the Event Loop

1. Timers Phase: Executes callbacks scheduled by expired timers configured via setTimeout() and setInterval().
2. Pending Callbacks Phase: Executes I/O callbacks deferred to the next loop iteration (e.g., specific TCP socket system errors like ECONNREFUSED).
3. Idle, Prepare Phase: Used internally by libuv for housekeeping prior to polling.
4. Poll Phase:
   • Calculates how long it should block and wait for new I/O events.
   • Processes events in the I/O polling queue (reading incoming network sockets, file buffers).
   • If the poll queue is empty and setImmediate() scripts have been scheduled, the loop exits the poll phase and proceeds directly to the Check phase.
5. Check Phase: Dedicated exclusively to executing callbacks scheduled via setImmediate().
6. Close Callbacks Phase: Executes cleanup handlers if a socket or handle is abruptly closed (e.g., socket.on('close', ...)).

2.2 Microtask Prioritization: process.nextTick() vs. Promise.then()

In Node.js, microtasks do not belong to libuv; they are managed directly by V8.

• The Microtask Queues:
  1. process.nextTick Queue: Highest priority in the entire runtime.
  2. Promise Job Queue (Promise.resolve().then(), queueMicrotask): Secondary microtask priority.

Execution Invariant

Whenever the C++ boundary transitions back to JavaScript, the engine completely empties the process.nextTick queue, followed immediately by the Promise microtask queue, before the Event Loop is permitted to transition to the next phase.

javascriptCopy Code
// Demonstration of Phase & Microtask Execution Precedence
setTimeout(() => console.log('1. Timer Macrotask'), 0);
setImmediate(() => console.log('2. Check Macrotask (setImmediate)'));

Promise.resolve().then(() => console.log('3. Promise Microtask'));
process.nextTick(() => console.log('4. nextTick High-Priority Microtask'));

console.log('5. Synchronous Top-Level Execution');

// Output:
// 5. Synchronous Top-Level Execution
// 4. nextTick High-Priority Microtask
// 3. Promise Microtask
// 1. Timer Macrotask
// 2. Check Macrotask (setImmediate)

[!WARNING]
Recursively calling process.nextTick() causes I/O starvation because the event loop can never advance past the microtask queue to process timers or network requests.

2.3 Libuv Thread Pool Configuration

CPU-bound and blocking system operations are executed in libuv's thread pool:

• File system operations (fs.* except synchronous calls).
• DNS resolution (dns.lookup()).
• Cryptographic hashing and key generation (crypto.pbkdf2, crypto.scrypt, crypto.randomBytes).
• Compression/decompression (zlib.*).

The default thread pool size is 4 threads. In high-throughput servers handling heavy disk or cryptographic loads, the pool size should be tuned at application startup via the environment variable:

bashCopy Code
UV_THREADPOOL_SIZE=64 node server.js

3. Node.js Module Architectures: CommonJS vs. ECMAScript Modules

3.1 CommonJS (CJS) Internals

CommonJS is the historical default module system of Node.js.

The Module Wrapper Function

When Node.js loads a CommonJS file, it wraps the entire file content inside an invisible IIFE wrapper function before evaluation:

javascriptCopy Code
(function(exports, require, module, __filename, __dirname) {
    // Your actual file code lives here!
});

This wrapper provides local scope isolation and injects the 5 global-like identifiers:

• module: Reference to the current module object.
• exports: A convenience reference pointing initially to module.exports.
• require: The function used to import dependencies.
• __filename: The resolved absolute filesystem path of the current file.
• __dirname: The directory path containing the current file.

The require.cache Mechanism

Modules imported via require() are executed once and cached inside require.cache. Subsequent calls return the cached exported object from memory, guaranteeing singleton instances across an application.

3.2 CommonJS vs. ES Modules (ESM)

javascriptCopy Code
// Emulating __dirname in ES Modules (Node.js ESM)
import { fileURLToPath } from 'node:url';
import { dirname } from 'node:path';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

4. Event-Driven Architecture with EventEmitter

The events module exposes the EventEmitter class, which serves as the foundational backbone for all streaming, networking, and asynchronous primitives in Node.js (including net.Socket, http.Server, and fs.ReadStream).

CODECopy Code
+-----------------------------------------------------------------------------------+
|                       EVENTEMITTER INTERNAL DISPATCH MODEL                        |
|                                                                                   |
|   Publisher (Emitter)                            Subscriber (Listener)            |
|   +-----------------------+                      +-----------------------+        |
|   | emitter.emit('data',  │                      | emitter.on('data',    |        |
|   |              payload) ├─────────────────────►│            (payload)=>{       |
|   +-----------------------+   Synchronous Event  |              process(payload)  |
|                               Dispatch Pipeline  |            })                 |
|                                                  +-----------------------+        |
+-----------------------------------------------------------------------------------+

4.1 Key APIs and Methods

• emitter.on(eventName, listener): Registers a persistent event listener.
• emitter.once(eventName, listener): Registers a one-time listener that unregisters itself immediately after its first invocation.
• emitter.emit(eventName, ...args): Synchronously invokes all listeners registered for eventName in the order they were registered.
• emitter.removeListener(eventName, listener) / emitter.off(): Detaches a specific listener reference.
• emitter.removeAllListeners([eventName]): Detaches all listeners.

4.2 The Special 'error' Event

If an EventEmitter instance emits an 'error' event and no listener has been registered for 'error', Node.js treats it as an unhandled exception, prints the stack trace, and crashes the entire process immediately.

javascriptCopy Code
import { EventEmitter } from 'node:events';

class PaymentGateway extends EventEmitter {}
const gateway = new PaymentGateway();

// CRITICAL DEFENSIVE REQUIREMENT: Always register an 'error' listener!
gateway.on('error', (err) => {
    console.error('Handled payment error safely:', err.message);
});

gateway.emit('error', new Error('Card declined due to insufficient funds'));

4.3 Memory Leak Warnings: setMaxListeners()

By default, an EventEmitter prints a warning to stderr if more than 10 listeners are attached to a single event:
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 listeners added.
This default guards against unmanaged observer registrations in long-running services. If intentional, adjust the limit via emitter.setMaxListeners(n).

5. Binary Data Processing with Buffers

5.1 What is a Buffer?

JavaScript engines originally had no mechanism for reading or manipulating raw binary byte streams (e.g., TCP network packets, binary file headers, encrypted payloads).
The Node.js Buffer class provides fixed-size raw memory allocations allocated outside the V8 JavaScript heap in raw C++ memory pools.

CODECopy Code
+-----------------------------------------------------------------------------------+
|                        BUFFER MEMORY ALLOCATION TOPOLOGY                          |
|                                                                                   |
|   V8 MANAGED HEAP                         RAW C++ ALLOCATED MEMORY                |
|   +--------------------------+            +------------------------------------+  |
|   | JavaScript Objects       |            | Buffer (8KB Pool Allocation)       |  |
|   | Fast Small Primitives    |            | - Fixed Byte Length                |  |
|   | Buffer Wrapper Handles ──┼───────────►| - Raw Octet Binary Stream          |  |
|   +--------------------------+            +------------------------------------+  |
+-----------------------------------------------------------------------------------+

5.2 Allocation Methods and Security Risks

1. Buffer.alloc(size[, fill[, encoding]]):
   • Allocates zero-filled, sanitized memory of size bytes.
   • Completely secure. Prevents data leakage.
2. Buffer.allocUnsafe(size):
   • Extremely fast allocation because it does not overwrite previous memory contents with zeroes.
   • Severe Security Hazard: May contain sensitive residual data (passwords, encryption keys, personal data) from earlier memory operations. Must be overwritten immediately before reading or transmitting.
3. Buffer.from(array | string | buffer):
   • Creates a new Buffer copying bytes from an existing array, string, or buffer.

javascriptCopy Code
import { Buffer } from 'node:buffer';

// Secure Allocation
const safeBuf = Buffer.alloc(1024); // 1KB zeroed buffer

// Creating from String with Encoding
const utf8Buf = Buffer.from('Node.js Architecture', 'utf-8');
console.log(utf8Buf.toString('hex')); // Hexadecimal representation
console.log(utf8Buf.toString('base64')); // Base64 encoding

6. File System Operations (fs and fs/promises)

The node:fs module provides comprehensive file management through three API styles:

1. Synchronous APIs (fs.readFileSync): Blocks the entire main thread until I/O finishes. Use only during initial startup configuration loading.
2. Callback APIs (fs.readFile): Classical error-first callbacks (err, data) => {}.
3. Promise APIs (fs/promises): Modern async/await non-blocking interfaces.

javascriptCopy Code
import fs from 'node:fs/promises';

async function processAuditLog(filePath) {
    try {
        // Read file contents as UTF-8 string
        const content = await fs.readFile(filePath, { encoding: 'utf-8' });
        
        // Append entry
        await fs.appendFile(filePath, `\n[${new Date().toISOString()}] Healthcheck OK`);
        
        // Inspect metadata
        const stats = await fs.stat(filePath);
        console.log(`File Size: ${stats.size} bytes, Is File: ${stats.isFile()}`);
    } catch (error) {
        if (error.code === 'ENOENT') {
            console.error('File not found at specified path');
        } else {
            console.error('System I/O failure:', error);
        }
    }
}

7. Node.js Streams and Data Pipelining

7.1 Why Streams? Memory vs. Time Efficiency

Reading a 5GB video file into memory using fs.readFile() requires allocating 5GB of RAM, exceeding standard V8 heap limits and crashing the process with JavaScript heap out of memory.
Streams process data sequentially in small, discrete chunks (default buffer size: 64KB for files, 16KB for object streams):

• Memory Efficiency: Never consumes more memory than the single active chunk size, regardless of whether the source file is 1MB or 1 Terabyte.
• Time Efficiency: Downstream consumers can begin processing, transforming, or streaming the first chunk over the network before the rest of the file has even been read from disk.

CODECopy Code
+-----------------------------------------------------------------------------------+
|                        THE FOUR CORE STREAM CATEGORIES                            |
|                                                                                   |
|  1. READABLE STREAM: Source of data (fs.createReadStream, req in HTTP)            |
|     Events: 'data', 'end', 'error', 'pause', 'resume'                             |
|                                                                                   |
|  2. WRITABLE STREAM: Destination for data (fs.createWriteStream, res in HTTP)    |
|     Methods: write(chunk), end(), Events: 'drain', 'finish', 'error'              |
|                                                                                   |
|  3. DUPLEX STREAM: Both Readable & Writable (net.Socket, WebSocket connection)    |
|                                                                                   |
|  4. TRANSFORM STREAM: Duplex stream that modifies data (zlib.createGzip)          |
+-----------------------------------------------------------------------------------+

7.2 Backpressure: Resolving Producer-Consumer Mismatches

Backpressure occurs when a fast producer (e.g., a high-speed SSD reading at 500MB/s) generates data faster than a slow consumer (e.g., a slow mobile client downloading over 3G at 100KB/s) can process and transmit it.

• Without backpressure control, unconsumed chunks accumulate in system memory, leading to RAM exhaustion.
• Handling Backpressure:
  1. writable.write(chunk) returns false when internal buffer limit (highWaterMark) is exceeded.
  2. The producer must pause reading (readable.pause()).
  3. When the consumer empties its internal buffer, the writable stream emits the 'drain' event.
  4. The producer listens for 'drain' and resumes reading (readable.resume()).

7.3 Safe Pipelining: stream.pipeline vs. Legacy .pipe()

Legacy .pipe() does not forward errors automatically across pipeline stages. If a middle stream throws an error, file descriptors remain dangling open, causing resource leaks.
Modern Node.js mandates stream.promises.pipeline:

javascriptCopy Code
import { pipeline } from 'node:stream/promises';
import fs from 'node:fs';
import zlib from 'node:zlib';

async function compressServerLog(sourcePath, destinationPath) {
    // Automatically destroys all streams on completion or error, preventing leaks!
    await pipeline(
        fs.createReadStream(sourcePath),
        zlib.createGzip(),
        fs.createWriteStream(destinationPath)
    );
    console.log('Stream compression pipeline finished successfully.');
}