Unit 1: Getting Started with Node.js and Handling Data I/O

1. Introduction to Node.js and Server-Side JavaScript

JavaScript was originally conceived in 1995 by Brendan Eich as a lightweight, client-side scripting language constrained strictly within the sandbox of the Netscape Navigator web browser. For over a decade, server-side enterprise development was dominated by multithreaded runtimes such as Java (JVM), C# (.NET), and synchronous process-per-request runtimes like PHP and Python (via Apache/WSGI).

In 2009, software engineer Ryan Dahl transformed modern systems programming by creating Node.js. Dahl observed that conventional web servers (such as Apache HTTP Server) scaled poorly under massive concurrent connections because they allocated a dedicated operating system thread per connection. When thousands of clients engaged in slow network I/O or database queries, thousands of threads sat idle in memory, consuming megabytes of stack space and triggering catastrophic CPU context-switching overhead (the classic C10K problem).

CODECopy Code
+-------------------------------------------------------------------------------+
|                       CONVENTIONAL SERVER VS. NODE.JS ARCHITECTURE            |
+-------------------------------------------------------------------------------+
|  THREAD-PER-CONNECTION MODEL (e.g., Apache HTTP)                             |
|  Client 1 ----> [ Thread 1 ] ====> Blocks waiting for DB/Disk I/O             |
|  Client 2 ----> [ Thread 2 ] ====> Blocks waiting for Network I/O             |
|  Client N ----> [ Thread N ] ====> Heavy memory footprint (1-2 MB stack/thread)|
|                 * Bottleneck: RAM exhaustion & CPU context-switching thrashing|
|                                                                               |
|  SINGLE-THREADED EVENT-DRIVEN MODEL (Node.js)                                 |
|  Client 1 ---\                                                                |
|  Client 2 ----+--> [ Single Main Thread ] ===> Offloads I/O to Libuv Thread   |
|  Client N ---/       (Event Loop)              Pool / OS Kernel (epoll/kqueue)|
|                 * Non-blocking, asynchronous execution                        |
|                 * Scales efficiently to tens of thousands of concurrent I/O  |
+-------------------------------------------------------------------------------+

1.1 What Exactly is Node.js?

Node.js is not a programming language, and it is not a web framework.

Node.js is an open-source, cross-platform, single-threaded, asynchronous, event-driven JavaScript runtime environment built on Google Chrome's V8 engine that executes JavaScript code outside of a web browser.

1.2 The Core Anatomy of the Node.js Runtime

The Node.js runtime environment is an orchestrated composition of several low-level C/C++ libraries and JavaScript abstractions:

CODECopy Code
+-------------------------------------------------------------------------------+
|                             NODE.JS ARCHITECTURE                              |
+-------------------------------------------------------------------------------+
|  JavaScript Application Layer (User Code, NPM Modules, Express, etc.)        |
+-------------------------------------------------------------------------------+
|  Node.js Core API (fs, http, path, net, stream, buffer, events, crypto)       |
+---------------------------------------+---------------------------------------+
|  Node.js C++ Bindings (Node API)      | V8 JavaScript Engine (Google)         |
|  * Bridges JS calls to C/C++ libraries| * JIT compilation (Ignition & TurboFan)|
|                                       | * Compiles JS directly to machine code|
+---------------------------------------+---------------------------------------+
|                             LIBUV (C Library)                                 |
|  * Event Loop Implementation (Timers, Poll, Check, Close)                     |
|  * Thread Pool (Default: 4 threads, configurable up to 1024 via UV_THREADPOOL) |
|  * Cross-platform Non-Blocking Asynchronous I/O (epoll on Linux, kqueue on   |
|    macOS, IOCP on Windows)                                                    |
+---------------------------------------+---------------------------------------+
|  Supporting C/C++ Libraries:                                                  |
|  * OpenSSL (TLS/SSL encryption, cryptography)                                 |
|  * zlib (Fast asynchronous compression & decompression: gzip, deflate)        |
|  * llhttp / http-parser (High-performance HTTP message parsing)               |
|  * c-ares (Asynchronous DNS query resolution)                                 |
+-------------------------------------------------------------------------------+

1. Google V8 Engine:
   • Developed in C++ by Google for Chrome.
   • Compiles JavaScript directly into native machine code just-in-time (JIT) using its interpreter (Ignition) and optimizing compiler (TurboFan), eliminating interpreted bytecode lag.
   • Manages call stack execution, heap memory allocation, and generational garbage collection (Scavenge for Young Generation, Mark-Sweep-Compact for Old Generation).
2. Libuv:
   • A multi-platform, battle-tested C library initially written specifically for Node.js.
   • Provides the event loop and abstracts asynchronous I/O primitives across diverse operating systems:
     • epoll on Linux
     • kqueue on macOS / BSD
     • IOCP (Input/Output Completion Ports) on Microsoft Windows
   • Houses an internal worker Thread Pool (default: 4 background threads) used to offload operations that cannot be performed asynchronously by OS kernels (e.g., blocking file system I/O, synchronous cryptographic tasks, and DNS resolution).

2. The Node.js Event Loop and Asynchronous Execution Model

Understanding the Event Loop is the defining prerequisite for mastering Node.js. While Node.js executes user JavaScript code on a single thread (the main thread), it delegates system I/O operations to the OS kernel or Libuv worker threads whenever possible.

CODECopy Code
   ┌──────────────────────────────────────────┐
┌─>│                 TIMERS                   │ <--- Executes setTimeout(), setInterval()
│  └────────────────────┬─────────────────────┘
│                       │
│  ┌────────────────────┴─────────────────────┐
│  │             PENDING CALLBACKS            │ <--- Executes deferred I/O callbacks (e.g. TCP errors)
│  └────────────────────┬─────────────────────┘
│                       │
│  ┌────────────────────┴─────────────────────┐
│  │               IDLE, PREPARE              │ <--- Internal Libuv housekeeping only
│  └────────────────────┬─────────────────────┘
│                       │
│  ┌────────────────────┴─────────────────────┐
│  │                   POLL                   │ <--- Retrieves new I/O events; executes I/O callbacks;
│  └────────────────────┬─────────────────────┘      blocks here waiting for incoming connections
│                       │
│  ┌────────────────────┴─────────────────────┐
│  │                  CHECK                   │ <--- Executes setImmediate() callbacks
│  └────────────────────┬─────────────────────┘
│                       │
│  ┌────────────────────┴─────────────────────┐
│  │              CLOSE CALLBACKS             │ <--- Executes close events (e.g., socket.on('close'))
└──┴──────────────────────────────────────────┘

2.1 The Six Distinct Event Loop Phases

Each phase maintains a first-in, first-out (FIFO) queue of callbacks. When the event loop enters a given phase, it executes callbacks in that phase's queue until the queue is exhausted or the system maximum execution quota is reached, then moves to the next phase:

1. Timers Phase:
   • Executes callbacks scheduled by setTimeout() and setInterval().
   • Crucial note: Timer thresholds specify the minimum delay before execution, not the guaranteed exact millisecond.
2. Pending Callbacks Phase:
   • Executes system-level I/O callbacks deferred from previous iterations, such as certain types of TCP socket connection errors (ECONNREFUSED).
3. Idle, Prepare Phase:
   • Used strictly internally by Libuv for subsystem coordination and state synchronization.
4. Poll Phase:
   • The heart of the event loop. Calculates how long it should block and wait for new I/O events.
   • Processes events from the poll queue (incoming HTTP connections, disk read completions).
   • If the poll queue is empty:
     • If scripts were scheduled with setImmediate(), the loop terminates the poll phase and proceeds directly to the Check phase.
     • If no setImmediate() scripts exist, the loop waits for callbacks to be added to the queue, up to a calculated timeout.
5. Check Phase:
   • Exclusively reserved for callbacks scheduled via setImmediate().
   • Allows code to execute immediately after the poll phase finishes.
6. Close Callbacks Phase:
   • Executes socket and handle cleanup callbacks, such as socket.on('close', ...).

2.2 Microtask Queues: process.nextTick() and Promises

Beyond the main six event loop phases, Node.js features two special Microtask Queues that take unconditional precedence over macrotasks:

CODECopy Code
+-------------------------------------------------------------------------------+
|                             MICROTASK QUEUE TAXONOMY                          |
+---------------------------------------+---------------------------------------+
| 1. `process.nextTick()` Queue         | 2. Promise Microtask Queue            |
+---------------------------------------+---------------------------------------+
| * Highest execution priority in Node. | * Resolves standard JS Promise handlers|
| * Runs immediately after current      |   (`.then()`, `.catch()`, `.finally()`)|
|   operation completes, BEFORE the     |   and `async/await` resumptions.      |
|   event loop continues to any phase.  | * Evaluated immediately after the     |
| * Starvation Warning: Recursive calls |   `nextTick` queue is drained, before |
|   can completely starve the loop!     |   the event loop advances.            |
+---------------------------------------+---------------------------------------+

javascriptCopy Code
// Demonstration of Microtask vs. Macrotask Execution Order
console.log('1: Synchronous Script Start');

setTimeout(() => {
  console.log('5: Macrotask - setTimeout (Timers Phase)');
}, 0);

setImmediate(() => {
  console.log('6: Macrotask - setImmediate (Check Phase)');
});

Promise.resolve().then(() => {
  console.log('4: Microtask - Promise.then()');
});

process.nextTick(() => {
  console.log('3: Microtask - process.nextTick()');
});

console.log('2: Synchronous Script End');

// Expected Output Order:
// 1: Synchronous Script Start
// 2: Synchronous Script End
// 3: Microtask - process.nextTick()
// 4: Microtask - Promise.then()
// 5: Macrotask - setTimeout (Timers Phase)
// 6: Macrotask - setImmediate (Check Phase)

3. Node.js REPL, CLI Tools, and Execution Environment

Node.js provides a built-in interactive programming environment known as the REPL (Read-Eval-Print Loop).

3.1 The REPL Mechanics

• Read: Accepts user JavaScript input, parses it into memory, and verifies syntax validity.
• Eval: Evaluates the parsed data structure via the V8 engine.
• Print: Formats and prints the evaluated result to stdout with syntax highlighting.
• Loop: Loops back and waits for the next user input command.

bashCopy Code
# Launching the REPL in terminal
$ node
Welcome to Node.js v20.11.0.
Type ".help" for more information.
> 10 + 25
35
> const greeting = "Advanced Web Development";
undefined
> greeting.toUpperCase()
'ADVANCED WEB DEVELOPMENT'
> _
'ADVANCED WEB DEVELOPMENT'

• The Underscore (_) Special Variable: In the Node.js REPL, the special variable _ automatically stores the result of the most recently evaluated expression.
• Key Dot Commands in REPL:
  • .help: Displays all available REPL dot commands.
  • .break / .clear: Exits from multiline input mode.
  • .save <filename>: Saves current REPL session history into a local file.
  • .load <filename>: Loads and executes an external JavaScript file into the REPL.
  • .exit: Terminates the REPL session (equivalent to pressing Ctrl+C twice or Ctrl+D).

4. Node Package Manager (NPM) and Module Architecture

Modern JavaScript applications rely on modular architectures, decoupling complex software into cohesive, reusable packages.

4.1 Project Configuration: package.json vs. package-lock.json

Every standard Node.js project is rooted by a package.json file generated via npm init (or npm init -y for default values).

CODECopy Code
+-------------------------------------------------------------------------------+
|                       PACKAGE.JSON VS. PACKAGE-LOCK.JSON                      |
+---------------------------------------+---------------------------------------+
| `package.json`                        | `package-lock.json`                   |
+---------------------------------------+---------------------------------------+
| * Manifest of project metadata,       | * Automatically generated lockfile.   |
|   scripts, and dependencies.          | * Records EXACT resolved versions and |
| * Specifies version *ranges* using    |   cryptographic integrity hashes      |
|   semver prefixes (^, ~).             |   (SHA-512) for every nested package. |
| * Human-readable and editable.        | * Guarantees 100% deterministic builds|
|                                       |   across development and CI/CD.       |
+---------------------------------------+---------------------------------------+

4.2 Semantic Versioning (SemVer: MAJOR.MINOR.PATCH)

NPM strictly enforces the SemVer 2.0.0 specification (X.Y.Z):

• MAJOR (Breaking): Incremented when incompatible API changes or breaking modifications are introduced (e.g., 2.0.0).
• MINOR (Feature): Incremented when backward-compatible new functionality is added (e.g., 1.1.0).
• PATCH (Bugfix): Incremented when backward-compatible bug fixes are deployed (e.g., 1.0.1).

CODECopy Code
+-------------------------------------------------------------------------------+
|                          SEMVER PREFIX COMPARISON                             |
+-----------+-----------------------+-------------------------------------------+
| Prefix    | SemVer Meaning        | Behavior with `1.4.2`                     |
+-----------+-----------------------+-------------------------------------------+
| `^`       | Caret: Minor & Patch  | Accepts >= 1.4.2 and < 2.0.0              |
|           | Updates Allowed       | (Will update to 1.5.0, but NOT 2.0.0)     |
+-----------+-----------------------+-------------------------------------------+
| `~`       | Tilde: Patch Updates  | Accepts >= 1.4.2 and < 1.5.0              |
|           | Only                  | (Will update to 1.4.9, but NOT 1.5.0)     |
+-----------+-----------------------+-------------------------------------------+
| `*` or `` | Wildcard: Any Version | Installs latest available (Dangerous in   |
|           |                       | production; breaking changes can slip in) |
+-----------+-----------------------+-------------------------------------------+
| `1.4.2`   | Exact Version Match   | Installs strictly 1.4.2 and nothing else. |
+-----------+-----------------------+-------------------------------------------+

4.3 CommonJS (CJS) vs. ECMAScript Modules (ESM)

Node.js supports two distinct module formats:

CODECopy Code
+-------------------------------------------------------------------------------+
|                          COMMONJS (CJS) VS. ES MODULES (ESM)                  |
+-----------------------+-----------------------------------+-------------------+
| Feature               | CommonJS (CJS)                    | ES Modules (ESM)  |
+-----------------------+-----------------------------------+-------------------+
| **Syntax**            | `require()` and `module.exports`  | `import` / `export`
| **Loading Mechanism** | Synchronous, runtime evaluation.  | Asynchronous, static parse phase.
| **Scope Globals**     | Provides `__dirname`, `__filename`| Does NOT have `__dirname`. Uses   |
|                       | directly out of the box.          | `import.meta.url` instead.        |
| **File Extension**    | `.cjs` or `.js` (default).        | `.mjs` or `"type": "module"` in   |
|                       |                                   | `package.json`.                   |
| **Top-Level Await**   | Not supported natively in CJS.    | Supported natively out of the box.|
+-----------------------+-----------------------------------+-------------------+

javascriptCopy Code
// ==================== CommonJS (mathUtils.cjs) ====================
function add(a, b) { return a + b; }
function subtract(a, b) { return a - b; }
module.exports = { add, subtract };

// Consuming CJS:
const { add, subtract } = require('./mathUtils.cjs');

// ==================== ES Modules (mathUtils.mjs) ====================
export const add = (a, b) => a + b;
export const subtract = (a, b) => a - b;
export default function multiply(a, b) { return a * b; }

// Consuming ESM:
import multiply, { add, subtract } from './mathUtils.mjs';

5. Built-in Core Modules: Path, OS, and EventEmitter

Node.js ships with comprehensive built-in core modules that require no external installation.

5.1 The path Module

Cross-platform path normalization is essential because Windows uses backslashes (\) while POSIX systems (Linux, macOS) use forward slashes (/).

javascriptCopy Code
const path = require('path');

// Safe cross-platform path joining
const filePath = path.join('users', 'admin', 'reports', 'annual.pdf');
// Windows: 'users\admin\reports\annual.pdf'
// POSIX:   'users/admin/reports/annual.pdf'

// Resolving absolute paths
const absolutePath = path.resolve('public', 'images', 'logo.png');
// Output: 'C:\Users\...\public\images\logo.png'

// Inspecting path segments
const filename = 'C:/projects/app/server.js';
console.log(path.basename(filename)); // 'server.js'
console.log(path.dirname(filename));  // 'C:/projects/app'
console.log(path.extname(filename));  // '.js'
console.log(path.parse(filename));    // { root: 'C:/', dir: ..., base: ..., ext: '.js', name: 'server' }

5.2 The events Module and the EventEmitter Class

Much of the Node.js core architecture is built around an idiomatic Event-Driven Architecture. Objects that emit events are instances of the EventEmitter class.

javascriptCopy Code
const EventEmitter = require('events');

class UserRegistrationSystem extends EventEmitter {
  register(username, email) {
    console.log(`[System] Registering user: ${username}...`);
    // Business logic: save user to database...
    
    // Emit named event with payload data
    this.emit('userRegistered', { username, email, timestamp: new Date() });
  }
}

const userSystem = new UserRegistrationSystem();

// Register listener for event (executed on every emit)
userSystem.on('userRegistered', (data) => {
  console.log(`[Email Service] Welcome email dispatched to ${data.email}`);
});

// Register single-use listener (executed once, then automatically deregistered)
userSystem.once('userRegistered', (data) => {
  console.log(`[Metrics] First registration tracked for session at ${data.timestamp}`);
});

// Triggering the event
userSystem.register('rahul_sharma', 'rahul@example.com');
userSystem.register('priya_singh', 'priya@example.com');

6. Binary Data Manipulation: Buffers in Node.js

JavaScript in web browsers originally lacked native primitives for reading or manipulating raw streams of binary data. Because servers must interact with TCP streams, binary network packets, and disk files, Node.js introduced the global Buffer class.

CODECopy Code
+-------------------------------------------------------------------------------+
|                                BUFFER ARCHITECTURE                            |
+-------------------------------------------------------------------------------+
| * Represents a fixed-size chunk of memory allocated OUTSIDE the V8 heap       |
|   directly in raw C++ memory via `malloc()`.                                  |
| * Stores raw octets (bytes ranging from 0 to 255 / 0x00 to 0xFF).             |
| * Instances cannot be resized; length is immutable once allocated.            |
+-------------------------------------------------------------------------------+

javascriptCopy Code
// 1. Creating Buffers
// Allocate a zero-filled buffer of 10 bytes
const bufAlloc = Buffer.alloc(10);
console.log(bufAlloc); // <Buffer 00 00 00 00 00 00 00 00 00 00>

// Allocate uninitialized buffer (faster, but potentially contains sensitive old memory!)
// Always overwrite before exposing to users!
const bufUnsafe = Buffer.allocUnsafe(10);

// Create buffer from string or array
const bufFromString = Buffer.from('Hello LPU', 'utf-8');
console.log(bufFromString); // <Buffer 48 65 6c 6c 6f 20 4c 50 55>
console.log(bufFromString.length); // 9 bytes

// 2. Converting Buffer to String
console.log(bufFromString.toString('utf-8')); // 'Hello LPU'
console.log(bufFromString.toString('hex'));   // '48656c6c6f204c5055'
console.log(bufFromString.toString('base64'));// 'SGVsbG8gTFBV'

// 3. Writing and Slicing
const targetBuf = Buffer.alloc(15);
targetBuf.write('Node.js', 0, 'utf-8');
console.log(targetBuf.toString('utf-8')); // 'Node.js��...'

7. Node.js Streams and Backpressure Handling

Handling large datasets (e.g., streaming a 5 GB video file or processing a massive CSV dataset) by loading the entire file into RAM causes memory bloat and server crashes. Streams solve this by processing data piece-by-piece (chunks) sequentially.

CODECopy Code
+-------------------------------------------------------------------------------+
|                             THE FOUR TYPES OF STREAMS                         |
+-----------------------+-----------------------+-------------------------------+
| Stream Type           | Directional Property  | Canonical Examples            |
+-----------------------+-----------------------+-------------------------------+
| **1. Readable**       | Read-only source.     | `fs.createReadStream`,        |
|                       | Emits `data`, `end`.  | `http.IncomingMessage` (req). |
+-----------------------+-----------------------+-------------------------------+
| **2. Writable**       | Write-only destination| `fs.createWriteStream`,       |
|                       | Accepts `write()`.    | `http.ServerResponse` (res).  |
+-----------------------+-----------------------+-------------------------------+
| **3. Duplex**         | Both Readable and     | `net.Socket` (TCP connection),|
|                       | Writable concurrently.| WebSockets stream.            |
+-----------------------+-----------------------+-------------------------------+
| **4. Transform**      | Duplex stream that    | `zlib.createGzip()`,          |
|                       | modifies/transforms   | `crypto.createCipheriv()`.    |
|                       | data as it passes.    |                               |
+-----------------------+-----------------------+-------------------------------+

7.1 Piping and the Critical Concept of Backpressure

When reading from a high-speed disk and writing to a slow network client, the readable stream can overwhelm the writable stream's internal memory buffer (highWaterMark, default 16 KB for objects or 64 KB for buffers). This memory buildup is called Backpressure.

CODECopy Code
[ Fast Readable Stream ] ===(High Volume Data)===> [ Writable Buffer OVERFLOW ]
                                                          |
                                                    Backpressure Occurs!
                                                          |
[ Fast Readable Stream ] <--- PAUSE (drain false) ---------+
                         --- RESUME (drain event) --------+

javascriptCopy Code
// Idiomatic Stream Piping with pipeline() for safe error handling
const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream');

const sourceFile = 'server.log';
const compressedFile = 'server.log.gz';

// Stream pipeline automatically manages backpressure and cleans up open file handles
pipeline(
  fs.createReadStream(sourceFile),        // 1. Readable
  zlib.createGzip(),                     // 2. Transform (compress on the fly)
  fs.createWriteStream(compressedFile),   // 3. Writable
  (err) => {
    if (err) {
      console.error('Pipeline processing failed:', err);
    } else {
      console.log('File compressed successfully without memory bloat!');
    }
  }
);

8. The File System (fs) Module: Synchronous, Callback, and Promise APIs

Node.js offers three distinct paradigms for interacting with the operating system file system:

CODECopy Code
+-------------------------------------------------------------------------------+
|                          THE THREE FILE SYSTEM APIS                           |
+-----------------------+-----------------------+-------------------------------+
| API Paradigm          | Invocation Method     | Characteristics & Best Use    |
+-----------------------+-----------------------+-------------------------------+
| **1. Synchronous**    | `fs.readFileSync()`   | Blocks main thread. Use ONLY  |
|                       | `fs.writeFileSync()`  | during initial boot / config. |
+-----------------------+-----------------------+-------------------------------+
| **2. Callback-Based** | `fs.readFile(path, cb)`| Non-blocking; traditional Node|
|   (Error-First)       |                       | error-first callback style.   |
+-----------------------+-----------------------+-------------------------------+
| **3. Promise-Based**  | `fs/promises`         | Non-blocking; modern, clean   |
|   (Async/Await)       | `await fs.readFile()` | syntax using `async/await`.   |
+-----------------------+-----------------------+-------------------------------+

javascriptCopy Code
// ==================== 1. Error-First Callback API ====================
const fs = require('fs');

fs.readFile('config.json', 'utf-8', (err, data) => {
  if (err) {
    console.error('Error reading file:', err.message);
    return;
  }
  console.log('File content (Callback):', data);
});

// ==================== 2. Modern Promise-Based API ====================
const fsPromises = require('fs/promises');

async function manageFiles() {
  try {
    const data = await fsPromises.readFile('config.json', 'utf-8');
    console.log('File content (Promise):', data);

    await fsPromises.writeFile('audit.log', `Access logged at ${new Date()}\n`, { flag: 'a' });
    console.log('Audit record appended.');
  } catch (err) {
    console.error('Async File Error:', err.message);
  }
}

manageFiles();

9. Comprehensive Summary and Review Exercises

9.1 Core Knowledge Takeaways

9.2 Critical Review Questions

1. Deconstruct the difference between the Node.js main thread and the Libuv thread pool. Which operations execute on the main thread, and which are offloaded to Libuv worker threads?
2. Explain the execution priority between process.nextTick(), Promise.then(), setTimeout(), and setImmediate(). Provide a code snippet illustrating this hierarchy.
3. Compare CommonJS (require) with ES Modules (import). What happens under the hood when a module is imported in ESM versus required in CJS?
4. What is Backpressure in Node.js streams? How does the stream.pipeline() utility mitigate memory exhaustion and handle stream errors?
5. Write a Node.js script using fs/promises that safely reads an arbitrary JSON file, validates its structure, appends an audit timestamp, and writes the updated output without blocking the main event loop.