Unit 1: Introduction to Hadoop

1. Introduction to Big Data

1.1 What is Big Data?

Big Data refers to datasets whose size, complexity, velocity, and variety surpass the processing, storage, and analytical capacities of traditional relational database management systems (RDBMS) and data warehouse solutions within an acceptable operational timeframe. Rather than being defined merely by raw byte count (gigabytes, terabytes, petabytes, exabytes, or zettabytes), Big Data encompasses computational paradigms, distributed architectures, and processing frameworks capable of extracting high-value intelligence from vast, heterogeneous, and rapidly flowing information streams.

1.2 Limitations of Traditional Database Systems (RDBMS)

Traditional database systems were designed primarily for structured, transactional workloads (OLTP) characterized by:

• Strict Predefined Schemas: RDBMS relies on relational algebra, requiring data normalization and rigid table schemas before data ingestion (schema-on-write). Changing schemas across billions of rows requires costly database migrations.
• Vertical Scalability (Scale-Up): Expanding capacity requires upgrading server hardware—procuring faster CPUs, expanded RAM, and specialized high-speed enterprise storage arrays (SAN/NAS). This model hits severe physical and economic cost limits (diminishing returns).
• ACID Constraints Overhead: Maintaining strict Atomicity, Consistency, Isolation, and Durability (ACID) guarantees across distributed nodes requires distributed locking mechanisms (e.g., two-phase commit), degrading throughput exponentially as cluster size grows.
• High Latency for Unstructured Content: Relational stores struggle with high-throughput ingestion of unformatted text, semi-structured web logs, telemetry streams, and binary objects (images, video, audio).

2. Types of Data: Structured, Semi-Structured, and Unstructured

Data generated across enterprise and consumer domains falls into three distinct structural classifications:

3. The Core Dimensions: The V's of Big Data

Industry consensus initially defined Big Data through Three V's (Gartner/Doug Laney), which has since expanded to Five Core V's and beyond:

3.1 Volume

• Definition: The sheer physical scale and footprint of data generated and persisted across global enterprises.
• Key Characteristics: Exponential growth driven by mobile devices, IoT sensors, social media interactions, clickstreams, and automated machine transactions. Datasets regularly scale into terabytes, petabytes, exabytes, and zettabytes.
• Architectural Impact: Requires horizontal distributed file systems that distribute chunks across hundreds or thousands of nodes, eliminating single-point storage bottlenecks.

3.2 Velocity

• Definition: The rate and frequency at which data is created, ingested, transmitted, and required to be processed or acted upon.
• Key Characteristics: Transition from monthly or daily batch processing to near-real-time and real-time streaming architectures.
• Architectural Impact: High velocity requires distributed event messaging backbones (Apache Kafka, RabbitMQ) and stream compute engines (Apache Flink, Apache Spark Streaming) to capture perishable insights before data loses its operational value.

3.3 Variety

• Definition: The structural diversity, heterogeneity, and distinct formats of inbound data streams.
• Key Characteristics: Combines tabular structured records, hierarchical semi-structured files (JSON, XML), unformatted raw text, time-series telemetry, image arrays, and audio streams.
• Architectural Impact: Storage architectures must support schema-on-read—storing raw binary or textual data without upfront schema enforcement, deferring parsing until analytical query execution.

3.4 Veracity

• Definition: The trustworthiness, accuracy, consistency, and reliability of data.
• Key Characteristics: Big Data pipelines frequently capture noisy, uncalibrated sensor readings, missing attributes, duplicate records, spoofed requests, and conflicting timestamps.
• Architectural Impact: Requires automated data cleansing, validation rules, deduplication pipelines, anomaly detection, and data lineage tracking.

3.5 Value

• Definition: The measurable business, scientific, or strategic insights extracted from large-scale data processing.
• Key Characteristics: Storing massive volumes of diverse data provides zero business return unless actionable patterns, cost optimizations, revenue opportunities, or predictive models are realized.
• Architectural Impact: High data footprint with low signal-to-noise ratio necessitates advanced analytical models, machine learning algorithms, and interactive business intelligence engines.

4. Introduction to Apache Hadoop

4.1 Historical Context and Evolution

In the early 2000s, search engine pioneers Doug Cutting and Mike Cafarella faced severe scalability bottlenecks while building Apache Nutch, an open-source web crawler and search indexer. In 2003 and 2004, Google published two foundational white papers:

1. "The Google File System (GFS)" (Ghemawat et al., 2003): Outlined a scalable distributed file system for large data-intensive applications running on commodity hardware.
2. "MapReduce: Simplified Data Processing on Large Clusters" (Dean & Ghemawat, 2004): Outlined a software framework for processing huge datasets in parallel across clusters.

Cutting incorporated these distributed storage and processing concepts into Nutch. In 2006, the distributed architecture was spun out into an independent Apache top-level project named Apache Hadoop (named after Doug Cutting's son's toy elephant). Yahoo! played an instrumental role in scaling Hadoop across tens of thousands of cluster nodes.

4.2 Core Architectural Principles of Hadoop

Hadoop is built around four foundational design tenets:

1. Commodity Hardware: Operates on standard, cost-effective commercial servers rather than proprietary, fault-tolerant mainframe hardware. Hardware failures are treated as common, expected occurrences rather than catastrophic exceptions.
2. Hardware Fault Tolerance: The software layer automatically handles hardware crashes, network partitions, and disk corruption through automatic data block replication and task re-execution.
3. Horizontal Scalability (Scale-Out): Computational and storage capacity scales linearly by adding additional commodity nodes to the cluster, avoiding vertical scaling limits.
4. Moving Computation to Data (Data Locality): Traditional computing moves large datasets across network fabrics to processing servers. Hadoop reverses this by scheduling tasks directly on the physical nodes where the corresponding data blocks reside, minimizing costly inter-rack network bandwidth saturation.

5. Core Components of Apache Hadoop

The modern Apache Hadoop platform consists of four core modules:

CODECopy Code
+-----------------------------------------------------------+
|              Hadoop MapReduce (Execution Engine)          |
+-----------------------------------------------------------+
|         YARN (Yet Another Resource Negotiator)            |
+-----------------------------------------------------------+
|           HDFS (Hadoop Distributed File System)           |
+-----------------------------------------------------------+
|                   Hadoop Common Utilities                 |
+-----------------------------------------------------------+

5.1 Hadoop Common

Provides the fundamental Java libraries, utilities, RPC (Remote Procedure Call) protocols, operating system abstraction layers, and shell script wrappers required by all Hadoop submodules.

5.2 HDFS (Hadoop Distributed File System)

• Role: High-throughput, distributed user-space file system designed to reliably store multi-terabyte to petabyte files across clusters.
• Master/Slave Model:
  • NameNode (Master): Maintains file system namespace tree, directory structure, access permissions, and mapping of file paths to disk block IDs in volatile RAM.
  • DataNodes (Slaves/Workers): Store actual block replicas on local disks and periodically report block inventories and heartbeats back to the NameNode.
• Block-Based Architecture: Files are split into large fixed-size blocks (default 128 MB in Hadoop 2.x/3.x) replicated across distinct physical racks (default replication factor = 3).

5.3 Hadoop YARN (Yet Another Resource Negotiator)

Introduced in Hadoop 2.0 to decouple cluster resource allocation from the MapReduce execution engine:

• ResourceManager (Cluster Master): Orchestrates global resource allocation (CPU, RAM, GPUs) across the entire cluster.
• NodeManager (Per-Node Slave): Monitors resource utilization of local worker nodes and manages container lifecycles.
• ApplicationMaster (Per-Job Orchestrator): Coordinates execution of an individual application, negotiating containers with the ResourceManager and task execution with NodeManagers.

5.4 Hadoop MapReduce

A distributed software programming model and computational framework for processing vast amounts of raw data in parallel:

• Map Phase: Worker nodes execute user-defined map functions across split records, emitting intermediate <Key, Value> pairs.
• Shuffle and Sort Phase: Framework groups and sorts all intermediate values sharing the identical key across cluster nodes.
• Reduce Phase: Reducer tasks aggregate, summarize, or compute final statistics on grouped key-value collections and write results directly to HDFS.

6. The Hadoop Ecosystem at a Glance

The broader Apache Hadoop ecosystem includes high-level analytical tools that build on top of core storage and resource management:

• Apache Hive: Data warehouse infrastructure that facilitates reading, writing, and managing large datasets using a SQL-like dialect (HiveQL).
• Apache Pig: High-level platform for creating complex data transformation workflows using a procedural dataflow language called Pig Latin.
• Apache HBase: Distributed, scalable, column-oriented NoSQL database built directly on top of HDFS, providing real-time, random read/write access.
• Apache Spark: In-memory distributed computational engine capable of running batch, interactive, streaming, and machine learning workloads up to 100x faster than MapReduce.
• Apache Sqoop: Command-line tool designed for efficiently transferring bulk data between relational databases (MySQL, Oracle) and HDFS/Hive.
• Apache Flume: Distributed service for efficiently collecting, aggregating, and moving large amounts of streaming log data into HDFS.
• Apache ZooKeeper: Centralized coordination service providing distributed configuration management, naming registries, synchronization, and leader election.
• Apache Oozie: Server-based workflow scheduler system to manage and chain Apache Hadoop data processing jobs.

7. Hadoop Deployment Modes and Prerequisites

7.1 System Prerequisites

• Java Development Kit (JDK): Hadoop is written in Java and requires JDK 8 or JDK 11. Environment variable JAVA_HOME must be explicitly configured in hadoop-env.sh.
• SSH (Secure Shell): Passwordless SSH authentication between master and worker nodes is mandatory for cluster control scripts (start-dfs.sh, start-yarn.sh).
• Operating System: Production Hadoop clusters run on Linux (CentOS, RHEL, Ubuntu, Rocky Linux). Windows deployments typically rely on WSL or virtualized containers.

7.2 Deployment Modes

7.3 Essential Configuration Files and CLI Commands

• core-site.xml: Configures core parameters such as the default file system URI (fs.defaultFS).
• hdfs-site.xml: Configures block replication factor (dfs.replication) and storage directories for NameNode/DataNode.
• yarn-site.xml: Configures YARN ResourceManager hostname, memory limits, and auxiliary shuffle services.
• mapred-site.xml: Specifies framework execution provider (mapreduce.framework.name set to yarn).
• Essential Verification Commands:

  bashCopy Code
  # Check Hadoop binaries and version
  hadoop version
  
  # Check status of Java processes running
  jps
  
  # Verify HDFS cluster storage health and capacity
  hdfs dfsadmin -report
  
  # Format NameNode filesystem namespace before first cluster boot
  hdfs namenode -format