← Back to the chapter · Next detail →
Technical detail · Big Data Technologies
Hadoop storage and worker responsibilities
Hadoop is an open-source distributed data platform. Hadoop divides storage, processing, resource management, and shared utilities. Hadoop Common supplies shared libraries. Hadoop Distributed File System (HDFS) stores large files as replicated blocks across worker machines. MapReduce runs batch jobs over those blocks, and YARN allocates CPU, memory, and scheduling capacity to cluster applications.
In HDFS, the NameNode keeps filesystem metadata: file names, directories, permissions, block locations, and mappings from logical files to physical blocks. DataNodes store and serve actual block replicas, periodically reporting health and block inventory to the NameNode. Metadata locates evidence; replicated blocks preserve historical telemetry, logs, images, and model-training files for batch analysis.
Legacy Hadoop MapReduce v1 similarly separated coordination from execution. The JobTracker accepted jobs, split them into map and reduce tasks, assigned work, and tracked progress. TaskTrackers ran tasks on worker nodes near the data. Modern stacks often replace those exact roles with YARN, Spark, Flink, or managed services. The review questions remain: who owns scheduling, who owns data placement, and how does recovery handle a worker failure?
Figure 3 makes the metadata and storage ownership split concrete. Durable bytes and the metadata needed to find them fail in different ways.
In Figure 3, the replication diagram places NAMENODE - filesystem metadata above mappings from /users/sensors/part-0.log and part-1.log to numbered blocks and replica counts. Below, DataNode A through DataNode H hold the coloured block replicas themselves, while the heartbeat note shows DataNodes reporting inventory and health back to the NameNode. Replication therefore protects block bytes across storage workers; it does not make namespace and placement metadata irrelevant. The broader technology record must name both responsibilities, their recovery evidence, and the governance controls that make a replayable dataset findable.