Map-Reduce Framework
Fault-Tolerant Distributed Data Processing Framework
Highlights
- End-to-end word count pipeline completed in <1s across multi-file corpora
- 3-state coordinator tracking with 10-second heartbeat failure detection
- Atomic intermediate commits ensuring zero data loss during worker crashes
System Overview
A Go implementation of the MapReduce distributed execution model featuring an RPC-based coordinator, worker nodes, state tracking, and crash recovery mechanics.
The Problem & Architectural Rationale
Processing large datasets across worker nodes requires coordinating Map -> Shuffle -> Reduce phases while handling worker crashes, network partitions, and straggler delays.
Architecture & Components
1. Master Coordinator
Assigns map and reduce tasks to idle workers, tracking task states across Idle, In-Progress, and Completed with low-overhead RPC heartbeats.
2. Failure Detection & Straggler Handling
Detects worker failures within a 10-second heartbeat timeout window, re-queuing in-flight tasks to healthy workers.
3. Atomic Output Commits
Workers write intermediate chunks to temporary files and atomically commit them via os.Rename to ensure idempotency and prevent corrupted partial reads.
Outcomes & Results
- Processed Project Gutenberg text datasets with sub-second full word-count indexing.
- Verified deterministic results across simulated worker SIGKILL tests.