Hadoop in Practice, 2nd Edition (Alex Holmes) (Z-Library)

# Hadoop in Practice, 2nd Edition — Reading Guide ## 【One-Line Pitch】 A practical, recipe-driven handbook for engineers who need to get real work done with Hadoop 2.x — covering YARN, MapReduce 2, data serialization, HDFS optimization, and ecosystem integrations like Kafka, Impala, and Spark SQL. If you're a Java developer or data engineer who learns best by adapting tested code to your own problems, this is your working reference. ## 【Book Arc】 - **Opening (~0%–10%)**: The book opens with its core promise — 104 tested techniques for Hadoop 2 — and immediately establishes the "In Practice" format: each technique follows a Problem → Solution → Discussion structure. Early chapters introduce the YARN architecture (ResourceManager, NodeManager, ApplicationMaster, containers) and explain how MapReduce 2 differs from the deprecated MapReduce 1 model, including a detailed walkthrough of deprecated properties and their replacements. - **Early (~10%–23%)**: The focus shifts to hands-on MapReduce development. You'll see complete, annotated code examples — from a word/document-ID inverted index (with setup(), map(), and reduce() methods explained line-by-line) to understanding how InputFormat, RecordReader, Partitioner, and RecordWriter fit together in the data flow. This section also covers YARN log aggregation, showing how container logs are copied to HDFS and retrieved via the `yarn logs` command. - **Middle (~23%–39%)**: Data serialization becomes the central theme. The book walks through working with text formats (including XML via StAX parsing), then moves into SequenceFiles — explaining how to create custom Writable classes, why SequenceFiles are splittable (synchronization markers every ~6 KiB), and how to encode Protocol Buffers by registering custom serializers in the `io.serializations` property. - **Middle (~39%–48%)**: Avro takes center stage. Techniques cover choosing the right Avro usage pattern in MapReduce, mixing Avro and non-Avro data, working with Avro key/value pairs, controlling sort behavior, and integrating Avro with Hive and Pig. The book also introduces columnar storage with Parquet — reading files via command line, using Parquet with MapReduce, Hive/Impala, and leveraging pushdown predicates and projection for performance. - **Late (~48%–end)**: The remaining techniques cover custom file formats (writing input/output formats for CSV, with attention to output committing), HDFS data organization (directory layout, data tiers, partitioning via MultipleOutputs and custom partitioners), file compaction with FileCrush, storing small binary files in Avro, atomic data movement, and compression codec selection. ## 【Key Takeaways】 - **YARN is a fundamentally different resource model** (Early): MapReduce 1's static "slots" are gone; YARN uses dynamic containers managed by the ResourceManager and per-application ApplicationMasters. Understanding this shift is essential for tuning and debugging Hadoop 2 clusters. - **Deprecated properties will eventually break your jobs** (Early): Most `mapred.*` properties have been renamed (e.g., `mapred.cache.files` → `mapreduce.job.cache.files`). Hadoop 2 still supports both, but Hadoop 3 likely won't — run a job and check the deprecation dump on stdout to future-proof your configs. - **The InputFormat contract drives everything** (Early): Every MapReduce job must satisfy three InputFormat contracts — key/value type information, input splitting, and record reading. Master this and you can feed any data source into MapReduce. - **SequenceFiles solve the small-files and complex-types problem** (Middle): They support compression, are splittable via periodic sync markers, and can model nested data structures through custom Writable classes — a solid default when text files get awkward. - **Custom serialization requires framework registration** (Middle): To use Protocol Buffers (or any custom serializer) with Hadoop, you must implement Serializer/Deserializer classes and append them to the `io.serializations` property in core-site.xml — a pattern that generalizes to other serialization frameworks. - **Avro gives you schema evolution plus MapReduce integration** (Middle): The book shows multiple integration patterns — Avro-specific input/output formats, key/value pair formats, and mixing Avro with plain Writables — plus how to wire schemas into Hive and Pig. - **Parquet's columnar format is a performance lever** (Middle): Pushdown predicates and projection let you skip irrelevant data at the storage layer. Pairing Parquet with Hive/Impala can dramatically cut query times compared to row-oriented formats. - **Output committing is the hidden complexity in custom formats** (Middle): When writing custom OutputFormats (e.g., for CSV), the book stresses that proper output committing — handling partial failures and cleanup — is what separates production-ready code from prototypes. ## 【Reading Tips】 - **Skim the YARN architecture chapter** (~0%–10%) if you're already running Hadoop 2 — focus on the deprecated properties table and the log aggregation flow, which are the most practically useful parts. - **Deep-read the MapReduce data flow section** (~10%–13%): The annotated code walkthroughs (especially the inverted index example) are the best way to internalize how setup(), map(), reduce(), and the framework's object reuse work together. - **Treat serialization chapters as a decision guide** (Middle): Don't read every Avro/Parquet technique end-to-end. Instead, skim the "Problem" statements and jump to the technique that matches your data shape — text, key/value, nested records, or columnar. - **Watch for the GitHub source references**: Many techniques point to the `hiped2` repository (github.com/alexholmes/hiped2). Clone it before reading — you'll want to run and modify the code rather than just reading listings. - **Skip the Mahout chapter if you're not doing ML**: The excerpts don't cover it in detail, and if your work is data engineering rather than machine learning, the earlier serialization and HDFS chapters will serve you better. ## 【Coverage Limits】 This guide is based on sampled excerpts covering roughly the first half of the book (through ~48%). The later techniques on Flume, Sqoop, Mahout, Kafka, Impala, and Spark SQL integration are not covered in the source material provided. ##