Flink入门与实战 (徐　葳) (Z-Library)

# Flink入门与实战 — Reading Guide ## 【One-Line Pitch】 A hands-on, code-first introduction to Apache Flink 1.6 that walks you from installation through production-grade features like HA clusters, Table API/SQL, and real-world ETL projects — ideal for Java/Scala developers new to stream processing who want to build working Flink applications rather than just learn theory. ## 【Book Arc】 - **Opening (~0%–4%)**: Flink's origins (Berlin TU research project → Apache top-level project), core characteristics (distributed, high-performance, HA, accurate), and the key insight that batch is just a special case of streaming. Covers the three data transmission models and typical use cases like real-time ETL. - **Early (~4%–14%)**: Quick-start chapter with development environment setup (IntelliJ IDEA, Maven with Aliyun mirror), a socket-based word count streaming example, then moves into installation — local mode, Standalone cluster, and Flink on YARN with detailed command analysis and JAR packaging via Maven. - **Early (~14%–25%)**: Deep dive into High Availability — ZooKeeper cluster setup, Standalone HA configuration across multiple nodes, and Flink on YARN HA leveraging YARN's task recovery plus checkpoint metadata in ZooKeeper. Ends with the Scala Shell for interactive experimentation. - **Middle (~25%–46%)**: The API core — DataStream sources (socket, collection, custom, connectors like Kafka/RabbitMQ/Redis), transformations (Map, FlatMap, KeyBy, Reduce, Union, Connect, Split/Select), partitioning strategies, sinks (Redis, files), then DataSet APIs and the relational layer: Table API and SQL with DataStream/DataSet/Table conversions. - **Middle (~46%–50%+)**: Advanced features — serialization options (Avro, Kryo, custom), broadcast variables for distributing lookup data, and accumulators for reliable counters across parallel tasks. Excerpts indicate these are covered with both Java and Scala examples. ## 【Key Takeaways】 - **Batch is a special case of streaming** (Opening): Flink treats all data as streams, with batch being the limit case — this unified model simplifies architecture and lets you use one framework for both processing styles. - **Cache timeout is your latency/throughput dial** (Opening): The buffer timeout value (0 to infinity) trades latency against throughput — set to 0 for lowest latency, infinity for maximum throughput, or anywhere between for custom balance. - **YARN deployment handles resource negotiation automatically** (Early): Flink on YARN uploads config and JARs to HDFS, starts JobManager+AM in one container, then dynamically allocates TaskManager containers — enabling parallel YARN sessions with port offsets per application. - **HA requires ZooKeeper for both Standalone and YARN modes** (Early): Standalone HA uses ZooKeeper for JobManager failover; YARN HA additionally relies on YARN's recovery mechanism plus checkpoint metadata stored in ZooKeeper — so ZooKeeper is non-negotiable for production. - **Connectors cover the ecosystem, but not symmetrically** (Early): Kafka, Kinesis, RabbitMQ, and NiFi provide both Source and Sink support; Cassandra, Elasticsearch, HDFS, and Redis are Sink-only; Twitter is Source-only — plan your data flows accordingly. - **Table API and SQL lower the barrier dramatically** (Middle): You can operate on data like MySQL tables without writing Java functions or manual tuning — SQL is non-programmer friendly and easy to adopt, though the API was still actively evolving with some unsupported features. - **Broadcast variables and accumulators solve distributed data problems** (Middle): Broadcast variables distribute lookup data to all tasks via RichFunction's open() method; accumulators provide accurate cross-parallelism counting that plain local variables can't guarantee. ## 【Reading Tips】 - **Skim Chapter 1** if you already know stream processing concepts — the key insight is the batch-as-stream model and the cache timeout trade-off; the rest is standard framework positioning. - **Deep-read Chapter 3 (installation/deployment)** if you'll run Flink yourself — the HA sections are the most valuable, showing real multi-node ZooKeeper + Flink setup with verification steps; the YARN internals explanation is worth studying even if you use a different deployment. - **Use Chapter 4 as a reference, not a cover-to-cover read** — the API sections are organized by category (Source/Transformation/Sink), so jump to what you need; the Java/Scala dual examples make it easy to follow in your preferred language. - **Pay attention to the custom Source example** (Early ~25%): the run()/cancel() pattern with a loop is the template for all custom sources — master this and you can integrate any data source. - **The final chapters (real-time ETL and reporting projects)** are where it all comes together — if you're short on time, prioritize these to see how the pieces fit in production scenarios. ## 【Coverage Limits】 This guide covers the book's first half in detail (through advanced features like broadcast variables and accumulators). The final two project chapters (real-time ETL and real-time reporting) are mentioned in the preface but not covered in the excerpts — expect hands-on integration of everything learned earlier. ##