Share E-Book
Scan to open this page

Scan with your phone to open this page

AuthorBill Chambers, Matei Zaharia

Learn how to use, deploy, and maintain Apache Spark with this comprehensive guide, written by the creators of the open-source cluster-computing framework. With an emphasis on improvements and new features in Spark 2.0, authors Bill Chambers and Matei Zaharia break down Spark topics into distinct sections, each with unique goals. You’ll explore the basic operations and common functions of Spark’s structured APIs, as well as Structured Streaming, a new high-level API for building end-to-end streaming applications. Developers and system administrators will learn the fundamentals of monitoring, tuning, and debugging Spark, and explore machine learning techniques and scenarios for employing MLlib, Spark’s scalable machine-learning library. Get a gentle overview of big data and Spark Learn about DataFrames, SQL, and Datasets—Spark’s core APIs—through worked examples Dive into Spark’s low-level APIs, RDDs, and execution of SQL and DataFrames Understand how Spark runs on a cluster Debug, monitor, and tune Spark clusters and applications Learn the power of Structured Streaming, Spark’s stream-processing engine Learn how you can apply MLlib to a variety of problems, including classification or recommendation

AI Reading Assistant

Whole-book reading guide from stratified index samples; jump to passages in the text

Brief outline
【One-Line Pitch】 Learn how to use, deploy, and maintain Apache Spark with this comprehensive guide, written by the creators of the ope… 【Book Arc】 - **Opening (~0%–12%)**: This can be one of three core cluster managers: Spark’s standalone cluster manager, YARN, or Mesos.; d see drastically different runtimes. - **Early (~12%–35%)**: single structured API query from user code to executed code.; "Croatia") .show(2) # in Python df.where(col("count") 2).where(col("ORIGIN_COUNTRY_NAME") ! - **Middle (~35%–65%)**: In a DataFrame, each column has a unique ID within Spark’s SQL Engine, Catalyst.; l work in a similar way using MySQL or PostgreSQL, as well. - **Late (~65%–88%)**: This method is accessible within serialized functions without having to serialize the data.; is one big partition in our dataset, we will have one task. - **Ending (~88%–100%)**: You can find up-to-date information about how to use this tool in the Spark documentation.; d into a UDF or function, or if you’re working with strange data types that cannot be serialized. 【Key Takeaways】 - **This can be one of thr…** (Opening): This can be one of three core cluster managers: Spark’s standalone cluster manager, YARN, or Mesos. - **d see drastically diff…** (Opening): d see drastically different runtimes. - **Spark’s structured API…** (Opening): Spark’s structured APIs and run them in a streaming fashion. - **single structured API…** (Early): single structured API query from user code to executed code. - **"Croatia") .show(2) #…** (Early): "Croatia") .show(2) # in Python df.where(col("count") 2).where(col("ORIGIN_COUNTRY_NAME") ! - **----------------------…** (Early): ---------------------------------- We can also do this quite easily in Python. 【Reading Tips】 - Use Passage locations below to jump into the text and set reading anchors - If this is a brief outline, click Regenerate (top right) for a synthesized guide 【Coverage Limits】 Compressed outline without the model (~33 index chunks). Full structured guide needs AI available.
Excerpt 1
t. In this diagram, we removed the concept of cluster nodes. The user can specify how many executors should fall on each node through configurations. NOTE Sp...
View in text
Excerpt 2
k.sql.types.{StructField, StructType, StringType, LongType} val myManualSchema = new StructType(Array( new StructField("some", StringType, true), new StructF...
View in text
Excerpt 3
t way to explain this is to begin performing some groupings. The first will be a count, just as we did before. We will group by each unique invoice number an...
View in text
Excerpt 4
maximum number of concurrent numPartitions JDBC connections. If the number of partitions to write exceeds this limit, we decrease it to this limit by calling...
View in text
Excerpt 5
y a fixed-size random sample from your RDD. You can specify whether this should be done by using withReplacement, the number of values, as well as the random...
View in text
Excerpt 6
the intermediate results to memory or disk after each step. The same kind of pipelining happens for a DataFrame or SQL computation that does a select, filter...
View in text
Excerpt 7
This includes monitoring things like CPU, network, and I/O. These are often reported in cluster-level monitoring solutions; however, there are more specific...
View in text
Excerpt 8
such as the Vectorized UDF extension for Python that gives your code multiple records at once using a Pandas data frame. We discussed UDFs and their costs in...
View in text
Tags
AI categories
Framework
ISBN: 1491912219
Publisher: O’Reilly Media
Publish Year: 2018
Language: English
Pages: 606
File Format: PDF
File Size: 7.9 MB
Text Preview (First 20 pages)
Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Generating text preview…