AI Features

0% completed

All LessonsFree Lessons (2)

Course Introduction

About the Course Course Structure

Spark Introduction and Basics

Spark Fundamentals Components and Architecture Spark and Big Data Spark's Java Main Abstraction: The DataFrame Quiz Yourself: Spark Introduction

Getting Started with Spark

Running the First Spark Program Spark Maven Based Projects Enriching the Basic DataFrame Program Deep Dive: Transformations and Data Storage Quiz Yourself: Getting Started with Spark

DataFrame Basic Operations

Working with DataFrame's Schemas

Dataset: a DataFrame of POJOs

Transformations and Actions

Transformations (I): Map and Filter

Actions (I): Count, Take, and Collect

Deep Dive: Internals of Spark Execution

Transformations (II): FlatMap and Distinct

Actions (II): Reduce and Aggregate Functions: Max, Min, and Mean

Quiz Yourself: Schema of a DataFrame

DataFrame Advanced Operations

Data Partitioning and Shuffling The groupBy and groupByKey methods Joins Sort and OrderBy Union, UnionByName, and DropDuplicates Accumulators and Broadcast Variables UDFs: User-defined Functions Quiz Yourself: DataFrame Advanced Operations

Spark SQL and Other Functionalities

Spark SQL Goodness Schema Manipulation How to Ingest Files: Part I How to Ingest Files: Part II Ingesting Databases Exporting Information Serialization: Working through the Wire Quiz Yourself: SparkSQL and Other Functionalities

Building a Big Data Batch Application

The Application Architecture Ecosystem Driver Program Design and Project Structure Driver Program and Job Implementation: Part I Driver Program and Job Implementation: Part II Ingestion Job: Part I Ingestion Job: Part II Batch Pipelines and Other Types of Jobs Testing and Spark

Deployment and Cluster Execution

Local and Cluster-based Execution Deploying and Running a Spark Application Quiz Yourself: Deployment and Cluster Execution

Monitoring and Performance Fundamentals

Interpreting Spark Logs Cluster Monitoring and SparkUI Performance Fundamentals and Recipes Quiz Yourself: Monitoring and Performance Fundamentals

Conclusion

Where to Go from Here

Apendix

Tools to Develop in Spark Locally IntelliJ: Debug and Inspect Spark Execution

Mastering Big Data with Apache Spark and Java

Quiz Yourself: Schema of a DataFrame

Quick quiz on the operations of a DataFrame to inspect and modify its schema.

We'll cover the following...