Concurrent Data Processing in Elixir

Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

Intermediate

78 Lessons

11h 10min

Certificate of Completion

Gain insights into data processing and concurrency in Elixir. Explore GenServer, GenStage, Flow, and Broadway to enhance performance, set up pipelines, and handle increased workloads efficiently.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

49 Playgrounds

7 Quizzes

This course includes

49 Playgrounds

7 Quizzes

Course Overview

Most engineers don’t think that concurrency is distinct from programming. Data processing includes transforming information in some way, like reporting, data aggregation, or analytics. On the other hand, concurrency involves significantly improving performance. This course is designed to help you navigate the world of concurrency tools available in the Elixir ecosystem. You’ll explore popular modules and libraries like GenServer, GenStage, Flow, and Broadway. You’ll be able to ingest data, set up pipelines...Show More

What You'll Learn

Understand how concurrency works in Elixir

Learn about Elixir supervisors and GenServers

Learn to use data processing pipelines

Learn to use the Flow module in Elixir

Learn to set up data ingestion pipelines

What You'll Learn

Understand how concurrency works in Elixir

Course Content

Getting Started

Get familiar with concurrent data processing in Elixir and its prerequisites.

What Does This Course Cover?

Course Prerequisites and Intended Audience

Easy Concurrency with the Task Module

Unpack the core of concurrent programming in Elixir with the Task module and supervision.

Introduction to Concurrency

An Introduction to the Task Module

Create the Playground Start Tasks and Retrieve Results Synchronous vs. Asynchronous Programs Retrieve a Task’s Results Use the await and yield Functions to Retrieve Results Stop a Task and Async Task Manage a Series of Tasks Parameters of the async_stream Function Link Processes Meet the Supervisor Use Task.Supervisor Understand the “Let It Crash” Philosophy Wrapping Up Quiz: Easy Concurrency with the Task Module

Long-Running Processes Using GenServer

Work your way through GenServer callbacks, process management, and optimizing concurrency in Elixir.

Long-running Processes and Starting GenServer GenServer Callbacks: init GenServer Callbacks: handle_continue and handle_call Genserver Callbacks: handle_cast GenServer Callbacks: handle_info GenServer Callbacks: terminate Make Improvements to Our GenServer The blueprint of our job-processing system How to Use Anonymous Functions Introduce DynamicSupervisor Process Restart Values and Frequency Implement a Supervisor Supervisor Strategies Registry Processes Register New Processes and Query the Registry Limit Concurrency and Inspect the Supervisors Examine Running Processes and Stop the Supervisors Wrapping Up Quiz: Long-running Processes Using GenServer

Create a GenServer and Supervisor From Scratch

Project

Data Processing Pipelines with GenStage

Find out about GenStage's role in managing data pipelines, back-pressure, and concurrency.

Back-Pressure Introduce GenStage The Producer and Consumer Build the Data Processing Pipeline Understand Consumer Demand Revisiting the Producer Add More Consumers and Buffer Events Add Concurrency With ConsumerSupervisor Create Multi-stage Data Pipelines Scale Up a Stage with Extra Processes Choose the Right Dispatcher Wrapping Up Quiz: Data Processing Pipelines with GenStage

Process Collections with Flow

Take a closer look at concurrent data processing with Elixir's Flow library for large datasets.

Introduction to the Flow Library Read Files Improve Performance with Streams Run Streams Concurrently and Understand Flows The reduce Function Route Events with Partitions Group and Sort Windows and Triggers Get Snapshots from Slow-running or Infinite Flows Add Flow to a GenStage Pipeline Wrapping Up Quiz: Process Collections with Flow

Data Ingestion Pipelines with Broadway

15 Lessons

See how Broadway simplifies data-ingestion pipelines with concurrent processing, batching, and fault tolerance.

Concluding the Course

1 Lesson

Learn how to improve your ability to create efficient, scalable Elixir applications.

Course Author

Trusted by 1.4 million developers working at companies

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Evan Dunbar

ML Engineer

Carlos Matias La Borde

Software Developer

Souvik Kundu

Front-end Developer

Vinay Krishnaiah

Software Developer

Eric Downs

Musician/Entrepeneur

Kenan Eyvazov

DevOps Engineer

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!