Building Scalable Data Pipelines with Kafka

Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

Beginner

62 Lessons

Certificate of Completion

Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

8 Playgrounds

This course includes

8 Playgrounds

Course Overview

If you’re interested in Big Data, then Apache Kafka is a must-know tool. What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. This course introduces you to Kafka theory and ...Show More

What You'll Learn

Learn the theory behind Kafka

Interact with a Kafka cluster running in the browser-terminal

What You'll Learn

Learn the theory behind Kafka

Course Content

Basics

Step through the fundamentals of Kafka, distributed systems, messaging patterns, and core components.

Introduction

Characteristics of Distributed Systems

Messaging Patterns

Commit Log

Components of Kafka Partitions Partition Rebalancer

Kafka Producer

Unpack the core of Kafka Producers, message sending methods, configurations, and serialization techniques.

Producer Sending Messages Producer Configurations Producer Serialization

Kafka Consumer

Go hands-on with Kafka consumers, configurations, offsets, and partition rebalancing techniques.

Consumer and Consumer Groups Kafka Consumer Consumer Configurations Commits and Offsets Offset Commit Configuration Handling Rebalances Stopping a Consumer Consumer Deserializer Running a Single Consumer

Kafka Internals

Break down complex ideas of Kafka's replication, controller, request processing, and reliability.

Replication Kafka Controller Request Processing Partition Allocation Data Storage Reliability in Kafka Reliable Producers and Consumers

Conclusion

Compare Kafka's scalability, throughput, and real-time processing with other messaging systems.

Kafka vs Other Messaging Systems

Appendix

3 Lessons

Activate Zookeeper insights, practical API use, and common distributed system solutions.

Reference: Replication

14 Lessons

Master the principles of replica management, leader-based and leaderless replication strategies, and conflict resolution methods.

Reference: Partitioning

4 Lessons

Learn how to use partitioning strategies to enhance scalability and optimize data pipelines.

Reference: Transactions

9 Lessons

Discover the logic behind managing data transactions, isolation levels, and concurrent write challenges.

10.

Reference: Issues in Distributed Systems

4 Lessons

Examine the challenges in developing and maintaining distributed systems, including networking, time synchronization, and handling failures.

Course Author

Trusted by 1.4 million developers working at companies

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Evan Dunbar

ML Engineer

Carlos Matias La Borde

Software Developer

Souvik Kundu

Front-end Developer

Vinay Krishnaiah

Software Developer

Eric Downs

Musician/Entrepeneur

Kenan Eyvazov

DevOps Engineer

Anthony Walker

@_webarchitect_

Emma Bostian 🐞

@EmmaBostian

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!