Coordination in Distributed Systems

A monolithic application operates within a single memory space and maintains a unified source of truth. Distributed systems, on the other hand, gain scalability and resilience by spreading work across multiple machines. However, this introduces a core challenge: how can many independent machines maintain a consistent view of shared state?

Without a reliable way to coordinate, the system can run into serious problems:

• It may not know which machine should handle certain tasks.

• Different machines might overwrite each other’s data.

• Some machines might not even realize when others have been disconnected or stopped working.

This challenge is called the coordination problem.

It sits at the core of distributed systems and frequently appears in System Design interviews because it directly impacts a system’s reliability and correctness. In this lesson, we’ll examine the key building blocks that enable distributed services to coordinate, transforming a set of independent machines into a powerful, unified system.

Introduction to distributed system coordination

When we break an application into distributed services, we gain fault tolerance and scalability.

However, these services must still collaborate. For instance, a cluster of database replicas needs to perform a leader election to agree on which node is the primary writer. A set of workers processing a queue needs to avoid processing the same job twice, which requires careful collaboration.

This act of getting multiple nodes to agree on a state or a course of action is called coordination. It often involves state replication, which ensures that all nodes have the same data, and heartbeats, which are regular signals nodes send to confirm they are alive and reachable.

The primary challenge in distributed coordination lies in balancing consistency, availability, and partition ...