System Design: Sequencer

Motivation

Large distributed systems can process millions of events per second, such as user posts or financial transactions. Each event must be assigned a globally unique identifier. These identifiers typically serve as primary keys in storage systems. Single-node databases often use auto-increment columns, but this approach does not scale in distributed systems where multiple nodes generate identifiers independently. Distributed environments, including horizontally sharded tables, require a coordinated or decentralized strategy for generating a unique ID.

Additionally, unique IDs facilitate debugging and tracing. For example, Facebook’s CanopyAn End-to-End Performance Tracing and Analysis System uses a TraceID to track events across the hundreds of microservices involved in a single user request.

How do we design a sequencer?

We divide the comprehensive design of a sequencer into two lessons:

1. Design of a unique ID generator: Define system requirements and discuss three generation methods: UUIDs, databases, and range handlers.

2. Unique IDs with causality: Incorporate time as a factor in ID generation to handle causality.

Designing a unique ID generator for a distributed system is challenging. The next lesson examines the specific requirements for this system.