Design Sharded Counters
Explore the design of sharded counters to efficiently manage high-volume write and read requests in systems like Twitter. Understand shard creation, write distribution strategies, read aggregation, and how this approach solves contention and scalability challenges while enabling high availability and reliability in large-scale applications.
High-level solution sketch
Managing millions of Tweet likes requires many counters operating on many nodes. To manage these counters, we need an efficient system that can provide high performance and scalability as the number of users grows.
What will happen when a single Tweet on Twitter gets a million likes, and the application server receives a write request against each like to increment the relevant counter? These millions of requests are eventually serialized in a queue for data consistency. Such serialization is one way to deal with concurrent activity, though at the expense of added delay. Real-time applications want to keep the quality of experience high by providing as minimum as possible latency for the end user.
Let’s see the illustration below to help us understand this problem:
A single counter for each Tweet posted by a celebrity is not enough to handle millions of users. The solution to this problem is a sharded counter, also known as a distributed counter, where each counter has a specified number of shards as needed. These shards run on different computational units in parallel. We can improve performance and reduce contention by balancing the millions of write requests across shards.
First, a write request is forwarded to the specified Tweet counter when the user likes that Tweet. Then, the system chooses an available shard of the specified Tweet counter to increment the like count. Let’s look at the illustration below to see sharded counters having specified shards:
In the illustration above, the total number of shards per counter is . We’ll use an appropriate value for ...