Requirements of a Rate Limiter’s Design

Requirements

Our focus in this lesson is to design a rate limiter with the following functional and non-functional requirements.

Functional requirements

• To limit the number of requests a client can send to an API within a time window.

• To make the limit of requests per window configurable.

• To make sure that the client gets a message (error or notification) whenever the defined threshold is crossed within a single server or combination of servers.

Non-functional requirements

• Availability: Essentially, the rate limiter protects our system. Therefore, it should be highly available.

• Low latency: Because all API requests pass through the rate limiter, it should work with a minimum latency without affecting the user experience.

• Scalability: Our design should be highly scalable. It should be able to rate limit an increasing number of clients’ requests over time.

Types of throttling

A rate limiter can perform three types of throttling.

1. Hard throttling: This type of throttling puts a hard limit on the number of API requests. So, whenever a request exceeds the limit, it is discarded.

2. Soft throttling: Under soft throttling, the number of requests can exceed the predefined limit by a certain percentage. For example, if our system has a predefined limit of 500 messages per minute with a 5% exceed in the limit, we can let the client send 525 requests per minute.

3. Elastic or dynamic throttling: In this throttling, the number of requests can cross the predefined limit if the system has excess resources available. However, there is no specific percentage defined for the upper limit. For example, if our system allows 500 requests per minute, it can let the user send more than 500 requests when free resources are available.