Rate Limits

What is a Rate Limit?

Rate limiting is the process where an IP address is given a limited set of queries to make in a time period. Once that quota is exceeded the service will reject any further requests. The client should then sleep or refuse to make any more requests until a specified time period has passed.

A Rate Limit Response

Ensembl REST informs you of service state using HTTP headers. There is a subset starting with X-RateLimit- showing what the current rate limit state is. A normal response looks like the following snippit

X-RateLimit-Limit: 55000
X-RateLimit-Reset: 892
X-RateLimit-Period: 3600
X-RateLimit-Remaining: 54999

Here we can see we are allowed 55000 requests over an hour (3600 seconds) meaning we can perform an average of 15 requests per second. We can also see my client has used just 1 request and is 892 seconds (nearly 15 minutes) away from having its quota reset. Calculating the reset date/time is relatively easy by adding the epoch time and converting seconds to milliseconds.

var resetDate = new Date(Date.now()+(892*1000));

A Maxed Out Rate Limit Response

Once you exceed the rate limit you will be shown a response like so:

Retry-After: 40.0
X-RateLimit-Limit: 55000
X-RateLimit-Reset: 40
X-RateLimit-Period: 3600
X-RateLimit-Remaining: 0

The Retry-After header is a message from the REST API server telling the client you must not query the service for this allotted number of seconds. Any client should then refuse to query the service until this period has been exceeded. This header is accompanied with a 429 HTTP response code. Some clients like jQuery's ajax method support callbacks based upon HTTP response code. This is the best way to respond to such an event. Since the header value is a floating point second value it should be trivial to pass this into any sleep function. You can see an example of sleeping in Perl below.

# Need to bring in HiRes sleep as we can be told to sleep for a portion of a second
use Time::HiRes qw/sleep/;
sleep(1.013293);