Your Workflow Worked Fine in Staging. Then Production Hit a 429.
You shipped the integration on Friday. The staging tests passed. Monday morning the on-call engineer wakes to a Slack storm: the CRM sync pipeline is dead, the payment webhook processor is stuck, and the analytics dashboard shows zero new data since 3 a.m. The root cause? An HTTP 429 from the upstream API your entire data flow depends on.
This is the most common failure mode in production integrations, and it is entirely preventable. API rate limiting is not a bug you work around once. It is a permanent architectural constraint you design for from day one.
This runbook covers how rate limits work at the protocol level, how to handle 429 responses gracefully, and how to build workflows that degrade smoothly instead of collapsing when an upstream service enforces its quota.
How API Rate Limits Actually Work
Every public API enforces some form of throttling. Understanding the mechanism matters because the wrong mental model leads to wrong retry logic.
Fixed Windows vs. Sliding Windows
The simplest scheme is a fixed window: the provider resets your quota at the top of every minute (or hour, or day). If the limit is 100 requests per minute and you send request 101 at second 59, you wait one second and get a fresh quota.
A sliding window is smoother. The provider looks at a rolling time period, say, the last 60 seconds, and counts how many requests you made within it. You cannot game the reset boundary because there is no boundary.
Token Buckets and Burst Allowance
Many providers layer a token bucket on top. You get a steady refill rate (e.g., 10 requests/second) and a burst capacity (e.g., 50). If you idle for a few seconds, you accumulate tokens and can briefly exceed your steady rate. This is why a workflow that paces itself at 10 req/s sometimes succeeds with a 15-request burst, then fails 30 seconds later when the bucket runs dry.
Per-Key vs. Per-IP vs. Per-Endpoint
Limits can apply at different scopes:
- Per API key: most common. All requests from your key share one pool.
- Per IP address: less common, but relevant when you deploy behind a NAT gateway or shared proxy.
- Per endpoint: some providers rate-limit expensive endpoints (e.g.,
/search) more aggressively than cheap ones (e.g.,/status).
Check the provider's documentation for all three dimensions before writing any retry logic.
Reading a 429 Response Properly
A 429 is not just a status code. It carries metadata you must use.
Standard headers to inspect:
Retry-After: the number of seconds (or an HTTP-date) the provider wants you to wait. This is the single most important header in your integration.X-RateLimit-Remaining: how many requests you have left in the current window.X-RateLimit-Reset: Unix timestamp when your quota resets.X-RateLimit-Limit: the total quota for the current window.
Not all providers implement all headers, and names vary. Stripe returns Retry-After on 429s. The GitHub API returns X-RateLimit-Remaining and X-RateLimit-Reset on every response, not just 429s. OpenAI's API returns a x-ratelimit-reset-requests header with the exact reset timestamp.
The key insight: read the Retry-After header and respect it. Blindly retrying after a fixed delay is how you turn a brief hiccup into a sustained outage.
Retry Strategy: Exponential Backoff with Jitter
The classic retry pattern has three parts:
- Exponential backoff: double the delay after each attempt (1 s → 2 s → 4 s → 8 s…).
- Jitter: add randomness so that ten concurrent clients do not all retry at the same instant and trigger another round of 429s.
- Cap: set a maximum delay so you do not wait ten minutes for a transient error.
Here is a TypeScript implementation that follows these rules:
interface RetryOptions {
maxRetries: number;
baseDelayMs: number;
maxDelayMs: number;
}
async function fetchWithRetry(
url: string,
options: RetryOptions = { maxRetries: 5, baseDelayMs: 1000, maxDelayMs: 30_000 }
): Promise<Response> {
for (let attempt = 0; attempt <= options.maxRetries; attempt++) {
const response = await fetch(url);
if (response.status !== 429) {
return response;
}
// Respect the Retry-After header if present
const retryAfter = response.headers.get('Retry-After');
let delayMs: number;
if (retryAfter) {
const parsed = Number(retryAfter);
delayMs = isNaN(parsed)
? new Date(retryAfter).getTime() - Date.now()
: parsed * 1000;
} else {
const exponential = options.baseDelayMs * Math.pow(2, attempt);
delayMs = Math.min(exponential, options.maxDelayMs);
}
// Jitter: delay between 50 % and 100 % of computed value
delayMs = delayMs * (0.5 + Math.random() * 0.5);
if (attempt === options.maxRetries) {
throw new Error(`Rate limit exceeded after ${options.maxRetries} retries`);
}
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
throw new Error('Unexpected: retry loop exited without returning');
}
The jitter formula 0.5 + Math.random() * 0.5 produces a delay between 50 % and 100 % of the computed backoff. A common alternative is "full jitter" (delayMs * Math.random()), which spreads retries more broadly but risks very short delays. The exact formula matters less than having one, without jitter, N concurrent clients hit the API in lockstep after every cooldown period, perpetuating the 429 storm.
Batching: Ten Requests in One Call
The most effective way to avoid rate limits is to send fewer requests. Batching turns N API calls into one.
When Batching Applies
Not every API supports batch endpoints, but many do:
- Salesforce accepts composite requests up to 25 sub-requests.
- Google APIs support batch HTTP endpoints that bundle multiple operations.
- OpenAI lets you combine prompts into a single multi-turn request instead of issuing separate calls.
A Practical Batching Pattern
Imagine you need to update 500 contact records in a CRM. Sending 500 individual PUT requests at 10 req/s takes 50 seconds of sustained traffic. If the provider limits you to 300 requests per minute, you hit a wall at the five-minute mark.
A batch approach:
async function batchUpdateContacts(
contacts: Contact[],
batchSize: number = 50
): Promise<void> {
const batches: Contact[][] = [];
for (let i = 0; i < contacts.length; i += batchSize) {
batches.push(contacts.slice(i, i + batchSize));
}
for (const batch of batches) {
await fetchWithRetry('/api/contacts/batch', {
method: 'PUT',
body: JSON.stringify({ records: batch }),
});
// Pace between batches to stay under the per-minute limit
const isLastBatch = batches.indexOf(batch) === batches.length - 1;
if (!isLastBatch) {
await new Promise((r) => setTimeout(r, 2000));
}
}
}
This reduces 500 requests to 10 batch calls with 2-second pauses, completing in roughly 18 seconds while consuming 10 API calls instead of 500. That is a 98 % reduction in request count and an immediate improvement in rate-limit headroom.
Pacing: The Token Bucket in Your Own Code
Retrying after you hit a 429 is reactive. Pacing is proactive, you throttle yourself before the API throttles you.
A simple in-process rate limiter uses a token bucket:
class TokenBucket {
private tokens: number;
private lastRefill: number;
constructor(
private capacity: number,
private refillRate: number // tokens per second
) {
this.tokens = capacity;
this.lastRefill = Date.now();
}
async acquire(): Promise<void> {
this.refill();
if (this.tokens >= 1) {
this.tokens -= 1;
return;
}
const waitMs = ((1 - this.tokens) / this.refillRate) * 1000;
await new Promise((r) => setTimeout(r, waitMs));
this.tokens = 0;
}
private refill(): void {
const now = Date.now();
const elapsed = (now - this.lastRefill) / 1000;
this.tokens = Math.min(
this.capacity,
this.tokens + elapsed * this.refillRate
);
this.lastRefill = now;
}
}
const limiter = new TokenBucket(20, 10); // burst 20, steady 10/s
async function pacedFetch(url: string): Promise<Response> {
await limiter.acquire();
return fetchWithRetry(url);
}
Every call to pacedFetch waits for a token before making the HTTP request. If the bucket is empty, the caller sleeps just long enough for a token to refill. This converts 429 errors from a recurring surprise into a non-event.
Best Practices: Five Rules for Production-Grade Rate Limit Handling
-
Always read
Retry-Afterfirst. If the provider tells you how long to wait, honour it. Building your own backoff logic on top of a server-specified delay creates unnecessary complexity and risk. -
Set a retry ceiling. Five retries with exponential backoff covers most transient limits. After that, fail gracefully: log the error, push the task to a dead-letter queue, and alert an operator. Endless retries turn a temporary throttling event into a permanent resource leak.
-
Decouple expensive operations from user-facing latency. If your workflow syncs data every five minutes, a 429 retry that adds 30 seconds matters far less than a 429 that blocks a user request in real time. Use queues to separate work that can wait from work that cannot. This is the same principle that applies to messaging systems like Kafka, where isolating consumers on shared infrastructure prevents one noisy tenant from degrading everyone.
-
Log rate-limit events with correlation IDs. Include the endpoint, status code,
Retry-Aftervalue, and attempt number in your structured logs. When a downstream problem surfaces hours later, you need to trace it back to the exact API call that triggered the throttle. -
Test your retry logic in CI, not in production. Write integration tests that mock 429 responses at varying rates. Verify that your backoff timing, jitter range, and fallback queue behave correctly. Injecting 429 errors in staging is far cheaper than discovering the gap at 3 a.m.
Multi-API Orchestration: When the Compound Limit Bites
A single API with a clear rate limit is manageable. The complexity explodes when your workflow touches multiple APIs sequentially or in parallel.
Consider a workflow that enriches a customer record:
- Fetch company data from Clearbit (hypothetical limit: 600 req/min).
- Look up the contact in HubSpot (hypothetical limit: 100 req/10s).
- Generate a summary via an LLM API (limit varies by model tier, as token-based cost models make clear).
Each step has its own limit, its own 429 format, and its own retry semantics. If step 2 throttles and you retry, step 1 may also throttle by the time you loop back. These cascading 429s are the most common way multi-service integration workflows fail.
The mitigation is orchestration-aware parallelism control. Instead of running steps 1-3 concurrently for every record, process records in controlled waves with a global concurrency semaphore:
async function withConcurrency<T>(
maxConcurrent: number,
queue: Array<() => Promise<T>>
): Promise<T[]> {
const results: T[] = [];
let index = 0;
async function worker() {
while (index < queue.length) {
const current = index++;
results[current] = await queue[current]();
}
}
const workers = Array.from({ length: Math.min(maxConcurrent, queue.length) }, () => worker());
await Promise.all(workers);
return results;
}
This caps total in-flight requests to maxConcurrent regardless of how many records your pipeline processes. It is a blunt instrument, but a reliable one.
For teams running these kinds of multi-API workflows at scale, the infrastructure decisions matter as much as the code. A custom software partner experienced in API integration can design the rate-limit-aware architecture around your specific provider mix, rather than retrofitting it after the first production incident.
When a Retry Is Not Enough: Dead-Letter Queues and Graceful Degradation
Even with perfect backoff and pacing, some requests will exceed your retry budget. The question is: what happens to them?
Dead-letter queues (DLQs) capture failed work for later reprocessing. When a 429 persists beyond your retry ceiling, instead of crashing or silently dropping the request, push it to a queue with metadata: original request payload, number of attempts made, last Retry-After value, and timestamp of final failure.
A separate process can drain the DLQ when the rate limit resets, or an operator can investigate if the failure indicates a broader issue, expired API key, misconfigured quota, or upstream outage.
Graceful degradation goes one step further. Instead of blocking the entire workflow on a failed API call, return a cached or default response. For example, if the enrichment API returns 429, serve stale data from your database and mark it as "last updated 6 hours ago." If the AI summary API is unavailable, skip the summary step and present the raw data instead. Both patterns transform a hard failure into a soft degradation, users see slightly stale data instead of an error page.
Takeaway: Rate Limits Are a Feature, Not a Bug
API providers do not rate-limit to annoy you. They rate-limit because shared infrastructure requires fairness. Your job as an integration architect is to respect those boundaries while keeping your workflow running.
The patterns in this runbook, exponential backoff with jitter, batching, proactive pacing, and dead-letter queues, are not optional hardening. They are baseline requirements for any production integration that touches a third-party API.
Start with the simplest win: add Retry-After-aware exponential backoff to every HTTP call in your integration layer. Then instrument your retry metrics. Then layer on batching and pacing as your volume grows.
Weighing whether to build this in-house or bring in experienced help? See how ProjectMakers approaches custom software projects that integrate reliably at scale.