Your payment provider's webhook hits your endpoint at 2:17 AM. Then again at 2:18 AM, same event, same payload, same transaction ID. Your system creates two charges. At 9:00 AM, your support team starts processing refund requests while engineering scrambles to write a post-mortem.
This scenario is not hypothetical in the sense that it is rare. Stripe, PayPal, AWS SNS, all major event delivery platforms explicitly warn that webhooks will be delivered more than once. Network timeouts, server restarts, and load-balancer failovers make duplicate delivery an architectural constant, not an edge case.
The fix is not to prevent duplicate delivery. That is impossible. The fix is to design your API so that receiving the same request twice produces the same result as receiving it once. That concept is called idempotency, and this guide shows you exactly how to implement it in your own systems.
What Idempotency Actually Means (And What It Doesn't)
An API operation is idempotent when calling it N times with the same parameters has the same effect as calling it once. Note the emphasis on effects rather than responses. A GET endpoint that returns 200 with a result is naturally idempotent, it changes nothing server-side. A POST that creates a new record is non-idempotent by default, each call creates another row.
HTTP semantics give us a baseline:
- GET, PUT, DELETE: The spec requires these to be idempotent. Your implementation should honor that.
- PATCH: Idempotent if the operation is a full field replacement. Incrementing a counter by +1 is not idempotent.
- POST: Not idempotent by default. This is where idempotency keys come in.
A common engineering shortcut is adding a unique constraint on (user_id, order_reference) and calling it a day. That prevents duplicate records, but the caller still receives a 409 error on retry, which a consuming service may interpret as a failure and re-queue the event, creating a retry loop. True idempotency means the caller gets the same successful response on every retry, with zero side effects beyond the first call.
Why Duplicate Requests Arrive More Often Than Teams Expect
Duplicate operations enter your system through channels that are easy to overlook during initial design:
Webhook re-delivery. Event platforms retry failed deliveries automatically. Some retry for hours. GitHub, for instance, retries for up to several days if your endpoint returns a non-2xx status. Stripe retries for up to three days with exponential backoff. Your webhook handler must tolerate every one of those attempts.
Client-side retries. Mobile apps operating on flaky cellular connections send the same POST request multiple times when the first attempt times out at the network layer, the server may have processed it, but the client never received the acknowledgement.
Load-balancer timeouts. Your upstream server processes the request in 400ms, but the load balancer's idle timeout is 300ms. The LB retries. A second request hits your application. Two records for one logical action.
Infrastructure orchestration. Kubernetes restarts a crashed pod. That pod crashed after writing to the database but before sending a reply to the caller. The caller retries. Without idempotency logic, the database gets a second row.
Human factors. A user taps "Submit Order" twice because the UI did not provide immediate feedback. On slow 3G connections, this is not impatience, it is a rational response to uncertainty about whether the first tap registered.
Step-by-Step: Implementing Idempotency
The standard pattern has four parts: key acceptance, lookup, storage, and expiry management. We will walk through each one.
Step 1, Accept an Idempotency Key
Your API requires callers to pass a unique key per logical operation via a header:
Idempotency-Key: a1b2c3d4-e5f6-7890-abcd-ef1234567890
The key should be a UUID generated by the client, or a hash of the request's unique business identifiers (payment intent ID, order number, external reference). Do not generate this server-side, the caller decides which retries represent "the same operation."
Stripe requires this header on all POST requests and stores results for 24 hours. That is a strong model to follow.
Step 2, Look Up the Key Before Processing
When a request arrives with a key that already exists in your idempotency table and has not expired, return the stored status code and body verbatim. Do not re-execute the business logic. If the request body differs from the original that was stored, return a 422 Unprocessable Entity, same key with different payload is a caller error that should surface immediately.
Step 3, Store the Result After Processing
On first receipt, process the request normally, then write the key, the request body fingerprint, the response body, the status code, and an expiration timestamp to your idempotency table. Use your primary database, Redis, or any store with atomic read-write guarantees.
A critical pitfall lurks here: two concurrent requests with the same key can both pass the "key not found" check before either one writes to the table. You need a database-level lock (SELECT ... FOR UPDATE), an INSERT ... ON CONFLICT DO NOTHING pattern, or Redis's SET NX to resolve this race condition atomically.
Step 4, Expire Old Keys
Idempotency keys should not accumulate indefinitely. Stripe uses a 24-hour TTL. For internal webhooks, 24-72 hours is a reasonable range, depending on your upstream provider's retry window. Schedule a periodic cleanup job for database-backed stores, or rely on Redis TTLs if you are using an in-memory cache.
Worked Example: Idempotent Payment Endpoint in Python
Here is a complete, minimal implementation. It uses an in-memory dictionary for clarity, swap it for Redis in production.
import hashlib
import json
from datetime import datetime, timedelta
from functools import wraps
from typing import Dict, Any, Tuple
# In-memory store, replace with Redis in production
_idempotency_store: Dict[str, Dict[str, Any]] = {}
def require_idempotency_key(ttl_hours: int = 24):
"""Decorator: makes a POST endpoint idempotent via the Idempotency-Key header."""
def decorator(func):
@wraps(func)
async def wrapper(*args, request, **kwargs) -> Tuple[Any, int]:
key = request.headers.get("Idempotency-Key")
if not key:
return {"error": "Missing Idempotency-Key header"}, 400
# Fingerprint ties the key to the exact request body
body_fingerprint = hashlib.sha256(
json.dumps(request.body, sort_keys=True).encode()
).hexdigest()[:16]
store_key = f"{key}:{body_fingerprint}"
# Look up cached result
if store_key in _idempotency_store:
cached = _idempotency_store[store_key]
if datetime.utcnow() < cached["expires_at"]:
return cached["response"], cached["status_code"]
else:
del _idempotency_store[store_key]
# First time: execute real business logic
response, status_code = await func(*args, request=request, **kwargs)
# Cache the result
_idempotency_store[store_key] = {
"response": response,
"status_code": status_code,
"expires_at": datetime.utcnow() + timedelta(hours=ttl_hours),
}
return response, status_code
return wrapper
return decorator
# Usage on an endpoint
@require_idempotency_key(ttl_hours=24)
async def create_payment(request) -> Tuple[Any, int]:
"""Charge a customer, safe to retry with the same Idempotency-Key."""
payment = await charge_customer(
amount=request.body["amount"],
currency=request.body["currency"],
)
return {"payment_id": payment.id, "status": "completed"}, 201
What this implementation does:
- Rejects requests without an
Idempotency-Keyheader with a 400, callers get immediate, unambiguous feedback. - Hashes the request body and combines it with the key. If the same key arrives with a different body, it counts as a distinct request, preventing callers from accidentally reusing keys across unrelated operations.
- On the first call, processes the request normally and caches the response body plus status code.
- On any subsequent call within the TTL, returns the cached response verbatim with the original status code, no new business logic executes.
The Business Impact at Scale
Consider this hypothetical scenario: your payment endpoint processes 5,000 operations per hour. If just 2% of requests arrive as duplicates, from webhook retries, timeout-driven resubmissions, and user double-clicks combined, that is 100 duplicate calls per hour hitting your business logic unnecessarily. Each one executes a database write, potentially a third-party API call, and in the worst case, a duplicate customer charge. Over a month, that escalates to approximately 72,000 unnecessary operations.
With the idempotency layer in place, those 100 duplicates per hour return from cache in under 1 millisecond instead of executing real business logic. The caller gets the correct response. The database stays clean. Your support inbox stays empty.
Common Pitfalls That Break Idempotency in Production
1. Checking and inserting without an atomic lock.
Two requests with the same key arrive within 5ms. Both see "key not found." Both execute the business logic. Fix this with INSERT ... ON CONFLICT DO NOTHING, SELECT ... FOR UPDATE, or SET NX in Redis.
2. Not tying the key to the body fingerprint. A caller accidentally reuses a UUID from last week for a completely new request. If you only check the key without validating the body hash, you return a stale, irrelevant cached response. Always combine the key with a body fingerprint.
3. TTL shorter than your upstream's retry window. If your payment processor retries for up to 72 hours and your idempotency TTL is 12 hours, late retries bypass the cache and hit business logic again. Match your TTL to your upstream's retry window, not to what "feels reasonable."
4. Returning inconsistent status codes for cache hits. Returning 200 for the first request and 201 (or any other code) for cached hits breaks clients that branch on status codes. Always store and replay the original status code exactly.
5. Ignoring the "in-progress" state. If the server crashes after writing the idempotency key but before storing the response body, the next call finds the key in the database with no response attached. Your table needs a tri-state: pending, completed, or failed. Some implementations store a lock record with a short TTL and treat an expired lock as "safe to reprocess."
Verifying Your Idempotent API Works
An idempotency test suite is straightforward. Cover these four cases for every idempotent endpoint:
Test 1, First call creates the resource. Send a POST with a new Idempotency-Key and verify the response is 201 (or whichever status code your endpoint returns) and a new resource exists in the database.
Test 2, Second call returns the same response without side effects. Send the exact same request again. Verify the response body and status code are identical to Test 1. Verify no new record was created.
Test 3, Same key with a different body returns 422. Resend the Idempotency-Key from Test 1 but with a different request payload. Expect a 422 error, the key is already associated with a specific body.
Test 4, Expired key re-executes the logic. Manipulate the key's expiration time (or wait), then send the same request. Verify a new resource is created.
As a concrete integration test outline:
POST /payments Idempotency-Key: key-001 {"amount": 100}
→ 201 {"payment_id": "pay_abc", "status": "completed"}
✓ One payment record exists.
POST /payments Idempotency-Key: key-001 {"amount": 100}
→ 201 {"payment_id": "pay_abc", "status": "completed"}
✓ Still one payment record. Response is identical byte-for-byte.
POST /payments Idempotency-Key: key-001 {"amount": 200}
→ 422 {"error": "Idempotency key reuse with different payload"}
# After key expires:
POST /payments Idempotency-Key: key-002 {"amount": 50}
→ 201 {"payment_id": "pay_def", "status": "completed"}
✓ Two payment records total. New key, new resource.
For event-driven systems where you are also managing message broker topics, this verification becomes even more important. In architectures that route events through systems like Kafka, ensuring your consumers process each message exactly once is a core part of building reliable integration pipelines.
Best Practices for Production Systems
1. Make the idempotency key mandatory, not optional. If the header is optional, callers will skip it, and you will get duplicate operations in production. Enforce it at the API gateway or middleware level for all mutating (POST, PATCH) endpoints.
2. Match your TTL to your longest retry window. Audit every upstream system that calls your webhook. Find the one with the longest retry schedule. Set your idempotency TTL to at least that value plus a safety buffer of 20-30%.
3. Use Redis (or equivalent) for the idempotency store at scale. Database lookups on every request add 5-15ms of latency. Redis handles this in under 1ms, and built-in TTL management means you do not need a separate cleanup job. For APIs processing thousands of calls per minute, the difference compounds.
4. Log idempotency cache hits separately from misses. If your cache hit rate suddenly climbs from 5% to 40%, something upstream is hammering you with retries. Separate metrics let you alert on that anomaly, it might indicate a failing webhook consumer, a network partition, or a misconfigured retry policy in a third-party service.
5. Store the full response, not just a success flag. Storing {"status": "success"} is not enough. The caller needs the identical response body, resource IDs, timestamps, confirmation tokens, status codes, on every retry. Store the complete JSON and replay it character-for-character.
When the Integration Surface Outgrows a Homegrown Approach
The pattern described above works well for individual endpoints. In practice, production systems rarely have just one endpoint. When your integration layer spans dozens of endpoints across multiple services, payment processing, inventory synchronization, user provisioning, notification delivery, managing idempotency at each endpoint separately creates duplication in your codebase and inconsistency in behavior between endpoints.
Teams building this kind of infrastructure at scale often partner with an experienced custom software development team that has solved these patterns across multiple production systems. At ProjectMakers, we have built integration layers where idempotency, retry logic, and message deduplication are handled as cross-cutting concerns through middleware, rather than copy-pasted into every controller by developers who each interpret the pattern slightly differently.
The alternative, maintaining a bespoke idempotency framework that drifts across teams, leads to exactly the kind of inconsistent behavior that idempotency is supposed to prevent.
The Bottom Line: Start with One Endpoint
API idempotency is not a theoretical purity exercise. Every webhook-based integration, every retry-capable mobile client, and every orchestration workflow that touches a POST endpoint will eventually produce duplicate requests. The question is not whether your system will face duplicate operations, it is whether it handles them gracefully or charges your customer twice and wakes your support team at 9 AM.
Start small. Pick your most critical POST endpoint, the one that writes to the database and cannot safely run twice. Implement the four-step pattern. Write the four test cases. Deploy. Then expand the pattern to every mutating endpoint in your integration layer.
The upfront investment is a few hours of implementation per endpoint. The return is a system where retries are safe, webhooks are resilient, and every idempotency key you accept is a promise you keep, exactly once.
Source: How To Build Reliable Workflows With API Idempotency