What Is Idempotency? Idempotent APIs, Safe Retries and Duplicate-Proof Pipelines

Networks fail in the least convenient way: a request reaches the server, the server does the work, and the response is lost on the way back. The client cannot tell whether the work happened, so it retries. Payment providers, message queues, webhooks and AI agents all retry by design, which means every system that receives their requests will see duplicates sooner or later.

An idempotent system turns retries from a risk into a safety net, because each request carries an identifier the server remembers, and a repeated identifier returns the original result instead of performing the work again.

This article explains what idempotent means in HTTP, the four main ways to make operations idempotent, how idempotency keys work, how to handle duplicates in queues and data pipelines and how one production pipeline does it. It does not cover the mathematical theory, and it is not language-specific.

Key takeaways

• Idempotent means repeating an operation has the same effect as doing it once. Reading data is idempotent. Adding $10 to a balance is not. Setting a balance to $110 is.
• In HTTP, GET, HEAD, OPTIONS, TRACE, PUT and DELETE are defined as idempotent. POST is not, and PATCH is not guaranteed to be.
• Idempotency keys make POST requests safe to retry: the client sends a unique key, and the server stores and replays the first response for that key.
• Message queues and webhooks deliver at least once, so consumers must recognize and skip messages they have already processed.
• Database unique constraints are the last line of defense, because they hold even when two duplicates arrive at the same moment.
• In a production intelligence pipeline, a processing ledger and content-hash deduplication stopped at-least-once delivery from creating duplicate records.

Idempotent meaning in HTTP

RFC 9110, the current HTTP standard, defines a method as idempotent when sending the same request several times has the same intended effect on the server as sending it once.

Two clarifications prevent most confusion. Idempotency is about the effect on the server, not the response: a second DELETE can return 404 and still be idempotent. And a method being idempotent by definition does not make an implementation idempotent. A PUT handler that also sends an email on every call has a side effect that repeats.

Why retries make idempotency necessary

Consider a checkout that calls a payment API.

1. The client sends a request to charge $120.
2. The payment provider charges the card.
3. The connection drops before the response arrives.
4. The client sees a timeout and retries.
5. Without idempotency, the provider charges the card a second time.

The client did nothing wrong. Retrying after a timeout is the correct behavior, and the problem only appears when the receiving side cannot recognize the second request as a repeat. The same pattern appears with webhooks that providers resend, with queue messages redelivered after a consumer crash and with AI agents that retry a tool call after an error.

Four ways to make an operation idempotent

Most systems need several of these together. An order API might accept an idempotency key from the client, write the order with a unique constraint on that key and publish an event that downstream consumers deduplicate by event ID.

Idempotency keys: a design checklist

Stripe popularized the pattern. Its API accepts an Idempotency-Key header on POST requests, saves the status code and body of the first request with that key, returns the saved response to any retry and rejects a reuse of the key with different parameters. Keys can be removed after they are at least 24 hours old. The IETF is standardizing the header in its Idempotency-Key HTTP header draft, which reached version 08 in October 2026.

When you design your own:

• Let the client generate the key. A random UUID created before the first attempt and reused for every retry of the same operation.
• Scope keys by client or account. Two customers who happen to send the same key must not see each other's results.
• Store the key and the result together, in the same transaction as the work. If the work succeeds and storing the key fails, the next retry will repeat the work.
• Compare the request with the original. A reused key with a different body is a client bug. Return an error instead of replaying a result that does not match.
• Handle concurrent duplicates. When two requests with the same key arrive at the same time, lock on the key or return a conflict status to the second until the first finishes.
• Keep keys long enough. At least as long as clients might retry, often 24 hours to several days.
• Do not put personal data in keys. Keys appear in logs, so use random values, not email addresses or order details.

The AWS Builders' Library article on making retries safe with idempotent APIs describes how Amazon applies the same ideas across its services.

Idempotency in queues, webhooks and data pipelines

Message brokers and webhook senders almost always guarantee at-least-once delivery. A message is redelivered if the consumer crashes before acknowledging it, and a webhook is resent if the endpoint responds slowly. Exactly-once delivery across a network is very hard to guarantee, so the practical goal is exactly-once effect: each message may arrive several times, and processing it twice changes nothing.

• Give every message a stable ID at the source, not when it is consumed.
• Record processed IDs in a table, and check and write the record in the same transaction as the business change.
• Use natural keys for data. In ingestion pipelines, a URL, an external record ID or a hash of the content identifies the same item even when it arrives through different routes.
• Make writes upserts. Insert-or-update by natural key turns a duplicate into a harmless rewrite.
• Publish through an outbox. Write the outgoing event to an outbox table in the same transaction as the change, then publish it, so a crash cannot leave a change without its event or send an event for a change that rolled back.
• Send failing messages aside. Messages that fail repeatedly go to a dead-letter queue for inspection, rather than blocking the queue or being retried forever.

Kafka has made its idempotent producer the default since version 3.0, which prevents duplicates caused by producer retries, but consumers still need their own deduplication for the rest of the pipeline.

A production pipeline built for duplicates

A US cybersecurity media company needed a platform that collects news and advisories from hundreds of open sources, extracts entities with language models and stores them as a graph of vendors, products and security events. The message queue delivered at least once, so the same article could be processed several times, and the same story could appear at different URLs.

A .wrk team of six to eight engineers built idempotency into each stage.

• A processing ledger. Each item's canonical form and processing state are recorded, so a redelivered message is recognized and skipped.
• Two-level deduplication. Items are matched by URL and by a hash of their content, which catches the same story published at different addresses.
• A dead-letter queue. Messages that keep failing are moved aside for inspection, so one bad source cannot stall the pipeline.
• Entity resolution with versioning. The graph worker resolves new mentions to existing entities and keeps earlier versions, so a reprocessed article updates entities rather than creating new ones.
• Metrics on every stage. Prometheus metrics cover scraping, language model latency and each pipeline step, so duplicate rates and failures are visible.

The platform launched on schedule as an MVP and processes heterogeneous sources reliably. Read the NEXUS case study.

Idempotent tools for AI agents

AI agents retry tool calls when a call times out or returns an error, and language models sometimes call the same tool twice in one plan. Any tool that changes data, such as creating a ticket, issuing a refund or sending an email, should accept an idempotency key or a natural key, so a repeated call returns the first result. This belongs alongside the permission and approval controls covered in our guide to AI agent security.

The same applies to webhook receivers. Our guide to webhooks covers signature checks, fast responses and duplicate handling on the receiving side.

How to test idempotency

• Replay tests. Send every state-changing request twice in automated tests and assert that the second call changes nothing and returns the same result.
• Concurrency tests. Send the same request twice at the same moment and check that only one effect occurs.
• Crash tests. Stop the process after the work but before the acknowledgment, restart it and confirm the redelivered message is skipped.
• Reconciliation checks. Compare counts between source and destination regularly, for example orders against payments, to catch duplicates that slipped through.

Common mistakes

• Generating the key on the server. A key created per request on the server is different for every retry and protects nothing.
• Checking before acting without a lock or constraint. "Check if it exists, then insert" fails when two duplicates arrive together.
• Forgetting side effects. The database write is idempotent, but the confirmation email or the external API call is sent again.
• Deduplicating by timestamp. Two legitimate actions can share a timestamp, and one retried action can have two.
• Assuming the broker handles it. Delivery guarantees stop at the broker. Your consumer's effects are your responsibility.

For data platforms, our article on medallion architecture shows how incremental loads and layered storage keep reprocessing safe.