Idempotency Is a Data Problem, Not an Application Problem

Retries are inevitable. Duplicate delivery is inevitable. Partial failure is inevitable.

If correctness depends on application memory, your system will eventually produce duplicate side effects.

1. The Retry Illusion

Modern systems retry automatically:

• HTTP clients retry on 5xx
• Load balancers retry upstream
• Message brokers redeliver
• Users click twice

Retries are not edge cases. They are normal operating conditions.

Reference: AWS – Timeouts, Retries and Backoff

2. Real-World Failure Pattern

Client → POST /charge
Server → Charge credit card
Server → Save record
Server → Response lost
Client → Retry POST /charge

The first charge succeeded. The response was lost.

The second charge is now a duplicate.

Nothing failed. The network behaved normally.

3. Why Locks Don't Save You

A distributed lock prevents concurrent execution. It does not prevent sequential duplication.

Request A processed at 10:00:00
Request B retried at 10:00:05

The lock is already released. The system executes again.

Concurrency control ≠ idempotency.

4. At-Least-Once Delivery

Most message brokers guarantee at-least-once delivery:

• Kafka
• RabbitMQ
• Azure Service Bus
• AWS SQS

At-least-once means duplicates are expected.

Reference: Kafka Delivery Semantics

5. The Probability Math

Suppose:

• Traffic: 3,000 RPS
• Retry rate: 0.5%
• Failure window: 1 hour

3,000 × 0.5% = 15 duplicate attempts per second
15 × 3600 = 54,000 duplicate attempts per hour

Even if only 1% become real duplicate side effects:

54,000 × 1% = 540 duplicated transactions

Under scale, tiny percentages become operational incidents.

6. Idempotency at the Data Layer

The only reliable solution: make duplication impossible at the storage layer.

CREATE UNIQUE INDEX ux_payment_idempotency
ON payments(idempotency_key);

INSERT INTO payments (idempotency_key, amount)
VALUES (@key, @amount)
ON CONFLICT (idempotency_key)
DO NOTHING;

If the same request arrives twice, the database enforces correctness.

The application becomes stateless.

7. Proven Production Patterns

1. Idempotency Key Pattern

• Client generates UUID
• Server persists key with result
• Duplicate key returns previous response

SELECT response_payload
FROM idempotency_records
WHERE key = @key;

2. Outbox Pattern

Persist event and state change in same transaction. Publish asynchronously.

Reference: Transactional Outbox Pattern

3. Exactly-Once Is Mostly Marketing

Exactly-once delivery requires distributed consensus. Most production systems simulate it using:

• Idempotency
• Deduplication tables
• Compensating transactions

Reference: Martin Kleppmann on Distributed Systems Guarantees

8. Conclusion

Locks control concurrency. Retries increase duplication. Queues guarantee redelivery.

Only the database can enforce uniqueness.

Idempotency is not an application concern. It is a data integrity constraint.