Field note / Fintech · Distributed systems

Exactly once is a product decision

Retries are infrastructure behavior. Whether repeating an action repeats its consequence is a product and domain decision.

A person taps “Pay.” The request crosses a network. The response does not return. What happened?

The honest answer is the caller does not know yet. The payment may have failed before reaching the server. It may have succeeded while the response was lost. It may still be running. A timeout describes the caller’s patience, not the world’s final state.

This small ambiguity is why “just retry it” can become “just charge them twice.” A retry is a duplicate wearing optimism.

Transport cannot decide business sameness

Infrastructure can redeliver a request. It cannot, by itself, know whether two requests express the same human intent. That knowledge belongs to the domain.

For a consequential command, the system needs a stable identity for the operation, a durable record of what happened, and a response that can be replayed without replaying the side effect. The implementation might use an idempotency key, a unique business reference, a command log, or a transaction boundary. The pattern matters less than the guarantee:

The distinction becomes especially important in fintech. “Create a payment” and “query a payment” are different operations. A browser redirect and a provider callback are different evidence. A user retry and a settlement reconciliation happen at different altitudes. Collapsing them into one convenient success boolean makes the interface simple by moving complexity into support, finance, and customer anxiety.

Exactly once is a story the whole product must maintain

No single queue flag can provide a universal exactly-once business guarantee across an unreliable network and arbitrary side effects. Useful systems build a narrower promise: at-least-once delivery plus deduplication, idempotent commands, recorded outcomes, and reconciliation where external truth can arrive late.

This affects the interface. If the system cannot yet prove the outcome, the UI should say that confirmation is pending. It affects support. An operator needs the operation reference and recorded transitions, not only the last screen the customer saw. It affects product language. “Try again” is only kind when trying again is safe.

The invariant is “one human intent produced no more than one business consequence.”

That is a stronger and more useful claim. It also makes the tradeoff visible. Deduplication records need retention. Operation identities need scope. A key reused forever can suppress a legitimate future action; a key forgotten too early can permit a duplicate. Reconciliation needs ownership. None of those choices belong exclusively to infrastructure.

A review prompt for consequential commands

Before shipping a retryable action, ask:

  1. What identifies the human intent rather than the HTTP request?
  2. Where is the first accepted outcome recorded?
  3. Can the same response be returned without repeating the side effect?
  4. What does the person see while the final state is unknown?
  5. Which process reconciles a late or contradictory answer?

If the answers span product, backend, provider integration, operations, and support, that is not accidental complexity. That is the real product boundary finally becoming visible.

Research notes

The AWS Builders Library article Making retries safe with idempotent APIs explains why semantic equivalence must be explicit and why caller-provided request identifiers matter. Stripe’s idempotent request documentation shows the idea in a public payment API, while its PaymentIntent lifecycle demonstrates why payment progress requires explicit, queryable states.

The philosophical lesson is modest: certainty is not created by repeating a request. It is created by preserving enough evidence to recognize the same intent, explain the observed result, and recover when the network leaves the sentence unfinished.

Verification ledger

Public references used to pressure-test the argument. Read the source before trusting the summary.

  1. 01 — Making retries safe with idempotent APIs
  2. 02 — Idempotent requests
  3. 03 — PaymentIntent lifecycle