On the cost of a retry
Every queue we have ever operated had the same line in its documentation: handlers must be idempotent. It is written like a rule of nature, something you comply with once and then stop thinking about. In practice it is a suggestion that quietly turns into a bug five years later, when the handler has been refactored six times and the idempotency key is now derived from something that is not stable.
The mistake is thinking of the key as a property of the message. It is not. It is a property of the effect you are trying to have exactly once. Those are different things, and the gap between them is where duplicate charges come from.
What the key is actually for
An idempotency key exists to answer one question at the storage layer: have I already applied this effect? That means the key has to be computed from the inputs that determine the effect, and stored in the same transaction that applies it. If your hander writes to a database and then publishes to a queue, you now have two effects and one key, and no amount of retrying will make that consistent.
If the key is generated before the effect it protects, it is a correlation id wearing a disguise.
The fix is unglamorous: make the effect and the key land in one row. A unique constraint on (account_id, request_id) plus an insert that returns the existing row on conflict gives you real idempotency, because the database is the arbiter rather than your process memory.
Where it costs more than it saves
The expensive case is the one nobody writes about: when the key is too specific. Include a timestamp, a retry counter, or a fresh UUID and every retry looks like a new request. The system keeps its promise — each effect applied once — while applying the effect four times, because four different keys described the same intent.
We have seen this cost a customer a weekend and a small amount of money. The retries were correct. The key was correct. The definition of "the same request" was the thing nobody had written down.
What we do now
Every handler in Relay is required to declare its deduplication scope in a comment above the function: the exact tuple it treats as identical. It is three lines of documentation that have caught two real bugs, which is a better return than most of our tests.