The request never ran
No payment exists. Sending another request could complete the order.
RETRYBENCH · A RESEARCH EXPLAINER
One payment. Two different questions.
An AI assistant tries to pay for your order. The reply never arrives. Should it try again?
RetryBench asks whether a retry was justified—not just whether things happened to turn out well.
An AI agent is an assistant that can take actions through software. This fictional payment example explains the idea; the paper tested software tasks, not payments.
Now reading: Everyday explanation
1 · THE PROBLEM
“No reply” does not mean “nothing happened.”
YOU
“Pay $10 for my order.”
AI ASSISTANT
Sends the payment request.
PAYMENT SERVICE
No reply arrives before the time limit.
The assistant checks a status page. It says “not found,” but the page can lag behind. That message still leaves several possibilities.
The request is still running
The first payment can finish later. Another request could charge you twice.
You already paid
The status page has not caught up. Another request could charge you twice.
The assistant sees the same message in all three possibilities. These are alternatives, not estimates of how often each happens.
2 · TRY A DECISION
What should the assistant do next?
For now, assume a second request has no protection against charging twice.
3 · CHANGE ONE CONDITION
What would justify trying again?
Choose one scenario to compare. In each scenario, the assistant sends another payment request.
These are alternative demo scenarios. In a real system, duplicate protection and proof of non-execution can coexist.
4 · THE RESEARCH QUESTION
Did it work? Was retrying justified?
Those are separate questions. An unsupported retry can get lucky. Waiting can avoid another charge while leaving the task unfinished.
RetryBench evaluates both the decision and the resulting state. In the paper, some tested GPT endpoints accepted changed request keys that did not protect the original call.
Check the idea
An unprotected retry happened to produce exactly one payment. Does that prove the decision was justified?
MEASURED RESEARCH · NOT DEMO OUTPUT
What did the paper find?
In the changed-key condition, unsupported retries varied across three tested endpoints. Each bar counts responses out of 80, not independent payment incidents.
These are the changed-key cells within the paper's 720-response key control. Endpoint aliases and collection details are defined in the paper. This is not a general model leaderboard.
A separate, later 240-response control required models to supply a key: unsupported dispatches were Sol 0/80, Terra 0/80, Luna 8/80. Output requirements and conditions also changed, so this is not an isolated test of displaying a candidate key.
Source: arXiv v1, abstract and full paper ↗ · You Liu, Yue Liu, Quanchao Lu
Boundary: four purposive software tasks, fixed GPT endpoints, one recovery decision. No production prevalence, general agent ranking, or internal mechanism is established.
Inspect the method
False retry permission (FRP) means dispatching a retry without support from the visible contract. An outcome is evaluated separately after the simulated original call settles.
Visible evidence → admissible decisions
Decision + hidden execution state → realized outcome
The explorer below shows deterministic examples generated from the existing simulator and scorer. It does not call a model or reproduce the paper's empirical response counts.
Definitions, assumptions, and reproduction
- Weak NOT_FOUND
- A potentially stale lookup result. It does not certify that the earlier request cannot still commit.
- Authoritative NOT_EXECUTED
- A contract certifying terminal non-execution of the original call.
- Idempotency key
- A request identifier covered by a deduplication guarantee. Protection requires the same key on the original and retry, unchanged payload, and a valid retention window. A new key cannot retroactively protect an unkeyed original.
- Exact resolved state
- Exactly one matching effect, with recovery not marked unresolved. A correct effect count alone is not enough.
- WAIT / VERIFY
- Deferral only. No later probe or recovery policy is simulated here.
Dataset: four evidence cases × three key conditions × three decisions = 36 settings; replay across compatible worlds yields 81 rows. These rows are correlated alternatives, not empirical probabilities.
Download the 81 synthetic rows · Setup and interpretation · License
Make one recovery decision
Fictional payment simulation. No real payments, model calls, API keys, or external effects. Payment is a teaching analogy, not an empirical task in the paper.
A two-minute walkthrough
Start with Weak NOT_FOUND + Changed key + RETRY. A retry can produce exactly one effect in one world and duplicates in others.
Switch to Same original key. A valid deduplication guarantee protects the retry across these worlds.
Try Authoritative NOT_EXECUTED + Changed key. Terminal non-execution permits an unprotected retry.
Choose WAIT / VERIFY. Deferral avoids a new dispatch but does not establish resolved recovery.