RetryBench: Was that retry allowed?

A successful recovery does not necessarily mean the retry was justified.

An agent sends a payment request and receives a timeout. The operation might never have run, already have committed, or still be in flight. Should the agent send it again?

This interactive demo lets you change the visible evidence, the idempotency-key condition, and the next action. It then shows whether retrying was permitted by the simulated contract and what happens in each compatible hidden execution state.

Read the revised manuscript (PDF, October 16, 2026) — RetryBench: Retry Decisions and Protected Recovery Across GPT Interfaces, by You Liu, Yue Liu, and Quanchao Lu.

This is the latest author revision, shared directly here. It has not yet replaced arXiv v1: Separating Retry Permission from Recovery Outcomes in GPT Tool Use. The demo's empirical summaries remain labeled as arXiv v1 results.

Read the blog walkthrough · Code and reproduction

Language and reading mode

Use 中文 / English at the top right. The full interface, explanations, choices, and generated results are bilingual. Switching languages preserves the selected scenario and visible results. The browser remembers your language preference when storage is available; the initial default is English. Raw example field names and the license remain in their original language.

A compact horizontal tab row switches between Overview / 通俗讲解 and Research & methods / 研究与方法. Bold text and an underline identify the active tab, directly above its content. Switching tabs preserves the page position. Arrow keys, Home, and End also switch the tabs.

The three story decisions use native radio buttons. One decision can be selected at a time; its radio dot and blue background remain visible while its result is shown. Changing language preserves the selection.

Two reading paths

• Overview / 通俗讲解: a fictional order, three possible hidden situations, plain-language decisions, payment outcomes, and a short understanding check. No knowledge of API contracts or idempotency is assumed.
• Research & methods / 研究与方法: sourced arXiv v1 results with response denominators, method definitions, explicit assumptions, downloadable examples, and the full contract explorer.

The story and technical explorer use the same frozen simulator rows. The measured paper results are separately labeled and sourced; the simulator does not generate those empirical counts.

Run locally

You need Python 3 and a modern web browser. No Python packages, model downloads, API keys, or paid inference are required.

Open a terminal in the folder containing index.html and examples.json, then run:

python3 -m http.server 7860 --bind 127.0.0.1

Open http://localhost:7860 in your browser. Press Ctrl+C in the terminal to stop the server.

You can also open index.html directly in a browser: the page embeds the same 81 rows, so the interaction needs no server or network. Keep the other files beside it for the download and documentation links.

Technical explorer walkthrough

Select Research & methods, then scroll to Make one recovery decision. The default story path provides a separate plain-language walkthrough.

1. Select Weak NOT_FOUND, Changed key, and RETRY, then click Reveal permission and outcomes. The retry is unsupported: one world ends with exactly one effect, while the other compatible worlds produce duplicates.
2. Change the key condition to Same original key (valid guarantee) and reveal the results again. Reusing the valid original key protects the retry across these worlds.
3. Select Authoritative NOT_EXECUTED with Changed key and RETRY. The contract certifies terminal non-execution, so a retry can proceed without original-key protection.
4. Choose WAIT / VERIFY. No new call is dispatched, but recovery remains unresolved at the one-decision horizon.

You can also try Timeout without status, Authoritative IN_FLIGHT, Original key absent, and STOP.

What the results mean

Weak NOT_FOUND is not proof of terminal non-execution. In the simulator, it is compatible with failure before execution, an original request still in flight, or a stale negative observation after commitment.

Same-key protection assumes that the original request carried that key, the payload is unchanged, and the provider's deduplication guarantee remains valid. Changing the key—or introducing a key after an unkeyed original—does not protect against duplication of the original call.

WAIT / VERIFY represents deferral. The demo does not execute a later verification policy. STOP sends no retry and can leave the intended operation undone.

Included examples

The demo contains four evidence cases, three key conditions, and three actions: 36 decision settings in total. Replaying each setting across its compatible hidden worlds produces 81 outcome rows in examples.json.

These rows were generated deterministically from the RetryBench simulator and scorer. They are synthetic teaching examples, not collected model responses or the full benchmark dataset. Hidden worlds are alternatives, not independent statistical observations or estimated probabilities; their row counts should not be interpreted as real-world failure rates.

Files

The static interface displays precomputed outcomes. It does not call a model, connect to a payment service, or perform real side effects. Its external research link points to the paper.

Scope

The payment scenario is fictional and serves as a teaching analogy. It is not an additional empirical task in the paper or a guarantee about any payment provider. The paper studies four selected software tasks, fixed GPT endpoints, and one recovery decision. Neither the demo nor those findings establish production error rates, general agent reliability, or episode-level safety.

Demo version: teaching-demo-v4. Open the hosted demo.

Citation

If you discuss or use the research, cite the paper:

@misc{liu2026retrybench,
  title         = {RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use},
  author        = {You Liu and Yue Liu and Quanchao Lu},
  year          = {2026},
  eprint        = {2610.15884},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.15884}
}

License

Code and generated teaching examples retain the PolyForm Noncommercial License 1.0.0 in LICENSE. No separate permissive data license is granted. See the license for permitted uses and conditions.