Retort

Evaluate the whole stack, not just the model. Retort applies statistical Design of Experiments (DoE) to measure how AI coding agents actually perform across the variables that decide a real project — programming language × model × context/serving/sampling × tooling — on the tasks you care about. Every run is scored for whether it provably implements the spec, plus how fast, how expensive, and how clean.

Why not just read a leaderboard? Sites like llm-stats.com compare many models across many benchmarks — but they hold the stack constant and ignore programming language, the surrounding tooling, and time taken. They can't tell you whether Opus 4.8 is worth its cost in Rust, how reliably a local model gets a Go MCP server completely right, or how long any of it takes. Retort answers exactly that: point it at your languages, models, and tasks (or your own codebase) and it finds the leading stack variant for your problem.

---

Contribute your experiment results

Ran retort on a model, language, or task combination we haven't tried? Open a pull request with the results — they're welcome and credited. Every contributor's runs land in their own namespace:

experiments/<your-github-id>/experiment-<name>/

Namespacing by GitHub id avoids collisions between contributors and makes every run attributable — the path itself is the provenance and the credit. Your experiment-<name>/ carries its own workspace.yaml, design.csv, per-run archives, retort.db, and a provenance.json recording the exact stack each run used, so the result is reproducible and traceable to you. retort aggregate folds your runs into the shared master.db alongside everyone else's, and retort report optimal picks them up in the leading-stack tables.

See experiments/README.md for the layout and the step-by-step (describe the experiment to Claude Code, or drive the CLI directly), and docs/future-experiments.md for the open queue if you want something worth running.

---

Features