daily nerdingHow to rank models for agentic engineering — five benches, not one index
How to rank models for agentic engineering — five benches, not one index
A ~40-minute framework for selecting frontier and workhorse models using variance, cost, speed, guardrails, abstention, and long-horizon SWE — distilled from IndyDevDan’s ranking walkthrough. Scores and UI labels are as shown on screen (approximate).
Start with the idea and the five leaderboards, then walk the twenty-four stages of the map.
Words on this page
Thirteen terms, in plain words. Skip this if you already read model leaderboards.
- Benchmark (bench)
- A fixed set of test tasks that every model runs, with scores posted on a leaderboard. This page keeps five of them and reads each one for how far apart the models land.
- SWE
- Short for software engineering. SWE-Bench and DeepSWE are benches built from real coding work.
- Agentic engineering
- Building with AI agents through structured systems such as harnesses, plans and factories, instead of chatting with a model. Dan calls it the ceiling and calls vibe coding the floor.
- Harness
- The loop that runs an agent on a task: it sends commands, watches what changes, and hands the result to a checker. Terminal-Bench runs this loop inside real containers.
- Verifier
- An automatic check that decides whether the agent’s work passes. Here it can be a pixel match against a target layout, or a fresh container that runs the tests on the agent’s patch.
- Token burn
- Tokens are the units a model reads, writes and is billed in. Token burn is how many it uses to finish one task, which drives both cost and time.
- SOTA and workhorse
- SOTA (state of the art) means the strongest models available. A workhorse is a cheaper model that can match them on some benches, as Gemini 3.8 Flash does on DeepSWE by spending more steps.
- model-stack
- Dan’s own local tool (at localhost:8787) that sorts models into tiers with prices on top. It is where he turns bench results into a plan for which model does which job.
- Hallucination
- When a model states something false as if it were true. AA-Omniscience takes points off for it, while saying “I don’t know” costs nothing.
- Saturated bench
- A leaderboard where the top models all score about the same, so it tells you little about which to pick. AA LCR v1.1, with models bunched near 80 to 86 percent, is the example here.
- Long-horizon
- Work that takes many steps over a long run, such as exploring a real codebase, making a change and testing it. DeepSWE measures this from short prompts.
- Swarm
- Many agents working on one job together. Dan warns that one agent’s hallucination can cascade through the whole swarm.
- Dark factory
- The last step in Dan’s path: agents, workflows, factories, swarms, then dark factories. Each step needs less of a human in the loop (HITL) as trust in the system grows.
Trade-off Triangle and the one number that matters
Every bench below is read against these two ideas.
Trade-off Triangle
Performance (apex) · Speed / Latency · Cost / Token burn
Useful Agent Output per Hour
Useful Agent Output per Hour = verified work / (time × spend)
Verifiable work, not raw tokens. Variance Principle: discard flat benches. Combine compute — don’t select compute.
Snapshot
Five Benches — what they measure & who led
Five benches with real spread between models, top eight on each. Chart snapshot from the video (approximate) — not a live leaderboard scrape.
Terminal-Bench v4.0
LeaderboardMeasures
Can the agent finish real work in a terminal/CLI harness (commands ↔ container state ↔ verifier pass/fail)? Headline: resolution / score %.
Who led
| # | Model | Score |
|---|---|---|
| 1 | GPT-6 Astra (Max) | 59.8% |
| 2 | GPT-6 Astra (High) | 59.7% |
| 3 | Claude Fable 5 (with fallback) | 54.0% |
| 4 | Claude Fable 5.1 (High with fallback) | 52.0% |
| 5 | Claude Fable 5.1 | 52.0% |
| 6 | Claude Opus 5 (Max) | 50.5% |
| 7 | Claude Fable 5 | 49.0% |
| 8 | GPT-6 Astra (Low) | 42.4% |
Open stage 03 Band B · stages 03–06
APEX-Agents
LeaderboardMeasures
Can the agent do long-horizon professional knowledge work (IB, consulting, corporate law) across files and apps, graded on expert rubrics? Headline: mean task score %.
Who led
| # | Model | Overall |
|---|---|---|
| 1 | GPT-6 Astra | 62.4% |
| 2 | Claude Fable 5.1 Max | 62.0% |
| 3 | Claude Opus 5 | 60.2% |
| 4 | Claude Fable 5 | 59.2% |
| 5 | Muse Spark 1.1 | 58.1% |
| 6 | Grok 4.6 | 57.5% |
| 7 | GPT-5.6 Sol | 56.7% |
| 8 | Claude Opus 4.8 | 56.2% |
Open stage 07 Band C · stages 07–09
AutomationBench-AA
LeaderboardCompliant (guardrails on)
Measures
Can the agent run cross-SaaS business workflows while obeying guardrails? Prefer compliant Score (violations fail the task), not raw objectives-completed.
Who led
| # | Model | Compliant Score |
|---|---|---|
| 1 | GPT-6 Astra (Max) | 68.9% |
| 2 | GPT-6 Astra (High) | 67.2% |
| 3 | Grok 4.6 (High) | 67.0% |
| 4 | GLM-5.3 | 65.5% |
| 5 | GLM-5.3 Flash | 64.9% |
| 6 | GPT-5.6 Sol | 63.8% |
| 7 | Claude Fable 5.1 | ~58.2% |
| 8 | Claude Opus 5 | ~58.0% |
Open stage 11 Band E · stages 11–14
AA-Omniscience
LeaderboardMeasures
Does the model know facts and stay calibrated — reward correct, punish hallucinations, no penalty for abstaining? Headline: Omniscience Index (−100…+100); zero ≈ 50/50 when it answers.
Who led
| # | Model | Index |
|---|---|---|
| 1 | GPT-6 Astra (High) | 44 |
| 2= | Tied for second at 43. Claude Fable 5.1 | 43 |
| 3= | Tied for third at 43. Claude Opus 5 | 43 |
| 4= | Tied for fourth at 42. GPT-5.6 Sol | 42 |
| 5= | Tied for fifth at 42. Claude Fable 5 | 42 |
| 6 | Gemini 3.8 Flash | 30 |
| 7 | Muse Spark 1.3 | 25 |
| 8 | Soul | 22 |
Open stage 15 Band F · stages 15–17
DeepSWE v1.1
LeaderboardPass-rate tie · cost discriminates
Measures
Can the agent do long-horizon SWE on real repos from short prompts (explore → patch → isolated verifier)? Headline: pass rate %; cost/steps matter for workhorse routing.
Who led
| # | Model | Pass rate | Cost |
|---|---|---|---|
| 1 | GPT-6 Astra (High) | 74%±3% | ~$6.52 per task |
| 2 | Gemini 3.8 Flash (High)cost leader among the 74% tie | 74%±3% | ~$2.36 per task |
| 3 | Claude Opus 5 (Max) | 74%±3% | ~$11.84 per task |
| 4 | GPT-5.6 Sol (Max) | 73%±3% | ~$5.66 per task |
| 5 | Claude Fable 5 (High) | 70%±3% | ~$13.41 per task |
| 6 | Kimi K3 (Max) | 69%±3% | ~$4.65 per task |
| 7 | Grok 4.6 (High) | 67%±3% | ~$3.45 per task |
| 8 | GLM-5.3 (Max) | 67%±3% | ~$7.33 per task |
Open stage 18 Band G · stages 18–19
Teaching map
Twenty-four stages · eight bands
Bands A–H are for readability only. Every node is first-class. Click a stage — full title and callouts open under that row. Arrow keys move along the path.
Band A · Frame the problem. Stage 01: Intelligence Index overload & moving targets
A · Frame
A · Frame the problem · Stage 01
01Intelligence Index overload & moving targets
Thesis
Catch-all indices such as the Artificial Analysis Intelligence Index v4.3 flatten conflicting metrics into one bar. Frontier releases blur the signal further: a new model can look decisive on a compressed index and still fail a real agent loop. Benches themselves are moving targets — MMLU gave way to SWE-Bench, which is already giving way to long-horizon DeepSWE. Architecture decisions need variance, not a single average.
Takeaway
Macro-indices compress away the variance needed for architecture decisions.
On screen
- Fable / Mythos 5.1 announce.
- GPT Astra portal.
- Intelligence Index bar plus cost scatter.
- On-screen: THIS BENCHMARK ITSELF IS A BIT OVERLOADED.
- On-screen: BENCHMARKS ARE A MOVING TARGET.
Metrics / criteria
- Intelligence Index v4.3
- MMLU → SWE-Bench → DeepSWE
- cost scatter
Models called out
- GPT Astra
- Fable / Mythos 5.1
Evidence note
Screencast of announce slides, AA index bar, and cost scatter — as shown, approximate.
B · Terminal-Bench
C · APEX
D · Interlude
E · AutomationBench
F · Omniscience
G · DeepSWE
H · Synthesis
Keys: ← → along the path. Home and End jump to the first and last nodes.
Secondary
Appendix
Secondary. Not a substitute for the 24-node path. Scores as shown, approximate.
Model comparison snapshot
Scores as shown on screen. Approximate. Not a live leaderboard.
- Astra / Astra Max / Astra HighTB Max ~59.8%; APEX ~62.4%; Omniscience High ~44; DeepSWE ~74% (as shown).
- Fable 5 / 5.1 MaxMore tokens than Astra (~2.5–2.7×); APEX Max ~62.0%; Automation raw ~88–89% with ~8–9% gated drop.
- Opus 5Automation raw ~88–89%; gated drop ~8–9% (as shown).
- Gemini 3.8 FlashDeepSWE ~74% at ~⅓ Astra cost by spending more steps/tokens.
- GLM-5.3 / FlashTB workhorse vs Kimi K3 cliff; Flash ultra-cheap background; disciplined Automation workhorse.
- Kimi K3TB: low versus GLM-5.3 (as shown).
- Grok 4.6Automation: disciplined workhorse; Gmail heatmap strong.
- Muse SparkOmniscience: strong general / weak SE coding Qs.
- DeepSeekGmail heatmap degraded (as shown).
- QwenGmail heatmap strong (as shown).
What this map is (and isn’t)
Screencast of leaderboards, slides, Harbor/DeepSWE/Mercor portals, and Dan’s local model-stack UI (localhost:8787) — not a live harness-code walkthrough. Scores are as reported from on-screen charts (approximate).
- Missing from industry / called out as gaps: Small Agent Teams (SATs) coordination, delegation/handoff fidelity, self-healing / swarm fault-tolerance benches.
- Not covered: local VRAM/hardware profiling for open-weights, image/audio/video/robotics modalities, private unmaintained benches.
- Do not treat the Stage 23 wishlist as shipping evaluations. Stage 23 is labeled wishlist / gap — a spoken hole, not a shipping eval.
Not in this map (on purpose): SAT / swarm / self-healing as live benches; local VRAM / quantization profiling; image / audio / video / robotics; a code-level walkthrough of Dan’s swarm or harness scripts; Demo beats; a timestamp scrubber. Teaching covers twenty-four stages from one ranking walkthrough.
Video source, attribution only: youtu.be/9weiIHy9T_0 (IndyDevDan). Author site: agenticengineer.com/tactical-agentic-coding.