Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization

1University of Washington 2Allen Institute for AI 3University of Pennsylvania 4Pyromind Dynamics 5Stanford University 6RadixArk 7The University of Texas at Austin 8Northwestern University

† Project Leader   * Corresponding Authors

Real-world tasks
25
Agent runs
150
Hours per run
12–24
GPU-hours
10,000+

Leaderboard

Scoring details ↗

17 / 25

Claude Opus 5.5 leads all six models with a mean rank score of 91.2.

Details ↗
Stage Level
Rank-score leaderboard
#ModelRank score95% CI WinsFirst valid scoreVisible output tokens
100 = first on every task.
Scoring and token definitions

Rank the six models by each task’s best valid within-budget score, then average 100 × (6 − rank) / 5 equally across tasks. Exact ties share average rank; a run without a valid scored submission receives zero. The 95% task-bootstrap confidence intervals use 10,000 resamples for the full suite and 3,000 for filtered groups.

Visible tokens use o200k_base for agent text and tool-call content, excluding tool responses, repeated code-diff displays, and hidden reasoning. Full-suite totals aggregate all 25 tasks. First-score time is the median across scored runs in the selected tasks.

All Results

Click a task below to view its details.

Task documentation ↗
Stage Level
Show more tasks

Infrathon can support knowledge discovery

Details ↗

6.82× → 19.86×

Claude designs a new compression method that increases the compression ratio from the human codec’s 6.82× to 19.86×, while improving video quality.

Two video cases compare the BF16 reference, human codec, and Claude Opus 5.5 codec. Below, Claude's quality-weighted Quant-VideoGen KV Codec score improves over 24 hours, with five submissions annotated by their code changes and compression ratios.
An example of optimization with Infrathon. On Quant-VideoGen KV Codec, agents compress the KV cache to reduce memory use during video generation. Claude Opus 5.5 develops a codec with higher compression ratios and better video quality than the human codec. Top: two video examples; red values show compression ratios. Bottom: Claude's scores over time combine compression ratio and video quality, with the Human Solution as the dashed baseline. Five numbered points mark selected submissions; boxes summarize their code changes.

Long Horizon vs. Short Horizon

Details ↗

9 / 12

Longer horizons are more effective for module and system tasks, while short-horizon TTS (test-time scaling) remains competitive on operators.

Loading optimization trajectories…

Longer horizons are more effective for module and system tasks, while short-horizon TTS remains competitive on operators.

Best-of-four (T/4) achieves 102.6% of long-horizon performance on operators, 96.9% on modules, and 77.3% on systems. Best-of-two (T/2) follows the same pattern.

Long-horizon optimization (T) wins 9 of 12 comparisons, including all four system comparisons.

Figure