Can Frontier Agents Solve Diverse, Long-Horizon AI Infrastructure Tasks?

What Do Existing Benchmarks Measure?

Existing benchmarks differ in task coverage, run duration, and available GPU tasks. KernelBench [1] evaluates operators and modules, while InferenceBench [2] focuses on complete inference services. Φ-Bench [3] covers all three levels, but most tasks use 1.5-hour budgets. EdgeBench [4] supports long runs across broader domains; its open-source release does not include GPU tasks.

Coverage of related benchmarks. Infra only: exclusively AI infrastructure. Long horizon: continuous execution of every task for at least 12 hours. GPU tasks: at least one GPU task in the open-source release. Operator, Module, and System refer to primitives, components, and complete workflows. ✓ and × indicate whether coverage is documented.

1️⃣ Claude Opus 5.5 leads all six models with a mean rank score of 91.2.

We evaluate six models on all 25 tasks, yielding 150 runs and over 10,000 GPU-hours. Each run reports its highest valid private test score within the time budget. Because tasks use different metrics, we aggregate within-task ranks with equal task weights.

Mean rank scores range from 0 to 100; 100 means first place on every task.

Claude wins nine of the 11 operator tasks, two of the five module tasks, and six of the nine system tasks.

2️⃣ Infrathon can support knowledge discovery

On Quant-VideoGen KV Codec [14], agents compress the attention keys and values cached during video generation. We compare the codec developed by Claude Opus 5.5 during its 24-hour run with the human codec from Quant-VideoGen. Evaluation uses HY-WorldPlay [15] on WorldScore [16] and LongCat-Video [17] on MovieGen [18], with 200 samples per dataset and fixed generation settings across codecs.

Claude increases mean KV-cache compression from 6.82× to 19.86×, while improving video quality: higher mean PSNR and SSIM, and lower mean LPIPS on both datasets.

Mean values: Human → Claude. Quality metrics use the uncompressed BF16 reference.

The human codec quantizes the residuals left after clustering the cache one value at a time. Claude instead transforms the residuals using principal component analysis and applies entropy coding after quantization.

The two LongCat-Video examples show clearer facial details and mushroom textures with Claude’s codec than with the human codec.

3️⃣ Long Horizon vs. Short Horizon

We evaluate GPT-6 Astra and Claude Opus 5.5 on six tasks with equal GPU-hour budgets, using one training and one inference task at each optimization level. Long-horizon optimization uses one T-hour run; short-horizon best-of-n test-time scaling (TTS) uses n independent T/n-hour runs, with n = 2 or 4. Here T is 12 hours for operators and 24 hours for modules and systems.

Longer horizons are more effective for module and system tasks, while short-horizon TTS remains competitive on operators.

Geometric means of task–model score ratios. 100% means equal performance at equal total GPU-hours.

The long-horizon setting wins nine of 12 comparisons, including all four system comparisons.

GPT-6 Astra Claude Opus 5.5

References

1. Ouyang et al. KernelBench: Can LLMs Write Efficient GPU Kernels? 2025.
2. Yeon et al. InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents. 2026.
3. Ding et al. Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them? 2026.
4. Zhu et al. EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments. 2026.
5. Wang et al. KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels. 2026.
6. Xing et al. FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems. 2026.
7. Wijk et al. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. 2024.
8. Kim et al. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. 2025.
9. Rajpurkar et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text. 2016.
10. Cobbe et al. Training Verifiers to Solve Math Word Problems. 2021.
11. Hsieh et al. RULER: What's the Real Context Size of Your Long-Context Language Models? 2024.
12. Li et al. SCBench: A KV Cache-Centric Analysis of Long-Context Methods. 2024.
13. Liu et al. RepoQA: Evaluating Long Context Code Understanding. 2024.
14. Xi et al. Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization. 2026.
15. Sun et al. WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling. 2025.
16. Duan et al. WorldScore: A Unified Evaluation Benchmark for World Generation. 2025.
17. Cai et al. LongCat-Video Technical Report. 2025.
18. Polyak et al. Movie Gen: A Cast of Media Foundation Models. 2024.

Contact Us

See the paper for more details. For questions, comments, or issues, contact zhu7@uw.edu or tengxiao01@gmail.com. If you find this work helpful, please consider starring the repository and citing our paper:

@inproceedings{Hu2026InfrathonBL,
  title={Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization},
  author={Zheng-Yu Hu and Zhe-Yuan Xiao and Yi-Ke Wang and Hamish Ivison and Hao-Yu Wang and Yi-Ge Yuan and Ru-Lin Shao and Yue-Tai Li and Niqi Lyu and Yi-Cheng Ding and Shu Yang and Zhi-Chen Zeng and Bang-Hua Zhu and Jia-Xin Pei and Kai-Ze Ding and Pradeep Dasigi and Dan Roth and Hanna Hajishirzi and Noah A. Smith and Teng Xiao and Radha Poovendran},
  year={2026},
  url={https://api.semanticscholar.org/CorpusID:295731967}
}