InferenceBench

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

This is the official repository for the paper "InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents".

Authors: Jehyeok Yeon, Ben Rank, Maksym Andriushchenko

Overview

InferenceBench measures whether autonomous CLI agents can act as ML systems engineers in a genuinely open-ended setting. Each run gives the agent a base LLM, a single NVIDIA H100, a wall-clock budget, and a scenario-specific objective; the agent must deliver a running, OpenAI-compatible inference server that maximizes the scenario's primary metric while passing both a quality gate and an integrity gate.

Unlike narrower benchmarks where the action space collapses to hyperparameter tuning over a known recipe, inference systems engineering forces real composition choices such as inference framework, attention backend, quantization format, KV-cache layout, scheduler tuning under brittle infrastructure where wrong combinations crash on launch rather than degrading gracefully. The benchmark is designed to test whether agents search an open engineering space or retrieve memorized configurations from it.

Headline Result

Across 15 frontier agent configurations on Mistral-7B-Instruct-v0.3 with a 2-hour budget per run, agents reliably beat a naïve PyTorch reference and often match or exceed default-configuration serving engines, but non-agent search (Random / SMAC3 / TPE) given the same 2-hour budget on vLLM beats every agent on every scenario. Behavioral analysis shows the bottleneck is not domain knowledge:

• 93.9% of agent runs ship a vLLM-based final launcher, even though SGLang, TGI, and TensorRT-LLM are explicitly mentioned in the prompt.
• The median run launches exactly one non-default vLLM configuration over the full 2 h budget.
• 58.3% of runs pass both gates while launching at most one distinct non-default configuration, so continuation does not imply adaptation.

Of 180 recorded runs: 65.0% pass both gates, 18.9% fail or do not complete the quality gate, 6.1% are integrity-flagged, and 10.0% fail final-server reachability or runtime checks.

Leaderboard

Per-scenario speedup over the naïve PyTorch baseline, mean ± SEM across three held-out seed pairs (21, 1337), (248, 428), (999, 777). Runs that fail a gate or final-server reachability contribute 1.00×. The aggregate column is the geometric mean across A–D.