AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups