Official Review · ICLR 2026

The paper introduces LiveNewsBench, a scalable and regularly updated benchmark for evaluating the web search capabilities of large language models (LLMs). The authors argue that existing benchmarks fail to distinguish between retrieval and memorized knowledge because they rely on static question sets that models may already have seen during training. To overcome this, they construct LiveNewsBench from freshly published news articles, automatically generating question–answer pairs that require real-time retrieval and multi-step reasoning. The benchmark includes training, validation, and test sets, as well as a human-verified subset and a longitudinal component that tracks how answers evolve over time. Evaluations show that LiveNewsBench effectively differentiates models' search abilities: leading proprietary models such as GPT-5 and Grok 4 perform best, while smaller open models lag behind. Disabling internet access causes large performance drops, which confirms that the dataset minimizes contamination from pretraining data. The authors conclude that LiveNewsBench provides a reliable, contamination-limited foundation for measuring and improving LLMs' capacity to reason over dynamic, real-world information. - The paper identifies and mitigates a major flaw in existing web-search benchmarks by using recent news events that occur after model training cutoffs, which ensures minimal contamination from pretraining data - The benchmark's fully automated pipeline produces large quantities of question-answer pairs with limited human input while maintaining reasonable quality - The benchmark is designed to refresh quarterly, which helps maintain relevance and tests models on genuinely new information - The results demonstrate clear differentiation among proprietary, open-source, and smaller models, which confirms the benchmark's ability to measure genuine differences in search performance - The paper overstates the limitations of static benchmarks. Static questions remain informative when their correct answers have changed since a model's training cutoff, as they still require retrieval rather than recall. The authors have not convincingly shown that they need to keep generating new benchmarks every few months -- a well-designed static benchmark with time-sensitive questions could already achieve the same goal. - The benchmark conflates retrieval ability with reasoning accuracy. A model may successfully retrieve the correct information yet fail due to flawed reasoning, so the reported performance does not isolate retrieval skill from broader reasoning capability. - The human-verified subset cannot guarantee the reliability of the full dataset. Since about 20% of automatically generated items are rejected during manual review, a similar error rate may persist in the unverified portion, which undermines claims of overall data quality. - The reported accuracies of 85-91% suggest that the benchmark questions may not be sufficiently difficult. The small