What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents

Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the agent by what it does. The result tells us about the original task only if the changed test case is still a valid test of that task. Many evaluations de