arXiv CS open access

Documentation vs. Code Patterns: What Drives LLM-Based Exception Oracle Generation?

2026 · arxiv_cs

arXiv CS · Papers · 2026

software-architecture software-engineering testing

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

arXiv:2608.00884v1 [cs.SE] 1 Aug 2026

Documentation vs. Code Patterns: What Drives LLM-Based Exception Oracle Generation? Soneya Binta Hossain✉

Matthew B. Dwyer

Tasfia Tasnim

University of Texas at Dallas Richardson, USA sbhossain@utdallas.edu

University of Virginia Charlottesville, USA matthewbdwyer@virginia.edu

University of Texas at Dallas Richardson, USA tasfia.tasnim@utdallas.edu

Abstract

LLM-based test oracle generation (TOG) methods report high accuracy on exception oracle generation, but it remains unclear what evidence drives these predictions. In particular, do models use explicit exceptional-behavior documentation such as Javadoc @throws clauses, or do they rely on recurring patterns in tests, code, and documentation? We investigate this question through a large-scale interventionbased study of three TOG systems spanning classifier-based and generative architectures and model sizes from roughly 110M to 7B parameters, evaluated on three real-world benchmarks comprising two generated-test datasets and a new benchmark of developerwritten tests. We first remove Javadoc @throws clauses and find that accuracy changes only marginally, with the largest drop below one percentage point. This indicates that structured exception documentation is not the primary driver of exception-oracle prediction. We then apply attribution-guided substitution ablations to identify the signals that predictions depend on. The results show that high accuracy can be driven by shortcut signals: some models are highly sensitive to a small number of structural tokens, while others distribute reliance across many lexical cues. These findings challenge the assumption that strong exceptionoracle accuracy reflects robust use of exception semantics. Future TOG systems should therefore be evaluated not only by whether they predict the correct oracle type, but also by whether their predictions are grounded in meaningful exception-triggering evidence.

Exceptions are fundamental in languages such as Java, C#, Python, and C++, where they signal abnormal conditions, enforce API contracts, and help preserve correct execution. Because software must behave correctly in both normal and exceptional executions, specifying and testing exceptional behavior is critical to reliability [3, 21]. Yet exceptional behavior is often harder to observe than valuereturning behavior [6, 18]: return values appear in ordinary executions, whereas exceptions arise only under specific runtime conditions. Developers and testing tools therefore often rely on external artifacts, especially documentation, to understand when and why exceptions should occur. In Java, Javadoc @throws clauses, including the synonymous @exception tag, provide one of the most structured forms of exception-related documentation. Similar mechanisms appear in other ecosystems, such as XML documentation in C#, Python docstrings, and contract-based specifications, making exceptional-behavior documentation a broader softwareengineering concern. This makes testing exceptional behavior a central part of software validation [1]. A system should not only return correct values in normal executions, but also detect, signal, and propagate errors in abnormal ones. This requirement is captured through exception oracles: test-oracle specifications that state whether a test should expect an exception. In Java, such oracles are commonly encoded using JUnit idioms such as @Test(expected=...) in JUnit 4, assertThrows in JUnit 5, or explicit try–catch blocks, as shown in Listing 1 [14, 15]. Similar constructs exist in other ecosystems, such as Python’s assertRaises and NUnit’s Assert.Throws [19, 20]. Despite their importance, exception oracles remain labor-intensive to write, error-prone, and dependent on developers anticipating failure modes that may be only partially documented [3]. Documentation can help bridge this gap. Javadoc often states the conditions under which exceptions should occur, making it a natural source for constructing exception oracles. For example, in Listing 2 (Javadoc 1), the documentation for ListOrderedMap.remove states that an IndexOutOfBoundsException should be thrown when the index is invalid. This directly suggests tests that pass invalid indices, such as −1 or an index equal to the map size, and expect an IndexOutOfBoundsException. Listing 2 (Javadoc 2) illustrates the same idea for a null-triggered exception. More generally, when a Javadoc clause identifies the condition that triggers an exception, it provides actionable specification evidence that can be translated into concrete inputs and expected exceptional behavior. In this sense, Javadoc can support automated exception-oracle generation rather than merely describe exceptional behavior.

CCS Concepts • Software and its engineering → Software testing and debugging.

Keywords automated software testing, test oracle generation, exception oracles, large language models, Javadoc, model attribution ACM Reference Format: Soneya Binta Hossain, Matthew B. Dwyer, and Tasfia Tasnim. 2026. Documentation vs. Code Patterns: What Drives LLM-Based Exception Oracle Generation?. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3832783. 3837464

This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3837464

Introduction

ASE ’26, October 12–16, 2026, Munich, Germany

Listing 1: Different ways of writing exception oracles in Java. // (1) Using the JUnit 4 @Test annotation @Test ( expected = IOException . class ) public void test0 () throws IOException { MyFileReader r = new MyFileReader ( " none . txt " ) ; r. open () ; } // (2) Using JUnit 5 ' s assertThrows @Test public void test1 () { assertThrows ( IOException . class , () -> { MyFileReader r = new MyFileReader ( " none . txt " ) ; r. open () ; }) ; } // (3) Using explicit try - catch with a fail () @Test public void test2 () { try { MyFileReader r = new MyFileReader ( " none . txt " ) ; r. open () ; fail (" Expected IOException not thrown " ) ; } catch ( IOException e ) { // test passes if exception is thrown } }

Recent TOG work has advanced from assertion-oriented learning approaches [24], to documentation-driven techniques for exceptional behavior [2, 23], to neural and LLM-based systems such as TOGA, ChatAssert, TOGLL, and Doc2OracLL [5, 7, 11, 13]. However, the basis of reported exception-oracle performance remains unclear. Prior TOG work has mainly focused on assertion oracles or on treating documentation as a broad context source, leaving the drivers of exception-oracle prediction underspecified. This matters because high accuracy alone does not show whether a model used meaningful exception evidence, such as a documented precondition, or relied on recurring shortcut cues in code, documentation, or test prefixes. In this paper, we investigate what actually drives exceptionoracle prediction in neural and LLM-based TOG. We answer this through a large-scale intervention-based study of three representative TOG systems on three real-world Java datasets: two generatedtest benchmarks (OE25 and Sf110) and OE25dev , a newly curated benchmark of developer-written tests from 25 systems. We define exception-oracle accuracy as an oracle-type prediction metric: for an exception-labeled instance, a prediction is correct when the instance is classified as Exception, not Assertion. This metric does not measure semantic equivalence, compilability, oracle style, or exact exception-type matching. Section 2.4 gives the formal definition. To identify the signals behind these predictions, we compare model behavior before and after removing Javadoc @throws clauses, and then apply attribution-guided substitution ablations with cue categorization across the test prefix, focal code, and documentation. Removing these exception clauses (ECs) causes no change or only very small drops in accuracy: the largest observed overall drop is 0.16 percentage points, and the largest drop among samples that originally contained exceptional-behavior clauses is 0.54 percentage points. This suggests that structured exception documentation is not the main driver. Instead, attribution-guided ablation shows that

Soneya Binta Hossain, Matthew B. Dwyer, and Tasfia Tasnim

Listing 2: Examples of Javadoc exception clauses and corresponding exception-oracle tests. Javadoc (1) /* * * Removes the element at the specified index . * @param index the index of the object to remove * @return the removed value , or { @code null } if none existed * @throws IndexOutOfBoundsException if the index is invalid */ public void test0 () { resetEmpty () ; resetFull () ; final ListOrderedMap <K ,V > lom = getMap () ; assertThrows ( IndexOutOfBoundsException . class , () -> lom . remove ( -1) ) ; } Javadoc (2) /* * * Creates a parameterized type instance . * @param rawClass the raw class to create a parameterized type instance for * @param typeArguments the types used for parameterization * @return { @link ParameterizedType } * @throws NullPointerException if { @code rawClass } is { @code null } * @since 3.2 */ public void test1 () { final Class <? > rawClass = null ; final Map < TypeVariable <? > , Type > tvMap = Collections . emptyMap () ; assertThrows ( NullPointerException . class , () -> TypeUtils . parameterize ( rawClass , tvMap )); }

models rely on alternative input signals. For TOGLL, correct predictions often flip after only 3–4 high-attribution substitutions and are dominated by structural cues such as newlines. For Doc2OracLL, predictions are harder to flip, but still depend on distributed lexical cues rather than on @throws/@exception clauses alone. Overall, this work argues that LLM-assisted task automation should be evaluated beyond accuracy alone. For TOG, this means checking whether exception predictions are grounded in meaningful evidence—such as the triggering precondition, method under test, documentation, or feasible throw path. It also means using controlled contexts, diverse tests, hard negatives, and counterfactual cases to expose shortcut cues such as formatting patterns or recurring lexical tokens. In summary, this paper makes the following contributions: • We present a large-scale intervention-based study of the signals that drive exception-oracle prediction in neural and LLMbased TOG. • We introduce OE25dev , a benchmark of developer-written Java tests that complements prior generated-test datasets with more realistic test structures and oracle patterns. • We show that Javadoc @throws/@exception clauses have limited effect, while alternative input cues play a much larger role in exception-oracle prediction. • We derive practical guidance for future TOG systems: evaluate beyond accuracy, control for shortcut cues, and require predictions to be grounded in exception-triggering evidence.