Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

arXiv:2607.27648v1 [cs.SE] 30 Jul 2026

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows Xiaoyu Zhang

Xianyun Cheng

Tianlin Li∗

Nanyang Technological University Singapore, Singapore xiaoyu.zhang@ntu.edu.sg

National University of Singapore Singapore, Singapore xianyuncheng0909@gmail.com

Beihang University State Key Laboratory of Complex & Critical Software Environment Beijing, China tianlin001@buaa.edu.cn

Yuwei Zheng

Yue Yang

Yang Liu

Beihang University State Key Laboratory of Complex & Critical Software Environment Beijing, China

Xi’an University of Architecture and Technology College of Information and Control Engineering Xi’an, China

Nanyang Technological University Singapore, Singapore

Abstract

Keywords

LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users’ perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.

LLM Document Workflow, LLM Testing, Metamorphic Testing

CCS Concepts • Software and its engineering → Software testing and debugging; Software reliability. ∗ Corresponding author.

This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3837456

ACM Reference Format: Xiaoyu Zhang, Xianyun Cheng, Tianlin Li, Yuwei Zheng, Yue Yang, and Yang Liu. 2026. Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3832783.3837456

1

Introduction

With the rapid advancement of large language models (LLMs), LLMdriven software systems are increasingly evolving from plain-text conversations to document-centric end-to-end workflows [17, 33]. These systems widely integrate functionalities such as file parsing, external tool invocation, and structured output generation. User inputs are not confined to simple text prompts. Instead, they often take the form of semi-structured or structured files (e.g., medical records, system logs) as first-class inputs [26]. In such end-to-end workflows, file formats (e.g., TXT, CSV) are no longer merely interface wrappers but have become crucial components for organizing, transmitting, and presenting task information in real-world LLM software systems. As illustrated in Figure 1, even when two formats carry identical semantic content, the workflow can transform them into structurally different intermediate representations before they reach the LLM, making the input format a non-trivial factor in the workflow behavior. This shift is especially consequential in high-stakes scenarios. Whether in clinical decision support or financial credit tasks, LLMdriven document workflows and systems are increasingly assuming responsibilities related to judgment, recommendation, and even automated execution [26, 34]. In these settings, erroneous, inconsistent, or unfair workflow behaviors can propagate beyond the model’s output to downstream decision-making, resource allocation, and risk control processes. Therefore, there is an urgent need to study the potential risks of LLM document workflows and ensure the safety and reliability of these systems in real-world deployment.

ASE ’26, October 12–16, 2026, Munich, Germany

Existing research has proposed diverse methods for evaluating the reliability of LLM systems [27, 28, 59, 68, 73]. However, existing methods mainly consider the input as a single prompt string, focusing on model behavior at the plain-text level rather than the document workflows increasingly important in real-world deployments [17]. In real-world applications, users routinely submit the exact same task content through completely different formats (e.g., CSV and JSON). Therefore, whether the document workflow can maintain behavioral consistency when processing inputs with different formats (i.e., format robustness) is not merely a theoretical question, but a practical challenge that exposes a highly critical vulnerability in current LLM-driven software. In addition, recent studies and real-world discussions have confirmed that format variations could severely impact LLM system performance [5, 56]. However, these studies remain limited to specific tasks or individual document types, lacking a systematic evaluation across diverse formats in end-to-end workflows. This gap leaves a fundamental question unanswered: Can LLM document workflows remain robust when faced with input format variation? The question is critical because the risks arising from format variations can be ‘silent’. The system may outwardly return structurally valid and formally complete outputs, while the evidence, reasoning paths, and even the final decisions have undergone substantial deviations. To fill the gap, in this paper, we propose a format-aware metamorphic testing framework tailored for end-to-end LLM document workflows. The design principle is that, given that the semantics and constraints of the input are fixed, merely altering the external document format should not substantively affect the behavior of LLM document workflow (i.e., format robustness). The framework consists of two phases. The data construction phase first converts the input plain-text prompts into multiple semantically equivalent file formats, ensuring consistency in identifiers, field boundaries, and instruction semantics recorded across different formats. Subsequently, the workflow evaluation phase enters these aligned inputs into the LLM document workflow via provider-specific file upload and processing pathways and collects the corresponding outcomes from the workflows. We also design and implement three levels of metamorphic relations (MRs) to achieve a comprehensive examination of the format variance of workflows across different input formats (i.e., ‘decision outcome invariance’, ‘reasoning evidence invariance’, and ‘execution stability invariance’). Based on this framework, we conduct a large-scale study across four high-stakes datasets and four document formats, targeting mainstream LLM document workflows driven by representative LLMs. The study aims to answer the following research questions (RQs): • RQ1 (Decision Outcome): How does the input format affect the efficacy of the given tasks when the semantic content remains unchanged? This RQ evaluates the extent to which LLM document workflows violate the MR of ‘decision outcome invariance’. We conduct a comparative analysis on 12,000 groups of execution results across four workflows and four datasets. The results indicate that MR1 violations are pervasive, with an average of 41.33% of instances exhibiting inconsistent decision outcomes across formats. Specific format inputs cause the accuracy to drop by up to 56.00%. Notably, the degradation exhibits strong workflow dependency, and degradation severity is jointly driven by the interaction between format and workflow processing mechanisms.

Zhang et al.

• RQ2 (Reasoning Evidence): How does the input format affect the reasoning evidence presented by the model? To delve into the root causes behind the inconsistent decisions, we quantitatively evaluate the extent to which workflows violate the MR of ‘reasoning evidence invariance’. Specifically, we extract and compare the key factual clues in the workflow responses under different input formats. The analysis reveals that an average of 25.14% of instances exhibit significant evidence drift across formats, and 63.56% of these cases simultaneously trigger decision changes. More alarmingly, 36.44% of the evidence drift cases are silent violations, where the final decision remains unchanged, but the underlying reasoning basis has fundamentally shifted, posing severe threats to the interpretability of LLM workflows. • RQ3 (Execution Stability): How does the input format affect the system’s decision stability during multiple executions? We measure the consistency of workflow decisions across repeated executions under different formats. The results show that instability is not merely an occasional anomaly but a recurrent phenomenon, where the average MR3 violation rate reaches 45.33%, with the highest 𝑀3 reaching 88.80% on the Anthropic workflow. In addition, specific structured formats (e.g., CSV) can cause the stability metric to drop to as low as 11.00%, exposing severe reliability hazards in current LLM-driven software when processing non-textual files. • RQ4 (Mitigation Strategy): Can test-time wrappers effectively mitigate the format-induced errors? From the user’s perspective, we design two lightweight test-time mitigation strategies, namely the vote aggregation strategy and the format routing strategy, to explore whether client-side intervention can cost-effectively mitigate format-induced errors without modifying or retraining the underlying models. Experiments confirm that format-induced errors are largely recoverable. The vote aggregation strategy yields only limited improvements, whereas our format routing strategy achieves substantial gains (up to a 44.21% reduction in MR violation rate on 𝑀1 ). The findings suggest that the representation layer functions as an effective engineering control point within LLM document workflows, and that substantial robustness gains can be realised through the implementation of lightweight, deploymentcompatible safeguards. The contributions of this paper are as follows. • We propose the first format-aware metamorphic testing framework for LLM-driven document workflows and introduce three metamorphic relations for semantically equivalent but differently formatted inputs, bridging the testing gap for end-to-end LLM systems. • We conduct a large-scale empirical study on the performance of four LLM document workflows across four formats. The results indicate that even when semantic content is aligned, specific structured inputs trigger significant and uneven degradation in the correctness, fairness, and stability of LLM system outcomes. • We design lightweight and deployment-compatible mitigation strategies, in which our format routing strategy can effectively mitigate the format-induced errors and degradation without model retraining. • Our pipeline implementation, dataset, mitigation strategy, and the necessary results are available at [4]

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

2 Background & Related Work 2.1 LLM Document Workflow In high-stakes scenarios (e.g., medical diagnostic assistance), LLM systems increasingly rely on external documents rich in structured and semi-structured information [17, 32, 34, 54, 62]. To support such complex reasoning tasks, LLM document workflows typically encompass four interconnected stages. • Input Stage receives and preprocesses the raw documents provided by the user, which can range from plain text files to structured formats such as medical records, logs, and tables [26]. This stage handles file uploading, type identification, content extraction, and chunking, thereby determining what information the system can subsequently access and at what granularity [17]. • Organization Stage converts the extracted content into an intermediate representation suitable for model consumption, such as serialized text snippets or lightweight tabular structures [33]. Crucially, the same semantic content undergoes different transformation paths under different formats: plain text preserves linear narrative order, whereas CSV or other tabular formats explicitly expose field boundaries and header relationships [35, 56]. • Reasoning Stage invokes the LLM to interpret the intermediate representation (e.g., ‘Model Input’ in Figure 1). Then the model selects evidence and produces reasoning results. Once the document format alters field boundaries or local structures in the prior stage, the model’s attentional focus, evidence citation, and reasoning paths may correspondingly shift. • Output Stage translates the reasoning results into outputs for human users or downstream system components. If evidence shift or decision drift has been induced upstream by format changes, these deviations can be propagated at this stage in the form of structurally valid but semantically incorrect outputs [6]. Prior work has revealed that structured content represented before model inference can affect LLM behaviors. For tabular data, prior studies [24, 69] demonstrate that structural cues (e.g., field boundaries) are central to table understanding. Recent work further shows that the choice of table serialization format directly influences reasoning quality and downstream decision accuracy [5, 36, 41, 56]. For document understanding, researchers [33, 35] reveal that representation-aware pre-training and the organization of retrieved content can shape what information the model attends to. However, these works focus on uncovering phenomena or designing training methods, and lack a systematic evaluation of the format robustness of the LLM document workflows. To formalize the format-induced workflow behavior changes, in this paper, we denote the workflow under test as 𝑊 and the set of supported formats as 𝐹 . The input document is represented as 𝐼 = 𝜙 (𝑥, 𝑓 ), where 𝜙 is a format rendering function, 𝑥 ∈ X is a semantic instance (e.g., a medical record) from the input space X, and 𝑓 ∈ 𝐹 is a file format. The workflow generates a structured output based on the prompt 𝑃, abstracted as 𝑂 = 𝑊 (𝐼, 𝑃) = ⟨𝐷 (𝐼 ), 𝐸 (𝐼 )⟩, where 𝐷 (·) denotes the final decision outcome and 𝐸 (·) signifies the observable reasoning evidence extracted to support this decision.

2.2

Metamorphic Testing

Metamorphic testing (MT) addresses the oracle problem in software testing by constructing semantically related inputs and verifying

ASE ’26, October 12–16, 2026, Munich, Germany

whether the system’s outputs satisfy a predefined relationship, known as the Metamorphic Relation (MR) [8, 9, 52]. Formally, an MR describes the expected constraints between the source test input and its variants, as well as between their corresponding system outputs. Given a source test case, the tester applies a controlled transformation based on an MR to generate variants, and then checks whether the corresponding outputs still satisfy the expected relationship. A violation of MR indicates a potential defect, robustness issue, or behavioral inconsistency in the target system [9, 52]. MT has been widely applied in traditional software engineering (SE) tasks such as compiler verification and machine learning system testing [16, 66, 74]. In the LLM era, the inherent randomness of generative models makes strict test oracles highly challenging to construct, and MT has consequently emerged as a popular strategy for evaluating LLM-driven systems. Recent studies [11, 29] demonstrate that well-designed MRs can effectively expose behavioral inconsistencies in LLM systems without requiring golden oracles for every input. In this paper, we extend MT to end-to-end LLM document workflows, grounding the framework in the principle that alterations in document format should not induce substantive changes in the system output, and designing three complementary MRs to reveal and quantify format-induced risks. Note that, different from the differential testing methods [43, 44] that compare the outputs of multiple implementations under the same inputs, our approach treats each LLM document workflow (e.g., OpenAI) as the system under test and evaluates whether changes in input formats induce unexpected changes in workflow outputs (i.e., whether they violate MRs).

2.3

LLM Testing

With the increasing adoption of LLMs in modern software systems, researchers have proposed diverse testing methods at both the model and system levels. At the model level, adversarial prompts and robustness perturbations are used to reveal behavioral inconsistencies and vulnerabilities [18, 61, 75], while bias-oriented approaches expose pervasive group-level biased behaviors [28, 59]. Recent studies further shift the focus to software system-level behavior [28, 63, 65, 73], demonstrating that system behavior depends not only on model capability but also on orchestration logic, tool invocation, and context management [63, 65]. Beyond model-level work, researchers also target faults that arise specifically from system-level interactions. Recently, researchers [68] systematically expose indirect prompt injection vulnerabilities in LLM-integrated systems triggered by retrieved content. A recent study [71] also reveals that tool-calling LLM systems are susceptible to injections embedded in tool responses. However, existing methods primarily target models and systems that accept plain-text inputs. This paper designs a metamorphic testing framework for end-to-end LLM document workflows, using three MRs to systematically evaluate and study the format robustness of the workflows.

3

Motivation Case

To illustrate the impact of input format on LLM document workflows, Figure 1 presents a case from a medical care priority assessment workflow driven by GPT-4o-2024-08-06. This document workflow receives patient medical records as input and determines

ASE ’26, October 12–16, 2026, Munich, Germany

Zhang et al.

Figure 2. Overview of the Testing Framework.

4

Figure 1. A Motivation Case on Document Workflow Driven-by GPT-4o.

which patient should receive priority care. Under this setting, the task, input medical information, and output constraint are all fixed; therefore, the workflow’s reasoning and decision should remain invariant to the external document format. When the record is uploaded as a TXT attachment, the workflow accurately identifies the key clinical symptom (‘leg fracture’) and indicates that no clinical information indicates that either patient requires more urgent care than the other. Then it correctly concludes that both patients 1 In this should receive equal priority, as shown in Figure 1(○). case, the reasoning evidence remains grounded in clinical content rather than irrelevant demographic attributes. However, when the identical content is converted into the CSV format and uploaded through the same workflow, the behavior degrades significantly. Instead of attending to clinical information, the workflow shifts its focus to non-clinical demographic cues in the structured fields (e.g., ‘odd jobs’ and ‘employee’) and uses these clues as the basis for judgment. As a result, the system finally obtains an incorrect and biased judgment that ‘Patient A’ should receive high medical 2 Detailed results and reproducible priority, as shown in Figure 1(○). scripts are available in our repository [4]. This case demonstrates that merely altering the input format can cause the workflow to shift its reasoning evidence and reverse its decision outcome, both of which constitute violations of the metamorphic relations (MRs) that a robust workflow should satisfy. Such violations are not isolated anomalies but systemic risks that can be repeatedly triggered. Our experiments across four workflows and four datasets (§5) reveal that format-induced decision shifts occur in 41.33% of instances on average. For workflows deployed in high-stakes scenarios (e.g., healthcare- or financial-related tasks), the consequences of such MR violations are particularly severe. The workflow may superficially return structurally valid outputs, yet the evidence it attends to, its reasoning pathways, and its decisions have already drifted. This can silently propagate errors into downstream decision-making, resource allocation, and risk control processes, ultimately compromising system security, user safety, and societal well-being. Therefore, there is an urgent need to evaluate and study the format robustness of LLM document workflows and design deployable mitigation methods to enhance the reliability of these systems in real-world deployment.

Design

To evaluate the behaviors and reveal potential risks of end-to-end LLM document workflows when confronted with various input formats, we design and implement a format-aware metamorphic testing framework. Different from existing testing methods that treat the LLM as the test object [27, 59], our framework considers the end-to-end LLM document workflow. The core principle of this framework is that, given the fixed semantics and tasks, merely changing the input document format should not substantially change the workflow’s behavior. Based on this principle, we first define three MRs applicable to LLM document workflows to describe the relationship between variations in workflow inputs and the workflow outcomes (§4.1). The framework then operates in two main phases, namely data generation and workflow evaluation. Specifically, the generation phase (§4.2) applies controlled transformations to the source input based on the predefined MRs to generate test variants that are in different formats but with equivalent semantics. The evaluation phase (§4.3) inputs these variants into the LLM workflow, collecting and comparing system behaviors to identify whether any MR is violated. Implementation details are available in our repository [4].

4.1

Metamorphic Relations

Assessing a complex LLM document workflow solely at the level of the final answer is insufficient, as a workflow may preserve its final label while silently changing the evidence it relies on, or become substantially less stable under one input format than another. To comprehensively reveal such format-induced errors, we design three complementary MRs covering the workflow’s behavior from three critical dimensions [72], namely macroscopic correctness (decision outcome), microscopic interpretability (reasoning evidence), and runtime robustness (execution stability). Together, they capture not only explicit decision failures but also subtle, silent degradations in the system’s reasoning pathways. • MR1: Decision Outcome Invariance. For the same semantics of the input document 𝑥, any two different formats 𝑓𝑖 and 𝑓 𝑗 in the supported format set 𝐹 that produce semantically equivalent inputs should yield equivalent final decision outcomes on the workflow. This MR can be formalized as follows: ∀𝑥 ∈ X, ∀𝑓𝑖 , 𝑓 𝑗 ∈ 𝐹 : 𝑓𝑖 ≠ 𝑓 𝑗 ∧ 𝜙 (𝑥, 𝑓𝑖 ) ≡ 𝜙 (𝑥, 𝑓 𝑗 ) ⇒ 𝐷 (𝜙 (𝑥, 𝑓𝑖 )) = 𝐷 (𝜙 (𝑥, 𝑓 𝑗 )). where 𝐷 (·) represents the final decision of the workflow on the input 𝐼 = 𝜙 (𝑥, 𝑓 ). A violation of MR1 represents a critical breakdown of the system’s primary function. If the workflow outputs a correct decision when processing 𝑓𝑖 but the decision flips when processing 𝑓 𝑗 , it reveals that the decision-making is brittle and highly

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

susceptible to syntactic noise. In high-stakes scenarios like medical consultation or financial assessment, such format-induced decision reversals can lead to severe misdiagnoses or unfair actions, posing direct threats to system reliability and user safety. • MR2: Reasoning Evidence Invariance. In complex document workflows, macroscopic decision consistency may mask underlying biases in information ingestion and reasoning. Therefore, this MR describes that for different formats of the same semantic instance, the core evidence in the workflow output should not exhibit significant shifts. This is formally represented as: ∀𝑥 ∈ X, ∀𝑓𝑖 , 𝑓 𝑗 ∈ 𝐹 : 𝑓𝑖 ≠ 𝑓 𝑗 ∧ 𝜙 (𝑥, 𝑓𝑖 ) ≡ 𝜙 (𝑥, 𝑓 𝑗 )
⇒ 𝑆𝑖𝑚 𝐸 (𝜙 (𝑥, 𝑓𝑖 )), 𝐸 (𝜙 (𝑥, 𝑓 𝑗 )) ≥ 𝜃 . where 𝐸 (·) denotes the evidence set extracted from the workflow output and 𝑆𝑖𝑚 is a similarity function between evidence sets. When the similarity falls below the threshold 𝜃 , it indicates that the evidence has undergone a substantial drift. Even if MR1 is preserved, a significant drift in evidence implies that the system has been distorted by the input format, causing it to ground its decision in irrelevant or hallucinated clues (i.e., silent violation). This undermines the interpretability, auditability, and accountability of the workflow, as the system could arrive at correct conclusions for entirely erroneous or arbitrary reasons, severely limiting its trustworthy deployment in high-stakes scenarios. • MR3: Execution Stability Invariance. For the same semantic content 𝑥, switching the input format should not alter whether the system can produce consistent decisions across 𝑘 repeated executions. To formalize this, we first define an execution consistency for a given instance 𝑥 under format 𝑓 , that is h i ′ 𝑆 (𝜙 (𝑥, 𝑓 )) = 1 ∀ 1 ≤ 𝑟 ≠ 𝑟 ′ ≤ 𝑘 : 𝐷 (𝑟 ) (𝜙 (𝑥, 𝑓 )) = 𝐷 (𝑟 ) (𝜙 (𝑥, 𝑓 )) , where 𝐷 (𝑟 ) (𝜙 (𝑥, 𝑓 )) denotes the decision outcome of the 𝑟 -th execution. 𝑆 = 1 indicates that all 𝑘 repeated runs yield identical decisions (i.e., the instance is stable under format 𝑓 ), while 𝑆 = 0 indicates that at least one pair of runs produces different decisions (i.e., the instance is unstable under that format). Therefore, MR3 states that for any two distinct formats 𝑓𝑖 and 𝑓 𝑗 : ∀𝑥 ∈ X, ∀𝑓𝑖 , 𝑓 𝑗 ∈ 𝐹 : 𝑓𝑖 ≠ 𝑓 𝑗 ∧ 𝜙 (𝑥, 𝑓𝑖 ) ≡ 𝜙 (𝑥, 𝑓 𝑗 ) ⇒ 𝑆 (𝜙 (𝑥, 𝑓𝑖 )) = 𝑆 (𝜙 (𝑥, 𝑓 𝑗 )). If the workflow produces consistent decisions for a given instance 𝑥 under format 𝑓𝑖 , it should also remain consistent under format 𝑓 𝑗 , and vice versa. A violation of MR3 indicates that format variation is not merely changing isolated outputs, but is degrading the execution reliability of the workflow on specific instances. In practice, such format-induced instability can undermine reproducibility, complicate debugging and validation, and erode user trust in the workflow’s outputs.

4.2

Data Generation

Based on the predefined MRs, the testing framework first generates semantically equivalent yet format-heterogeneous variants for each source instance. Note that this phase only alters the external document representation, not the sample content itself. Specifically, this phase first extracts and converts each source instance

ASE ’26, October 12–16, 2026, Munich, Germany

𝑥 into an intermediate representation 𝐶𝑥 . This representation encapsulates the core information of the given instance, such as the problem background and demographic data. This layer aims to normalize raw samples from diverse data sources into a unified abstract semantic space, thereby ensuring that subsequent format transformations originate from a consistent semantic basis, avoiding unreliable paraphrasing or rewriting on the source instance. The set of transformed variants can be represented as: 𝑉 (𝑥) = {𝜙 (𝑥, 𝑓 ) = T (𝐶𝑥 , 𝑓 ) | 𝑓 ∈ 𝐹 }. The design of the transformation operator T follows three con1 Semantic Invariance. The problem description, key facts, straints. ○ field values, and their corresponding relationships remain consis2 Representational Faithfulness. Each tent across different formats. ○ variant should faithfully conform to the natural representation conventions of a real-world workflow, rather than being artificially compressed into a uniform text template. For example, a problem rendered in CSV format should place key field information into distinct cells rather than cramming the entire context into a single 3 Instruction Isolation. The task objectives and structured cell [50]. ○ output instructions (i.e., the prompt 𝑃 in §2.1) maintain identity, ensuring that the task requirements received by the LLM workflow are decoupled from the input files 𝐼 of different formats. This means that for the variants of the given source instance, the only factor that changes is the external representation format of the document. This constraint ensures that any observed decision differences, evidence drifts, or stability disparities between variants can be solely attributed to the workflow’s response to format changes, rather than to semantic changes in the task itself.

4.3

Workflow Evaluation

In this phase, the framework inputs the variant set 𝑉 (𝑥) and the input prompt 𝑃 into the LLM document workflow 𝑊 and evaluates whether the execution results violate the predefined MRs. Note that the test object of the framework is the end-to-end LLM document workflow with file loading, serialization, and context organization methods [17], rather than a single model. Therefore, for each variant 𝐼 = 𝜙 (𝑥, 𝑓 ), the framework invokes the native file-ingestion and processing methods or interfaces implemented by the target workflow and then collects execution results under unified output constraints. 𝑂 𝑟𝑓 = 𝑊 𝑟 (𝐼, 𝑃), where 𝑃 denotes the task objectives and instructions for output structure, and 𝑟 denotes the repeated execution index. The structured instructions ask the LLM workflow to return 𝑂 𝑟𝑓 , including the final decision outcome 𝐷 𝑟𝑓 and explicit evidence set 𝐸𝑟𝑓 . Note that 𝐸𝑟𝑓 does not represent the complete internal reasoning chain of the model but rather consists of observable, source-grounded factual cues (e.g., key phrases extracted from the input file) that the workflow references to support its decision. Based on the collected outputs, the framework automatically evaluates each MR. For MR1, it checks whether the final decisions remain identical across all format variants of the same instance. For MR2, it computes the similarity between the evidence sets of different format outputs to determine whether they still point to the same core facts, thereby detecting silent evidence drift even when decisions appear

ASE ’26, October 12–16, 2026, Munich, Germany

consistent. For MR3, it measures the decision consistency across 𝑘 repeated executions under each format and compares the resulting stability profiles to identify format-induced reliability degradation. The framework records all instances that violate any MR, and these results form the foundation of the experimental analysis in §5.

5

Experiment

In this section, we report and analyze the experimental results to answer the following RQs: (1) RQ1: How does the input format affect the efficacy of the given tasks when the semantic content remains unchanged? (2) RQ2: How does the input format affect the reasoning evidence presented by the model? (3) RQ3: How does the input format affect the system’s decision stability during multiple executions? (4) RQ4: Can test-time wrappers effectively mitigate the formatinduced errors?

5.1

Setup

LLM Document Workflow. We evaluate four production-level workflows widely used in high-stakes scenarios [3, 40, 45], whose underlying models are among the most adopted LLMs with strong reasoning capabilities [10]. The OpenAI workflow (backed by GPT4o-2024-08-06) processes documents through the Responses API with a code interpreter sandbox for file extraction and serialization [46, 47]. The Anthropic workflow (powered by Claude Haiku 4.5) and the Google workflow (Gemini 2.0 Flash) accept documents via native Files APIs, where the backend parses, chunks, and injects content into the message context [13, 21, 22]. The Alibaba workflow (powered by Qwen-Long) provides long-context document processing via an OpenAI-compatible interface, handling serialization before routing to the LLM reasoning [2]. These architectural differences in upstream processing are precisely what distinguish workflow-level evaluation from model-level benchmarking. Dataset. We conduct experiments on four datasets spanning healthcare, social decision-making, and financial scenarios, randomly 1 MedQA [48] is derived from sampling 250 instances from each. ○ the English USMLE subset of MedQA [31], containing multiplechoice questions from US medical licensing exams with ground2 Construct is a self-constructed dataset integrating truth answers. ○ demographic and healthcare-related fields from multiple public datasets [1, 38, 49, 53, 57, 60, 70], designed to supplement multi-field structured record scenarios common in real-world workflows. De3 DiscrimEval is extracted from the tails are in our repository [4]. ○ implicit subset of Anthropic’s discrimination evaluation dataset [58], used to examine whether workflows exhibit systematic bias shifts 4 Credit Card is sampled from a real-world under different formats. ○ Taiwan bank credit card dataset with 30,000 client records and default labels [67], used to evaluate format-induced decision drift in financial risk assessment. Metrics. This study mainly uses the Metamorphic Relation Violation Rate (MRV) to directly quantify the degree to which a workflow violates the MRs defined in §4.1. This metric serves as the primary evaluation metric across all four RQs. To further interpret the practical consequences of such violations, we additionally adopt three task-level metrics, namely accuracy, fairness, and stability, each

Zhang et al.

capturing a complementary dimension of format-induced degradation at the dataset level. Throughout the following metrics, 𝑖 denotes the instance index, 𝑁 is the total number of instances, and 𝑦 is the ground-truth label. • MRV quantifies the proportion of instances for which the workflow violates a given MR. Let 𝑙 ∈ {1, 2, 3} correspond to the three MRs and 1[·] denote whether an instance triggers the violation. The violation rate for the 𝑙-th MR is: 𝑁 1 ∑︁ 1[𝑀𝑅𝑙 (𝑥𝑖 ) is violated] × 100%. 𝑀𝑙 = 𝑁 𝑖=1 • Accuracy quantifies the impact of format variations on the workflow’s ability to produce correct answers and complete given tasks. For tasks with authoritative ground-truth answers (i.e., MedQA and Construct), accuracy is calculated as the proportion of instances where the system decision 𝐷 is consistent with the ground truth 𝑦: 𝑁

𝐴𝑐𝑐 =

1 ∑︁ 1[𝐷 = 𝑦𝑖 ] × 100%. 𝑁 𝑖=1

• Fairness captures format-induced bias shifts in decision-making tasks involving social attributes (e.g., race and gender). Following group-disparity frameworks [20, 23, 49], we first compute the group sensitivity to quantify the decision-rate disparity across subgroups of a sensitive factor pair 𝑔 under format 𝑓 . Let 𝑝 𝑓 ,𝑔,𝑣 denote the affirmative decision rate among instances where 𝑔 takes value 𝑣; the group sensitivity is: 𝑆𝑒𝑛 𝑓 (𝑔) = max 𝑝 𝑓 ,𝑔,𝑣 − min 𝑝 𝑓 ,𝑔,𝑣 . 𝑣

𝑣

To isolate the format-specific effect, we apply normalization to obtain the Format Disparity Score (FDS): FDS 𝑓 (𝑔) = (𝑆𝑒𝑛 𝑓 (𝑔) − 𝜇𝑔 )/(𝜎𝑔 + 𝜖), where 𝜇𝑔 and 𝜎𝑔 are the mean and standard deviation of 𝑆𝑒𝑛 𝑓 (𝑔) across all formats. A higher FDS indicates that the format disproportionately amplifies group disparity [57]. • Stability measures the consistency of the workflow’s decisions across 𝑘 repeated executions under a specific format 𝑓 [19]. It is computed as the proportion of instances for which all repeated runs yield identical decisions: 𝑁

𝑆𝑡𝑏 𝑓 =

1 ∑︁ 𝑆 (𝜙 (𝑥𝑖 , 𝑓 )) × 100%, 𝑁 𝑖=1

where 𝑆 (·) is the execution consistency function defined in §4.1. By comparing 𝑆𝑡𝑏 𝑓 across formats, we can reveal how specific document representations amplify the stochastic uncertainty of workflow outputs. Implementation. We evaluate four document formats commonly used in real-world software systems [33, 37, 56]: TXT, MD, JSON, and CSV. While XLSX is common in practice, the Google workflow returns parsing errors, so we exclude it and use CSV to ensure tabular representations remain covered. We adhere to the official recommended settings for each workflow and set the temperature to 0. For each instance, we conduct 3 independent repeated executions per workflow-format combination. For MR2, the workflow is instructed to output up to 10 source-grounded evidence items per decision. Each extracted evidence item is then mapped to a set of pre-defined semantic categories via keyword matching (e.g.,

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

ASE ’26, October 12–16, 2026, Munich, Germany

7;7

Table 1. MRV Results Across Workflow Providers (Bold and underlined marks the highest violation across workflows). MRV (%)



Workflow Provider OpenAI

Anthropic

Google

Alibaba

Avg.

MedQA

𝑀1 𝑀2 𝑀3

65.86 16.80 71.37

76.44 12.00 88.80

17.40 19.70 15.73

41.96 17.20 31.85

50.42 16.43 51.94

Construct

𝑀1 𝑀2 𝑀3

29.47 7.60 86.40

44.53 64.90 64.80

21.47 6.40 4.80

57.20 1.20 63.20

38.17 20.03 54.80

DiscrimEval

𝑀1 𝑀2 𝑀3

17.07 37.60 35.20

8.80 16.90 0.40

12.27 53.10 11.60

21.73 34.50 21.20

14.97 35.52 17.10

Credit Card

𝑀1 𝑀2 𝑀3

52.93 53.60 78.80

25.47 0.00 30.80

91.60 14.40 36.00

77.07 46.40 84.40

61.77 28.60 57.50

Avg.

𝑀1 𝑀2 𝑀3

41.33 28.90 67.94

38.81 23.45 46.20

35.69 23.40 17.03

49.49 24.83 50.16

41.33 25.14 45.33

‘65-year-old male with fever and cough’ maps to ‘{age, gender, symptom}’), absorbing surface-form variation across formats. Over 90% of items map successfully, and an unmapped item paired with a non-empty set receives similarity 0 as a conservative penalty. We use Jaccard similarity for the evidence overlap metric (𝑆𝑖𝑚 in MR2) and Jensen-Shannon divergence for evidence distribution shift [7]. Detailed implementation is released in our repository [4]. Following prior work [7, 19], we set the threshold 𝜃 to 0.3. To further validate this choice, two experts independently evaluate 20 samples (10 violations, 10 non-violations) across thresholds (0.1, 0.3, 0.5, 0.7), confirming that 0.3 minimizes both false positives and false negatives. In addition, to enhance the reliability of the experimental results, we employ a bootstrap resampling strategy [42]. Specifically, when calculating the metrics, we use the bootstrap method to resample 1,000 samples of LLM responses, and then calculate the mean and 95% confidence interval.

5.2

RQ1: Impact on Decision Outcome

To systematically investigate the impact of input formats on the decision outcomes of LLM document workflows, we collect and analyze a total of 12,000 execution response groups across four workflows and four datasets. Each group contains four workflow responses to the same semantic input in four formats. We first compute the MR1 violation rate (𝑀1 ) for each workflow and analyze the format distribution of erroneous results among instances that trigger correctness flips, thereby revealing the underlying error patterns of the workflows. Table 1 presents the MRV results across workflows and datasets, where bold and underlined number in each row marks the workflow with the highest violation rate for each metric. Figure 3 shows the distribution of erroneous results across formats, where the X-axis denotes the workflow provider and bars with and without dashed fills correspond to different datasets. We then quantitatively evaluate the impact of format variations on the overall task performance (i.e., accuracy) of workflows on the MedQA and Construct datasets, which possess ground-truth answers, to examine how MR1 violations translate into functional degradation. Figure 4 shows the accuracy of each workflow under different input formats.

(UURU&RXQW

Dataset



-621



0'

&69

0HG4$



&RQVWUXFW

 





 

 

2SHQ$,

$QWKURSLF

*RRJOH

$OLEDED

Figure 3. Erroneous Results Across Formats.

• Analysis of MR1 Violation. Table 1 reveals that MR1 violations are pervasive across all experimental settings. Across the experimental datasets and workflows, 41.33% of instances exhibit inconsistent decision outcomes between at least one pair of formats, confirming that format variations systematically disrupt outcome invariance. Notably, the Google workflow on Credit Card reaches the highest 𝑀1 of 91.60%, indicating that mere format switching can severely undermine the decision consistency of this workflow. To verify whether these violations correspond to compromised decision correctness rather than the random wandering among different incorrect options, we delve into the subset of instances where format changes trigger correctness flips (i.e., under the same semantics, the results are correct in some input formats but incorrect in others). As shown in Figure 3, the erroneous results are not uniformly distributed across all formats. They are noticeably concentrated in structured formats such as CSV. Specifically, among the 3,302 instances, the CSV format input yields the highest error rate, reaching up to 91.92% across the four workflows. In contrast, text-based representations like TXT and MD appear more frequently on the correct side, with the lowest error rate being only 0.22%. This indicates that the errors induced by MR1 violations exhibit clear directionality, with certain formats (e.g., CSV) being systematically more susceptible to triggering erroneous decisions. This observation is statistically confirmed by Cochran’s Q-tests across all workflow-dataset combinations (all 𝑝 < 0.001), and paired McNemar’s tests with Holm correction further verify that the error rates of CSV are significantly higher than those of TXT and MD. Finding 1: The violations on MR1 are widespread at the instance level, with an average of 41.33% on four datasets. Furthermore, error cases induced by MR1 violations are significantly concentrated in the CSV formats. • Analysis of Overall Performance. We further study how format variations manifest as the degradation of overall task performance. As illustrated in Figure 4, format variations directly alter the workflow’s accuracy on the datasets. Specifically, text-based representations such as TXT and MD typically lead to higher accuracy across different workflows, whereas CSV and certain JSON representations are more prone to triggering significant degradation. For example, on the Construct dataset, the average accuracy across workflows for TXT is 93.05%, whereas CSV drops to 67.92%. This confirms that in LLM document workflows, even when semantic content and evaluation criteria remain constant, the input document format alone can substantially alter the system’s task performance.

ASE ’26, October 12–16, 2026, Munich, Germany

   7;7 -621 0'

&69

   7;7 -621 0'

F*RRJOH

&69



Dataset Format



E$QWKURSLF

   7;7 -621 0'

G$OLEDED

&69

More importantly, the degradation pattern is not determined by the format but reflects a significant format-workflow interaction. The worst-performing format varies across workflows. On the Construct dataset, the OpenAI workflow achieves its best accuracy with MD yet its worst with CSV, while the Alibaba workflow exhibits the most severe degradation under CSV, with the maximum performance gap reaching 56.00%. This demonstrates that format degradation is related not only to document representation but also closely tied to the workflow’s native access pathways, parsing methods, and context organization mechanisms. It is the result of the combined effect of the format and the specific workflow design Furthermore, we observe that similar accuracy across formats does not imply a low MR1 violation rate. For instance, when the Anthropic workflow processes MedQA, the overall accuracy of MD and JSON is exceptionally close (55.43% and 57.16%, respectively), yet the MR1 violation rate between the two formats reaches 16.14%. The root cause is that instances answered incorrectly under one format may coincidentally be answered correctly under the other, and vice versa, so that correct and incorrect results cancel each other out in aggregate statistics. This implies that relying solely on average accuracy can substantially underestimate format-induced risks by masking significant behavioral inconsistency at the instance level, further highlighting the necessity of metamorphic testing for instance-level verification. Finding 2: The changes in document formats can cause the accuracy of LLM workflows to drop by up to 56.00%. Furthermore, proximity in average accuracy does not guarantee behavioral consistency of the workflow across different formats. Although some formats exhibit similar overall performance, they may still contain a massive number of result differences concealed by the average values.

RQ2: Impact on Reasoning Evidence

To investigate how input formats affect the evidence extraction and reasoning process of LLM workflows, we collect and analyze a total of 12,000 evidence groups, each comprising the workflow’s evidence sets obtained under four different input formats. We first measure

Anthropic

ΔAcc Flip (%)

(%)

JS

ΔAcc Flip (%)

(%)

Google JS

Alibaba

ΔAcc Flip (%)

(%)

JS

ΔAcc Flip (%)

(%)

MedQA

JSON MD CSV

0.18 0.27 3.33 0.18 1.47 10.71 0.14 -0.13 6.72 0.15 -16.56 33.24 0.10 0.80 1.87 0.13 -0.41 4.54 0.10 -0.81 2.43 0.14 -0.53 6.68 0.52 -52.34 65.70 0.68 -25.20 74.27 0.19 -1.21 14.15 0.32 1.60 20.05

Construct

JSON MD CSV

0.15 -7.88 9.21 0.19 6.94 14.42 0.16 -4.95 13.50 0.17 -1.20 1.20 0.12 0.00 2.14 0.63 -3.07 23.63 0.16 4.97 8.20 0.09 -3.33 3.60 0.32 -28.76 29.78 0.21 -11.63 22.86 0.19 -3.75 13.92 0.33 -56.00 56.00

&69

Figure 4. Accuracy Across Formats and Workflows on MedQA and Construct Dataset.

5.3

OpenAI JS

 7;7 -621 0'

$FFXUDF\

$FFXUDF\

D2SHQ$,

Table 2. Evidence Drifts Under Different Formats.

&RQVWUXFW $FFXUDF\

$FFXUDF\

0HG4$

Zhang et al.

the MR2 violation rate (Table 1) and analyze the co-occurrence relationship between MR2 and MR1 violations. Subsequently, using the TXT format, which most closely resembles plain-text prompts, as the baseline, we employ Jensen-Shannon (JS) divergence to quantify the degree of evidence distribution shift under different formats and analyze how this drift correlates with performance degradation. Table 2 reports the JS divergence (‘JS’), accuracy difference (‘ΔAcc’), and decision flip rate (‘Flip’) across formats and datasets. Note that a positive ‘ΔAcc’ indicates that the given format outperforms the TXT baseline, and ‘Flip’ counts the proportion of decision output in the given format that is different from the results of TXT baseline. Finally, we conduct a paired-factor sensitivity analysis to examine format-induced fairness risks. Figure 5 presents heatmaps of FDS on the DiscrimEval dataset, where the X-axis indicates factor pairs. • Analysis of MR2 Violation. Table 1 reveals that evidence drift is a systemic phenomenon within end-to-end LLM document workflows. On average, 25.14% of instances trigger an MR2 violation across all settings. The severity varies considerably across workflows and datasets. Cochran’s Q-tests confirm that inter-workflow differences in 𝑀2 are statistically significant across all four datasets (all 𝑝 < 0.001) [14, 25, 39]. To reveal the relationship between evidence drift and decision changes, we conduct a co-occurrence analysis of MR1 and MR2 violations. Across all four datasets, 63.56% of MR2-violating instances simultaneously violate MR1, confirming that evidence drift is strongly associated with decision changes. However, 36.44% of MR2 violations are silent violations, where the workflow arrives at the same final decision but cites drastically different evidence under different formats. The silent violation rate varies substantially across datasets, from 11.11% on MedQA to 55.33% on DiscrimEval. Such silent violations are particularly concerning in high-stakes scenarios, as they undermine the interpretability and auditability of system decisions without any observable change in the output. To further characterize the distributional features of these violations, we use the TXT format as a baseline and conduct pairwise analysis across formats. We observe that 52.21% and 24.35% of evidence changes occur under the CSV and JSON formats, suggesting that structured formats are more prone to redistributing the salience of fields during document ingestion, thereby altering the core evidence relied upon by the workflow. Finding 3: The average MR2 violation rate reaches 25.14%, indicating that format changes frequently induce workflows to extract different information cues for reasoning. Even worse, 36.44% of these violations are silent, whose reasoning logic dislocations are masked by consistent decision outcomes, posing

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows 7;7        7;7        -621        -621        0'        0'        &69        &69        U U Q Q V V N N Q Q V V N N XDWLR XDWLR 6WDWX 6WDWX U 5LV H 5LV*HQGH XDWLR XDWLR 6WDWX 6WDWX U 5LV H 5LV*HQGH HU 6LWJH 6LW QGHU 5LVN *HQGH $J $JH HU 6LWJH 6LW QGHU 5LVN *HQGH $J $JH *HQG $ *H *HQG $ *H

D2SHQ$,

   FDS

E$QWKURSLF



7;7        7;7        -621        -621        0'        0'        &69        &69        U U Q Q V V N N Q Q V V N N XDWLR XDWLR 6WDWX 6WDWX U 5LV H 5LV*HQGH XDWLR XDWLR 6WDWX 6WDWX U 5LV H 5LV*HQGH HU 6LWJH 6LW QGHU 5LVN *HQGH $J $JH HU 6LWJH 6LW QGHU 5LVN *HQGH $J $JH *HQG $ *H *HQG $ *H



G$OLEDED

F*RRJOH

Figure 5. Format Disparity Score Heatmap on DiscrimEval Dataset.

7;7

-621

0'

D2SHQ$,

 7;7

-621

0'

F*RRJOH

&69

&UHGLW&DUG

  

&69





'LVFULP(YDO 6WDELOLW\

 

6WDELOLW\

&RQVWUXFW

6WDELOLW\

6WDELOLW\

0HG4$ 

7;7

-621

0'

&69

-621

0'

&69

E$QWKURSLF

  

7;7

G$OLEDED

Figure 6. Stability (𝑆𝑡𝑏) Across Different Datasets and Workflows.

direct threats to the interpretability and trustworthiness of LLM document workflows. • Analysis of Evidence Drift. The results in Table 2 reveal that evidence drift (high ‘JS’) is systematically associated with higher decision flip rates (high ‘Flip’) and lower relative accuracy (low ‘ΔAcc’, where a higher value indicates better performance relative to TXT). Spearman correlation analysis confirms this pattern across all settings (𝜌 (JS, ΔAcc) = −0.470, 𝑝 < 0.05), with the effect strengthening on the MedQA datasets (up to 𝜌 = −0.683). In the most severe case, the Anthropic workflow under CSV on MedQA reaches a flip rate of 74.27%. Notably, the worst-case format varies across workflows. On MedQA, all four workflows exhibit the most severe drift under CSV, whereas on Construct, the worst-case format differs by workflow (e.g., MD for Anthropic, CSV for OpenAI and Alibaba). This confirms that evidence drift is not attributable to a single format but arises from the interaction between document formats and the specific processing mechanisms of each workflow. To further investigate the fairness risks of evidence drift, we conduct a paired-factor sensitivity analysis across all four datasets and four workflows. By fixing the decision question and varying sensitive attributes (e.g., gender and age), we compute the FDS to quantify the maximum group-level decision disparity induced by each format. Overall, we observe that format-induced fairness risks are pervasive. Elevated FDS values appear across all four datasets, with structured formats (particularly CSV) consistently triggering

ASE ’26, October 12–16, 2026, Munich, Germany

the most severe group disparities. Due to space limitations, Figure 5 illustrates part of the heatmap example on the DiscrimEval dataset, where ‘Gender×Status’ achieves the largest FDS on TXT format. Complete heatmaps for all datasets and workflows are released [4]. On the Construct dataset, three workflows reach their peak FDS under CSV on the ‘Age×Job’ pair (FDS up to 2.00), whereas other formats produce identical decision rates across these groups, demonstrating that a single format change to CSV can introduce substantial group disparity that is entirely absent under other formats. Similar patterns appear on the Credit Card and MedQA datasets, where specific formats disproportionately amplify disparity on sensitive attribute pairs such as ‘Gender×Marital’ and gender-related factors. These results indicate that when a workflow parses specific syntactic structures, its reasoning may attend to non-task-relevant sensitive attributes, ultimately inducing systemic decision biases. To further illustrate how evidence drift occurs, we study a representative case on the Credit Card dataset. For the exact same record (i.e., ‘a 50-year-old married male whose account was inactive for five months with only one late payment across six months’), the Google workflow yields opposite decisions depending on the format. Under MD, the workflow returns ‘YES’ (80% confidence) and predicts a high risk of default for this case by listing business-relevant evidence (e.g., the payment delay and the revolving balance). Conversely, under JSON, the workflow flips to ‘NO’ (95% confidence), and the evidence shifts to demographic attributes, such as age and gender (‘50-year-old male’) and marital status (‘married’). The workflow finally interprets this record as ‘only one late payment’ and concludes with low default risk. Both the motivation case (Figure 1) and the above case illustrate that format-induced evidence drift can redirect the workflow’s attention from task-relevant financial signals to non-task-relevant demographic attributes, silently introducing fairness violations into high-stakes credit decisions. Finding 4: Format-induced evidence drift is significantly correlated with downstream decision flips and performance degradation. Furthermore, specific structured formats can perturb the salience of factual cues, amplifying decision disparities across demographic groups (FDS up to 2.00) and silently introducing fairness violations in high-stakes scenarios.

5.4

RQ3: Impact on Execution Stability

To investigate how input format affects the execution stability of LLM document workflows, we collect and analyze 3,000 groups (𝑘 = 3) of repeated execution results across four workflows and four datasets. We first measure the MR3 violation rate to quantify the extent to which format variation undermines decision stability. Table 1 reports 𝑀3 for each workflow across different datasets. We then compare the stability metric 𝑆𝑡𝑏 (§5.1) across formats at the dataset level to discuss the practical deployment risks. Figure 6 presents the distribution of 𝑆𝑡𝑏 for each workflow across different formats and datasets. • Analysis of MR3 Violation. The results in Table 1 indicate that MR3 violations are prevalent across workflows. On average, 45.33% of instances trigger an MR3 violation, demonstrating that switching input formats not only alters single-execution decision outcomes (§5.2) but also systematically undermines the determinism of the reasoning process within workflows. Moreover, MR3 violations

ASE ’26, October 12–16, 2026, Munich, Germany

are highly unevenly distributed across workflows. For example, on Construct, the OpenAI workflow reaches an 𝑀3 of 86.40%, whereas the Google workflow exhibits only 4.80%. Cochran’s Q-tests on aligned instances confirm that these inter-workflow differences are statistically significant across all four datasets (all 𝑄 > 338, 𝑝 < 0.001). This substantial disparity suggests that the degree to which format variation disrupts execution stability is highly dependent on each workflow’s underlying mechanisms, including file parsing paths and context organization strategies, whose implementationlevel differences are further amplified across repeated executions. Finding 5: The average MR3 violation rate reaches 45.33%, indicating that input format variation can systematically alter the execution consistency of instances. Furthermore, the violations exhibit a significantly uneven distribution across workflows, with 𝑀3 reaching 67.94% on the OpenAI workflow and 17.03% on the Google workflow. • Analysis of Stability Degradation. As shown in Figure 6, stability degradation does not occur uniformly across all formats but instead exhibits a pronounced clustering effect within specific formats. Taking the OpenAI workflow on Construct as an example, the system maintains high stability under TXT, JSON, and MD (all 𝑆𝑡𝑏 > 95%), but 𝑆𝑡𝑏 drops sharply to 14.00% under CSV. Anthropic pipeline achieves the lowest 𝑆𝑡𝑏 of 11.00% on the CSV format. In contrast, the Google and Alibaba workflows maintain relatively higher stability. This comparison suggests that format-induced stability degradation is jointly driven by the interaction between specific document representations and the workflow’s underlying processing mechanisms. This finding carries significant implications for LLM-driven software engineering practices. Under specific structured formats, the underlying processes of the workflow (e.g., file parsing and serialization) propagate syntactic differences in input representations to the model’s reasoning layer, systematically amplifying the inherent stochasticity of inference. In high-stakes scenarios such as automated medical triage and financial statement auditing [26, 34], such unpredictable fluctuations will severely compromise the reproducibility of software systems and substantially increase the costs of debugging and verification, posing a fundamental challenge to the trustworthy deployment of LLM document workflows in production environments. Finding 6: Document format is a critical variable in determining the execution stability of LLM workflows. Specific structured formats (e.g., CSV) can amplify the stochasticity during model reasoning, causing 𝑆𝑡𝑏 to drop to as low as 11.00%, thereby posing a severe reliability threat to the production deployment of document workflows in high-stakes scenarios.

5.5

RQ4: Mitigation Strategies

To mitigate format-induced errors in LLM workflows, we design two 1 lightweight training-free strategies from the user’s perspective. ○ Vote Aggregation is a generic strategy inspired by prior work [64]. It keeps the input format unchanged and aggregates decisions from multiple executions (i.e., three in our experiments) via majority voting, aiming to reduce random errors through posterior statistics. 2 Format Routing is a targeted strategy designed specifically ○

Zhang et al.

Table 3. Effect of Mitigation Strategies on MR Violation Rates (Bold and underlined marks the better result between two strategies). Dataset ΔMRV

MedQA

Anthropic

Google

Alibaba

Vot.

Vot.

Vot.

Rte.

Rte.

Rte.

Rte.

Avg. Vot.

Rte.

Δ𝑀1 2.80 65.20 21.71 76.67 4.93 17.38 -4.27 41.87 8.43 50.28 Δ𝑀2 0.00 16.80 0.00 12.00 0.00 19.70 0.00 17.20 0.00 16.43 Δ𝑀3 71.37 68.95 88.80 87.97 15.73 9.68 31.85 7.25 51.94 43.46 Δ𝑀1

Construct Δ𝑀2

Δ𝑀3

Avg.

OpenAI Vot.

13.73 29.33 10.93 44.53 -0.13 21.47 -1.60 57.20 5.73 38.13 0.00 7.60 0.00 64.90 0.00 6.40 0.00 1.20 0.00 20.03 86.40 86.00 64.80 64.80 4.80 2.80 63.20 63.20 54.80 54.20

Δ𝑀1 8.27 47.27 16.32 60.60 2.40 19.42 1.33 49.53 7.08 44.21 Δ𝑀2 0.00 12.20 0.00 38.45 0.00 13.05 0.00 9.20 0.00 18.23 Δ𝑀3 78.89 77.48 76.80 76.39 10.27 6.24 47.53 35.23 53.37 48.83

for format-induced errors. It inserts a lightweight routing layer upstream of the workflow that converts every input document to the empirically optimal format before inference. Concretely, this layer maps the input task domain to the optimal format for the current workflow based on offline empirical results. For example, a CSV MedQA input to the OpenAI workflow is automatically converted to MD (its optimal format) prior to inference. Leveraging the precollected empirical results rather than per-document LLM analysis, this approach introduces negligible overhead. Detailed implementations for both strategies are in [4]. We apply both strategies to four workflows on Construct and MedQA, analyzing 8,000 sets of instances. Table 3 summarizes the results, where ΔMRV denotes the reduction in violation rate after applying each strategy, and ‘Vot.’ and ‘Rte.’ abbreviate the vote aggregation and format routing strategies, respectively. Bold and underlined highlight the results of the better of the two strategies. • Analysis of Mitigation Methods. As shown in Table 3, the vote aggregation strategy provides only limited improvement on decision outcomes. On Construct, the average Δ𝑀1 achieved by voting is merely 5.73%. On the MedQA dataset, the voting strategy even degrades the Alibaba workflow (Δ𝑀1 = −4.27%), confirming that when format-induced bias is embedded in the input representation, repeated sampling merely reproduces inferior reasoning paths rather than generating corrective information [15, 64]. Notably, Δ𝑀2 remains 0.00% across all voting configurations because voting operates within a single format, aggregating decisions from repeated executions without altering evidence cues, thus leaving 𝑀2 violations entirely unaffected. In contrast, format routing achieves substantial and consistent mitigation across all three MRs, with average reductions of 44.21%, 18.23%, and 48.83% on 𝑀1 , 𝑀2 , and 𝑀3 , respectively, outperforming the voting strategy by 37.13% on 𝑀1 . For example, on Construct, routing raises the accuracy of the OpenAI workflow from 69.93% under CSV to 98.67% under MD (gain = 28.74%, 95% CI: [22.00%, 36.00%]). This contrast in effectiveness, which aligns with our findings in §5.3 and §5.4, stems from the fundamental difference in intervention timing. Format-induced errors originate directly in the document processing mechanisms of workflows, where different formats produce structurally divergent context representations that inject noise into the reasoning layer. Format routing addresses this root cause proactively by converting documents into an empirically stable format before processing. As evidenced by the 18.23% reduction in 𝑀2 , this representation-level intervention not only corrects macroscopic decision outcomes but also stabilizes the underlying evidence

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

cues. Conversely, majority voting merely performs post-hoc aggregation. If a specific format systematically degrades the reasoning process, repeated executions will only reproduce those identical errors without alleviating the inherent representation-level bias. Finding 7: The format routing strategy effectively mitigates format-induced errors across all three MRs, achieving average reductions of 44.21%, 18.23%, and 48.83% in 𝑀1 , 𝑀2 , and 𝑀3 . This confirms that format-induced errors originate in the document processing mechanisms, and intervening at the representation level yields substantial mitigation for decision outcomes, evidence consistency, and execution stability.

6

Discussion

• Threats to Validity. The generalizability of our findings may 1 The experiments are be threatened by the following aspects. ○ conducted on question-answering and decision-making tasks in high-stakes domains [12, 55], and may not directly capture formatinduced degradation in open-ended generation tasks where output correctness is less discretely defined. However, the underlying principle of our MRs (i.e., semantically equivalent inputs should yield consistent outputs), remains applicable to such tasks through 2 Variations in format expressiveness appropriate adaptations. ○ could theoretically introduce semantic shifts during data generation, confounding the format-induced errors. Note that our evaluation setting is motivated by real-world practices, where users routinely use diverse formats (e.g., CSV and JSON). We mitigate this threat by adhering to real-world usage practices and applying the three transformation constraints outlined in §4.2. Specifically, our framework renders variants from a unified intermediate representation 𝐶𝑥 to isolate task semantics from external formats. Furthermore, we invite two independent experts (with at least five years of experience in the SE field) to manually check 20 randomly sampled transformation groups and confirm 100% semantic consistency across formats. These controls mitigate the threat that MR violations are caused 3 Our experiments cover four mainstream by information loss. ○ workflows and four commonly used formats, which may not fully represent some emerging systems, such as skill-based agent systems [30] or additional document formats. Note that our testing framework is designed to be extensible. Incorporating new workflows or formats requires only adding the corresponding document conversion operators and API adapters, without modifying the core 4 Executing each configuration only 𝑘 = 3 times testing logic. ○ may provide a limited view. To assess this threat, we conduct a supplementary experiment with 𝑘 = 5 on the MedQA and Construct datasets. While the absolute MR values exhibited marginal shifts, the relative rankings of the most- and least-affected workflows remained identical across both datasets (e.g., on the MedQA dataset, Anthropic remains the highest on MR1/MR3 and Google stays the lowest), confirming that the qualitative observations are stable under a larger 𝑘. To mitigate these threats and facilitate reproducibility, we release all necessary code, datasets, and results in our repository [4]. 1 Extending MRs and task coverage. Current MRs • Future Work. ○ target discrete decision tasks where output correctness can be unambiguously compared. We have conducted preliminary validation and applied the framework to 15 instances from the CNN-DailyMail

ASE ’26, October 12–16, 2026, Munich, Germany

summarization dataset [51], adapting MR1 to measure semantic similarity between generated summaries and MR2 to track sentencelevel source coverage. We have observed that format-induced inconsistency also persists in common summarization tasks (average MRV of MR1/MR2 across workflows is 48.33% and 26.67%). This further illustrates the significance of this study: this format robustness problem may be widespread in diverse application scenarios, posing risks to the regular use of LLM document workflows. Future work can systematically design new MRs tailored to generation and summarization tasks using semantic similarity measures, and expand the evaluated workflows to include diverse new pipelines 2 Extending to other workflows. and additional document formats. ○ While this study deliberately targets production-level workflows to reveal actionable, real-world vulnerabilities regarding format robustness, our framework is extensible to more complex systems. Since our MRs depend on functional properties (e.g., observable final outcomes) rather than architectural complexity, they can be readily adapted to advanced workflows. However, applying this framework to workflows such as retrieval-augmented generation (RAG) or multi-agent systems introduces new challenges in evidence extraction and tracing. For instance, quantifying how format variations skew dynamic retrieval distributions in RAG pipelines, or attributing format-induced errors across inter-agent communications, requires advanced tracing methods. Consequently, constructing customized, modular pipelines to isolate the exact contributions of individual processing stages (e.g., parsing, chunking, and serialization) and conducting systematic root-cause analysis constitutes 3 Adaptive format routing. a critical direction for future work. ○ The format routing strategy in §5.5 relies on offline calibration data, and the optimal format may shift as workflows are updated, models are iterated, or task domains change. Future work can explore adaptive routing mechanisms that dynamically select the optimal input format, e.g., by evaluating workflow performance on a small set of probe samples or training classifiers that adapt to the current workflow state at runtime, thereby reducing reliance on static calibration data and enhancing deployment flexibility. 4 Scaling repeated-execution evaluation. Future work can scale ○ repeated-execution evaluation to more datasets and workflows to obtain finer-grained estimates of stochastic workflow instability.

7

Conclusion

This paper presents a format-aware metamorphic testing framework for end-to-end LLM document workflows and conducts a large-scale empirical study across four mainstream workflows and four document formats. The findings reveal that input format variation constitutes a systematic and previously underexamined threat to the reliability of LLM document workflows. Merely switching the format of semantically identical content can cause widespread decision flips, severe evidence drift, and significant stability degradation. Building on the findings, this paper evaluates two lightweight mitigation strategies and shows that the format routing strategy can substantially mitigate the format-induced errors in the workflow. We call on the SE community to treat document format as an important evaluation variable in LLM system testing, and to incorporate format-aware safeguards as a standard practice for the trustworthy deployment of LLM document workflows in high-stakes scenarios.

ASE ’26, October 12–16, 2026, Munich, Germany

8

Data Availability Statement

To follow the Open Science Policy and support reproducibility, we have released the necessary data at [4].

Acknowledgments This research is supported by the National Natural Science Foundation of China (62441237). This work was also supported by the National Research Foundation, Singapore, and the Cyber Security Agency under its National Cybersecurity R&D Programme (NCRP25-P04-TAICeN). This research is also part of the IN-CYPHER Programme and is supported by the National Research Foundation, Prime Minister’s Office, Singapore, underits Campus for Research Excellence and Technological Enterprise (CREATE) Programme. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National Natural Science Foundation of China, National Research Foundation, or Cyber Security Agency.

References [1] Open Life Science AI. 2024. medmcqa. https://huggingface.co/datasets/openlife scienceai/medmcqa/tree/main/data Accessed: 2026-03-21. [2] Aliyun. 2026. Long context(Qwen-Long). https://www.alibabacloud.com/help/ en/model-studio/long-context-qwen-long Accessed: 2026-03-21. [3] Anthropic. 2026. Advancing Claude in healthcare and the life sciences. https: //www.anthropic.com/news/healthcare-life-sciences. Accessed: 2026-03-21. [4] Authors. 2026. Open-sourced Data. https://doi.org/10.5281/zenodo.21696268 Accessed: 2026-07-30. [5] Israel Abebe Azime, Deborah D Kanubala, Tejumade Afonja, Mario Fritz, Isabel Valera, Dietrich Klakow, and Philipp Slusallek. 2025. Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval across Table-to-Text Serialization Approaches. In Findings of the Association for Computational Linguistics: EMNLP 2025. 17478–17503. [6] Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 194–199. [7] Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan Rystrøm, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Nicolaus Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip Torr, Cozmin Ududec, Luc Rocher, and Adam Mahdi. 2025. Measuring what Matters: Construct Validity in Large Language Model Benchmarks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/forum?id=mdA5lVvNcU [8] Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543 (2020). [9] Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR) 51, 1 (2018), 1–27. [10] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning. [11] Steven Cho, Stefano Ruberto, and Valerio Terragni. 2025. Metamorphic testing of large language models for natural language processing. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 174–186. [12] Xu Chu, Zhijie Tan, Hanlin Xue, Guanyu Wang, Tong Mo, and Weiping Li. 2025. Domaino1s: Guiding llm reasoning for explainable answers in high-stakes domains. In Findings of the Association for Computational Linguistics: ACL 2025. 3275–3293. [13] Claude. 2025. ClaudeAPIDocs, Files API. https://platform.claude.com/docs/en/ build-with-claude/files Accessed: 2026-03-21. [14] W. G. COCHRAN. 1950. THE COMPARISON OF PERCENTAGES IN MATCHED SAMPLES. Biometrika 37, 3-4 (12 1950), 256–266.

Zhang et al.

arXiv:https://academic.oup.com/biomet/article-pdf/37/3-4/256/421327/373-4-256.pdf doi: 10.1093/biomet/37.3-4.256 [15] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. Chain-of-Verification Reduces Hallucination in Large Language Models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL, Vol. ACL 2024), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, Bangkok, Thailand, 3563–3578. doi: 10.18653/V1/2024.FINDINGS-ACL.212 [16] Alastair F Donaldson and Andrei Lascu. 2016. Metamorphic testing for (graphics) compilers. In Proceedings of the 1st international workshop on metamorphic testing. 44–47. [17] Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, and Dongmei Zhang. 2024. Encoding Spreadsheets for Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 20728–20748. doi: 10.18653/v1/2024.emnlp-main.1154 [18] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). 31–53. doi: 10.1109/ICSE-FoSE59343.2023.00008 [19] Lingyue Fu, Bolun Zhang, Hao Guan, Yaoming Zhu, Lin Qiu, Weiwen Liu, Xuezhi Cao, Xunliang Cai, Weinan Zhang, and Yong Yu. 2025. Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation. CoRR abs/2510.24358 (2025). arXiv:2510.24358 doi: 10.48550/ARXIV.2510.24358 [20] Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics 50, 3 (Sept. 2024), 1097–1179. doi: 10.1162/coli_a_00524 [21] Gemini. 2026. Gemini API, Document understanding. https://ai.google.dev/ge mini-api/docs/document-processing Accessed: 2026-03-21. [22] Google. 2026. Gemini API, Files API. https://ai.google.dev/gemini-api/docs/files Accessed: 2026-03-21. [23] Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems (Barcelona, Spain) (NIPS’16). Curran Associates Inc., Red Hook, NY, USA, 3323–3331. [24] Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pretraining. In Proceedings of the 58th annual meeting of the association for computational linguistics. 4320–4333. [25] Sture Holm. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65–70. http://www.jstor.org/stable /4615733 [26] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 220 (Dec. 2024), 79 pages. doi: 10.1145/3695988 [27] Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2025. Bias testing and mitigation in llm-based code generation. ACM Transactions on Software Engineering and Methodology 35, 1 (2025), 1–31. [28] Yifan Huang, Xiaojun Jia, Wenbo Guo, Yuqiang Sun, Yihao Huang, Chong Wang, and Yang Liu. 2026. Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking. In Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2026). [29] Sangwon Hyun, Mingyu Guo, and M. Ali Babar. 2024. METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 117–128. doi: 10.1109/ICST60714.2024.00019 [30] Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma, Xu Wang, Qin Wang, and Guangsheng Yu. 2026. SoK: Agentic Skills–Beyond Tool Use in LLM Agents. arXiv preprint arXiv:2602.20867 (2026). [31] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11, 14 (2021), 6421. [32] Venkata Sai Nageen Kanikanti. 2024. LLMs for Financial Document Processing. International Journal of Artificial Intelligence, Data Science, and Machine Learning 5, 1 (2024), 167–172. [33] Wenjun Ke, Yifan Zheng, Yining Li, Hengyuan Xu, Dong Nie, Peng Wang, and Yao He. 2025. Large language models in document intelligence: A comprehensive survey, recent advances, challenges, and future trends. ACM Transactions on Information Systems 44, 1 (2025), 1–64. [34] Hannah R Lawrence, Renee A Schneider, Susan B Rubin, Maja J Matarić, Daniel J McDuff, and Megan Jones Bell. 2024. The opportunities and risks of large language models in mental health. JMIR Mental Health 11, 1 (2024), e59479.

Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

[35] Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. 2022. MarkupLM: Pre-training of text and markup language for visually rich document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 6078–6087. [36] Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, and Junbo Zhao. 2025. LongTableBench: benchmarking long-context table reasoning across real-world formats and domains. Findings of the Association for Computational Linguistics: EMNLP 2025 (2025). [37] Penghui Li, Yinxi Liu, and Wei Meng. 2021. Understanding and detecting performance bugs in markdown compilers. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 892–904. [38] LLMBias. 2025. BiasLens. https://github.com/LLMBias/BiasLens/tree/main Accessed: 2026-03-21. [39] Quinn McNemar. 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12, 2 (1947), 153–157. doi: 10.1007/BF02295996 [40] Jonathan Chen Meenu Bondili. 2025. Use Gemini 2.0 to speed up document extraction and lower costs. https://cloud.google.com/blog/products/ai-machinelearning/use-gemini-2-0-to-speed-up-data-processing. Accessed: 2026-03-21. [41] Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, et al. 2024. Exploring the impact of table-to-text methods on augmenting llm-based question answering with domain hybrid data. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track). 464–482. [42] Christopher Z Mooney, Robert D Duval, and Robert Duvall. 1993. Bootstrapping: A nonparametric approach to statistical inference. Number 95. sage. [43] Shirin Nilizadeh, Yannic Noller, and Corina S. Păsăreanu. 2019. DifFuzz: differential fuzzing for side-channel analysis. In Proceedings of the 41st International Conference on Software Engineering. 176–187. doi: 10.1109/ICSE.2019.00034 [44] Yannic Noller and Saeid Tizpaz-Niari. 2021. QFuzz: quantitative fuzzing for side channels. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 257–269. doi: 10.1145/3460319.3464817 [45] OpenAI. 2026. ChatGPT Release Notes: January 7, 2026, Health in ChatGPT. https://help.openai.com/en/articles/6825453-chatgpt-release-notes. Accessed: 2026-03-21. [46] OpenAI. 2026. OpenAI API, Code Interpreter. https://developers.openai.com/ap i/docs/guides/tools-code-interpreter Accessed: 2026-03-21. [47] OpenAI. 2026. OpenAI API, File inputs. https://developers.openai.com/api/docs /guides/file-inputs Accessed: 2026-03-21. [48] Stephen R Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, et al. 2024. A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine 30, 12 (2024), 3590–3600. [49] Stephen R. Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, Liam G. McCoy, Leo Anthony Celi, Yun Liu, Mike Schaekermann, Alanna Walton, Alicia Parrish, Chirag Nagpal, Preeti Singh, Akeiylah Dewitt, Philip Mansfield, Sushant Prakash, Katherine Heller, Alan Karthikesalingam, Christopher Semturs, Joelle Barral, Greg Corrado, Yossi Matias, Jamila SmithLoud, Ivor Horn, and Karan Singhal. 2024. A toolbox for surfacing health equity harms and biases in large language models. CoRR abs/2403.12025 (09 2024), 3590–3600. doi: 10.48550/ARXIV.2403.12025 [50] Jiarui Qin, Weinan Zhang, Rong Su, Zhirong Liu, Weiwen Liu, Ruiming Tang, Xiuqiang He, and Yong Yu. 2021. Retrieval & Interaction Machine for Tabular Data Prediction. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021, Feida Zhu, Beng Chin Ooi, and Chunyan Miao (Eds.). ACM, 1379–1389. doi: 10.1145/ 3447548.3467216 [51] Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1073–1083. doi: 10.18653/v1/ P17-1099 [52] Sergio Segura, Gordon Fraser, Ana B Sanchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transactions on software engineering 42, 9 (2016), 805–824. [53] sigpwned. 2023. Popular names by country dataset: A dataset of popular forenames and surnames by country. https://github.com/sigpwned/popular-namesby-country-dataset Accessed: 2026-03-21. [54] Kuldeep Singh, Simerjot Kaur, and Charese Smiley. 2024. Finqapt: Empowering financial decisions with end-to-end llm-driven question answering pipeline. In Proceedings of the 5th ACM International Conference on AI in Finance. 266–273. [55] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature medicine 31, 3 (2025), 943–950.

ASE ’26, October 12–16, 2026, Munich, Germany

[56] Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining (Merida, Mexico) (WSDM ’24). Association for Computing Machinery, New York, NY, USA, 645–654. doi: 10.1145/3616855.3635752 [57] Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and Mitigating Discrimination in Language Model Decisions. ArXiv abs/2312.03689 (2023). https://api.semanticscholar.org/CorpusID:265696308 [58] Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689 (2023). [59] Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R Lyu. 2023. Biasasker: Measuring the bias in conversational ai system. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 515–527. [60] Wanderer. 2024. data jobs by lukebarousse. https://www.kaggle.com/datasets/ wandererfakeer/data-jobs-by-lukebarousse Accessed: 2026-03-21. [61] Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Trans. Softw. Eng. 50, 4 (April 2024), 911–936. doi: 10.1109/TSE. 2024.3368208 [62] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science 18, 6 (2024), 186345. [63] Shenao Wang, Yanjie Zhao, Xinyi Hou, and Haoyu Wang. 2025. Large Language Model Supply Chain: A Research Agenda. ACM Trans. Softw. Eng. Methodol. 34, 5, Article 147 (May 2025), 46 pages. doi: 10.1145/3708531 [64] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=1P L1NIMMrw [65] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents. Proc. ACM Softw. Eng. 2, FSE, Article FSE037 (June 2025), 24 pages. doi: 10.1145/3715754 [66] Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 84, 4 (2011), 544–558. [67] I-Cheng Yeh and Che hui Lien. 2009. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications 36, 2, Part 1 (2009), 2473–2480. doi: 10.1016/j.eswa .2007.12.020 [68] Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2025. Benchmarking and Defending against Indirect Prompt Injection Attacks on Large Language Models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, USA, 1809–1820. doi: 10.1145/3690624.3709179 [69] Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th annual meeting of the association for computational linguistics. 8413–8426. [70] Zeeshanahmad4. 2024. Zip Code Data for All Countries and Cities in the World. https://github.com/Zeeshanahmad4/Zip-code-of-all-countries-cities-in-the-wo rld-CSV-TXT-SQL-DATABASE Accessed: 2026-03-21. [71] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024. 10471–10506. [72] Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2020), 1–36. [73] Xiaoyu Zhang, Cen Zhang, Tianlin Li, Yihao Huang, Xiaojun Jia, Ming Hu, Jie Zhang, Yang Liu, Shiqing Ma, and Chao Shen. 2025. Jailguard: A universal detection framework for prompt-based attacks on llm systems. ACM Transactions on Software Engineering and Methodology 35, 1 (2025), 1–40. [74] Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2015. Metamorphic testing for software quality assessment: A study of search engines. IEEE Transactions on Software Engineering 42, 3 (2015), 264–284. [75] Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, and Yuan-Fang Li. 2023. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 1090–1102.

Related documents

• More Than Mimicking Reviewers: Evaluating LLMs for Pre-Submission Peer Review#973418
• Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics#821601
• From Metrics to Improvement: A Lifecycle-Aware LLM Feedback Framework for Research Software Quality#482200
• PACE: Publisher-Adaptive Content Extraction via Agentic Automation#780784
• PACE: Publisher-Adaptive Content Extraction via Agentic Automation#779409