arXiv:2607.27648v1 [cs.SE] 30 Jul 2026
Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows Xiaoyu Zhang
Xianyun Cheng
Tianlin Li∗
Nanyang Technological University Singapore, Singapore [email protected]
National University of Singapore Singapore, Singapore [email protected]
Beihang University State Key Laboratory of Complex & Critical Software Environment Beijing, China [email protected]
Yuwei Zheng
Yue Yang
Yang Liu
Beihang University State Key Laboratory of Complex & Critical Software Environment Beijing, China
Xi’an University of Architecture and Technology College of Information and Control Engineering Xi’an, China
Nanyang Technological University Singapore, Singapore
Abstract
Keywords
LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users’ perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.
LLM Document Workflow, LLM Testing, Metamorphic Testing
CCS Concepts • Software and its engineering → Software testing and debugging; Software reliability. ∗ Corresponding author.
This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3837456
ACM Reference Format: Xiaoyu Zhang, Xianyun Cheng, Tianlin Li, Yuwei Zheng, Yue Yang, and Yang Liu. 2026. Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3832783.3837456
1
Introduction
With the rapid advancement of large language models (LLMs), LLMdriven software systems are increasingly evolving from plain-text conversations to document-centric end-to-end workflows [17, 33]. These systems widely integrate functionalities such as file parsing, external tool invocation, and structured output generation. User inputs are not confined to simple text prompts. Instead, they often take the form of semi-structured or structured files (e.g., medical records, system logs) as first-class inputs [26]. In such end-to-end workflows, file formats (e.g., TXT, CSV) are no longer merely interface wrappers but have become crucial components for organizing, transmitting, and presenting task information in real-world LLM software systems. As illustrated in Figure 1, even when two formats carry identical semantic content, the workflow can transform them into structurally different intermediate representations before they reach the LLM, making the input format a non-trivial factor in the workflow behavior. This shift is especially consequential in high-stakes scenarios. Whether in clinical decision support or financial credit tasks, LLMdriven document workflows and systems are increasingly assuming responsibilities related to judgment, recommendation, and even automated execution [26, 34]. In these settings, erroneous, inconsistent, or unfair workflow behaviors can propagate beyond the model’s output to downstream decision-making, resource allocation, and risk control processes. Therefore, there is an urgent need to study the potential risks of LLM document workflows and ensure the safety and reliability of these systems in real-world deployment.
ASE ’26, October 12–16, 2026, Munich, Germany
Existing research has proposed diverse methods for evaluating the reliability of LLM systems [27, 28, 59, 68, 73]. However, existing methods mainly consider the input as a single prompt string, focusing on model behavior at the plain-text level rather than the document workflows increasingly important in real-world deployments [17]. In real-world applications, users routinely submit the exact same task content through completely different formats (e.g., CSV and JSON). Therefore, whether the document workflow can maintain behavioral consistency when processing inputs with different formats (i.e., format robustness) is not merely a theoretical question, but a practical challenge that exposes a highly critical vulnerability in current LLM-driven software. In addition, recent studies and real-world discussions have confirmed that format variations could severely impact LLM system performance [5, 56]. However, these studies remain limited to specific tasks or individual document types, lacking a systematic evaluation across diverse formats in end-to-end workflows. This gap leaves a fundamental question unanswered: Can LLM document workflows remain robust when faced with input format variation? The question is critical because the risks arising from format variations can be ‘silent’. The system may outwardly return structurally valid and formally complete outputs, while the evidence, reasoning paths, and even the final decisions have undergone substantial deviations. To fill the gap, in this paper, we propose a format-aware metamorphic testing framework tailored for end-to-end LLM document workflows. The design principle is that, given that the semantics and constraints of the input are fixed, merely altering the external document format should not substantively affect the behavior of LLM document workflow (i.e., format robustness). The framework consists of two phases. The data construction phase first converts the input plain-text prompts into multiple semantically equivalent file formats, ensuring consistency in identifiers, field boundaries, and instruction semantics recorded across different formats. Subsequently, the workflow evaluation phase enters these aligned inputs into the LLM document workflow via provider-specific file upload and processing pathways and collects the corresponding outcomes from the workflows. We also design and implement three levels of metamorphic relations (MRs) to achieve a comprehensive examination of the format variance of workflows across different input formats (i.e., ‘decision outcome invariance’, ‘reasoning evidence invariance’, and ‘execution stability invariance’). Based on this framework, we conduct a large-scale study across four high-stakes datasets and four document formats, targeting mainstream LLM document workflows driven by representative LLMs. The study aims to answer the following research questions (RQs): • RQ1 (Decision Outcome): How does the input format affect the efficacy of the given tasks when the semantic content remains unchanged? This RQ evaluates the extent to which LLM document workflows violate the MR of ‘decision outcome invariance’. We conduct a comparative analysis on 12,000 groups of execution results across four workflows and four datasets. The results indicate that MR1 violations are pervasive, with an average of 41.33% of instances exhibiting inconsistent decision outcomes across formats. Specific format inputs cause the accuracy to drop by up to 56.00%. Notably, the degradation exhibits strong workflow dependency, and degradation severity is jointly driven by the interaction between format and workflow processing mechanisms.
Zhang et al.
• RQ2 (Reasoning Evidence): How does the input format affect the reasoning evidence presented by the model? To delve into the root causes behind the inconsistent decisions, we quantitatively evaluate the extent to which workflows violate the MR of ‘reasoning evidence invariance’. Specifically, we extract and compare the key factual clues in the workflow responses under different input formats. The analysis reveals that an average of 25.14% of instances exhibit significant evidence drift across formats, and 63.56% of these cases simultaneously trigger decision changes. More alarmingly, 36.44% of the evidence drift cases are silent violations, where the final decision remains unchanged, but the underlying reasoning basis has fundamentally shifted, posing severe threats to the interpretability of LLM workflows. • RQ3 (Execution Stability): How does the input format affect the system’s decision stability during multiple executions? We measure the consistency of workflow decisions across repeated executions under different formats. The results show that instability is not merely an occasional anomaly but a recurrent phenomenon, where the average MR3 violation rate reaches 45.33%, with the highest 𝑀3 reaching 88.80% on the Anthropic workflow. In addition, specific structured formats (e.g., CSV) can cause the stability metric to drop to as low as 11.00%, exposing severe reliability hazards in current LLM-driven software when processing non-textual files. • RQ4 (Mitigation Strategy): Can test-time wrappers effectively mitigate the format-induced errors? From the user’s perspective, we design two lightweight test-time mitigation strategies, namely the vote aggregation strategy and the format routing strategy, to explore whether client-side intervention can cost-effectively mitigate format-induced errors without modifying or retraining the underlying models. Experiments confirm that format-induced errors are largely recoverable. The vote aggregation strategy yields only limited improvements, whereas our format routing strategy achieves substantial gains (up to a 44.21% reduction in MR violation rate on 𝑀1 ). The findings suggest that the representation layer functions as an effective engineering control point within LLM document workflows, and that substantial robustness gains can be realised through the implementation of lightweight, deploymentcompatible safeguards. The contributions of this paper are as follows. • We propose the first format-aware metamorphic testing framework for LLM-driven document workflows and introduce three metamorphic relations for semantically equivalent but differently formatted inputs, bridging the testing gap for end-to-end LLM systems. • We conduct a large-scale empirical study on the performance of four LLM document workflows across four formats. The results indicate that even when semantic content is aligned, specific structured inputs trigger significant and uneven degradation in the correctness, fairness, and stability of LLM system outcomes. • We design lightweight and deployment-compatible mitigation strategies, in which our format routing strategy can effectively mitigate the format-induced errors and degradation without model retraining. • Our pipeline implementation, dataset, mitigation strategy, and the necessary results are available at [4]
Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows
2 Background & Related Work 2.1 LLM Document Workflow In high-stakes scenarios (e.g., medical diagnostic assistance), LLM systems increasingly rely on external documents rich in structured and semi-structured information [17, 32, 34, 54, 62]. To support such complex reasoning tasks, LLM document workflows typically encompass four interconnected stages. • Input Stage receives and preprocesses the raw documents provided by the user, which can range from plain text files to structured formats such as medical records, logs, and tables [26]. This stage handles file uploading, type identification, content extraction, and chunking, thereby determining what information the system can subsequently access and at what granularity [17]. • Organization Stage converts the extracted content into an intermediate representation suitable for model consumption, such as serialized text snippets or lightweight tabular structures [33]. Crucially, the same semantic content undergoes different transformation paths under different formats: plain text preserves linear narrative order, whereas CSV or other tabular formats explicitly expose field boundaries and header relationships [35, 56]. • Reasoning Stage invokes the LLM to interpret the intermediate representation (e.g., ‘Model Input’ in Figure 1). Then the model selects evidence and produces reasoning results. Once the document format alters field boundaries or local structures in the prior stage, the model’s attentional focus, evidence citation, and reasoning paths may correspondingly shift. • Output Stage translates the reasoning results into outputs for human users or downstream system components. If evidence shift or decision drift has been induced upstream by format changes, these deviations can be propagated at this stage in the form of structurally valid but semantically incorrect outputs [6]. Prior work has revealed that structured content represented before model inference can affect LLM behaviors. For tabular data, prior studies [24, 69] demonstrate that structural cues (e.g., field boundaries) are central to table understanding. Recent work further shows that the choice of table serialization format directly influences reasoning quality and downstream decision accuracy [5, 36, 41, 56]. For document understanding, researchers [33, 35] reveal that representation-aware pre-training and the organization of retrieved content can shape what information the model attends to. However, these works focus on uncovering phenomena or designing training methods, and lack a systematic evaluation of the format robustness of the LLM document workflows. To formalize the format-induced workflow behavior changes, in this paper, we denote the workflow under test as 𝑊 and the set of supported formats as 𝐹 . The input document is represented as 𝐼 = 𝜙 (𝑥, 𝑓 ), where 𝜙 is a format rendering function, 𝑥 ∈ X is a semantic instance (e.g., a medical record) from the input space X, and 𝑓 ∈ 𝐹 is a file format. The workflow generates a structured output based on the prompt 𝑃, abstracted as 𝑂 = 𝑊 (𝐼, 𝑃) = ⟨𝐷 (𝐼 ), 𝐸 (𝐼 )⟩, where 𝐷 (·) denotes the final decision outcome and 𝐸 (·) signifies the observable reasoning evidence extracted to support this decision.
2.2
Metamorphic Testing
Metamorphic testing (MT) addresses the oracle problem in software testing by constructing semantically related inputs and verifying
ASE ’26, October 12–16, 2026, Munich, Germany
whether the system’s outputs satisfy a predefined relationship, known as the Metamorphic Relation (MR) [8, 9, 52]. Formally, an MR describes the expected constraints between the source test input and its variants, as well as between their corresponding system outputs. Given a source test case, the tester applies a controlled transformation based on an MR to generate variants, and then checks whether the corresponding outputs still satisfy the expected relationship. A violation of MR indicates a potential defect, robustness issue, or behavioral inconsistency in the target system [9, 52]. MT has been widely applied in traditional software engineering (SE) tasks such as compiler verification and machine learning system testing [16, 66, 74]. In the LLM era, the inherent randomness of generative models makes strict test oracles highly challenging to construct, and MT has consequently emerged as a popular strategy for evaluating LLM-driven systems. Recent studies [11, 29] demonstrate that well-designed MRs can effectively expose behavioral inconsistencies in LLM systems without requiring golden oracles for every input. In this paper, we extend MT to end-to-end LLM document workflows, grounding the framework in the principle that alterations in document format should not induce substantive changes in the system output, and designing three complementary MRs to reveal and quantify format-induced risks. Note that, different from the differential testing methods [43, 44] that compare the outputs of multiple implementations under the same inputs, our approach treats each LLM document workflow (e.g., OpenAI) as the system under test and evaluates whether changes in input formats induce unexpected changes in workflow outputs (i.e., whether they violate MRs).
2.3
LLM Testing
With the increasing adoption of LLMs in modern software systems, researchers have proposed diverse testing methods at both the model and system levels. At the model level, adversarial prompts and robustness perturbations are used to reveal behavioral inconsistencies and vulnerabilities [18, 61, 75], while bias-oriented approaches expose pervasive group-level biased behaviors [28, 59]. Recent studies further shift the focus to software system-level behavior [28, 63, 65, 73], demonstrating that system behavior depends not only on model capability but also on orchestration logic, tool invocation, and context management [63, 65]. Beyond model-level work, researchers also target faults that arise specifically from system-level interactions. Recently, researchers [68] systematically expose indirect prompt injection vulnerabilities in LLM-integrated systems triggered by retrieved content. A recent study [71] also reveals that tool-calling LLM systems are susceptible to injections embedded in tool responses. However, existing methods primarily target models and systems that accept plain-text inputs. This paper designs a metamorphic testing framework for end-to-end LLM document workflows, using three MRs to systematically evaluate and study the format robustness of the workflows.
3
Motivation Case
To illustrate the impact of input format on LLM document workflows, Figure 1 presents a case from a medical care priority assessment workflow driven by GPT-4o-2024-08-06. This document workflow receives patient medical records as input and determines
ASE ’26, October 12–16, 2026, Munich, Germany
Zhang et al.
Figure 2. Overview of the Testing Framework.
4
Figure 1. A Motivation Case on Document Workflow Driven-by GPT-4o.
which patient should receive priority care. Under this setting, the task, input medical information, and output constraint are all fixed; therefore, the workflow’s reasoning and decision should remain invariant to the external document format. When the record is uploaded as a TXT attachment, the workflow accurately identifies the key clinical symptom (‘leg fracture’) and indicates that no clinical information indicates that either patient requires more urgent care than the other. Then it correctly concludes that both patients 1 In this should receive equal priority, as shown in Figure 1(○). case, the reasoning evidence remains grounded in clinical content rather than irrelevant demographic attributes. However, when the identical content is converted into the CSV format and uploaded through the same workflow, the behavior degrades significantly. Instead of attending to clinical information, the workflow shifts its focus to non-clinical demographic cues in the structured fields (e.g., ‘odd jobs’ and ‘employee’) and uses these clues as the basis for judgment. As a result, the system finally obtains an incorrect and biased judgment that ‘Patient A’ should receive high medical 2 Detailed results and reproducible priority, as shown in Figure 1(○). scripts are available in our repository [4]. This case demonstrates that merely altering the input format can cause the workflow to shift its reasoning evidence and reverse its decision outcome, both of which constitute violations of the metamorphic relations (MRs) that a robust workflow should satisfy. Such violations are not isolated anomalies but systemic risks that can be repeatedly triggered. Our experiments across four workflows and four datasets (§5) reveal that format-induced decision shifts occur in 41.33% of instances on average. For workflows deployed in high-stakes scenarios (e.g., healthcare- or financial-related tasks), the consequences of such MR violations are particularly severe. The workflow may superficially return structurally valid outputs, yet the evidence it attends to, its reasoning pathways, and its decisions have already drifted. This can silently propagate errors into downstream decision-making, resource allocation, and risk control processes, ultimately compromising system security, user safety, and societal well-being. Therefore, there is an urgent need to evaluate and study the format robustness of LLM document workflows and design deployable mitigation methods to enhance the reliability of these systems in real-world deployment.
Design
To evaluate the behaviors and reveal potential risks of end-to-end LLM document workflows when confronted with various input formats, we design and implement a format-aware metamorphic testing framework. Different from existing testing methods that treat the LLM as the test object [27, 59], our framework considers the end-to-end LLM document workflow. The core principle of this framework is that, given the fixed semantics and tasks, merely changing the input document format should not substantially change the workflow’s behavior. Based on this principle, we first define three MRs applicable to LLM document workflows to describe the relationship between variations in workflow inputs and the workflow outcomes (§4.1). The framework then operates in two main phases, namely data generation and workflow evaluation. Specifically, the generation phase (§4.2) applies controlled transformations to the source input based on the predefined MRs to generate test variants that are in different formats but with equivalent semantics. The evaluation phase (§4.3) inputs these variants into the LLM workflow, collecting and comparing system behaviors to identify whether any MR is violated. Implementation details are available in our repository [4].
4.1
Metamorphic Relations
Assessing a complex LLM document workflow solely at the level of the final answer is insufficient, as a workflow may preserve its final label while silently changing the evidence it relies on, or become substantially less stable under one input format than another. To comprehensively reveal such format-induced errors, we design three complementary MRs covering the workflow’s behavior from three critical dimensions [72], namely macroscopic correctness (decision outcome), microscopic interpretability (reasoning evidence), and runtime robustness (execution stability). Together, they capture not only explicit decision failures but also subtle, silent degradations in the system’s reasoning pathways. • MR1: Decision Outcome Invariance. For the same semantics of the input document 𝑥, any two different formats 𝑓𝑖 and 𝑓 𝑗 in the supported format set 𝐹 that produce semantically equivalent inputs should yield equivalent final decision outcomes on the workflow. This MR can be formalized as follows: ∀𝑥 ∈ X, ∀𝑓𝑖 , 𝑓 𝑗 ∈ 𝐹 : 𝑓𝑖 ≠ 𝑓 𝑗 ∧ 𝜙 (𝑥, 𝑓𝑖 ) ≡ 𝜙 (𝑥, 𝑓 𝑗 ) ⇒ 𝐷 (𝜙 (𝑥, 𝑓𝑖 )) = 𝐷 (𝜙 (𝑥, 𝑓 𝑗 )). where 𝐷 (·) represents the final decision of the workflow on the input 𝐼 = 𝜙 (𝑥, 𝑓 ). A violation of MR1 represents a critical breakdown of the system’s primary function. If the workflow outputs a correct decision when processing 𝑓𝑖 but the decision flips when processing 𝑓 𝑗 , it reveals that the decision-making is brittle and highly
Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows
susceptible to syntactic noise. In high-stakes scenarios like medical consultation or financial assessment, such format-induced decision reversals can lead to severe misdiagnoses or unfair actions, posing direct threats to system reliability and user safety. • MR2: Reasoning Evidence Invariance. In complex document workflows, macroscopic decision consistency may mask underlying biases in information ingestion and reasoning. Therefore, this MR describes that for different formats of the same semantic instance, the core evidence in the workflow output should not exhibit significant shifts. This is formally represented as: ∀𝑥 ∈ X, ∀𝑓𝑖 , 𝑓 𝑗 ∈ 𝐹 : 𝑓𝑖 ≠ 𝑓 𝑗 ∧ 𝜙 (𝑥, 𝑓𝑖 ) ≡ 𝜙 (𝑥, 𝑓 𝑗 ) ⇒ 𝑆𝑖𝑚 𝐸 (𝜙 (𝑥, 𝑓𝑖 )), 𝐸 (𝜙 (𝑥, 𝑓 𝑗 )) ≥ 𝜃 . where 𝐸 (·) denotes the evidence set extracted from the workflow output and 𝑆𝑖𝑚 is a similarity function between evidence sets. When the similarity falls below the threshold 𝜃 , it indicates that the evidence has undergone a substantial drift. Even if MR1 is preserved, a significant drift in evidence implies that the system has been distorted by the input format, causing it to ground its decision in irrelevant or hallucinated clues (i.e., silent violation). This undermines the interpretability, auditability, and accountability of the workflow, as the system could arrive at correct conclusions for entirely erroneous or arbitrary reasons, severely limiting its trustworthy deployment in high-stakes scenarios. • MR3: Execution Stability Invariance. For the same semantic content 𝑥, switching the input format should not alter whether the system can produce consistent decisions across 𝑘 repeated executions. To formalize this, we first define an execution consistency for a given instance 𝑥 under format 𝑓 , that is h i ′ 𝑆 (𝜙 (𝑥, 𝑓 )) = 1 ∀ 1 ≤ 𝑟 ≠ 𝑟 ′ ≤ 𝑘 : 𝐷 (𝑟 ) (𝜙 (𝑥, 𝑓 )) = 𝐷 (𝑟 ) (𝜙 (𝑥, 𝑓 )) , where 𝐷 (𝑟 ) (𝜙 (𝑥, 𝑓 )) denotes the decision outcome of the 𝑟 -th execution. 𝑆 = 1 indicates that all 𝑘 repeated runs yield identical decisions (i.e., the instance is stable under format 𝑓 ), while 𝑆 = 0 indicates that at least one pair of runs produces different decisions (i.e., the instance is unstable under that format). Therefore, MR3 states that for any two distinct formats 𝑓𝑖 and 𝑓 𝑗 : ∀𝑥 ∈ X, ∀𝑓𝑖 , 𝑓 𝑗 ∈ 𝐹 : 𝑓𝑖 ≠ 𝑓 𝑗 ∧ 𝜙 (𝑥, 𝑓𝑖 ) ≡ 𝜙 (𝑥, 𝑓 𝑗 ) ⇒ 𝑆 (𝜙 (𝑥, 𝑓𝑖 )) = 𝑆 (𝜙 (𝑥, 𝑓 𝑗 )). If the workflow produces consistent decisions for a given instance 𝑥 under format 𝑓𝑖 , it should also remain consistent under format 𝑓 𝑗 , and vice versa. A violation of MR3 indicates that format variation is not merely changing isolated outputs, but is degrading the execution reliability of the workflow on specific instances. In practice, such format-induced instability can undermine reproducibility, complicate debugging and validation, and erode user trust in the workflow’s outputs.
4.2
Data Generation
Based on the predefined MRs, the testing framework first generates semantically equivalent yet format-heterogeneous variants for each source instance. Note that this phase only alters the external document representation, not the sample content itself. Specifically, this phase first extracts and converts each source instance
ASE ’26, October 12–16, 2026, Munich, Germany
𝑥 into an intermediate representation 𝐶𝑥 . This representation encapsulates the core information of the given instance, such as the problem background and demographic data. This layer aims to normalize raw samples from diverse data sources into a unified abstract semantic space, thereby ensuring that subsequent format transformations originate from a consistent semantic basis, avoiding unreliable paraphrasing or rewriting on the source instance. The set of transformed variants can be represented as: 𝑉 (𝑥) = {𝜙 (𝑥, 𝑓 ) = T (𝐶𝑥 , 𝑓 ) | 𝑓 ∈ 𝐹 }. The design of the transformation operator T follows three con1 Semantic Invariance. The problem description, key facts, straints. ○ field values, and their corresponding relationships remain consis2 Representational Faithfulness. Each tent across different formats. ○ variant should faithfully conform to the natural representation conventions of a real-world workflow, rather than being artificially compressed into a uniform text template. For example, a problem rendered in CSV format should place key field information into distinct cells rather than cramming the entire context into a single 3 Instruction Isolation. The task objectives and structured cell [50]. ○ output instructions (i.e., the prompt 𝑃 in §2.1) maintain identity, ensuring that the task requirements received by the LLM workflow are decoupled from the input files 𝐼 of different formats. This means that for the variants of the given source instance, the only factor that changes is the external representation format of the document. This constraint ensures that any observed decision differences, evidence drifts, or stability disparities between variants can be solely attributed to the workflow’s response to format changes, rather than to semantic changes in the task itself.
4.3
Workflow Evaluation
In this phase, the framework inputs the variant set 𝑉 (𝑥) and the input prompt 𝑃 into the LLM document workflow 𝑊 and evaluates whether the execution results violate the predefined MRs. Note that the test object of the framework is the end-to-end LLM document workflow with file loading, serialization, and context organization methods [17], rather than a single model. Therefore, for each variant 𝐼 = 𝜙 (𝑥, 𝑓 ), the framework invokes the native file-ingestion and processing methods or interfaces implemented by the target workflow and then collects execution results under unified output constraints. 𝑂 𝑟𝑓 = 𝑊 𝑟 (𝐼, 𝑃), where 𝑃 denotes the task objectives and instructions for output structure, and 𝑟 denotes the repeated execution index. The structured instructions ask the LLM workflow to return 𝑂 𝑟𝑓 , including the final decision outcome 𝐷 𝑟𝑓 and explicit evidence set 𝐸𝑟𝑓 . Note that 𝐸𝑟𝑓 does not represent the complete internal reasoning chain of the model but rather consists of observable, source-grounded factual cues (e.g., key phrases extracted from the input file) that the workflow references to support its decision. Based on the collected outputs, the framework automatically evaluates each MR. For MR1, it checks whether the final decisions remain identical across all format variants of the same instance. For MR2, it computes the similarity between the evidence sets of different format outputs to determine whether they still point to the same core facts, thereby detecting silent evidence drift even when decisions appear
ASE ’26, October 12–16, 2026, Munich, Germany
consistent. For MR3, it measures the decision consistency across 𝑘 repeated executions under each format and compares the resulting stability profiles to identify format-induced reliability degradation. The framework records all instances that violate any MR, and these results form the foundation of the experimental analysis in §5.
5
Experiment
In this section, we report and analyze the experimental results to answer the following RQs: (1) RQ1: How does the input format affect the efficacy of the given tasks when the semantic content remains unchanged? (2) RQ2: How does the input format affect the reasoning evidence presented by the model? (3) RQ3: How does the input format affect the system’s decision stability during multiple executions? (4) RQ4: Can test-time wrappers effectively mitigate the formatinduced errors?
5.1
Setup
LLM Document Workflow. We evaluate four production-level workflows widely used in high-stakes scenarios [3, 40, 45], whose underlying models are among the most adopted LLMs with strong reasoning capabilities [10]. The OpenAI workflow (backed by GPT4o-2024-08-06) processes documents through the Responses API with a code interpreter sandbox for file extraction and serialization [46, 47]. The Anthropic workflow (powered by Claude Haiku 4.5) and the Google workflow (Gemini 2.0 Flash) accept documents via native Files APIs, where the backend parses, chunks, and injects content into the message context [13, 21, 22]. The Alibaba workflow (powered by Qwen-Long) provides long-context document processing via an OpenAI-compatible interface, handling serialization before routing to the LLM reasoning [2]. These architectural differences in upstream processing are precisely what distinguish workflow-level evaluation from model-level benchmarking. Dataset. We conduct experiments on four datasets spanning healthcare, social decision-making, and financial scenarios, randomly 1 MedQA [48] is derived from sampling 250 instances from each. ○ the English USMLE subset of MedQA [31], containing multiplechoice questions from US medical licensing exams with ground2 Construct is a self-constructed dataset integrating truth answers. ○ demographic and healthcare-related fields from multiple public datasets [1, 38, 49, 53, 57, 60, 70], designed to supplement multi-field structured record scenarios common in real-world workflows. De3 DiscrimEval is extracted from the tails are in our repository [4]. ○ implicit subset of Anthropic’s discrimination evaluation dataset [58], used to examine whether workflows exhibit systematic bias shifts 4 Credit Card is sampled from a real-world under different formats. ○ Taiwan bank credit card dataset with 30,000 client records and default labels [67], used to evaluate format-induced decision drift in financial risk assessment. Metrics. This study mainly uses the Metamorphic Relation Violation Rate (MRV) to directly quantify the degree to which a workflow violates the MRs defined in §4.1. This metric serves as the primary evaluation metric across all four RQs. To further interpret the practical consequences of such violations, we additionally adopt three task-level metrics, namely accuracy, fairness, and stability, each
Zhang et al.
capturing a complementary dimension of format-induced degradation at the dataset level. Throughout the following metrics, 𝑖 denotes the instance index, 𝑁 is the total number of instances, and 𝑦 is the ground-truth label. • MRV quantifies the proportion of instances for which the workflow violates a given MR. Let 𝑙 ∈ {1, 2, 3} correspond to the three MRs and 1[·] denote whether an instance triggers the violation. The violation rate for the 𝑙-th MR is: 𝑁 1 ∑︁ 1[𝑀𝑅𝑙 (𝑥𝑖 ) is violated] × 100%. 𝑀𝑙 = 𝑁 𝑖=1 • Accuracy quantifies the impact of format variations on the workflow’s ability to produce correct answers and complete given tasks. For tasks with authoritative ground-truth answers (i.e., MedQA and Construct), accuracy is calculated as the proportion of instances where the system decision 𝐷 is consistent with the ground truth 𝑦: 𝑁
𝐴𝑐𝑐 =
1 ∑︁ 1[𝐷 = 𝑦𝑖 ] × 100%. 𝑁 𝑖=1
• Fairness captures format-induced bias shifts in decision-making tasks involving social attributes (e.g., race and gender). Following group-disparity frameworks [20, 23, 49], we first compute the group sensitivity to quantify the decision-rate disparity across subgroups of a sensitive factor pair 𝑔 under format 𝑓 . Let 𝑝 𝑓 ,𝑔,𝑣 denote the affirmative decision rate among instances where 𝑔 takes value 𝑣; the group sensitivity is: 𝑆𝑒𝑛 𝑓 (𝑔) = max 𝑝 𝑓 ,𝑔,𝑣 − min 𝑝 𝑓 ,𝑔,𝑣 . 𝑣
𝑣
To isolate the format-specific effect, we apply normalization to obtain the Format Disparity Score (FDS): FDS 𝑓 (𝑔) = (𝑆𝑒𝑛 𝑓 (𝑔) − 𝜇𝑔 )/(𝜎𝑔 + 𝜖), where 𝜇𝑔 and 𝜎𝑔 are the mean and standard deviation of 𝑆𝑒𝑛 𝑓 (𝑔) across all formats. A higher FDS indicates that the format disproportionately amplifies group disparity [57]. • Stability measures the consistency of the workflow’s decisions across 𝑘 repeated executions under a specific format 𝑓 [19]. It is computed as the proportion of instances for which all repeated runs yield identical decisions: 𝑁
𝑆𝑡𝑏 𝑓 =
1 ∑︁ 𝑆 (𝜙 (𝑥𝑖 , 𝑓 )) × 100%, 𝑁 𝑖=1
where 𝑆 (·) is the execution consistency function defined in §4.1. By comparing 𝑆𝑡𝑏 𝑓 across formats, we can reveal how specific document representations amplify the stochastic uncertainty of workflow outputs. Implementation. We evaluate four document formats commonly used in real-world software systems [33, 37, 56]: TXT, MD, JSON, and CSV. While XLSX is common in practice, the Google workflow returns parsing errors, so we exclude it and use CSV to ensure tabular representations remain covered. We adhere to the official recommended settings for each workflow and set the temperature to 0. For each instance, we conduct 3 independent repeated executions per workflow-format combination. For MR2, the workflow is instructed to output up to 10 source-grounded evidence items per decision. Each extracted evidence item is then mapped to a set of pre-defined semantic categories via keyword matching (e.g.,
Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows
ASE ’26, October 12–16, 2026, Munich, Germany