JTA: Joint Testability Architecture for Scenario-Based Validation of Safety-Critical Software

JTA: Joint Testability Architecture for Scenario-Based Validation of Safety-Critical Software Wenyao Xue1 , Jiandi Wang1 , and Yichen Wang1,∗ 1

arXiv:2608.05594v1 [cs.SE] 6 Aug 2026

Beihang University, Beijing, China xuewenyao@buaa.edu.cn, 22374306@buaa.edu.cn, wangyichen@buaa.edu.cn *corresponding author

Abstract—Validation adequacy in safety-critical software depends on more than the system under test. Critical scenarios must be constructed under controlled conditions, execution evidence must be aligned into verdict-ready form, and abnormal outcomes must be attributable to actionable causes. Existing testability research remains largely artifact-centric and offers little architectural support for reasoning about the combined capability of the scenario, the test system, and the system under test. Joint Testability Architecture (JTA) addresses this gap by treating those three elements as a single object of analysis and design. It characterizes validation capability along three dimensions—controllability, observability, and isolability—and organizes them through three domains, three bridges, and an analysis-design-evaluation-refinement loop. JTA also introduces scenario contracts, joint capability assessment, validation blind-spot identification, and bridgeoriented design actions that map capability gaps to concrete improvements in control points, evidence organization, and attribution boundaries. An illustrative analysis of ArduPilot failsafe validation shows that link-loss scenarios are comparatively mature, whereas state-estimation anomaly scenarios remain harder to validate because evidence alignment and attribution semantics are weaker. JTA is not a replacement for existing testing or safety-analysis techniques; it provides an architectural basis for modeling, designing, and assessing scenario-based validation in safety-critical software. Keywords–Joint Testability Architecture, safety-critical software, test system, controllability, observability, isolability 1. I NTRODUCTION In safety-critical systems such as flight control, automated driving, rail transit, and industrial control, validation problems rarely arise from a simple lack of test cases. More often, the difficulty lies in constructing critical scenarios stably, collecting enough execution evidence to support a verdict, or attributing abnormal outcomes to an actionable engineering boundary. This suggests that validation capability is not solely a property of the code under test; it is a system-level capability shaped jointly by the scenario, the test system, and the system under test (SUT). Traditional research on software testability has accumulated substantial knowledge on controllability, observability, structural complexity, and design-for-testability [1], [2], [3], [6].

However, such research typically interprets testability as a static or semi-static property of the software artifact itself. When the target of analysis shifts to safety-critical scenarios such as failsafe activation, abnormal recovery, mode migration, or multi-component coordinated control, an artifact-centric view is no longer sufficient to answer the following questions: Can the test system reliably construct the target scenario? Can system outputs be organized into verdict-ready evidence? Can validation failures be narrowed down to engineeringactionable responsibility boundaries? In other words, “code being testable” does not imply that “scenario-based validation is adequate.” This gap becomes particularly visible in mature open-source projects. ArduPilot, for example, already provides a relatively complete ecosystem of software-in-the-loop (SITL) simulation, regression testing, and logging infrastructure, together with long-evolved flight-control and failsafe logic. Yet once the validation target shifts from ordinary functional behavior to scenarios such as remote-control (RC) link loss, groundcontrol-station (GCS) heartbeat timeout, and extended Kalman filter (EKF) state-estimation anomalies, the central question is no longer whether a script exists. It is whether existing scripts, simulation environments, and exposed system hooks can jointly support an adequate validation system. We therefore introduce Joint Testability Architecture (JTA), which treats testability in safety-critical software as a scenarioconditioned capability of a joint system. The paper addresses three research questions: 1) RQ1: How can scenarios, the test system, and the system under test be unified into a joint object that is amenable to analysis and design? 2) RQ2: How can controllability, observability, and isolability be operationalized into a repeatable adequacy judgment for scenario-based validation? 3) RQ3: How can joint capability gaps be translated into executable architectural design actions, so that validation blind spots can be systematically reduced? Our contributions are as follows: 1) We introduce the joint testability system model JT Sσ = ⟨σ, Tσ , Sσ ⟩, which expands the object of validation in safety-critical software from an isolated software artifact to a scenario-instantiated joint entity composed of the scenario, the test system, and the system under test.

2) We define a scenario contract SC(σ) with an explicit faultsource set r, thereby turning isolability from a generic slogan into a concrete design constraint centered on distinguishable fault sources. 3) We formulate joint capability and scenario adequacy, and propose a bridge-oriented design method organized as “three domains, three bridges, and a closed loop,” so that capability gaps can be mapped directly to concrete design actions on the control bridge, evidence bridge, and attribution bridge. 4) We present an illustrative analysis of ArduPilot failsafe validation, showing how JTA identifies capability blind spots in link-failure and state-estimation anomaly scenarios and derives corresponding bridge-level design recommendations. The remainder is organized as follows. Section 2 discusses the background and related work. Section 3 presents the Joint Testability Architecture and its formal model. Section 4 describes the capability-gap-driven analysis and design method. Section 5 provides an illustrative evaluation using ArduPilot. Section 6 discusses the novelty, implications, and limitations of the work. Section 7 offers concluding remarks. 2. BACKGROUND AND R ELATED W ORK 2.1 Artifact-Centric Testability Research Research on software testability has long focused on whether a program is easy to test. Early studies examined how program behavior can be stimulated and observed through controllability and observability [1]. ISO/IEC 25010 includes testability in the software quality model and emphasizes a product’s ability to support the establishment of test criteria and the execution of tests [6]. More recent work extends the discussion to designlevel testability measurement and architectural transformation for testability improvement [2], [3], [4]. Model-based systems engineering (MBSE) work on design-for-testability (DFT) similarly shows that testability can be shaped during system design [5]. Two points from this literature matter most here: controllability and observability remain central dimensions of testability, and testability can be improved through design intervention. That literature also has a clear scope. The object of analysis is usually the software artifact itself, and the test system and scenario constraints are seldom treated as co-equal analysis objects alongside the system under test. Once the focus shifts from “whether a piece of code is easy to test” to “whether a safety-critical scenario is validated adequately,” the test system’s driving capability, evidence organization capability, and attribution capability become indispensable variables. 2.2 Safety-Critical Verification and Fault-Driven Assessment In safety-critical systems, fault injection, scenario analysis, system safety analysis, and assurance-based validation have all been studied extensively. Fault-injection research uses controlled perturbations to validate fault tolerance and protection mechanisms [9], [10]. System safety approaches such as System-Theoretic Process Analysis (STPA), Failure Mode and Effects Analysis (FMEA), and Fault Tree Analysis (FTA)

focus on hazards, failure modes, unsafe control actions, and causal scenarios [11], [12]. More recent work on model-based verification and assurance-case engineering emphasizes organizing evidence and validation activities early in the lifecycle [13], [14]. Studies of safety-critical test assessment have also examined test assets through safety tactics and mutation-based fault sensitivity [19]. IEEE Std 1012 frames verification and validation activities from a lifecycle perspective [8]. Across these lines of work, one point is consistent: validation in safety-critical systems is organized around scenarios, evidence, and accountability chains, not just execution. JTA is complementary to this literature rather than a replacement for it. Methods such as STPA are effective for identifying critical scenarios and safety constraints, but they do not directly answer whether an existing test system and system under test can jointly support adequate validation for those scenarios. Fault-injection techniques are useful for constructing abnormalities, but they do not by themselves guarantee evidence structuring or attribution convergence. JTA addresses this gap by providing an architectural perspective on joint adequacy across these adjacent methods. 2.3 Test Architecture, Model-Based Testing, and UTP Model-Based Testing (MBT) and the UML Testing Profile (UTP) provide standardized ways to represent test contexts, test components, test behavior, and verdict relations [15], [16]. ISO/IEC/IEEE 29119 offers a general framework for testing processes and documentation [7]. Contract-based design validation further shows that test conditions can be expressed explicitly through preconditions and postconditions [17]. Together, these efforts move the test system beyond an informal collection of scripts and toward an object that can be modeled explicitly. Our recent work on modeling testing requirements for safety-critical software likewise indicates that scenarios, constraints, and testing requirements can be organized systematically [18]. JTA differs from these approaches in what it takes as the main problem. MBT and UTP focus on how to represent test artifacts and generate testing activities, whereas JTA focuses on how to identify joint capability gaps around critical scenarios and map those gaps to architectural design actions. UTP can serve as one possible representation path for JTA, but it is not the methodological core. The main concern in JTA is joint adequacy analysis, not the completeness of modeling notation. 2.4 Research Positioning Existing work provides useful foundations in artifact-level testability, safety-scenario identification, and test modeling. However, there is still no approach that simultaneously satisfies the following three conditions: 1) it takes the joint entity composed of the scenario, the test system, and the system under test as the object of analysis; 2) it operationalizes controllability, observability, and isolability into an adequacy judgment for scenario-based validation; and 3) it maps validation blind spots directly to executable archi-

tectural design actions. JTA is proposed as a unified response to these three requirements. 3. A RCHITECTURE AND F ORMAL M ODEL 3.1 Architecture Definition The Joint Testability Architecture (JTA) is formalized as JT A = ⟨DΣ , DT , DS , BC , BO , BI , L⟩

(1)

where DΣ denotes the Scenario Domain, DT the Test-System Domain, and DS the System-Under-Test Domain; BC , BO , and BI denote the control bridge, evidence bridge, and attribution bridge, respectively; and L denotes the closedloop process of analysis, design, evaluation, and refinement. Instead of focusing on an isolated software artifact or a standalone testing tool, JTA takes a scenario-centered joint testability system as its primary object of analysis. Plainly, the architecture asks three practical questions for each critical scenario: what must be driven, what must be observed, and what must be distinguished. 3.2 Joint Testability System For project-level validation problems, the joint testability system is defined as JT S = ⟨Σ, T, S⟩ (2) where Σ is the set of critical scenarios, T is the test system, and S is the system under test. For any scenario σ ∈ Σ, the scenario-instantiated joint testability system is defined as JT Sσ = ⟨σ, Tσ , Sσ ⟩

(3)

where Tσ ⊆ T denotes the subset of testing capabilities that actually participate in stimulation, collection, verdict, and diagnosis for scenario σ, and Sσ ⊆ S denotes the boundary of the system under test that is directly relevant to the scenario. This definition emphasizes that the basic object of joint testability is not an isolated software artifact, but a scenario-centered validation assembly. 3.3 Scenario Contracts Inspired by contract-based formulations that make preconditions and acceptable outcomes explicit in advance [17], each critical scenario is represented by the following scenario contract: SC(σ) = ⟨g, p, u, e, o, r, κ, Θ⟩ (4) where: • g: validation objective of the scenario; • p: preconditions and environmental constraints; • u: stimulation mode and stimulus set; • e: minimum evidence set; • o: verdict rules; • r: set of fault sources to be distinguished; • κ: criticality of the scenario; • Θ = ⟨θC , θO , θI ⟩: minimum requirements on controllability, observability, and isolability.

This eight-tuple specifies the minimum semantic boundary required to support valid scenario-based verification. Specifically, g, p, and u answer “what is to be validated,” “under which constraints,” and “by what means the scenario should be activated.” The pair e and o defines “which minimum evidence is required” and “under what rules a verdict is produced.” The set r states which potential fault sources must be distinguishable. The threshold vector Θ specifies the minimum adequacy requirements on controllability, observability, and isolability for the scenario, while κ is used to weight risk importance and improvement priority at the project level. Therefore, the scenario contract serves both as the input for deriving capability requirements and as the prerequisite for computing JT Cap(JT Sσ ), Adeq(σ), and Gap(σ). Compared with scenario descriptions that mainly serve as testing specifications or requirement statements [18], JTA explicitly incorporates the fault-source set r into the scenario contract. Safety-critical validation usually cannot stop at determining whether an abnormality has occurred; it must also clarify which candidate sources the abnormality should be distinguished from, so that the validation conclusion can be translated into actionable evidence organization, verdict rules, and rectification boundaries. Without r, isolability remains a generic statement and is difficult to transform into explicit constraints on attribution obligations, evidence design, and bridge-level actions [9], [10], [12], [14]. This is the step that turns a scenario from a testing description into a design constraint. 3.4 Operationalizing the Three Capability Dimensions Assume that the scenario contract SC(σ) induces three requirement sets: the control-obligation set PCreq (σ), the evidence-obligation set POreq (σ), and the attribution-obligation set PIreq (σ). In practice, these sets are a review checklist derived from the scenario contract. For example, a control obligation may be a parameter, injection point, or harness entry; an evidence obligation may be a log field, state snapshot, or event timestamp; and an attribution obligation may be a cause code, source label, or responsibility boundary. In particular, PIreq (σ) is jointly determined by r and o: if a verdict rule requires the distinction among several fault sources, the system must provide cause points, boundary points, or semantic markers that support such a distinction. Let the control points, evidence points, and attribution points actually exposed by the system under test be denoted by PCexp (σ), POexp (σ), and PIexp (σ), respectively. The systemside capabilities are then defined as |PCreq (σ) ∩ PCexp (σ)| |PCreq (σ)| |P req (σ) ∩ POexp (σ)| OS (σ) = O |POreq (σ)| |P req (σ) ∩ PIexp (σ)| IS (σ) = I |PIreq (σ)| CS (σ) =

(5) (6) (7)

where CS reflects how completely the system under test provides controlled activation points for the scenario, OS

reflects the extent to which the system exposes key evidence required for a verdict, and IS reflects how explicitly it exposes responsibility boundaries and cause semantics. Observability and isolability are therefore related but not identical: observability asks whether relevant facts can be seen, whereas isolability asks whether those facts can separate candidate fault sources into actionable causes. For cases where a single evidence item is insufficient, JTA represents isolability through conditional distinguishability. For two candidate fault sources ri , rj ∈ r, define Dσ (ri , rj | E, τ ) ∈ {0, 12 , 1}

TABLE I. Operational Meaning of the Three Capability Dimensions Dimension

Controllability C

Observability O

(8)

where E is the available evidence bundle and τ is the relevant time window. A value of 1 means that the pair can be distinguished directly; 12 means that it is distinguishable only when combined evidence or timing constraints are satisfied; and 0 means that the current joint system cannot distinguish the pair. This treatment captures cases such as mode transitions that require both a cause code and a preceding timing pattern before attribution is valid. The capability of the test system should not be described merely in terms of tool presence or absence. Instead, it should be characterized through repeatable behavioral indicators. Accordingly, nine basic indicators are defined, each taking values in [0, 1]: • A: scenario activation success rate; R: repeatability consistency; Tp : temporal control precision; • Ec : evidence completeness; La : evidence alignment quality; Va : verdict automation rate; • Ds : ratio of distinguishable fault sources; Be : explicitness of responsibility boundaries; Qa : attribution consistency. The test-system-side capabilities are then defined as CT (σ) = α1 A + α2 R + α3 Tp

(9)

OT (σ) = β1 Ec + β2 La + β3 Va

(10)

IT (σ) = γ1 Ds + γ2 Be + γ3 Qa

(11)

where αi , βi , γi ∈ [0, 1] and each weight set sums to 1. The significance of these equations is that test-system capability depends not only on whether a particular tool exists, but on whether available tools can reliably activate scenarios, produce verdict-ready evidence, and support consistent attribution across executions. Table I summarizes the operational meaning of the three capability dimensions. 3.5 Scoring and Review Protocol To make JT Cap(JT Sσ ) reproducible, the review follows a trace-matrix protocol. Required points are extracted from requirements, scenario contracts, STPA/FMEA/FTA results, verdict rules, and safety constraints; exposed points are extracted from SUT interfaces, parameters, harnesses, log schemas, event streams, cause codes, and test-system adapters. The system-side coverage ratios are then mapped to the ordinal scale used in the case study: H means that at least 80% of required points are exposed and no critical point is missing; M means 50–80% coverage or a manual/partly ambiguous

Isolability I

SUT-side concern

Test-system concern

Whether the test system Whether sufficient control can stably construct the points are exposed to scenario, repeat support controlled executions, and finely scenario activation control critical timing Whether the test system Whether key events, can collect, align, states, and logs required structure, and for a verdict are emitted automatically adjudicate evidence Whether the test system Whether explicit can distinguish fault responsibility boundaries sources and produce and cause semantics are consistent attribution exposed outcomes

evidence chain; L means below 50% coverage or a missing critical point. For the test-system indicators, equal weights are the default. Non-uniform weights are allowed only when justified by scenario criticality or a safety artifact, and they should be reported with a sensitivity check. This protocol does not remove expert judgment, but it makes the judgment auditable. 3.6 Joint Capability, Scenario Adequacy, and Validation Blind Spots Because scenario-based validation capability is constrained jointly by the test system and the system under test, the joint capability is defined as JT Cap(JT Sσ ) = ⟨JC(σ), JO(σ), JI(σ)⟩ and

(12)

JC(σ) = min(CS (σ), CT (σ)) JO(σ) = min(OS (σ), OT (σ))

(13)

JI(σ) = min(IS (σ), IT (σ)) The minimum operator, rather than an average, is used because joint capability is subject to a strong bottleneck constraint. If the system under test does not expose the necessary control points, even a powerful test infrastructure cannot reliably construct the scenario. Conversely, if the test system cannot organize evidence and produce a verdict, the raw logs emitted by the system under test still cannot be converted into a validation conclusion. Scenario adequacy is defined as ( 1, JC(σ) ≥ θC ∧ JO(σ) ≥ θO ∧ JI(σ) ≥ θI Adeq(σ) = 0, otherwise (14) The capability-gap vector is defined as Gap(σ) = max(0, Θ(σ) − JT Cap(JT Sσ ))

(15)

where the three components of Gap correspond to controllability, observability, and isolability gaps, respectively. The set of validation blind spots is further defined as Blind(Σ) = {σ ∈ Σ | Adeq(σ) = 0}

(16)

To summarize the validation status of critical scenarios at the project level, the system-level joint testability is defined as P κ(σ) · Adeq(σ) (17) JT (JT S) = σ∈ΣP σ∈Σ κ(σ) where JT (JT S) ∈ [0, 1] denotes the criticality-weighted adequacy coverage over the set of critical scenarios. These definitions are not meant to collapse scenario analysis into a single abstract score. Their immediate role is to answer a more practical question: for a concrete scenario-based validation task, does the current joint testability system meet the minimum adequacy requirements? At the project level, JT (JT S) then provides a traceable summary view. 3.7 Three Domains, Three Bridges, and a Closed Loop JTA organizes the joint testability system around three domains, three categories of bridges, and a continuously evolving closed loop. The “three” structure is intentional rather than decorative. The three capability dimensions are necessary because a valid scenario result must be constructed, observed, and attributed; omitting any one of them leaves a different type of blind spot. The three domains are necessary because scenario intent, test-system capability, and SUT exposure are owned by different engineering artifacts. The three bridges are necessary because each cross-boundary mismatch requires a different design response. The Scenario Domain DΣ organizes scenario contracts, validation objectives, evidence requirements, verdict rules, and criticality, and serves as the organizing center of the joint design space. • The Test-System Domain DT carries capabilities such as stimulation orchestration, simulation and replay, fault injection, evidence collection, verdict generation, and diagnosis. • The System-Under-Test Domain DS carries not only functional behavior but also the responsibility of supplying validation capability; it therefore needs to expose sufficient control points, evidence points, and responsibility boundaries during design. •

The three bridges play distinct roles. The control bridge BC translates the stimulation requirements in the scenario contract into reproducible parameter settings, message injections, temporal control, and constrained anomaly construction. The evidence bridge BO organizes logs, states, events, and mode transitions into structured evidence and supports automated verdict generation. The attribution bridge BI maps anomalous outcomes to trigger sources, mode causes, and component boundaries, so that failures can be translated into engineeringactionable remediation tasks. These bridges are architectural patterns, not necessarily domain-independent tools: in concrete systems they may be realized by harnesses, adapters, logging instrumentation, monitors, replay controllers, verdict scripts, or responsibility-boundary annotations. From the perspective of design operations, the three bridges can be further refined as follows:

Control bridge BC : its inputs are the scenario preconditions p, the stimulation mode u, and the available set of control points; its output is a repeatable sequence of parameterized stimuli. It is activated with priority when GapC (σ) > 0. Typical actions include a scenario orchestrator, a timing controller, fault-injection adapters, and injection-enabled replay mechanisms. • Evidence bridge BO : its inputs are raw log streams, state streams, and the minimum evidence set e; its output is structured, temporally aligned evidence that supports the verdict rules o. It is activated with priority when GapO (σ) > 0. Typical actions include structured logging, a state snapshot bus, evidence structuring components, and automated verdict rules. • Attribution bridge BI : its inputs are failure outcomes, candidate trigger sources, and cause semantics; its output is an attribution result that aligns source, cause, and boundary. It is activated with priority when GapI (σ) > 0. Typical actions include cause-code extension, responsibilityboundary mapping, multi-trigger discrimination rules, and semantic annotations for mode transitions. The overall method follows an analysis-design-evaluationrefinement loop: critical scenarios and their capability gaps are first identified; bridge mechanisms are then added according to the gap types; scenario adequacy is subsequently re-evaluated; and any remaining gaps are fed back into the next round of modeling. •

4. M ETHODOLOGY 4.1 Five-Step Analysis Workflow JTA uses a five-step workflow to analyze critical scenarios. Steps 1–2 instantiate the joint object in RQ1, Step 3 operationalizes the adequacy judgment in RQ2, and Steps 4–5 translate capability gaps into bridge-oriented design actions for RQ3. Step 1: Identify critical scenarios. Based on requirements, failure reports, domain experience, or the results of system safety analysis such as STPA, FMEA, or FTA, determine the scenario set Σ to be assessed and assign a criticality value κ to each scenario. Step 2: Instantiate scenario contracts. For each σ ∈ Σ, explicitly fill in the objective, preconditions, stimulation mode, minimum evidence set, verdict rules, fault-source set, and threshold vector in SC(σ). The focus of this step is not to write an additional test case, but to make explicit the minimum joint conditions required for a valid verification conclusion. Step 3: Assess bilateral capabilities. Assess CapS (σ) = ⟨CS , OS , IS ⟩ and CapT (σ) = ⟨CT , OT , IT ⟩ separately. When data are sufficiently available, numerical indicators may be used. In illustrative analyses, each dimension may also be mapped to a three-level ordinal scale—high (H), medium (M), and low (L)—according to a unified rubric. Table II provides a set of unified criteria for illustrative assessment, intended to reduce arbitrariness in the H/M/L mapping. Step 4: Compute capability gaps and identify blind spots.

�∑ Scenario Domain Scenario Contract ��(�)

�� Test System Domain Simulator

Fault Injector Decision Maker

Replayer

Minimal Evidence �

Candidate Failure Sources r

Controlled Activation

Decision Rule �

Verification Objective �

Criticality �

Fault Injection

Temporal Orchestration

System Logs

System Events

�� Evidence Bridge

System States

System Snapshots

Cause Codes

Responsibility Boundaries

�� Attribution Bridge

Failure Sources

Analysis

Design

Environmental Constraints �

Driving Mode �

Parameter Configuration

�� Control Bridge

Monitor

……

Minimum Threshold Θ

�� Systerm Under Test Domain Core Functions

State Machine

Control Points

Evidence Points

Safety Mechanisms

……

Decision Associations

Evaluate

Refine

Figure 1. The “three domains, three bridges, and a closed loop” architecture of JTA. The Scenario Domain organizes contracts and thresholds; the three bridges connect the Test-System Domain and the System-Under-Test Domain; and the closed loop turns capability gaps into evolving bridge-level design actions. According to the definitions in Section 3, compute joint capability, scenario adequacy, and capability gaps. Any scenario with Gap(σ) ̸= 0 is treated as a validation blind spot and must enter the design stage. Step 5: Perform bridge design and re-evaluation. Map the gap components to design actions on the control bridge, evidence bridge, and attribution bridge. After the additions are made, re-evaluate scenario adequacy until the minimum requirements are met for critical scenarios or an explicit residual-risk explanation is provided. 4.2 Capability Gaps and Bridge Design Patterns Table III summarizes representative symptoms of the three gap categories and their corresponding design patterns. 4.3 Two Adoption Modes JTA supports two adoption modes. The retrofitting mode targets existing systems and supplements scenario contracts and bridge mechanisms around critical scenarios on top of available test assets. The co-construction mode, in contrast, uses scenario contracts to constrain the design of control points, evidence points, and responsibility boundaries from the early stage of system design, thereby making validation capability an architectural quality attribute on a par with functional capability. The case study reported here adopts the former mode, but the method itself supports both. 5. C ASE : A RDU P ILOT 5.1 Case Objectives and Analysis Protocol We use the open-source flight-control stack ArduPilot as an illustrative case [20]. ArduPilot is a useful case for three reasons. First, the project provides publicly accessible code for

flight control, failsafe mechanisms, and logging infrastructure. Second, it already includes SITL, automated testing, and replay capabilities, making it a realistic carrier of an “existing test system.” Third, the three scenarios of RC link loss, GCS heartbeat timeout, and EKF state-estimation anomalies span different levels of validation difficulty, from external link failures, to communication-timeout failures, to internal stateestimation degradation. This ordering is intentional: it lets the case show how the bottleneck moves from control, to evidence, to attribution as scenario semantics become more complex. Beyond illustrating how JTA organizes analysis and design, the case reuses the paper’s research questions in a concrete setting: whether JTA can instantiate a joint object for each failsafe scenario (RQ1), whether it can judge adequacy through controllability, observability, and isolability (RQ2), and whether the resulting gaps can be mapped to concrete bridge-level actions (RQ3). To avoid presenting the case as a runtime evaluation without runtime evidence, the following analysis protocol is adopted: • A0 : based on the existing ArduPilot codebase and test assets, identify the current joint capability in a structured manner; • A1 : based on the gaps identified in A0 , formulate targeted JTA bridge-level design actions and analyze the expected path for reducing blind spots. We do not report quantitative performance gains after bridge actions are carried out in the system. The case study instead reports the capability assessment and the resulting designoriented improvement paths around critical scenarios. That choice fits the paper’s methodological focus and avoids over-

TABLE II. Illustrative Assessment Criteria for the Three Capability Dimensions (H/M/L) Dimension

Controllability C

Observability O

Isolability I

SUT side (CapS )

Test-system side (CapT )

H: perturbation strength, H: all critical fault timing, and duration can be conditions can be triggered configured independently independently with and repeated reliably. configurable timing. M: the scenario can be M: some conditions are driven, but parameters are controllable but timing or coupled or repeatability is interfaces are constrained. moderate. L: no practical control L: the target scenario entry exists for the target cannot be constructed condition. reliably. H: key states, events, and H: evidence can be mode transitions are all collected, temporally emitted with explicit aligned, structured, and semantics. adjudicated automatically. M: key information is M: information can be partially visible but aggregated but manual evidence is incomplete. alignment is still needed. L: effective logs or state L: effective exposure are missing for evidence-processing critical behavior. capability is missing. H: cause codes cover major trigger paths and H: consistent responsibility boundaries source-cause-boundary are explicit. attribution can be produced. M: partial cause semantics M: major sources can be exist but boundaries remain distinguished but stability ambiguous. is insufficient. L: cause codes are missing L: failure sources cannot or major sources cannot be be effectively distinguished. distinguished.

TABLE III. Capability Gaps and Bridge Design Patterns Gap dimension

Representative symptoms

Controllability C

The target scenario cannot be reproduced stably; critical timing is uncontrollable; perturbations cannot be repeated

Observability O

Isolability I

Representative design actions

Add a scenario orchestrator, parameterized control interfaces, a timing controller, constrained fault-injection adapters, and injection-enabled replay mechanisms to BC Only fragmented logs are Add structured logging, a available; events and states state snapshot bus, evidence are not aligned; verdicts structuring, timeline depend on manual alignment, and automated inspection verdict rules to BO Add cause codes, source Failure sources are unclear; labels, multi-component responsibility-boundary boundaries are ambiguous; mapping, mode-transition conclusions cannot be semantics, and turned into remediation trigger-discrimination rules tasks to BI

committing to quantitative claims that the available material does not yet support. The case maps JTA concepts to concrete ArduPilot assets as follows. The Test-System Domain contains SITL, sim_vehicle.py, autotest.py, MAVLink heartbeat control, replay assets, and log-processing scripts. The SystemUnder-Test Domain contains failsafe logic, mode management,

EKF modules, DataFlash logging, event messages, and modereason enumerations. The Scenario Domain contains the three failsafe contracts in Table IV. Thus the case is not an analysis of the whole repository; it is a scenario-scoped joint testability analysis over the assets that participate in failsafe validation. The case assessment adopts a unified three-level ordinal rubric: H indicates that the dimension can stably support the core requirements of the scenario contract; • M indicates that the dimension partially satisfies the requirements, but still relies on manual supplementation or contains local uncertainty; • L indicates that the dimension cannot stably support the core requirements of the contract. •

To reduce subjectivity in the illustrative assessment, the bilateral capabilities for all scenarios are judged according to the unified criteria in Table II. To highlight the logic of minimum adequacy, the minimum threshold for all three scenarios is uniformly set to Θ = ⟨M, M, M ⟩. 5.2 Scenario Contract Instantiation Table IV lists the key components of the scenario contracts for the three scenarios. Unlike conventional scenario descriptions, the table gives not only the validation objective, stimulation mode, and minimum evidence, but also the explicit faultsource set r that must be distinguished. In this case, κS1 = 1.0, κS2 = 1.0, and κS3 = 1.5 are used to reflect the higher risk significance and validation difficulty of the EKF stateestimation anomaly scenario. 5.3 Joint Capability Assessment Using the code and test-asset mapping established above, the scenario-relevant artifacts can be grouped as follows. On the test-system side, they include sim_vehicle.py, autotest.py, SITL, the MAVLink heartbeat channel, replay assets, and log-processing assets. On the system-undertest side, they include failsafe logic, mode management, EKF modules, the logging mechanism, and mode-reason enumerations. Following the three-dimensional capability model in Section 3 and the illustrative criteria in Table II, the three scenarios are assessed ordinally, and the results are shown in Table V. The rating basis is the presence or absence of scenario-specific required/exposed points: controllable parameters and message channels for C, aligned logs and mode events for O, and cause codes plus source-boundary mappings for I. Table V yields three observations. First, S1 shows that JTA does not assume every scenario is inadequate. For link-related scenarios with comparatively mature test assets, minimum adequacy can already be achieved. Second, the main problem in S2 is not scenario construction itself, but the weakness of the attribution bridge: the current logs and cause semantics are insufficient to distinguish a genuine GCS heartbeat timeout from control error introduced by the testing toolchain. Third, S3 captures a typical joint-testability challenge. Compared with link failures, state-estimation anomalies depend more heavily on fine-grained perturbation control and cross-component

TABLE IV. Key Components of the Scenario Contracts for the Three Scenarios Stimulation mode u

Minimum evidence set e

Sources to be distinguished r

S1 RC

Validate whether the expected protective action (RTL/LAND/SMART RTL) is triggered after RC link loss

Validation objective g

SIM_RC_FAIL=1, coordinated with parameters such as FS_THR_ENABLE

mode-switch records, failsafe events, ModeReason:: RADIO_FAILSAFE, and recovery path

genuine RC loss; operator misuse at the ground side; test-script timeout or timing drift

S2 GCS

Validate whether the specified protective action is triggered after GCS heartbeat timeout

interruption of MAVLink HEARTBEAT, combined with FS_GCS_TIMEOUT settings

timeout events, mode-change logs, ModeReason::GCS_FAILSAFE, and trigger timeline

S3 EKF

Validate whether ALT HOLD/LAND is triggered when EKF variance or health degrades beyond the threshold

simulation perturbations combined with parameters such as SIM_GPS1_ENABLE and FS_EKF_THRESH

EKF variance/health status, failsafe events, ModeReason:: EKF_FAILSAFE, and mode-transition timeline

Scenario

genuine GCS heartbeat timeout; communication-stack jitter; abnormal heartbeat control in the test toolchain EKF variance beyond threshold; EKF health-bit failure; concurrent GPS quality degradation; mode transition caused by factors other than EKF

TABLE V. Joint Capability Assessment Results for the Three ArduPilot Scenarios (H/M/L as Ordinal Ratings) Scenario

CS

OS

IS

CT

OT

IT

JC

JO

JI

Adeq

S1 RC

H

H

M

H

M

M

H

M

M

1

S2 GCS

H

H

M

H

M

L

H

M

L

0

S3 EKF

M

H

L

L

M

L

L

M

L

0

RC link loss can be triggered reliably. The remaining bottlenecks lie mainly in evidence structuring and coarse-grained cause semantics, but the scenario already meets the minimum adequacy threshold Heartbeat timeout can be triggered, but attribution remains weak across timeout occurrence, action trigger, and externally visible cause semantics Repeatable construction of EKF anomalies remains weak. Mode transitions are observable, but trigger-source discrimination and attribution consistency are still limited

level joint testability, the current result is 1.0 × 1 + 1.0 × 0 + 1.5 × 0 JT (JT S) = = 28.6% 1.0 + 1.0 + 1.5 This value is a criticality-weighted adequacy coverage computed from the ordinal judgments in Table V; it is not an empirical performance-improvement metric. It shows that the current test assets cover the RC link-loss scenario reasonably well, but substantial blind spots remain in the joint adequacy of the GCS-timeout and EKF-anomaly scenarios. It also shows that unresolved gaps in high-criticality scenarios can significantly lower the project-level validation status, making JT (JT S) a useful indicator for prioritizing improvement efforts.

Step 1: Identify critical scenarios set Σ and κ Step 2: Instantiate scenario contracts fill in SC(σ) Step 3: Assess bilateral capabilities compute CapS , CapT Step 4: Compute gaps obtain JT Cap, Gap, Adeq

Gap ̸= 0?

Main observations

no

Critical scenarios meet adequacy threshold

yes

Step 5: Bridge design augment BC /BO /BI

Figure 2. The five-step analysis workflow of JTA. Steps 3–5 form an iterative bridge-design loop around capability gaps. semantic attribution, and therefore reveal bottlenecks in both controllability and isolability. If the above criticality settings are used to compute system-

5.4 Representative Capability Gaps and Bridge Design Actions Building on the above results, Table VI presents representative capability gaps and the corresponding A1 design actions. The table shows how bridge-oriented design turns validation difficulty into explicit engineering action. Table VI shows where JTA is most useful. It does not stop at labeling a system “hard to test” with a single score; it turns vague engineering impressions into gap items that are dimension-traceable, object-localizable, and action-executable. For S2, the key improvement is not whether the scenario can be triggered one more time, but whether attribution semantics are made explicit. For S3, the key improvement is not whether more logs are added indiscriminately, but whether anomaly construction becomes controllable and cross-component responsibility boundaries become clearer.

TABLE VI. Representative Capability Gaps and Corresponding Bridge-Level Design Actions Scenario

S2

S2

S3

S3

S3

Gap dimension

Corresponding bridge-level design action

Gap description

The current ModeReason::GCS_FAILSAFE indicates that a GCS failsafe has occurred, but does not provide an attribution view directly linked to heartbeat counts, timeout Isolability I thresholds, and link status, which makes it difficult to distinguish genuine timeout from control error in the test environment A timing gap remains in the evidence chain between “heartbeat interruption” and “mode transition,” so verdicts Observability O still depend heavily on manual inspection of multi-source logs Constructing EKF anomalies depends on simulation perturbations and parameter combinations; perturbation Controllability C strength, duration, and injection timing are difficult to decouple, leading to insufficient execution consistency Existing replay mechanisms are mainly intended for replaying historical logs and make it difficult to superimpose Controllability C new perturbations during replay, which limits counterfactual-scenario validation When ALT HOLD/LAND mode transitions occur, the current cause semantics do not clearly distinguish whether Isolability I the trigger originated from EKF variance beyond threshold, health-status failure, or concurrent GPS quality degradation

5.5 Findings and Discussion The case suggests three points. First, validation difficulty often comes from a mismatch in joint capability: many problems arise from misalignment among scenario requirements, testsystem capability, and exposed system hooks, rather than from flight-control code complexity alone. Second, the RC, GCS, and EKF scenarios exhibit different bottleneck structures, which supports treating testability as a scenario-conditioned property rather than a single system-level attribute. Third, existing SITL, AutoTest, and logging infrastructure already provide a substantial starting point; JTA helps reorganize those assets so that they move from merely “being able to run” toward “being able to adjudicate, attribute, and refine.” 6. D ISCUSSION 6.1 Novelty and Research Positioning The main novelty lies in three shifts. First, the object of testability analysis moves from an isolated software artifact to the triplet JT Sσ composed of the scenario, the test system, and the system under test. Second, a scenario is treated not merely as a descriptive item in a test specification, but as a minimal design unit that carries objectives, evidence requirements, verdict rules, a fault-source set, and minimum thresholds through the scenario contract. Third, architectural action becomes an explicit outcome of analysis: capability gaps are mapped directly to design actions on the control bridge, evidence bridge, and attribution bridge. The novelty is therefore not the isolated idea that fault sources can be traced, which also appears in safety analysis and MBSE practice; it is the use of fault-source obligations inside a scenarioconditioned joint adequacy review that covers the scenario, the test system, and the SUT together.

Build a triadic attribution view of “timeout event–cause code–link status” in the attribution bridge BI ; add heartbeat-control logs and timestamp alignment on the test-system side Add heartbeat-state snapshots and event alignment in the evidence bridge BO to structure timeout events, mode transitions, and textual alerts into a coherent timeline Add parameterized perturbation templates and a timing controller to the control bridge BC so that perturbation amplitude, duration, and injection timing can be configured independently Add an injection-enabled replay adapter to the control bridge BC so that real flight data can be combined with controlled perturbations Add multi-trigger mapping in the attribution bridge BI and record EKF variance, health bits, GPS quality scores, and mode reasons jointly as structured cause tuples

6.2 Implications for Safety-Critical Software Engineering JTA treats dependability testing as an architectural issue rather than a purely execution-level one. In practical terms, validation capability should not be checked only after functional implementation is complete; it should be organized proactively through scenario contracts during design. For safety-critical software, this shift matters in at least four ways. First, it upgrades the test system from an auxiliary collection of scripts to a formal design object. Second, it turns evidence organization and responsibility boundaries into first-class design concerns. Third, it provides a finer-grained common language for assurance-case engineering, shift-left validation, and crossteam collaboration [13], [14]. Fourth, it complements the lifecycle-oriented view of verification and validation required by IEEE Std 1012 [8]. 6.3 Limitations and Threats to Validity This study also has clear boundaries. First, it is a method-andarchitecture study. The purpose of the case is to demonstrate how JTA organizes analysis and design, not to report performance gains or certification benefits after all bridge actions are carried out. Second, the H/M/L ordinal assessment adopted here is suitable for illustrative analysis under the current material conditions, but it still involves engineering judgment. Future work may refine it into automatically extractable numerical indicators and more rigorous review rubrics. Third, the case comes from the open ArduPilot ecosystem, whose openness and testing-infrastructure maturity are project-specific. The applicability of JTA in closed industrial systems, especially under restricted control interfaces, restricted logging, or restricted evidence disclosure, still requires further case-based validation. The main validity risks of the present study fall into four categories.

Construct validity: the current H/M/L ordinal assessment is still an illustrative discretization of the numerical model in Section 3. In future work, indicators such as controlinterface completeness, log coverage, and cause-code coverage should be further automated. • Internal validity: the assessment process inevitably contains researcher judgment, especially in defining scenario boundaries and minimum evidence sets. • External validity: the openness and testing-infrastructure maturity of ArduPilot are higher than those of many industrial projects; the conclusions therefore cannot be generalized directly to all safety-critical software. • Method boundary: JTA is intended as a meso-level architectural analysis and design method. It does not replace formal verification, certification processes, or concrete testing frameworks; rather, it provides a unified perspective for organizing validation capability across them. •

7. C ONCLUSION Joint Testability Architecture (JTA) reframes testability in safety-critical software as a scenario-conditioned system capability carried jointly by the scenario, the test system, and the system under test. The paper defines the joint testability system, scenario contracts, joint capability, scenario adequacy, validation blind spots, and the system-level joint testability metric JT (JT S), and uses the “three domains, three bridges, and a closed loop” formulation to map capability gaps to concrete design actions on the control bridge, evidence bridge, and attribution bridge. The illustrative analysis of ArduPilot failsafe validation shows that the main bottlenecks of link-related scenarios lie in evidence structuring and the explicitness of cause semantics, whereas the main bottlenecks of EKF anomaly scenarios lie in the controllability of anomaly construction and the consistency of cross-component attribution. This result indicates that “difficulty of testing” in safety-critical software is not merely a code-level issue, but is caused by joint mismatches among scenario requirements, testing infrastructure, and exposed system hooks. The broader implication is that, in safety-critical software, validation capability should be designed, not merely checked after the fact. That means bringing the test system into architectural decisions, treating the scenario as the organizing unit of validation, and evaluating testability as a joint system capability. Future work will refine capability indicators in additional industrial cases, validate the benefits of implementing bridge-level actions, and explore deeper integration with UTP, assurance cases, and model-driven engineering. R EFERENCES [1] J. M. Voas and K. W. Miller, “Software testability: The new verification,” IEEE Software, vol. 12, no. 3, pp. 17– 28, 1995. [2] V. Garousi, A. M. M. Safi, A. Hacaloglu, and B. Felderer, “A survey on software testability,” Information and Software Technology, vol. 108, pp. 35–64, 2019. [3] M. Zakeri-Nasrabadi, S. Parsa, and S. Jafari, “Mea-

suring and improving software testability at the design level,” Information and Software Technology, vol. 174, p. 107511, 2024. [4] F. Bagheri-Galle, S. Parsa, and M. Zakeri, “A systematic literature review on transformation for testability techniques in software systems,” Information and Software Technology, vol. 185, p. 107788, 2025. [5] J. Ramirez et al., “Application of design for testability (DFT) strategies based on a model-based systems engineering (MBSE) approach in the development of highly complex safety-critical systems,” in Proc. IEEE International Symposium on Systems Engineering (ISSE), 2024, doi: 10.1109/ISSE63315.2024.10741151. [6] ISO, “ISO/IEC 25010:2023 Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Product quality model,” 2023. [7] ISO/IEC/IEEE, “ISO/IEC/IEEE 29119-1:2022 Software and Systems Engineering – Software Testing – Part 1: General Concepts,” 2022. [8] IEEE, “IEEE Std 1012-2016: Standard for System, Software, and Hardware Verification and Validation,” IEEE, New York, NY, 2017. [9] J. Arlat et al., “Fault injection for dependability validation: A methodology and some applications,” IEEE Transactions on Software Engineering, vol. 16, no. 2, pp. 166–182, 1990. [10] M. C. Hsueh, T. K. Tsai, and R. K. Iyer, “Fault injection techniques and tools,” IEEE Computer, vol. 30, no. 4, pp. 75–82, 1997. [11] N. G. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety. Cambridge, MA, USA: MIT Press, 2012. [12] N. Leveson and J. Thomas, STPA Handbook. Cambridge, MA, USA: MIT, 2018. [Online]. Available: https://psas.scripts.mit.edu/home/get file.php?name= STPA Handbook.pdf [13] J. Cederbladh, A. Cicchetti, and J. Suryadevara, “Early validation and verification of system behaviour in modelbased systems engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, Art. no. 81, pp. 1–67, 2024. [14] R. Wei et al., “ACCESS: Assurance case centric engineering of safety-critical systems,” Journal of Systems and Software, vol. 213, Art. no. 112034, 2024. [15] A. Bertolino, “Software testing research and practice,” in Abstract State Machines 2003: Advances in Theory and Practice, E. Börger, A. Gargantini, and E. Riccobene, Eds., vol. 2589 of Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, 2003, pp. 1–21. [16] Object Management Group, “UML Testing Profile 2 (UTP 2), Version 2.2,” formal/24-06-05, 2024. [Online]. Available: https://www.omg.org/spec/UTP2/2. 2/About-UTP2/ [17] B. Meyer, “Applying ‘design by contract’,” IEEE Computer, vol. 25, no. 10, pp. 40–51, 1992.

[18] W. Xue, Q. Ren, and Y. Wang, “An Empirical Study on Modeling Testing Requirements for Safety-Critical Software,” in Proc. IEEE QRS-C, 2024, pp. 125–134. [19] H. G. Gurbuz, B. Tekinerdogan, C. Catal, and N. P. Er, “Test suite assessment of safety-critical systems using

safety tactics and fault-based mutation testing,” Cluster Computing, vol. 27, no. 4, pp. 5377–5401, 2024. [20] ArduPilot, “ArduPilot/ardupilot,” GitHub repository. [Online]. Available: https://github.com/ArduPilot/ ardupilot

Related documents

• MAGS: Multi-agent Auto-formalization Guarantees Safety for Agentic Outputs#1013986
• MAGS: Multi-agent Auto-formalization Guarantees Safety for Agentic Outputs#978375
• Executable World Models for ARC-AGI-3 in the Era of Coding Agents#158542
• No Red Lines: The Impossibility of Formal Safety Guarantees in Large Language Models V2.0#80379
• Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety#1077870