A Fully Automated, Deployment-Aware Testing Pipeline for IoT-Based Automotive Applications

A Fully Automated, Deployment-Aware Testing Pipeline for IoT-Based Automotive Applications

arXiv:2608.19752v1 [cs.SE] 20 Aug 2026

Denesa Zyberaj1,2 [0009-0009-4207-8723] , Roman Vintonyak1,2 [0009-0008-1361-2205] , Pascal Hirmer1 [0000-0002-2656-0095] , and Marco Aiello2 [0000-0002-0764-2124] Mercedes-Benz AG, Bela-Barenyi-Straße, 71059 Sindelfingen, Germany {denesa.zyberaj, roman.vintonyak, pascal.hirmer}@mercedes-benz.com 2 Institute for Architecture of Application Systems, University of Stuttgart, Universitätsstraße 38, 70569 Stuttgart, Germany marco.aiello@iaas.uni-stuttgart.de 1

Abstract. Testing embedded software in modern vehicles is challenging due to system complexity, decentralized architectures, and strict safety and performance constraints. In this work, we present an end-to-end, deployment-aware testing pipeline for IoT-based automotive applications. The pipeline combines requirement-driven test and code generation with large language model (LLM) and vision-language model (VLM) assistance, and human-in-the-loop curation to reduce manual effort and improve consistency. Using Eclipse openDuT, it supports flexible, distributed deployment across geographically separated cyber-physical and IoT infrastructures, optimizing for node availability and cross-organizational coordination. For validation, we conduct a case study using a Child Presence Detection System (CPDS), achieving full functional requirement coverage across all 9 requirements and 100% Gherkin generation accuracy on the controlled requirement set. Distributed test execution across geographically separated ECUs via Eclipse openDuT confirms the pipeline’s applicability to OEM–supplier testing workflows. Keywords: Test Automation · Continuous Integration · Cyber-Physical Systems · Automotive Testing · Decentralized Testing · IoT Testing

1

Introduction

The automotive industry is rapidly evolving toward software-defined, sensor-rich vehicles embedded within complex Internet of Things (IoT) ecosystems. The vehicles rely on distributed Electronic Control Units (ECUs) connected through diverse network topologies and are responsible for coordinating safety-critical functionalities [38,28]. These ECUs are provisioned globally and require early testing while still at their production facilities, effectively being organized as a distributed cyber-physical architecture. Such architecture demands robust, scalable, and automated testing processes to address geographic dispersion,

2

Zyberaj et al.

heterogeneity, and strict real-time constraints [4,5]. As vehicle functionality increasingly relies on interconnected ECUs and over-the-air software updates, testing has become a significant cost driver in automotive development [28,27].

Fig. 1. Global distribution of automotive testing components. Original Equipment Manufacturer (OEM) collaborates with geographically distributed suppliers providing ECUs, sensors, and software components.

Ensuring correctness, safety, and performance across these decentralized systems is particularly challenging, especially given the organizational fragmentation in the automotive supply chain. Hardware and software components are often developed independently by globally distributed suppliers and integrated late at the Original Equipment Manufacturer’s (OEM) site (see Figure 1). This asynchronous development process limits traceability and hinders test coordination. Validation is typically conducted by suppliers or external test labs, often with manually tracked test plans and inconsistent documentation. As a result, correlating test results across systems is difficult, valuable data remains underutilized, and comprehensive end-to-end validation is often postponed until all physical components are delivered. This delay increases iteration costs and prevents early-stage integration testing [19,33,34,38]. Traditional industry standards, such as ISO 26262 [22], AUTOSAR [2], and ASPICE [1], provide useful guidance but do not fully support automated, infrastructure-aware testing for IoT-based architectures. Manual and semi-automated workflows still dominate, slowing feedback and integration into CI/CD pipelines.

Deployment-Aware Testing Pipeline for IoT Automotive Applications

3

In this paper, we introduce a fully automated testing pipeline for IoT-based automotive applications that generates Gherkin [9] scenarios and executable test scripts from natural-language requirements and UML diagrams and links each scenario to its source requirement, using a large language model (LLM) and visionlanguage model (VLM) assistance with human-in-the-loop curation. The pipeline performs deployment-aware orchestration based on lightweight network profiling. It provides distributed execution via Eclipse openDuT [14], emitting machinereadable results and Universally Unique Identifier (UUID)-based identifiers for end-to-end traceability and reproducibility. We implement a distributed Child Presence Detection System (CPDS) and demonstrate that it reduces manual authoring and deployment effort while enabling higher coverage across multiple nodes. The remainder of the paper is structured as follows: Section 2 provides background knowledge on testing of software and ECUs in the automotive industry. Section 3 illustrates our proposed approach and implementation. Section 4 discusses results and evaluation. Section 5 reviews related works. Finally, Section 6 summarizes our findings and outlines future research directions.

2

Software and ECU Testing in the Automotive Industry

Continuous integration (CI) enables developers to integrate changes frequently and validate them through automated builds and tests, reducing integration risk and surfacing defects early [13]. Testing is central to CI/CD pipelines and provides continuous feedback on software quality and readiness for deployment. Automated tests at unit, component (integration), and acceptance levels are triggered by application, configuration, or environment changes. Unit tests give rapid feedback on isolated code. Component tests exercise interactions with databases, file systems, or external services. Acceptance tests validate the fulfillment of requirements and are often written in a behavior-driven development style using human-readable, executable specifications, for example, with Cucumber [8] or JBehave [24]. This alignment helps keep requirements, tests, and implementation synchronized and supports automatic regeneration of documentation with each build. Manual and exploratory testing remain important for usability checks, visual consistency, and open-ended scenarios that are hard to automate. Continuous delivery (CD) automates bringing software changes to a productionready state, while continuous deployment also automates the release step. A deployment pipeline moves each change through build, test, and deploy stages and coordinates developers, testers, and operations [21]. A well-designed pipeline validates every change, improves release predictability, and increases transparency across the process. Standard CI/CD services (for example, Jenkins or GitLab CI) facilitate automated execution and management of these pipelines. Adopting CI/CD and automated analysis in automotive contexts correlates with reduced defect rates, improved quality, and faster development cycles [10,6]. Dakić et al. [10] report lower project costs and higher return on investment when these practices are integrated into automotive workflows. Beyond economic

4

Zyberaj et al.

Fig. 2. Overview of the simplified Child Presence Detection System (CPDS). Adapted from [39].

outcomes, research explores mechanisms for maintaining high coverage and intelligent test selection within CI/CD. For example, Fehlmann and Kranich [15] propose a framework based on combinatory algebra and recursive arrow terms to recombine tests and map them to user stories via a coverage matrix, enabling relevance filtering and metric-driven suite management such as convergence gap and SVM-based optimization, which is relevant for Advanced Driver Assistance Systems (ADAS) systems with large operational spaces.

3

Distributed Testing Pipeline

The automated testing of automotive software systems is organized here as an executable workflow. We implemented the pipeline as a working prototype and evaluated it on the CPDS use case, which reflects timing-sensitive, multi-sensor, distributed testing in an automotive IoT setting. 3.1

Design of the Pipeline

The pipeline consists of requirements definition, test case generation, deployment, execution, reporting, and undeployment. Figure 3 shows the conceptual flow and the concrete tools used in the prototype, adapted from our prior work [39]. The goal is fully automated, distributed testing across ECUs with end-to-end traceability. Step 1: Requirements definition. Requirements are authored in natural language and, where helpful, with diagrams such as UML activity or sequence diagrams that clarify interactions and timing. Clear requirements are critical in supplier settings where subsystems are developed independently. They form the basis for automated test case generation and later traceability. Step 2: Test case generation. Test cases are automatically derived from naturallanguage requirements with LLM assistance and from UML diagrams with VLM assistance, with human-in-the-loop curation. We generate Gherkin scenarios and link each to its source requirement. A comprehensive evaluation of LLM and VLM models on this task and beyond (Vehicle Signal Specification signal mapping and Python test script generation) is provided in our companion study [40],

Deployment-Aware Testing Pipeline for IoT Automotive Applications

5

which benchmarks both commercial models (GPT-4o-mini, GPT-4.1, GPT-5, Gemini 2.5 Pro) and locally deployable alternatives (Qwen 2.5 VL 72B, Vicuna7B) on the same CPDS case. The results show that a GPT-4 class model with Chain-of-Thought and Few-Shot prompting provides the best trade-off between generation quality and false-positive rate; accordingly, the present pipeline uses a GPT-4 class model for Gherkin generation. Details of the prompting strategy, guardrails, and iteration process are described in [40]. The resulting .feature files can embed deployment metadata, timing constraints, and scenario logic that guide script generation for distributed ECU testing. Step 3: Deployment. Before deployment, the pipeline profiles device connectivity to account for heterogeneous networks. It gathers round-trip time, deviation, and packet loss and aggregates them into a normalized connection_strength score used to rank devices for placement and filtering. Deployment then uses scripts derived from the .feature files to configure sensors, start components, and set up logging. Orchestration is performed with Ansible [35] using an auto-generated inventory. Each deployment, execution, and undeployment task is tagged with a UUID to maintain traceability across CI/CD runs. Distributed communication across ECUs is provided by Eclipse openDuT [14], which offers secure connectivity and a hardware abstraction layer. In our setup, connectivity can be backed by systems such as NetBird [30] or WireGuard [11]. This enables realistic OEM–supplier communication topologies and reduces integration overhead prior to in-vehicle tests. Step 4: Execution. A central controller dispatches test instructions derived from the scenarios and coordinates simultaneous execution across ECUs. Tests are represented as JavaScript Object Notation (JSON) objects that define step logic, expected outcomes, and time constraints. The controller exposes a simple Representational State Transfer (REST) interface for triggering and monitoring runs, while lightweight gRPC messaging supports inter-ECU events for timely sensor–actuator signaling. Progress is streamed during execution and consolidated afterward into machine-readable summaries that support downstream analysis. Step 5: Reporting. Results are aggregated into structured JSON with per-step outcomes, traces, and timings. Each item is mapped back to its Gherkin step and originating requirement to enable coverage analysis and faster root-cause investigation. A traceability matrix summarizes which scenarios validate which functional requirements. Step 6: Undeployment. Post-test cleanup mirrors deployment. Inverse Ansible playbooks remove software components and temporary artifacts and reset devices to a clean state so subsequent runs start from a known baseline. Table 1 provides an overview of all tools and components used in the pipeline, their respective pipeline steps, and the specific roles they play in the workflow. 3.2

CPDS Use Case Implementation

The Child Presence Detection System (CPDS) use case demonstrates the pipeline on a safety-critical scenario aligned with Euro NCAP guidance and related patents [29,36,7]. Direct sensors, such as cabin camera, Ultra-Wideband (UWB)

6

Zyberaj et al.

Table 1. Tools and components used in the pipeline Tool

Pipeline Step

Role

GPT-4 class LLM [40]

Test case generation

Generates Gherkin scenarios and executable Python test scripts from natural-language requirements Extracts signals and behavioral logic from UML diagrams to enrich scenario generation

VLM (e.g., Test case Gemini 2.5 Pro, generation Qwen 2.5 VL) [40] Gherkin [9] Test case definition Eclipse Deployment, openDuT [14] Execution, Management Ansible [35] Deployment, Undeployment NetBird / Deployment WireGuard [30,11] gRPC [18] Execution REST

Execution

Python (unittest) JSON

Execution

MongoDB

Reporting

GitHub Actions [17]

CI/CD orchestration

Reporting

Human-readable BDD scenario format linking requirements to executable test logic Provides secure connectivity and hardware abstraction across distributed ECUs Orchestrates automated deployment and teardown of software components via auto-generated inventories Establishes encrypted overlay networks between ECUs to simulate OEM–supplier topologies Supports low-latency inter-ECU messaging for timely sensor–actuator signaling Exposes the central controller interface for triggering and monitoring test runs Runtime for generated test scripts; maps directly from Gherkin steps to test methods Universal artifact format for test definitions, execution results, and traceability data Stores test results, connectivity profiles, and pipeline metadata during prototype runs Triggers and manages the automated pipeline on self-hosted runners

Deployment-Aware Testing Pipeline for IoT Automotive Applications

7

Fig. 3. Overview of the distributed testing pipeline prototype, with main steps, supporting tools, and ECU communication via openDuT.

radar, and microphone, detect presence, while indirect sensors, such as seat pressure and belt sensors, provide corroboration. Detected events trigger escalating interventions up to emergency notification, as illustrated in Figure 2. The CPDS prototype was implemented as a functional end-to-end demonstrator covering the full pipeline from requirements ingestion to post-test undeployment. Three ECUs were instantiated as software processes running on virtual nodes, each hosting a dedicated Python service that simulates its hardware component (door control, Heating, Ventilation, and Air Conditioning (HVAC), and infotainment). Sensor inputs were emulated via software-injected signals rather than physical hardware. The implementation was sufficient to validate all nine functional requirements and to exercise all six pipeline stages. It represents an early-integration prototype targeting OEM–supplier testing workflows rather than a production-grade deployment. We defined functional and technical requirements in Tables 2 and 3. The CPDS escalation logic is visualized in Figure 2. The combination of multi-sensor inputs, tight timing, and distributed orchestration makes CPDS a representative stress test for the pipeline. We used a GPT-4 class model to generate Gherkin scenarios from the requirements and then produced executable Python test scripts. Listing 1.1 shows a scenario for emergency notification, and Listing 1.2 illustrates an executable excerpt. 1 Scenario: Child detected, no driver reaction, emergency services ,→ alerted 2 Given the vehicle ignition is switched off 3 And direct and indirect sensors detect a child alone in the vehicle 4 When the system proceeds through all escalation steps 5 And there is no reaction from the driver 6 Then the vehicle system calls emergency services 7 And allows emergency access to the vehicle Listing 1.1. Exemplary Gherkin test case for CPDS.



8

Zyberaj et al. Table 2. Functional requirements for the CPDS Req. ID

Requirement description

FR1

The CPDS shall detect an unattended child’s presence using direct and indirect sensors. The CPDS shall notify the driver of the forgotten child in the vehicle. The CPDS shall initiate an initial warning upon detection if the driver does not react. The CPDS shall escalate warnings using exterior lights and the horn if the initial warning fails. The CPDS shall activate the HVAC to regulate interior conditions if previous warnings fail. The CPDS shall notify all registered carers if the driver does not react. The CPDS shall unlock vehicle doors to allow emergency access if prior interventions are unsuccessful. The CPDS shall autonomously contact emergency services as a final safety measure. The CPDS shall remain in standby if no unattended child is detected.

FR2 FR3 FR4 FR5 FR6 FR7 FR8 FR9

Table 3. Technical requirements for the CPDS Req. ID

Requirement description

TR1 TR2 TR3 TR4 TR5

Use camera, UWB radar, and microphone for direct detection. Use seat sensors and door or window sensors for indirect detection. Support gRPC for inter-ECU communication. Allow real-time monitoring and status updates during escalation. Enable automated generation and execution of deployment scripts.

 1 def test_emergency_services_alerted_when_no_driver_reaction(self): 2 self.vehicle_system.initiate_initial_warning() 3 self.assertFalse(self.vehicle_system.driver_reacted()) 4 self.vehicle_system.escalate_warning() 5 self.assertFalse(self.vehicle_system.driver_reacted()) 6 emergency_called = self.vehicle_system.call_emergency_services() 7 self.assertTrue(emergency_called) Listing 1.2. Executable test script excerpt for CPDS escalation.

Three ECUs were distributed across virtual or physical nodes interconnected through openDuT, simulating OEM–supplier arrangements. The devices and

Deployment-Aware Testing Pipeline for IoT Automotive Applications

9

roles are listed in Table 4. Connectivity profiling produced a normalized connection_strength score to guide placement, as shown in Listing 1.3. Table 4. Devices including IP address and core functionality Device name

IPv4 address

Software component

Functionality

ecu_1 ecu_2 ecu_3

192.168.0.101 192.168.0.102 192.168.0.103

door_ecu.py ac_ecu.py infotainment_ecu.py

Doors Air conditioning Notifications and infotainment

 1 { 2 "availability": 100, 3 "availability_status": "available", 4 "average_rtt": 0.06, 5 "connection_strength": 0.999, 6 "device_type": "heating_ventilation_and_airconditioning_ecu", 7 "device_type_short": "HVAC", 8 "hardware_version": "v1.0", 9 "ip": "192.168.1.102", 10 "max_rtt": 0.084, 11 "mdev": 0.023, 12 "min_rtt": 0.037, 13 "name": "ecu_2", 14 "normalized_packet_loss": 0.0, 15 "sensors": [ 16 {"status": "available", "type": "temperature"}, 17 {"status": "available", "type": "status"} 18 ], 19 "software_packages": [], 20 "status": "online" 21 } Listing 1.3. ECU connectivity profiling result.

The CPDS scenarios are expressed as JSON to drive execution and to enable machine-readable reporting. A single representation is used throughout, ensuring consistent log parsing and traceability. A central controller executes scenarios across ECUs, streams progress, and consolidates results. Execution results are stored as JSON with per-signal checks and per-step outcomes.

10

Zyberaj et al.

 1 { 2 "tests": [{ 3 "id": 1, 4 "name": "Test car is locked", 5 "teststeps": [1, 2, 3, 4], 6 "logic_blocks": [{ 7 "id": 1, 8 "logic_type": "TIME_CONSTRAINT", 9 "condition_test_id": 1, 10 "time_constraint_s": 3, 11 "next_test_true": null, 12 "next_test_false": null, 13 "description": "Time constraint: 3 seconds passed" 14 }] 15 }] 16 } Listing 1.4. Test JSON file for CPDS.

Table 5 summarizes requirement coverage through a traceability matrix that maps scenarios to functional requirements. This supports faster root-cause analysis and requirements-driven regression selection.  1 { 2 "test_id": "1", 3 "status": "completed", 4 "result": { 5 "status": "passed", 6 "teststeps": [ 7 {"door_fl": {"expected_value": "0", "observed_value": "0", ,→ "status": "passed"}}, 8 {"door_fr": {"expected_value": "0", "observed_value": "0", ,→ "status": "passed"}}, 9 {"door_bl": {"expected_value": "0", "observed_value": "0", ,→ "status": "passed"}}, 10 {"door_br": {"expected_value": "0", "observed_value": "0", ,→ "status": "passed"}} 11 ] 12 } 13 } Listing 1.5. Successful test result JSON for scenario “Test car is locked”.

Post-test resource management uses inverse Ansible playbooks to remove components, reset configurations, and restore ECUs to a clean baseline so subsequent runs remain reproducible.

Deployment-Aware Testing Pipeline for IoT Automotive Applications

11

Table 5. Traceability Matrix: Mapping Test Scenarios to Functional Requirements Test Scenario

FR1 FR2 FR3 FR4 FR5 FR6 FR7 FR8 FR9

Verification of unattended child detection Child detected, driver reacts to initial warning Child detected, driver reacts after escalation warning Child detected, driver reacts after climate control intervention Child detected, driver reacts after notifying carers Child detected, no driver reaction, emergency services alerted No child detected

✓

4

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓ ✓

Results and Discussion

We evaluated the pipeline on the CPDS use case with three ECUs distributed across separate nodes. The evaluation covers three dimensions: test case generation quality, requirement coverage, and deployment feasibility. 4.1

Test Case Generation Quality

In the present pipeline, the 9 functional and 5 technical CPDS requirements (Tables 2 and 3) were deliberately scoped and phrased to avoid ambiguity, e.g., by avoiding compound OR conditions and ensuring each requirement maps to a single, observable behavior. Within this controlled set, the GPT-4 class model produced executable Gherkin scenarios for all 14 requirements without any manual correction, achieving 100% generation accuracy. To contextualize this result against a more realistic, industry-scale requirement set, we draw on the companion evaluation in [40], which applies the same generation approach to the full CPDS escalation logic expressed as 36 fine-grained natural-language requirements. The CPDS requirement set in [40] expands the escalation logic into 36 finegrained requirements; the generation quality figures cited here therefore reflect the requirement granularity used in that study and are directionally applicable to the coarser functional requirements used in the present pipeline. Of 36 naturallanguage CPDS requirements, 32 (89%) were automatically transformed into executable Gherkin scenarios without modification. The remaining four required targeted corrections, primarily for requirements containing underspecified OR conditions or ambiguous edge cases [40]. Human review time averaged 1–5 minutes per requirement for Gherkin scenarios and 5–20 minutes per scenario for code adaptation, depending on complexity. These figures assume a reviewer with

12

Zyberaj et al.

basic familiarity with the CPDS domain and practical test automation skills in Python [40]. 4.2

Requirement Coverage

The traceability matrix in Table 5 maps all seven test scenarios to their corresponding functional requirements. All nine functional requirements (FR1–FR9) are covered by at least one scenario, yielding full functional requirement coverage in the automated pipeline. Each requirement is traceable to its originating scenario via UUID-tagged JSON artifacts, supporting root-cause analysis and requirements-driven regression selection. Structured JSON-based test artifacts explicitly linked to requirements improve traceability and documentation consistency, overcoming common limitations of manual documentation. 4.3

Deployment Feasibility

The successful implementation of the CPDS use case demonstrated the pipeline’s practical applicability. Connectivity profiling produced a normalized connectivity score for each ECU prior to deployment; for instance, ecu_2 achieved a connection_strength of 0.999, an average round-trip time of 0.06 ms, and zero packet loss (Listing 1.3), enabling automated ranking and filtering of candidate nodes without manual configuration. Automated deployment via Ansible, test execution across all three ECUs, result aggregation, and post-test undeployment were all completed successfully, validating the core functionality and confirming the operational feasibility of the approach. By using Eclipse openDuT, the pipeline scales effectively across geographically distributed, heterogeneous ECU environments, simplifying the complexity of decentralized systems. Leveraging industry standards such as Ansible, REST, and gRPC enables the pipeline to integrate smoothly into existing automotive testing infrastructures, minimizing retooling effort. 4.4

Limitations

Despite these results, several limitations arose during the implementation. The CI/CD pipeline implemented via GitHub Actions tightly couples the automation framework to GitHub, making it difficult to migrate to other version control systems or to pipeline tools such as Jenkins or GitLab CI. The self-hosted automation runners were developed explicitly for Windows environments, posing compatibility challenges with Linux systems. Additionally, the current implementation executes deployment, testing, and undeployment cycles sequentially, resulting in inefficient resource utilization and unnecessary wait times for developers. Scalability concerns were also noted regarding database performance and network constraints when expanding to larger ECU networks. MongoDB, used for prototyping, may encounter performance bottlenecks due to frequent concurrent access in large-scale deployments. Future improvements should focus on parallel execution capabilities, smarter scheduling algorithms informed

Deployment-Aware Testing Pipeline for IoT Automotive Applications

13

by connection_strength and data dependencies, efficient resource management, enhanced network fault tolerance, and migration to time-series databases better suited for high-frequency IoT telemetry. The pipeline is currently aligned with functional safety verification as defined under ISO 26262. Extending the approach to cover the Safety of the Intended Functionality (SOTIF, ISO 21448) [23], for instance, by generating test scenarios for operational design domain boundary cases, remains a direction for future work. The inclusion of fault-injection scenarios, by simulating sensor failures or network dropout during execution, would further strengthen confidence in the pipeline’s test validity and is planned as a future extension. To improve platform independence, replacing the current Windows-based self-hosted runners with a containerized execution environment (e.g., Docker-based runners) is a near-term engineering goal.

5

Related Work

Several studies address CI implementations tailored to automotive software development. For instance, Nithnin et al. introduce the Contest Tool, an internal application designed to automate the downloading of source files, loading of software onto ECUs, execution of smoke tests, and generation of reports [31]. The tool enables rapid defect detection in ECU software, though it is not publicly available. A more comprehensive and unified CI infrastructure is described by Du et al. [12]. Their proposal incorporates software emulation, fault injection, and Hardware-in-the-Loop (HiL) platforms. By integrating these stages into the CI pipeline, their approach enhances early fault detection and improves the overall reliability of system verification. Obergfell et al. [32] suggest a transformation for automotive software engineering. They propose a dual-pipeline model that integrates architectural-level functionality with source-code-level implementation. Emphasizing agile methods, architectural extensibility, and enhanced supplier collaboration, the authors challenge the rigidity of the traditional V-model development approach. While the paper remains theoretical, it outlines key drivers and process-level shifts critical for OEMs adopting software-centric, continuous development practices. Broadening beyond purely automotive contexts but maintaining relevance to CI/CD practice, Soni [37] presents an end-to-end DevOps automation solution leveraging cloud technologies. The approach incorporates automated application lifecycle management, CI processes using Jenkins, configuration management via Chef, and infrastructure provisioning with VMware or public cloud environments3 , providing a valuable perspective applicable to automotive contexts seeking scalable, cloud-based solutions. Hilpert and Tetz [20] address the scalability challenges of automotive software testing by developing a fully automated prototype using Kubernetes-based cloud infrastructure. Their framework enables the scalable execution of Model-in-theLoop (MiL) and Software-in-the-Loop (SiL) tests, demonstrating how cloud 3

Chef: https://www.chef.io/ and VMWare: https://www.vmware.com/.

14

Zyberaj et al.

resources can be efficiently allocated for large-scale test automation. In the domain of model-based testing, Krejčí and Novák [26] introduce a framework for automated test prioritization in distributed automotive systems. Their approach uses classification techniques to assign relevance scores to test cases, reducing redundant tests and focusing effort on the most critical tests for integration quality. Complementing this, Kim et al. [25] present the MAESTRO framework, which automates test generation from model specifications to achieve high coverage while minimizing manual input. Recent work has also addressed integrating machine learning into automotive CI/CD pipelines. Ferreira et al. [16] describe a Machine Learning Operations (MLOps) process for ML-enhanced automotive applications that automates the full machine learning lifecycle within a CI/CD context. Berglund et al. [3] explore test maintenance strategies for machine learning systems in the automotive industry, identifying key challenges in test stability and reproducibility as models evolve. Compared to the approaches above, the present pipeline is distinguished by the combination of three properties that none of the reviewed works provide jointly: LLM and VLM assistance for test artifact generation starting directly from natural-language requirements and UML diagrams, deployment-aware orchestration driven by live network profiling, and full requirements-to-result traceability across a distributed, IoT-grade automotive ECU topology. Nithnin et al. and Du et al. support distributed ECU execution but rely on manually authored test cases without requirements-driven generation. Hilpert and Tetz achieve scalable multi-ECU execution in the cloud but do not address OEM–supplier geographic distribution, LLM-assisted generation, or end-to-end traceability. MAESTRO automates test generation from requirements, but requires pre-existing formal model specifications as input rather than operating from heterogeneous natural-language and diagram artifacts. Obergfell et al. address requirements-driven pipeline design at an architectural level, but remain theoretical and do not provide a working prototype.

6

Conclusion and Outlook

In this paper, we present a fully automated, deployment-aware testing pipeline for IoT-based automotive applications and validated it on a distributed CPDS case. The pipeline converts natural-language requirements into Gherkin scenarios and executable scripts with LLM and VLM assistance and human-in-the-loop curation, orchestrates distributed execution via Eclipse openDuT, and emits machine-readable results with stable identifiers for end-to-end traceability. In our evaluation, it reduced manual authoring and deployment effort and supported higher coverage across multiple nodes. Within the controlled CPDS requirement set, the pipeline achieved 100% Gherkin generation accuracy; at industry-scale requirement granularity, 89% of scenarios required no manual correction [40]. Distributed execution across three ECUs confirmed the pipeline’s operational feasibility in an OEM-supplier network topology, with automated deployment and undeployment completing successfully across all test runs.

Deployment-Aware Testing Pipeline for IoT Automotive Applications

15

The current prototype has limits. It is realized on GitHub Actions with selfhosted Windows runners, executes scenarios sequentially, and uses a convenience data store that may not scale. Future work focuses on scaling to larger multi-ECU settings, adding parallel execution with scheduling informed by connection_strength and data dependencies, and strengthening fault tolerance under variable network conditions. A systematic comparison of LLMs and VLMs for test generation is provided in our companion study [40]; future work will extend this to larger, multi-OEM requirement sets and diverse ECU configurations. To support reuse and collaboration, we will release the prototype and configuration templates as open source.

References 1. ASPICE: Aspice - automotive software process improvement and capability determination (2024), https://vda-qmc.de/en/automotive-spice/ 2. AUTOSAR: Autosar - automotive open system architecture (2024), https://www. autosar.org/ 3. Berglund, L., Grube, T., Gay, G., de Oliveira Neto, F.G., Platis, D.: Test maintenance for machine learning systems: A case study in the automotive industry. In: 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). pp. 410–421 (2023). https://doi.org/10.1109/ICST57152.2023.00045 4. Broy, M., Krüger, I., Pretschner, A., Salzmann, C.: Engineering automotive software. Proceedings of the IEEE 95, 356–373 (2007). https://doi.org/10.1109/JPROC. 2006.888386 5. Chakraborty, S., Al Faruque, M.A., Chang, W., Goswami, D., Wolf, M., Zhu, Q.: Automotive cyber–physical systems: A tutorial introduction. IEEE Design & Test 33(4), 92–108 (2016). https://doi.org/10.1109/MDAT.2016.2573598 6. Chatterjee, T.: Test lead time and cost improvement of automotive embedded project-a new perspective with automation and ci/cd. In: 2024 International Conference on Vehicular Technology and Transportation Systems (ICVTTS). vol. 1, pp. 1–6. IEEE (2024) 7. Cole, C.J.: System to detect the presence of an unattended child in a vehicle (Jan 2007), united States Patent US7,170,401 B1 8. Cucumber: Cucumber BDD Framework (2026), https://cucumber.io/, accessed: 2026-03-01 9. Cucumber: Gherkin reference (2026), https://cucumber.io/docs/gherkin/, accessed: 2026-03-01 10. Dakić, P., Todorović, V., Vranić, V.: Financial justification for using ci/cd and code analysis for software quality improvement in the automotive industry. In: 2022 IEEE Zooming Innovation in Consumer Technologies Conference (ZINC). pp. 149–154 (2022). https://doi.org/10.1109/ZINC55034.2022.9840702 11. Donenfeld, J.A.: WireGuard: Next Generation Kernel Network Tunnel. In: Proceedings of the Network and Distributed System Security Symposium (NDSS) (2017), https://www.wireguard.com/papers/wireguard.pdf 12. Du, B., Azimi, S., Moramarco, A., Sabena, D., Parisi, F., Sterpone, L.: An automated continuous integration multitest platform for automotive systems. IEEE Systems Journal 16(2), 2495–2506 (2022). https://doi.org/10.1109/JSYST.2021.3069548

16

Zyberaj et al.

13. Duvall, P.M., Matyas, S., Glover, A.: Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley, Upper Saddle River, NJ, 8th printing edn. (2013) 14. Eclipse Foundation: Eclipse openDuT (2026), https://opendut.eclipse.dev, accessed: 2026-03-01 15. Fehlmann, T., Kranich, E.: A framework for automated testing. In: Yilmaz, M., Niemann, J., Clarke, P., Messnarz, R. (eds.) Systems, Software and Services Process Improvement. pp. 275–288. Springer International Publishing, Cham (2020) 16. Ferreira, A., Fernandes, J.: Mlops for developing machine-learning-enhanced automotive applications. IEEE Software PP, 1–8 (01 2024). https://doi.org/10. 1109/MS.2024.3415750 17. GitHub, Inc.: GitHub Actions (2026), https://github.com/features/actions, accessed: 2026-03-01 18. Google LLC: gRPC: A High Performance, Open Source Universal RPC Framework (2026), https://grpc.io, accessed: 2026-03-01 19. Grimm, K.: Software technology in an automotive company - major challenges. 25th International Conference on Software Engineering, 2003. Proceedings. pp. 498–503 (2003). https://doi.org/10.1109/ICSE.2003.1201228 20. Hilpert, T., Tetz, O.: Developing a fully automated prototype for scaling automotive software tests in the cloud. In: 23. Internationales Stuttgarter Symposium. Springer Fachmedien Wiesbaden (2023). https://doi.org/10.1007/978-3-658-41057-9_ 32, https://link.springer.com/chapter/10.1007/978-3-658-41057-9_32 21. Humble, J., Farley, D.: Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. A Martin Fowler Signature Book, Addison-Wesley, Upper Saddle River, NJ (2011) 22. International Organization for Standardization: ISO 26262-1:2018 Road vehicles – Functional safety – Part 1: Vocabulary. International Standard (2018), https: //www.iso.org/standard/68383.html, edition 2, 33 pages 23. International Organization for Standardization: ISO 21448: Road vehicles – Safety of the Intended Functionality (SOTIF). Tech. Rep. ISO 21448:2022, ISO (2022) 24. JBehave: JBehave: BDD for Java (2026), https://jbehave.org/, accessed: 202603-01 25. Kim, Y., Lee, D., Baek, J., Kim, M.: Maestro: Automated test generation framework for high test coverage and reduced human effort in automotive industry. Information and Software Technology 123, 106221 (11 2019). https://doi.org/10.1016/j. infsof.2019.106221 26. Krejčí, L., Novák, J.: Framework and automated prioritization procedure for modelbased testing of automotive distributed systems. Proceedings of the Dateso 2017 Workshop on Databases, Texts Specifications, and Objects, DATESO 2017 pp. 31–42 (2017) 27. Kushwaha, D.S., Misra, A.: Software test effort estimation. ACM SIGSOFT Softw. Eng. Notes 33 (2008). https://doi.org/10.1145/1360602.1361211 28. Lami, G., Biscoglio, I., Falcini, F.: An empirical study on software testing practices in automotive. In: Clarke, P.M., O’Connor, R.V., Rout, T., Dorling, A. (eds.) Software Process Improvement and Capability Determination. pp. 301–315. Springer International Publishing, Cham (2016) 29. NCAP, E.: European new car assessment programme - test and assessment protocol - child presence detection version 1.2 (Mar 2023), https://www.euroncap.com/ media/79888/euro-ncap-cpd-test-and-assessment-protocol-v12.pdf 30. NetBird: NetBird: Open-Source Zero-Configuration VPN (2026), https://netbird. io, accessed: 2026-03-01

Deployment-Aware Testing Pipeline for IoT Automotive Applications

17

31. Nithin, S., S J, H., S, P.: Enhancing the automotive software test environment using continuous integration and validation pipeline. In: 2023 Innovations in Power and Advanced Computing Technologies (i-PACT). pp. 1–6 (2023). https://doi. org/10.1109/i-PACT58649.2023.10434454 32. Obergfell, P., Kugele, S., Segler, C., Knoll, A., Sax, E.: Continuous software engineering of innovative automotive functions: An industrial perspective. In: 2019 IEEE International Conference on Software Architecture Companion (ICSA-C). pp. 127–128 (2019). https://doi.org/10.1109/ICSA-C.2019.00030 33. Pretschner, A., Broy, M., Krüger, I.H., Stauner, T.: Software engineering for automotive systems: A roadmap. In: Briand, L.C., Wolf, A.L. (eds.) International Conference on Software Engineering, ISCE 2007, Workshop on the Future of Software Engineering, FOSE 2007, May 23-25, 2007, Minneapolis, MN, USA. pp. 55– 71 (2007). https://doi.org/10.1145/1253532.1254710, http://doi.acm.org/10. 1145/1253532.1254710 34. Rajkumar, R., Lee, I., Sha, L., Stankovic, J.: Cyber-physical systems: The next computing revolution. In: Design Automation Conference. pp. 731–736 (2010). https://doi.org/10.1145/1837274.1837461 35. Red Hat, Inc.: Ansible: Automation for Everyone (2026), https://www.ansible. com/, accessed: 2026-03-01 36. Solutions, I.S.S.I.S.S.: European patent ep1 669 251 a (Jun 2006), inventor: Orlewski, Pierre 9021, ETTELBRUCK (LU) 37. Soni, M.: End to end automation on cloud with build pipeline: The case for devops in insurance industry, continuous integration, continuous testing, and continuous delivery. In: 2015 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM). pp. 85–89 (2015). https://doi.org/10.1109/CCEM.2015.29 38. Zyberaj, D., Hirmer, P., Aiello, M.: Towards a globally distributed testing network in the automotive industry. In: Proceedings of the 4th European Symposium on Software Engineering. p. 46–51. ESSE ’23, Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3651640.3651646, https: //doi.org/10.1145/3651640.3651646 39. Zyberaj, D., Hirmer, P., Aiello, M.: Using eclipse opendut for distributed automotive testing. In: Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering. EASE ’25, Association for Computing Machinery, New York, NY, USA (2025). https://doi.org/https: //doi.org/10.1145/3756681.3757013, https://doi.org/./ 40. Zyberaj, D., Mazur, L., Hirmer, P., Petrovic, N., Aiello, M., Knoll, A.: Req2road: A genai pipeline for sdv test artifact generation and on-vehicle execution. In: Advanced Information Systems Engineering. Springer, Verona, Italy (2026), cAiSE 2026, to appear

Related documents

• Cluster-Aware Dual-Level Test Specification Generation for Large-Scale Automotive Software Requirements#282862
• System Prompt Architecture: The Operational Engine of the MVE#270349
• Vulnerability Management for Containers across Continuous Integration and Continuous Delivery (CI/CD) Pipelines#621150
• Virtual Data in CMS Production#153459
• M 1.4 - 10 km SSE of Newport Beach, CA#681496