Characterizing Tests in IoT Software: Practices, Challenges and Opportunities

1

Characterizing Tests in IoT Software: Practices, Challenges and Opportunities

arXiv:2606.12592v1 [cs.SE] 10 Jun 2026

Rufeng Chen ∗ , Hengcheng Zhu † , Wuqi Zhang † , Zixu Zhou ∗ , and Lili Wei ∗ ∗ Department of Electrical and Computer Engineering, McGill University, Montreal, Canada † The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong SAR Emails: ∗ {rufeng.chen, zixu.zhou, lili.wei}@mcgill.ca † {hzhuaq, wzhangcb}@connect.ust.hk

Abstract—The Internet of Things (IoT) is experiencing rapid growth. Smart devices are emerging in smart homes and industrial applications, performing mission-critical tasks. Bugs in IoT software can lead to severe consequences. For example, a buggy smart lock can allow unauthorized access to a private property. Testing is a primary practice to expose software bugs and ensure software quality. However, little is known about how IoT software is tested. To bridge this gap, we conducted the first empirical study on test cases in open-source IoT software. Specifically, we evaluated the effectiveness of test cases in IoT software, explored the challenges inherent in testing IoT software, and analyzed the usage of mock objects. Our results indicate that while IoT software often contains a considerable number of tests, their effectiveness remains limited. We identified the primary challenges in testing IoT software as managing complex interactions with various external dependencies, such as other network-reliant IoT components, file systems, operating systems, and databases. We also observed that the use of mock objects in IoT software closely aligns with our identified testing challenges. This alignment demonstrates the potential of mocking as a solution to enhance test coverage and address the complexities of IoT software testing. Index Terms—Internet of Things, Testing, Empirical Study

I. I NTRODUCTION HE Internet of Things (IoT) is emerging—it is estimated that the global IoT market size will grow from $300.3B in 2021 to $650.5B by 2026 [1]. IoT interconnects various smart devices equipped with sensors and actuators to perform mission-critical tasks that demand high reliability, security, and responsiveness in real-world environments. For example, IoT systems can manage smart devices in a household, including light bulbs, heaters, and door locks. Software is the key bridge that connects devices to form the Internet of Things. Bugs in IoT software can significantly compromise the reliability and security of IoT systems and cause unprecedented consequences. For instance, a bug in the software of a smart lock can allow unauthorized access to a private property. Given the importance of IoT software, an increasing number of studies have been conducted to understand the challenges in developing reliable IoT software. These studies identified interesting IoT topics [2], shared development experiences [3], reviewed testing strategies [4], investigated IoT bugs and vulnerabilities [5]–[7], and examined the perspective of developers in IoT platforms [8]. Although these studies offer valuable perspectives on what makes IoT development and testing

T

Corresponding author: Lili Wei (lili.wei@mcgill.ca).

challenging, there remains a lack of empirical knowledge on how IoT software is tested in practice. Little is known about the characteristics and effectiveness of existing test cases: how well they cover the code, and what factors contribute to test gaps. These are the questions this study seeks to answer. The heterogeneous and dynamic interdependencies of IoT systems require thorough testing to ensure reliability and security. Such systems face challenges that are rarely encountered in traditional software operating under more homogeneous and predictable conditions. For example, unlike general software that predominantly relies on HTTP, IoT systems operate using a diverse range of communication protocols, including Zigbee, MQTT, or device-specific ones. This diversity requires specialized testing approaches to accurately simulate and test each protocol’s intricate behaviors and interactions. In addition, IoT software must function in both digital and physical domains, leading to a vast input space and numerous execution scenarios. Finally, the interplay between cloud, edge, and devicelevel code creates a multi-layered execution context, where bugs may emerge only through cross-layer interactions. In practice, such interactions are particularly difficult to test, since IoT systems are often developed as separate components (e.g., device firmware, gateways, and cloud services) maintained in different repositories, and developers frequently test individual software parts without full access to the complete end-toend system. However, the unique challenges of testing in such environments remain largely underexplored by existing research, leaving a gap in enhancing IoT software testing. To bridge this gap, we conducted the first empirical study by investigating the tests in open-source IoT software to answer the following research questions. RQ1 (Test Effectiveness): How effective are existing tests in IoT software? What statement coverage, branch coverage, and mutation scores can be achieved by the existing tests? Motivation: RQ1 can establish a basic understanding of the state of testing in IoT software. By evaluating the coverage metrics, we can estimate how thoroughly IoT systems are being tested and identify specific components or functionalities where tests remain limited. Result: We used static analysis on 824 IoT projects and conducted in-depth coverage and mutation analysis on 37 projects with runnable tests. The analysis of RQ1 revealed that the tests in IoT software achieve an average statement coverage of 65.2%, branch coverage of 53.4%, and mutation score of 39.9%. Such insufficient coverage results suggest the

2

need for a more in-depth exploration to identify the barriers to testing uncovered code (RQ2). RQ2 (Factors Contributing to Uncovered Code): What factors contribute to the presence of uncovered code by IoT software tests? Motivation: RQ2 aims to discover the underlying factors that prevent certain code segments in IoT projects from being adequately tested. Understanding these factors can guide the development of tools and methods to improve the effectiveness of IoT software testing. Result: Over half of the analyzed code segments (51.4%) remain untested primarily due to overlooked interactions with external dependencies, with untested communication scenarios representing the most significant barrier. These scenarios include network operations such as sending requests to external APIs, retrieving data from remote servers, and messaging between IoT devices. This highlights the critical need for new testing strategies tailored to the challenges posed by the excessive external dependencies of IoT software. RQ3 (Mocking in IoT Testing): How are mock objects used in IoT software testing? What are their roles and what dependencies do they simulate? Motivation: IoT systems highly rely on external dependencies, and mock objects are commonly used to simulate dependencies. This question investigates the use of mock objects in IoT software testing and explores the potential to leverage mock objects to address testing challenges for IoT software. Result: The vast majority (76.04%) of the mock objects in the test cases are simulating communication APIs such as sending and handling device interactions and network messages. This is in line with our previous finding that the major hurdle to improving code coverage in IoT software is untested communication scenarios. The prevalent use of mock objects emphasizes their potential application in enhancing test coverage and addressing the challenges of IoT software testing. Meanwhile, we observe that 99.89% of the analyzed mock objects involve defining or verifying specific dependency behaviors rather than simply acting as placeholders, making it challenging to effectively generate meaningful mock objects and enhance IoT software testing. Collectively, these research questions outline the status quo of IoT testing practices, identify gaps and challenges, and suggest future research directions. Our key observations suggest a promising future research direction to migrate mock objects between IoT projects by reusing the large number of readily available mock objects in open-source IoT projects. To summarize, this paper makes the following contributions: • We conducted the first empirical study focusing on characterizing the testing practices in open-source IoT software. Our study examines how developers test IoT systems and how these practices differ from those of traditional software. • We evaluated the prevalence and effectiveness of the existing tests in open-source IoT software. Our analysis shows that while IoT projects frequently include test cases, these tests often fall short in effectively detecting faults, highlighting the need for improved test adequacy. • We conducted an in-depth analysis to summarize the key challenges in improving the code coverage for testing in IoT

Device

Edge

Cloud

App

Fig. 1. The Overview of IoT Layers

software and identified the key challenge as how to properly trigger the diverse communication scenarios between IoT software and its dependencies. • We analyzed the usage of mock objects in the test cases and proposed a promising future research direction to improve IoT software testing by migrating mock objects. Our dataset, per-project metadata, and all results for reproducing the coverage and mutation analyses are publicly available in our replication package on GitHub [9]. II. BACKGROUND Internet of Things. The Internet of Things refers to a network of interconnected physical devices that collect, exchange, and process data through the internet. These devices range from consumer products, such as smart home appliances, to industrial systems, such as manufacturing sensors. By integrating sensing, computation, and communication capabilities, IoT enables real-time monitoring, automation, and data-driven decision-making across diverse domains. IoT Software. IoT software refers to software across the device, edge (gateway), cloud, and application layers in IoT systems whose primary responsibility is to manage, coordinate, or encapsulate device communication and protocollevel interactions between physical IoT devices and networked services. IoT software may overlap with embedded firmware or traditional cloud applications. However, IoT software is distinguished by its device-facing responsibilities, such as handling protocol-specific communication, managing device identity and state, and coordinating interactions across heterogeneous physical devices and services. IoT software is commonly described using a layered architecture [5], [10] consisting of application, cloud, edge, and device layers, as illustrated in Figure 1. In the literature, these layers are sometimes referred to using alternative terminology, e.g., the edge layer is also described as the network or connectivity layer, while the cloud layer is often described as the service layer or platform layer [11], [12]. Unlike traditional software, functionality in IoT systems often spans multiple layers and relies on device communication, external services, and runtime configuration. First, IoT software must handle device-specific protocols and complex state transitions across layers, making it difficult for test suites to exercise various execution scenarios and achieve strong coverage or faultdetection effectiveness. This directly motivates our study of test effectiveness and uncovered code in IoT projects (RQ1 and RQ2). Second, IoT software frequently relies on external parties such as MQTT brokers, cloud APIs, gateways, or physical devices to fulfill its functionality. As these dependencies are often unavailable during testing, developers commonly require

3

isolation and emulation mechanisms (e.g., mocks) to enable reproducible execution. In IoT systems, mocks are used to simulate interactions with external components, such as devices, network services, and data sources, enabling controlled and reproducible testing. This motivates our investigation of mocking practices in IoT software testing (RQ3). Application Layer: Provides user services, such as mobile apps, dashboards, and user interfaces. Unlike conventional applications, IoT applications often handle heterogeneous devices, real-time monitoring, and continuous connectivity. In this layer, mocks are used to simulate external data sources or service responses so that application logic can be tested without relying on live systems. • Cloud Layer: Manages large-scale data storage, processing, and analytics. It hosts APIs, machine learning models, and integration with other systems. IoT clouds are distinct in that they need to ingest massive streams of sensor data, ensure low-latency responses for control operations, and integrate with domain-specific protocols and devices. Here, mocks are used to emulate incoming data streams or external service interactions to enable scalable and repeatable testing. • Edge Layer: Bridges devices and cloud by using lightweight protocols (e.g., MQTT, CoAP) to filter data and enable low-latency responses under limited connectivity. In this layer, mocks are used to simulate device communications or network conditions to support testing of data processing and protocol handling. • Device Layer: Consists of hardware such as sensors, actuators, and embedded controllers that collect data and execute commands. IoT devices must interact with the physical world in real time. At this layer, mocks are used to emulate device inputs or outputs when physical hardware is unavailable or impractical to use during testing. •

Our study does not exclude any of these layers. We analyze the testing artifacts available in open-source IoT repositories as they appear in practice. Since IoT systems are often decomposed into separately maintained components, individual repositories may implement only a subset of the architecture. We therefore report the layer distribution of the studied repositories in Section III to clarify the scope of our analysis. At the device layer, IoT testing may involve Software-inthe-Loop (SiL), Hardware-in-the-Loop (HiL), or Model-inthe-Loop (MiL) techniques. In the open-source IoT repositories analyzed in this study, testing practices primarily adopt Software-in-the-Loop approaches, such as protocol-level simulation and virtual device responses, and we did not observe Model-in-the-Loop infrastructure. Accordingly, our empirical results primarily reflect software-level testing. Prior secondary studies have examined software testing challenges at different IoT layers. Garousi et al. provide a comprehensive survey of embedded software testing techniques, including device-level validation and simulation-based testing [13]. More recent surveys analyze testing challenges in distributed IoT systems spanning device, edge, and cloud layers, highlighting issues such as heterogeneity, limited observability, and complex dependencies [4], [14]. These studies collectively indicate that IoT software testing faces stronger

environmental constraints and dependency complexity than general-purpose software systems. Unlike traditional software systems, IoT software often interacts with various communication protocols, operates across heterogeneous hardware platforms, and adapts to dynamic and distributed environments. These characteristics pose unique testing challenges, particularly in designing effective test inputs. Test cases must follow communication protocols across devices, networks, and cloud services, while reflecting realistic environmental conditions (e.g., temperature, motion, or location) and continuous interaction with the physical world. III. DATA C OLLECTION Figure 2 shows an overview of the workflow of our empirical study. We started by collecting a dataset of 824 GitHub projects related to IoT. Makhshari and Mesbah [5] also collected a dataset of IoT software. Although their dataset contains 91 IoT projects, many of these projects are not suitable for dynamic test execution: 29 projects had been abandoned for at least four years, and 65 do not build or run. Since answering our research questions requires executing the tests of IoT software, we collected this new dataset with more up-to-date IoT software repositories, which are more likely to build and run successfully [15]. Using static analysis, we extracted test cases from these 824 projects and summarized their distribution. Among the 317 projects that contained test cases, 88 projects with more recent updates and higher popularity were further considered for detailed analysis. Although only 37 of these projects could be successfully executed, they are representative of the larger set, accounting for 12.5% of the total lines of code and 20.4% of the test cases, and covering all four IoT layers. These 37 projects are representative of our dataset, given their substantial number of test cases and their coverage of all IoT layers. These account for 12.5% of total LOC and 20.4% of test cases observed across the 317 projects. At the 95% confidence level, this sample size corresponds to a margin of error of approximately ±5% when calculated with respect to the total number of test cases, indicating that our dataset provides a representative basis for analysis. We compared the 37 selected projects with the full set of 317 IoT repositories at an aggregate level. As shown in Table I, both groups exhibit similar project sizes, measured by the average number of source files per project. The selected projects contain more test cases and substantially lower P/T (Production-to-Test) ratios on average, reflecting the presence of active and executable test suites required to obtain meaningful coverage and mutation results. Projects with sparse or inactive tests tend to yield trivial or uninformative dynamic analysis outcomes. Such outcomes are uninformative because they primarily reflect missing or broken test activity rather than the challenges of achieving effective testing in IoT software. Therefore, we exclude them to avoid drawing misleading conclusions. We note that this selection criterion may bias the reported coverage and mutation scores toward better-tested IoT repositories. Accordingly, the effectiveness metrics reported in this study should be interpreted as characterizing test-active and runnable IoT projects, rather

4

Github

Search for IoT-Related Topic

Test Identification Static analysis to extract test cases in the projects

824 IoT Projects

RQ1: Test Effectiveness Measuring statement coverage, branch coverage and mutation score achieved by the test cases.

Selected projects: star >= 100 & executable & updated after Jan 1 2022

RQ2: Testing Challenges Analyzing the factors leading to uncovered code segments.

37 Projects

RQ3: Mocking in IoT Testing Analyzing usages of mock objects in test cases of IoT software.

Fig. 2. The Overview of Our Empirical Study

than the entire population of IoT repositories. In addition, the selected projects collectively cover all IoT layers with comparable distributions, preserving architectural diversity. To validate this observation, we performed two-proportion z-tests for each layer and found no statistically significant differences between the selected subset and the full dataset. In line with our motivation, we treat each repository as an independently developed IoT software component, reflecting the realistic setting where developers often have limited access to the full IoT system. Thus, the analyzed projects may implement only one or a few IoT layers, while their tests still need to account for cross-layer dependencies through communication, configuration, and external services. Specifically, among the 317 analyzed repositories, 7 projects implement all four layers, while 88 projects implement only a single layer, with the remaining projects covering multiple layers. The complete mapping between projects and their corresponding layers is available in our online repository. In this section, we detail the data collection process and describe our approach for each research question separately before discussing their results. TABLE I C OMPARISON B ETWEEN S ELECTED AND F ULL -S ET I OT R EPOSITORIES Metric Number of repositories Average source files per project Average test cases per project Average P/T ratio Device-layer Edge-layer Cloud-layer Application-layer

37 Selected Projects

317 IoT Repositories

37 499.4 813.3 23.0 13 (35.1%) 12 (32.4%) 20 (54.1%) 8 (21.6%)

317 518.7 466.0 356.9 112 (35.3%) 85 (26.8%) 199 (62.8%) 90 (28.4%)

Note: P/T ratio denotes the Production-to-Test ratio.

A. Project Collection To determine the selection criteria for our IoT project dataset, we initially reviewed the dataset collected by Makhshari and Mesbah [5], which includes 91 IoT projects from GitHub. We analyzed their primary languages, defined as the language with the most files in each repository. JavaScript, Python, Java, and C emerged as the top languages, accounting for 78.9% of the projects. Based on this insight, we focused on JavaScript, Python, and Java. C was excluded due to the lack of standardized testing frameworks, which prevents automatic test case identification for further analysis. We adopted a similar approach to Makhshari and Mesbah [5] in collecting IoT projects by using GitHub topic tags. We sourced our dataset from GitHub, adhering to three

selection criteria: (1) The primary language of the project is either JavaScript, Python, or Java. (2) The projects should be tagged with at least one of the following keywords: IoT, Internet of Things, IoT-platform, IoT-device, IoT-Application. These tags were the top five most popular topics on GitHub containing “IoT” at the time of the experiment. (3) The project must have more than 10 stars to exclude toy projects. We collected a dataset comprising 824 open-source GitHub projects. The dataset is characterized by a diverse representation of primary languages: 371 Python projects (45%), 182 Java projects (22%), and 271 JavaScript projects (33%). The dataset includes renowned smart home platforms such as Home Assistant (58.7k stars) [16] and ThingsBoard (13.2k stars) [17], protocol libraries like paho.mqtt.python (1.8k stars) [18], backend SDKs such as kuzzle (1.2k stars) [19], and command-line interfaces like keylime (300 stars) [20]. This variety ensures that the dataset reflects the multifaceted nature of IoT projects. Our study focuses on productionlevel IoT software repositories in which testing is part of normal development practice. Repositories primarily designed for testing frameworks or benchmarking are excluded, as they do not reflect how testing is integrated into real IoT software projects. We acknowledge such repositories as a complementary population and identify their analysis as an important direction for future work. B. Dataset Characterization The analyzed repositories are primarily software components rather than complete IoT systems. We do not distinguish whether a repository represents an entire IoT system. Instead, each repository is characterized based on the IoT layers it implements (application, cloud, edge, and device). This characterization enables us to analyze the dataset’s architectural distribution and include projects from different layers. To make this distribution explicit, Table I reports the layer coverage of both the 37 selected runnable projects and the full set of 317 IoT repositories. The results show that the selected projects preserve similar architectural diversity. Device-layer projects account for 35.1% of the selected dataset versus 35.3% in the full pool, while cloud-layer projects represent 54.1% versus 62.8%. Application-layer components are also present (21.6% of the selected projects). To further clarify what kinds of IoT software artifacts are included in our dataset, we provide examples from each IoT layer observed in our dataset. At the device layer, repositories often implement device-facing communication logic

5

that directly interacts with sensors, actuators, or embedded protocols. For example, i2c-bus provides low-level access to hardware devices over the I2C bus. At the edge layer, repositories typically provide gateway or protocol-mediation components that bridge local devices with external services. For instance, iotedgedev supports IoT Edge deployment workflows. At the cloud layer, many repositories implement backend services or coordination platforms that manage fleets of connected devices through IoT protocols. A representative example is azure-iot-sdk-python, which provides device–cloud communication support and end-to-end messaging workflows. Finally, at the application layer, some repositories include user-facing automation and management logic for configuring IoT devices and workflows. For example, home-assistant-cli provides command-line interfaces for IoT administration. C. Test Identification 1) Approach: Based on the collected dataset, we first analyzed the prevalence of test cases in IoT software via static analysis. We adopted a two-stage approach to identify the test code in our subjects. In the first stage, we identified code patterns for common testing frameworks in our subjects. In the second stage, we designed automated techniques to identify test code in these projects based on the code patterns. Stage I: Code Pattern Derivation. In the first stage, we aimed to derive code patterns of commonly used testing frameworks. For each project, we inspected the source files and directory structures to check if they contained any test cases. We then identified the testing frameworks by examining the test code and build configurations. To derive the patterns, we analyzed test cases and relevant documentation, identifying common configurations and method annotations that define test cases. The resulting code patterns are as follows: • JAVA S CRIPT : J EST [21], M OCHA JS [22], and JASMINE [23] are the commonly used frameworks in our JAVA S CRIPT subjects. We identified each function closure passed to the testing API test() or it() as a test case. Each of these closures is recognized as an individual test case. • P YTHON : P Y T EST [24] and U NIT T EST [25] are the two commonly used testing frameworks. We identified P Y T EST test cases by matching the functions whose names start with test in the source files named with the prefix test_ or postfix _test. We identified test cases using the U NIT T EST framework by detecting methods within classes that inherit unittest.TestCase. • JAVA : JU NIT [26] is the de facto testing framework for JAVA projects. We identified the methods with annotation @Test as JU NIT test cases. Stage II: Automated Test Identification. Using the derived code patterns, we implemented an automated tool to identify test cases by matching common testing frameworks patterns. The tool first locates candidate test files, those named with “test” or placed in “test” directories. Then it performs pattern matching on the AST of the source file. For example, in Java projects using JU NIT, our tool identifies methods annotated with @Test. Details of all patterns used by our tool are

TABLE II D ISTRIBUTION OF T EST C ASES BY P RIMARY L ANGUAGES

# Test Case

# Projects 317

P/T Ratio

Total

Mean

Median

Mean

Median

147,736

466.0

34

356.9

8.97

available on our project’s website [9]. In each project, we analyzed the production and test code written in its primary language. During the analysis, we ignored the directories that are commonly used to store external libraries (e.g., node_modules, platform, and third-party). 2) Results: Among the 824 projects analyzed, 317 (38.5%) contain tests. This proportion notably exceeds what is observed within the broader spectrum of GitHub projects, where only 17.2% are reported to include tests [27]. Furthermore, this rate is substantially higher than the 14.2% within Android applications as identified in previous research [28]. Such findings highlight the pronounced emphasis on testing within IoT software projects, surpassing general software development trends and specific domains like Android apps. Table II details the test case distribution across these 317 projects, with an average of 466 test cases per project and a median of 34. The absolute number of test cases may not reflect the actual test abundance since different projects can have different scales. To account for this variation, we introduce the Production-to-Test (P/T) ratio, defined as the ratio of lines of production code to lines of test code. A smaller P/T ratio corresponds to a larger proportion of test code, indicating that the underlying project is more extensively tested. P/T ratio =

Lines of Production Code Lines of Test Code

Figure 3 illustrates the distribution of the P/T ratio. In the context of projects with commendable code coverage, a P/T ratio lower than 1 is typically indicative of robust testing practices [29]. However, among our examined IoT projects, only 13 (4.1%) exhibited a P/T ratio below 1, while a substantial portion (124 out of 317, or 39.1%) have a P/T ratio exceeding 17.19, indicating that many IoT projects include very limited test code. The project with the highest P/T ratio, Istsos2 [30], an IoT server encompassing both a backend server and a GUI application, registered a P/T ratio of 68051.88, highlighting a critical lack of testing relative to its production code. These results suggest that despite the presence of test cases, testing in many IoT projects remains insufficient compared to code size. D. Test Granularity The collected test cases can have varying granularities as unit tests, integration tests, or system tests. To gain insight into this, we performed a closed coding analysis following the deductive approach by Pecorelli et al. [31], classifying 382 Python, 383 Java, and 379 JavaScript tests (95% CL, 5% ME). This classification was based on criteria such as

6

Frequency

30

6WDWHPHQW&RYHUDJH %UDQFK&RYHUDJH 0XWDWLRQ6FRUH

20 10 0 0.17

17.19

281.34 P/T Ratio (log scale)

4382.37





&RYHUDJH  





   0XWDWLRQ6FRUH

 

0HDQ



68051.88

Fig. 3. Distribution of P/T Ratio of IoT Projects

the tested components, dependency isolation, and documentation. Our analysis identified 779 (68.1%) as unit tests, 348 (30.4%) as integration tests, and 17 (1.5%) as system tests, highlighting the predominance of unit and integration testing in the analyzed IoT projects. Therefore, the insights derived from our subsequent analysis mainly reflect testing practices for unit and integration levels. IV. RQ1: T EST E FFECTIVENESS A. Approach RQ1 aims to investigate the effectiveness of the test cases in IoT software. To answer this research question, we evaluated the statement coverage, the branch coverage, and the mutation scores that can be achieved by the existing test cases of the selected IoT projects. Our approach unfolds as follows: Executing test cases of the collected projects requires tremendous manual effort to properly set up the execution environment, since some of the projects may not contain clear instructions to set up the runtime environment or the instructions are outdated. As a result, it is impractical to run the test cases of all 317 projects. To filter out very inactive projects while maintaining representativeness, we applied two basic selection criteria: (1) repositories must have at least 100 GitHub stars to ensure minimal community adoption, and (2) repositories must have at least one commit after January 1, 2022 to exclude projects with no recent development activity. We adopted these criteria because long-term inactive projects often contain outdated dependencies or unmaintained test setups, making dynamic execution unreliable and potentially leading to uninformative coverage and mutation outcomes. To further validate that the selected projects remain actively maintained beyond this cutoff, we examined their commit recency and commit frequency. Only one project in the final dataset has no commits after the cutoff date. Overall, 68% of the selected projects were updated within 2025, and the selected projects exhibit an average commit frequency of 13 commits per month after the cutoff date (median: 2), indicating sustained maintenance beyond the cutoff. These filters yielded 88 projects for further analysis. We then attempted to measure their code coverage, following each repository’s build and test instructions when available to replicate their testing setup. Nevertheless, some projects could not be built due to obsolete dependencies, specific

Fig. 4. Boxplots showing Statement Coverage, Branch Coverage, and Mutation Score across IoT projects. Coverage is based on 37 projects; mutation scores on 28 with successful analysis.

hardware requirements, and incomplete documentation. In addition, the tests of some successfully executed projects finished with errors. To ensure the integrity of our collected test coverage, we included only those projects whose tests were executed without errors. This exclusion was essential to eliminate potential biases arising from incorrect test configurations or environmental discrepancies. In the end, we successfully executed the tests in 37 projects (42%). To collect code coverage and mutation scores, we used established automated tools. For Python projects, we used MUTMUT to perform mutation testing with all default mutation operators enabled, and C OVERAGE . PY to collect statement and branch coverage. For Java projects, we used PIT with its standard mutation operators and JAC O C O for coverage measurement. For JavaScript projects, we employed S TRYKER with all builtin mutators enabled and NYC for coverage collection. All tools provide automated instrumentation and command-line execution, enabling consistent measurement across projects and facilitating replication of our experimental setup. B. Results Figure 4 shows the distribution of statement and branch coverage across analyzed projects. The average statement and branch coverage were 65.2% and 53.4%, respectively. These metrics characterize testing effectiveness among runnable and test-active IoT projects, rather than the entire population of IoT repositories. These averages are higher than those reported in previous studies on Maven projects, which showed average statement coverages of 37.8% and 42.0% [32], [33], suggesting relatively stronger testing effort in the runnable and testactive IoT projects we analyzed. However, most projects still lack completeness: 54.1% have statement coverage below 75%, and 75.7% have branch coverage below 75%. We also observe significant disparities between projects. The project vandium-node [34] had a balanced P/T ratio of 1.24 and achieved 100% coverage. In contrast, the lowest statement and branch coverage were both 0.9% for smart-socket [35] and UsbSerial [36], with high P/T ratios of 347.9 and 19.81. Despite extensive efforts, mutation analysis failed for 9 of the 37 projects due to tool limitations, including crashes caused by bugs in the tools and compatibility issues such as conflicts with the project’s build environment. Figure 4 shows that test effectiveness remains limited in many analyzed

7

IoT projects, with an average mutation score of 39.9% and a median of 34.4%. This suggests that many test cases fail to effectively detect faults introduced through mutation, highlighting gaps in test effectiveness. This indicates limited faultdetection ability in many test cases—18 out of 28 projects (64.3%) scored below 50%. The highest score, 100%, was achieved by Adafruit_IO_Python [37], which also had over 90% statement coverage. In contrast, smart-socket [35] scored just 2.8%, with only 5 test cases and 0.9% statement coverage. These observations suggest that substantial room for improvement remains in the testing practices of the runnable and actively tested IoT projects we analyzed, motivating our investigation in RQ2 to uncover the underlying challenges. Understanding these challenges is essential for advancing testing methodologies. Answer to RQ1: Analysis reveals average statement coverage of 65.2%, branch coverage of 53.4%, and mutation score of 39.9% among runnable and test-active IoT projects. For context, prior studies on Maven-based Java projects reported average statement coverages of 37.8% and 42.0%, while empirical studies on GitHub open-source systems reported average mutation scores around 40%. Compared with these baselines, IoT projects exhibit substantially higher coverage but similar mutation effectiveness, motivating further analysis in RQ2.

V. RQ2: FACTORS C ONTRIBUTING TO U NCOVERED C ODE A. Approach The goal of RQ2 is to analyze the factors that lead to uncovered code within IoT software testing, and thereby to pinpoint the challenges in testing for IoT software. To achieve this, we adopted an open-coding strategy to analyze the uncovered code segments identified in RQ1. For our manual analysis, we included the projects that are the top 75% in terms of both the highest statement coverage and the lowest P/T ratio. We adopted such a selection criterion to exclude projects with overly low coverage or excessively high P/T ratios. Projects with low statement coverage have extensive portions of untested code. Similarly, a high P/T Ratio suggests that a project where the volume of production code far outweighs the extent of test code. Limited efforts were put into testing in these projects. Code segments in such projects may not be covered simply because the developers did not write a test for them. Analyzing such projects cannot provide meaningful insights for this RQ. In contrast, the selected projects with higher code coverage and lower P/T ratio are more likely to contain well-maintained and comprehensive test cases. Inspecting the factors leading to uncovered code in these projects is more likely to provide useful insights into testing IoT software. This selection resulted in 11 Python projects, seven JavaScript projects, and seven Java projects. Table III summarizes the number of projects remaining at each filtering step for each language, from the initial pool to the final selected dataset.

TABLE III P ROJECT S ELECTION F LOW BY L ANGUAGE . Language

Initial

Have Tests

Star & Active Filter

Runnable

Coverage & P/T Filter

Python JavaScript Java

371 271 182

122 80 115

34 13 41

16 7 14

11 7 7

Subsequently, we sampled the uncovered code segments with a 95% confidence level and a 5% margin of error. Each entry represents a contiguous block of uncovered code, rather than isolated lines, allowing us to analyze meaningful untested segments. Due to differences in uncovered code volume across languages, we applied language-specific sampling: Python — 202 from 1,103 entries; JavaScript — all 131 entries analyzed; Java — 383 from 89,393 entries. Our open coding procedure involved two authors independently categorizing uncovered code segments based on source code and corresponding test reports. An initial round of independent analysis was followed by discussion to define a hierarchical taxonomy and establish consistent categorization criteria. The authors then applied these criteria to classify the entire dataset, holding iterative discussions to address discrepancies and refine the taxonomy as needed. A third author participated in resolving conflicts. We achieved a Cohen’s Kappa score of 0.87, indicating a high level of agreement. Although this research question aims to identify factors contributing to the existence of uncovered code segments in IoT software testing, we encountered a considerable number of code segments where no obvious testing challenges could be identified. For example, we found code segments where entire functions remained untested, yet the function arguments were straightforward to set up. To systematically address these ambiguous cases, we adapted our classification strategy to focus on the intrinsic functionalities of uncovered code. This approach helps reveal common, critical functionalities that are vital yet overlooked by developers in testing. B. Results To understand the factors leading to uncovered code segments, we derived a hierarchical taxonomy. All the code segments are first categorized into three primary categories as shown in Table IV. The first primary category (Uncovered Code Segments with Identified Hurdles) contains 368 code segments where we successfully observed the factors impeding their coverage. We identified the subcategories as the impeding factors. The second primary category (Uncovered Code Segments with No Identified Hurdles) contains 241 code segments without identifiable coverage hurdles. To gain a clearer understanding of these segments, we analyzed their functionalities. The third primary category (Unreachable Code Segments) contains 107 cases where the code segments cannot be covered, such as dead code or unimplemented features. Below, we introduce these primary categories and subsequently delve into their respective subcategories to summarize the key challenges in achieving better test coverage for IoT software. 1) Category 1: Uncovered Code Segments with Identified Hurdles: This category contains 368 code segments (51.4%) where we identified factors impeding their coverage. We

8

TABLE IV TAXONOMY OF U NCOVERED C ODE S EGMENTS Subcategory

Count

Uncovered Code Segments with Identified Hurdles Untested Communication Scenarios Misaligned File System Dependency Improper System Configuration Insufficient Database Setup Untested IoT-Event Handlers Uncovered Code Segments with No Discernible Hurdles CLI Usage Illegal Object Handling Arid Code Overridden Standard APIs Data Structure Processing Inherited and Overridden Functions Others

368 199 57 55 52 5

Listing 2. Example of Misaligned File System Dependency 1 if "CONNECTION_STRING" in os.environ: 2 # Uncovered block...

Listing 3. Example of Improper System Configuration

241 48 43 33 21 15 8 73

Unreachable Code Segments

107

Test Suite Anomalies Pending Implementations Dead Code

64 26 17

1 def subscribe_weather(self, weather_id, forecast_type): if forecast_type in forecast_types: 2 3 self._client.subscribe( ’{0}/integration/weather/{1}/{2}’.format( 4 self._username, weather_id, forecast_type 5 6 )) 7 else: 8 raise TypeError("Invalid Forecast Type Specified.")

1 self.json = json.loads( Utility.get_file_contents(path, expandvars=expand_vars) 2 3 ) 4 if "modulesContent" in self.json: return self.json["modulesContent"] 5

1 try { 2 writeBatch.put(..., rocksKey(row), ...); return null; 3 4 } catch (RocksDBException e) { 5 throw new StorageException(...); 6 }

Listing 4. Example of Insufficient Database Setup Interaction 1 if event == SUN_EVENT_SUNRISE: 2 brightness = lerp(...)

Listing 5. Example of Event-Driven Scenarios

This subcategory includes code segments involved in file processing, such as I/O operations and file system interactions. Testing these segments is challenging due to the need to simulate file system behavior or ensure specific file contents and formats. Listing 2 shows an example code snippet in Listing 1. Example of Communication Scenarios (highlighted lines indicate Iotedgedev [40] where Lines 4–5 are uncovered. Covering uncovered code, this also applies to the other listings) them requires a test case that places a JSON file containing the key "modulesContent" in the expected directory. This type of code requires careful consideration in testing to observe that the majority of them are not covered due to account for various file system states, such as the existence, the various constraints on dependencies and environments. accessibility, and correctness of specific files. Improper System Configuration (55 code segments): These constraints can arise from interactions with web services, database operations, file system manipulations, network This category includes code segments that rely on the underconnectivity issues, etc. We further categorize the factors into lying operating system, including dependencies on environsubcategories and Table IV shows their distribution. We detail ment variables, platform-specific characteristics, and system calls. These dependencies complicate testing, as they require the definition of each subcategory as follows: Untested Communication Scenarios (199 code seg- the test environment to replicate specific system conditions ments): This subcategory encompasses code segments that or permissions. Listing 3 shows an example code snippet require communication with various IoT components or web in Azure-IoT-SDK-Python [41], where Line 2 checks if services, often executed through network requests using vari- CONNECTION_STRING exists in the environment variables ous protocols (e.g. HTTP, MQTT [38], and LWM2M [39]). of the system. This environment variable is necessary for reIt involves operations such as sending requests to external mote connections for this project. Testing such code segments APIs, fetching data from remote servers, or handling responses requires the replication of specific environmental settings. Insufficient Database Setup (52 code segments): This subfrom web services and device interactions. Listing 1 shows an example in this category from Adafruit IO Python [37]. The category includes code segments that interact with databases. function subscribes to MQTT-based weather forecast topics Testing them is challenging due to the need to simulate provided by Adafruit IO, requiring both a valid weather ID connections, control database states, and validate both norand forecast type (Line 1). Line 2 checks whether the provided mal and exceptional behavior. Listing 4 shows an example forecast type is valid. The entire function was left untested from Ignite 3 [42], where Line 5 is uncovered. Lines 2–3 in the project. Covering this function requires constructing perform a database insertion via RocksDB, while Line 5 input strings (i.e., weather_id and forecast_type) handles a possible RocksDBException. Covering this line that match the predefined MQTT string format. Moreover, it requires configuring RocksDB with conditions that trigger also requires mocking the MQTT client and verifying that it RocksDBException. Effective testing must account for both successful operations and error handling scenarios. performs correct subscriptions for valid inputs. Untested IoT-Event Handlers (5 code segments): This Misaligned File System Dependency (57 code segments):

9

category includes code segments triggered by hardware-related or environmental stimuli events, such as temperature changes, motion detection, light levels, or other sensor-based inputs. Such scenarios are crucial for IoT software that reacts to environmental changes or user interactions. Testing these segments is challenging due to the need to simulate various realworld conditions. Listing 5 shows an example code snippet in Adaptive Lighting [43] where Line 2, which adjusts brightness based on a sunrise event, remains untested. 2) Category 2: Uncovered Code Segments with No Discernible Hurdles: This category contains code segments where we did not observe any challenges in writing test cases to cover them. These code segments are typically entire functions that are uncovered and arguments of the functions are straightforward to create. It is unclear why these code segments are not covered. As a result, we focused on categorizing the functionalities of these code segments. 241 out of the 716 (33.7%) analyzed code segments fall into this category. As shown in Table IV, the functionalities of code segments in this category fall into: (1) CLI Usage (48 code segments): Code segments that execute or interact with the Command Line Interface (CLI). (2) Illegal Object Handling (43 code segments): Code segments that deal with illegal or unexpected object states, including handling null references, invalid object configurations, or exceptions arising from improper use of objects. The illegal objects of these category are easy to construct (e.g., feeding in a null value). (3) Arid Code (33 Code Segments): Code segments classified as “arid" according to Petrovic and Ivankovic’s work [44], referring to code with limited value for testing. Arid code includes segments associated with logging, testing, non-functional properties, and axiomatic language features whose functionalities can be trivially tested (e.g., getters and setters). (4) Overridden Standard APIs (21 code segments): This category includes cases where standard methods (e.g., toString(), equals(), and hashCode()) are overridden. (5) Data Structure Processing (15 code segments): Code segments that perform operations on data structures like arrays, lists, maps, and trees, such as traversal, sorting, searching, insertion, and deletion. (6) Inherited and Overridden Functions (8 code segments): This category includes functions in class hierarchies that lack test coverage. (7) Others (73 code segments): Code segments that do not belong to the above categories. They are specific coding practices appearing only once or twice in our dataset. The code segments in these subcategories are either trivially testable or require minimal effort to construct the necessary inputs, suggesting that their lack of coverage is likely due to oversight rather than inherent testing complexity. 3) Category 3: Uncoverable Code Segments: Finally, 107 out of the 716 (14.9%) analyzed code segments cannot be covered due to (1) test suite anomalies (64 code segments), where errors and design flaws in test implementations or intentionally skipped tests prevent some tests from being executed; (2) pending implementations (26 code segments), where portions of the codebase are marked for future development with placeholders; and (3) dead code (17 code segments), where the code can never be reached (e.g., unused variables, methods, classes, or unsatisfiable conditions.)

Answer to RQ2: Over half of uncovered code segments (51.4%) are impeded by coverage hurdles, predominantly due to external dependencies (98.6%), especially network communications. Addressing these challenges requires advances in protocol-aware mocking and automated test environment setup.

VI. RQ3: M OCKING IN I OT T ESTING A. Approach Mocking is widely used in testing to isolate units from their dependencies. Developers can create mock objects (also known as test doubles) to replace the original dependency implementations and fully control their behaviors [45]. Studies have been conducted to characterize the usage of mock objects in different domains, such as Java programs [46]–[49] or Android apps [45]. In this research question, we aim to dissect the landscape of mock usage in IoT software testing. To answer this research question, we (1) leveraged dynamic analysis to extract mock objects in the code, (2) conducted a closed coding analysis to identify the role of each mock object, and (3) conducted an open coding analysis to categorize the dependencies that are simulated by each mock object. Extracting mock objects from test cases. As pointed out by Zhu et al. [46], extracting mock objects via static analysis suffers from significant limitations in precision. To avoid including extensive noise in our dataset, we leveraged a dynamic analysis approach to extract mock object instances from the test cases, following Zhu et al. [46]. Specifically, this approach identifies mock objects by inspecting the dependencies of each function call, including the types of its arguments and return value. Mock objects created by mocking frameworks typically follow certain patterns in their type names. For instance, in the commonly used JAVA mocking framework M OCKITO [50], mock objects typically have type names that include MockitoMock or EnhancerbyMockito. Similarly, for the P YTHON UNITTEST framework [25], relevant mock object types are unittest.mock.Mock or unittest.mock.MagicMock. JAVA S CRIPT typically does not have sufficient runtime information to determine whether an object is a mock object or not, so our analysis seeks to identify invocations of functions that create mock objects, e.g., sinon.stub() [51]. We can thereby identify mock objects by matching the type of the extracted function call dependencies with these framework-specific patterns. We collected such patterns for 18 mock frameworks. These frameworks were collected by identifying libraries that contain the keyword “mock” in our 317 projects and manually analyzing their documentation. The details of the frameworks and their mock object type patterns are available in our research artifact [9]. This dynamic analysis resulted in 1,056 mock objects from all 37 projects collected in Section IV. We used this dataset as the subject for our further analysis. Categorizing the mock objects. To understand the usage and purpose of mock objects, we conducted a manual analysis of the extracted mock objects. Specifically, we categorized them from two perspectives: (1) Roles of the mock objects.

10

TABLE V ROLES OF M OCK O BJECTS IN I OT S OFTWARE T ESTING

# Instances Percentage (%)

TABLE VI M OCKED D EPENDENCY T YPES

Mock

Stub

Spy

Fake

Type

716 67.80

320 30.30

18 1.70

2 0.19

Network Communication Logger System Call Domain Logic Scheduling File

Mock objects can play different roles in a test case. In practice, there are five common types of mock objects [45]: • Mock. A mock is a replacement for the test dependencies such that the method invocations can be verified (e.g., using the API verify() provided by M OCKITO). • Spy. Spies are similar to mocks but wrap real objects, allowing the actual method to execute while still enabling behavior verification. In M OCKITO, spies can be created using the spy() method. • Stub. A stub is a mock object with hard-coded behaviors (i.e., return values or exceptions) for its method invocations [47]. Such hard-coded behaviors usually respond to specific scenarios in the tests. • Fake. A fake is a lightweight working implementation suitable for testing purposes only. For example, a fake database can use an array to store its data. • Dummy. A dummy is an object which neither the test nor the code under test interacts with. They are usually used as a placeholder for the parameter regardless of a specific test. The roles of the mock objects can characterize how the mock objects are used. (2) Types of dependencies simulated by the mock objects. We also categorized mock objects based on the functionalities they simulate in the test environment. This perspective characterizes what the mock objects are used for. These two perspectives identify the challenges of mock usage in IoT testing and can guide future research on generating mocks for IoT-specific test cases. We adopted an iterative procedure to manually analyze the mock objects that is similar to the procedure we describe in Section V-A. Two authors independently analyzed the dataset to identify the roles and simulated dependencies of mock objects, forming and refining categorization criteria through discussion. Throughout the process, they regularly discussed disagreements and refined the categorization criteria. A third author was involved to resolve any conflicts between the initial two authors’ classifications. We achieved a Cohen’s Kappa score of 0.89 for the roles of the mock objects and 0.82 for the types of dependencies simulated by mock objects at the end, indicating a high level of agreement. B. Results 1) Roles of Mock Objects: Table V summarizes our analysis results of mock object roles in IoT software testing. 716 (67.80%) out of the 1,056 analyzed mock objects play the role of Mock, and 30.30% play the role of Stub. These two categories account for 98.10% of all the analyzed mock objects. Conversely, Spy (1.70%) and Fake objects (0.19%) are comparatively rare. We found no dummies in the dataset. These results show that the vast majority of mock usages in IoT software test cases involve defining (both Mock and Stub)

# Instances 803 (76.04%) 68 (6.44%) 31 (2.94%) 30 (2.84%) 29 (2.75%) 15 (1.42%)

Type

# Instances

User Interface Database Environment CLI API Key Others

15 (1.42%) 8 (0.76%) 8 (0.76%) 7 (0.66%) 6 (0.57%) 36 (3.40%)

or verifying (Mock and Spy) specific dependency behaviors instead of simply acting as placeholders for the corresponding dependencies (Dummy and Fake). As a result, it is nontrivial to generate mock objects to improve test generation for IoT software, since they need to encapsulate meaningful and context-specific dependency behaviors. 2) Types of Dependencies Simulated by Mock Objects: It is important to characterize the types of dependencies that are simulated by the mock objects to identify the frequently mocked APIs. Results from this analysis can guide future research on leveraging mock objects to improve testing for IoT software. Table VI shows the categorization results for the types of simulated dependencies. We identified 12 categories of different simulated dependencies as follows: Network Communication (803 mock objects). Mocks in this category simulate external communication interfaces, enabling tests to verify data exchange handling without real network calls. This facilitates controlled testing of error handling, serialization, and protocol compliance. Logger (68 mock objects). Mocking logging dependencies allows tests to verify event recording behavior without relying on the real logging system. System Call (31 mock objects). Mocking system calls allows tests to simulate responses from the operating system, enabling the validation of project behavior in response to system-level events without real system interactions. Domain Logic (30 mock objects). In this context, domain logic mocks represent internal logic systems, allowing tests to handle these rules within the project without needing the actual implementations present. Scheduling (29 mock objects). This category contains mock objects used to test event scheduling and management. It includes mocks that simulate scheduling logic, event queues, and event-driven behaviors. File (15 mock objects). This category uses mocks to simulate file systems, enabling tests of file operations such as reading and writing without accessing real files. This helps in verifying file manipulation behaviors under various conditions. User Interface (15 mock objects). User Interface mock objects simulate user interactions, which facilitates validating the response to user interactions without needing real users, leading to a higher degree of test automation. Database (8 mock objects). Mock objects simulating database interactions enable testing data handling logic—such as queries and transactions—independently from the actual database, ensuring correct implementation. Environment (8 mock objects). Environment mock objects simulate different runtime environments, testing the project’s

11

adaptability and correctness under various configurations without altering the real execution environment. Command Line Interface (CLI, 7 mock objects). Mock objects for CLI components simulate the behavior of the command line and terminal. Similar to mock objects for UI, CLI mock objects can verify the processing logic of CLI arguments and commands without actual user input. API Key (6 mock objects). Mocking API key management systems allows for the testing of key storage, retrieval, and usage, ensuring secure and appropriate use of API keys without real external service interactions. Others (36 mock objects). This category groups mock objects that simulate external dependencies not easily classified into existing categories. These mocks represent unique or infrequent dependencies with limited presence in typical IoT testing scenarios. Answer to RQ3: Mock objects are primarily used to test communication APIs in IoT software, and are also frequently applied to simulate system calls, files, databases, and environment APIs, thus addressing key code coverage challenges (RQ2). Mocks are essential for overcoming IoT testing barriers, but generating effective mocks remains difficult due to the requirement for realistic and contextspecific behaviors. VII. I MPLICATIONS In this section, we discuss the implications that can be drawn from the results of our research questions. Implication 1. The prevalent use of mock objects aligns with the core testing challenges in IoT software, with both showing a striking focus on network communication scenarios. Leveraging mock objects presents a promising approach to address these challenges. As shown in Table VI, among all the mocked dependencies, 76.04% simulate network APIs. This number is significantly higher than that in other types of software. For example, only 8.7% of the mock objects in Android apps target network APIs [45]. This striking difference reflects the unique demands of IoT systems, which heavily rely on network communications through various protocols such as Zigbee and MQTT. In the meantime, these results align closely with the categorization results in Table IV (RQ2): 363 out of 368 code segments in Category 1 involve external dependencies, while 199 segments (54.07%) are due to untested communication scenarios across diverse protocols. Our findings reveal a strong overlap between the challenges in covering IoT code and the types of dependencies simulated by mock objects. This overlap highlights network communication as the primary testing challenge for IoT software and suggests mocking as a de facto solution to address this challenge. It may seem contradictory that communication-related code remains largely untested while such mock objects are widely used. This is because the results are collected across different projects. While some well-maintained projects effectively mock complex network dependencies, many others still face difficulties. This observation also leads to our next implication. Implication 2. Despite the challenges of correctly mocking network communication in IoT software, the abundance

1 # Mock function in Home Assistant to simulate setting light state 2 def mock_light_set_state(... red=45, green=55, brightness =50...): 3 return {... "red": red, "green": green, "brightness": brightness...} 4 5 # Real implementation in openHAB’s ShellyLightHandler to control light 6 col.setRed(setColor(lightId, "red", command, 255)); 7 col.setGreen(setColor(lightId, "green", command, 255)); 8 ...

Listing 6. Example of Test Migration

of existing mocks in open-source projects enables mock migration as a potential solution. As demonstrated in RQ3, mock objects in IoT tests need to faithfully encapsulate the specific dependency behaviors rather than act as simple placeholders. Defining such mocks to replicate communication scenarios involving IoT devices remains a significant challenge. Existing mock generation techniques, which often depend on human-written test oracles [47] or runtime profiling [52], are not applicable since it is infeasible to predict or profile the behaviors of all IoT devices. Based on our empirical study results, we envision migrating mock objects across software as a solution to address this problem since: (1) IoT software systems often interact with a common set of IoT devices, (2) the behaviors of these devices are likely to remain consistent across different software contexts, and (3) as demonstrated by RQ3, a large number of mock objects are already available in open-source projects, providing a rich resource for migration. For example, Shelly is an IoT device producer providing a variety of IoT devices for smart homes [53]. Both Home Assistant [16] and OpenHAB [54] smart home platforms support integrations of Shelly devices. However, the Shelly component in Home Assistant contains substantial test cases while the component in openHAB remains untested. Lines 1-3 in Listing 6 demonstrate how Home Assistant uses a mock function to simulate setting the state of a Shelly light device. This function accepts structured parameters (e.g., red, green, blue, brightness) that conform to the Shelly protocol, enabling Home Assistant to test its Shelly integration logic without relying on real devices. In comparison, openHAB’s ShellyLightHandler performs similar operations to control device states (Lines 6-7). However, this logic remains untested due to the absence of a mocking infrastructure. Notably, both platforms interact with Shelly devices using the same protocol, and their commands share the same attributes (e.g., colors, brightness) and valid ranges. This alignment presents a clear opportunity: test code from Home Assistant can be adapted to simulate equivalent device states in openHAB. By reusing mock inputs and verifying method calls, it becomes feasible to migrate test cases and improve test coverage for ShellyLightHandler. Potential Solutions to Identified Coverage Hurdles. To make our findings more actionable, Table VII summarizes alternative solution directions for each identified hurdle for improving the test coverage. For some hurdles, existing techniques already provide

12

TABLE VII C OVERAGE H URDLES AND P OTENTIAL S OLUTIONS Coverage Hurdle

Potential Solution

Untested Communication Scenarios

Systematic mocking support and reuse of existing device/service mocks

Misaligned File System Dependency

Mocking file resources and filesystem states

Insufficient Database Setup

Mocking and seeding representative database states

Improper System Configuration

Systematic configuration-space exploration (e.g., iGen)

Untested IoT-Event Handlers

Automated event-sequence generation (e.g., IoT-TEG)

promising starting points. For Improper System Configuration, systematic exploration of configuration spaces can help exercise configuration-dependent branches, for example through configuration interaction inference and test generation techniques such as iGen [55]. For Untested IoT-Event Handlers, automated generation and mutation of event sequences can stimulate event-driven behaviors, as demonstrated by IoT event generation frameworks such as IoT-TEG [56]. For the remaining hurdles, including Untested Communication Scenarios, Misaligned File System Dependency, and Insufficient Database Setup, the core difficulty lies in reproducing realistic external dependencies during testing. While existing approaches such as fuzzing or automated input generation can partially address these issues, they often fail to capture the protocol semantics, data formats, and stateful behaviors required by IoT software. As discussed in Implications 1 and 2, developers therefore rely on mock objects to emulate external services, devices, files, and databases. However, manually constructed mocks are often incomplete and inconsistent across projects, leaving many interaction paths uncovered. Based on our findings, we suggest that improving the fidelity and reuse of mocks is a more practical and impactful solution direction. In particular, the abundance of existing mock objects in open-source IoT projects enables systematic mock improvement and migration across software that interacts with the same devices or protocols, providing an opportunity to reduce uncovered code in practice. Mocking Scope and Limitations. Mock-based testing is widely adopted in IoT software projects to isolate external dependencies, such as MQTT or HTTP endpoints, environment variables, databases, and file systems. Such mocks improve controllability and reproducibility in software-level tests and enable developers to exercise code paths that would otherwise depend on unavailable or non-deterministic conditions. However, mock-based isolation tests differ fundamentally from execution on real devices or hardware-in-the-loop setups. When IoT software is deployed on physical devices, factors such as timing constraints, network instability, and hardwarespecific behaviors may not be accurately reproduced by simulated components. As a result, while mocks are essential for scalable software testing, final validation of IoT behavior still requires execution on real devices or realistic testbeds. Therefore, our findings should be interpreted as reflecting

testing practices in software repositories, where mocks primarily support reproducible development testing, while devicelevel execution may remain necessary for confirming end-toend IoT behavior in practice. VIII. T HREATS TO VALIDITY External Validity. Our study focused on open-source IoT projects and the test cases within those projects. Devicelayer testing practices such as Model-in-the-Loop testing may be conducted using external environments and thus may not be reflected in repository contents. As a result, our findings primarily reflect the testing practices that are observable in open-source IoT software projects, while still providing useful empirical evidence on how developers test and validate IoT software in practice. Our analyzed testing and mocking frameworks may not cover all the frameworks used by developers. As a result, our findings may not generalize to closed-source IoT software or those using different frameworks for testing or mocking. However, the IoT projects we collected span various domains and sizes to ensure that our findings are as representative as possible of the open-source IoT ecosystem. Additionally, our analysis is conducted at the repository level and does not explicitly separate IoT-specific code from general-purpose utility or application code within the same project. In practice, IoT repositories often combine protocolhandling and device-facing logic with conventional components (e.g., configuration, storage, or web services). Therefore, some uncovered-code categories we observed may also arise in non-IoT software, although they manifest in IoT projects under stronger environmental and dependency constraints. Nevertheless, RQ2 results highlighted major IoT-relevant challenges, e.g., “Untested Communication Scenarios” and “Untested IoTEvent Handlers”, suggesting that the non-IoT-specific code in the projects did not significantly impact our key findings. Our study focuses on functional testing artifacts observable in open-source IoT repositories. Testing of non-functional properties (e.g., performance, security, or real-time constraints) and hardware-dependent validation is not systematically captured in such repositories. As a result, such testing cannot be analyzed in a uniform and reproducible manner across projects using repository-based evidence alone. Our findings should thus be interpreted as characterizing functional testing practices in open-source IoT software components across different IoT layers. Internal Validity. The categorization of uncovered code segments and mock objects involved manual open-coding and closed-coding processes. While these methods are grounded in rigorous qualitative research principles, they are susceptible to subjective bias. To reduce the impact of subjective bias, two authors independently coded a subset of the data, and discrepancies were discussed until a consensus was reached. Moreover, our approach included sampling when analyzing the uncovered code segments, which might have led to the omission of relevant data that could influence the results. We also ensured that our confidence level and sampling strategy were systematically applied to obtain a statistically representative sample size and avoid biases related to the selection of code segments.

13

Construct Validity. In this study, test effectiveness is defined at the source-code level and measured using statement coverage, branch coverage, and mutation score. These metrics reflect the extent to which tests exercise implementation logic and reveal faults, but do not capture system-level behavior or non-functional properties. We follow prior empirical software testing studies that adopt coverage and mutation analysis as widely accepted metrics for test-suite effectiveness [57]. These metrics enable objective comparison across projects and provide a basis for assessing the effectiveness of IoT test suites in practice. To measure test effectiveness, we used established coverage and mutation testing tools. Specifically, statement and branch coverage were collected using C OVER AGE . PY for Python, JAC O C O for Java, and NYC for JavaScript. Mutation scores were computed using MUTMUT (Python), PIT (Java), and S TRYKER (JavaScript). Static analysis and test artifact inspection were conducted using custom scripts built on top of language-specific abstract syntax trees and repository metadata. To mitigate potential inaccuracies in our tool implementation, we conducted tests of our implementation and manually checked the results. When using existing coverage or mutation testing tools, we strictly followed their instructions. However, it is possible that running the test cases in different environments could potentially lead to different code coverage results. To mitigate this threat, we strictly followed the instructions provided by the analyzed IoT software projects, aiming to replicate the coverage that can be achieved by the test cases. We also provide Docker images that can reproduce our evaluation results [9]. IX. R ELATED W ORK IoT Software. Testing Techniques and Frameworks. Research on IoT testing has explored a variety of testing techniques and tools, including several widely studied modelbased testing [58], smart home application testing [59], [60], and comparisons of IoT testing tools [61]. More recent work has proposed IoT-specific testing frameworks, such as event-driven test generation for realistic IoT behaviors [56], model-based generation of executable end-to-end IoT test scripts [62], combinatorial testing strategies for complex IoT systems [63], automated interoperability and integration testing platforms [64], scalable online conformance testing for IoT applications [65], a testing framework for remote IoT test execution [66], protocol-level testing under varying network conditions and traffic patterns [67], smart testing frameworks with resource-demand prediction in dynamic IoT environments [68], and model-driven Thing-in-the-Loop approaches for IoT verification and validation [69]. Protocol- and network-level robustness testing has also been studied using fuzzing-based approaches [70], [71]. Complementary to predeployment testing, runtime verification approaches monitor IoT executions during operation, and runtime testing has also been systematically reviewed [72], [73]. Failures and Diagnosis. Several studies have investigated challenges and bugs in IoT software, with a primary focus on security and reliability issues. Prior work has examined vulnerabilities arising from interactions with the physical

world [7], firmware-level defects [74], protocol implementations [75], and IoT applications [76]. Beyond security-oriented analyses, Makhshari et al. conducted a large-scale empirical study of IoT bug reports, analyzing 5,565 issues from 91 repositories to construct a taxonomy of failures, root causes, and fault locations [5]. More recent work has also investigated debugging, troubleshooting, and fault diagnosis in deployed IoT systems, including live debugging support for IoT applications [77], record-and-replay techniques for diagnosing COTS IoT devices [78], systematic fault handling mechanisms [79], and distributed fault diagnosis frameworks [80]. Reviews and Surveys. Several systematic literature reviews have surveyed IoT systems testing and related quality concerns, summarizing techniques, challenges, and research trends [4], [14], [81], [82]. In addition to technical approaches, survey studies have gathered practitioner perspectives on IoT testing challenges. Minani et al. [83] surveyed 49 practitioners and identified issues such as heterogeneity, lack of standards, and limited automation, while Rodriguez-Cardenas et al. [8] surveyed 80 developers and reported issues related to device compatibility, firmware updates, and documentation. Despite this growing body of work, existing studies largely focus on proposing testing techniques, tools, or studying practitioner perceptions. They do not empirically characterize test effectiveness, test coverage, or mocking practices based on executing tests in real IoT repositories. In contrast, our work provides a measurement-based analysis by executing and analyzing test suites from 37 runnable IoT projects, offering empirical evidence on how IoT software is tested in practice. Empirical Studies On Testing. Empirical studies have been conducted to characterize testing practices in different domains. For example, researchers have studied Android apps to explore automation [84] and GUI testing [85], and Java projects to correlate test coverage with bugs [32] and software metrics [33], such as the number of developers involved. While our data collection approach may share similarities with these previous efforts, our focus distinctly lies on identifying and understanding testing challenges within IoT software. Kochhar et al. [28] analyzed test case existence and computed code coverage across 600 Android apps, using surveys to understand the challenges faced by developers. Unlike the study that relied on surveys to gauge testing challenges, we directly analyzed uncovered code segments to reveal testing difficulties. These studies have explored testing practices across diverse software domains. However, testing in IoT remains underexplored. Our research specifically focuses on IoT software developed in Python, Java, and JavaScript, which distinguishes it from the Android applications targeted in previous studies. Empirical Studies On Mocking. Several studies have analyzed the use of mocking frameworks in software testing, highlighting the prevalence and implementation practices across various projects. For instance, Xiao et al. [48] analyzed 246 Apache Java projects to understand how mocking frameworks are utilized and the common usage patterns within these projects. Fazzini et al. [45] employed static analysis to extract and categorize mock objects from 1,000 Android applications, providing a broad overview of mocking practices in mobile app development. In contrast, our approach leverages

14

dynamic analysis to extract mock objects, which offers higher precision. Zhu et al. [46] conducted an extensive study on mocking decisions with 10,000 test cases across four opensource projects and proposed M OCK S NIFFER to recommend mocking decisions. More recently, Zhu et al. [49] further conducted an empirical study on the usage of mock assertions to reveal their practical usage and fault detection capabilities. These studies have primarily focused on static analysis and generic mocking recommendations, with some providing categorizations of mock usage. In comparison, our research goes beyond this by developing a taxonomy specifically tailored to the unique needs of IoT software. This taxonomy categorizes mock objects based on their functionalities, with particular emphasis on how they address the challenges of simulating external dependencies unique to IoT environments. Our study addresses gaps in the existing IoT literature by providing a focused empirical study of testing practices within IoT software, particularly in understanding the utilization of mocks and the challenges associated with testing. X. C ONCLUSION In this paper, we presented a large-scale empirical study of testing practices in open-source IoT software projects. Our investigation revealed that, although IoT software tends to include more tests than general-purpose software, its coverage and effectiveness remain limited. We identified significant challenges in IoT software testing, particularly in handling complex dependencies and interactions such as network communications. Furthermore, our study highlighted the substantial role of mock objects in IoT software testing and revealed a clear alignment between mock usage and the challenges faced by developers. Future research can build on these insights to enhance IoT software testing by improving the utilization of mock objects, developing techniques that can better simulate complex system interactions, and creating tools that facilitate more effective and automated testing for IoT software. We envisioned mock object migration as a promising direction for future research, leveraging the large number of mock objects available in well-maintained open-source projects to generate tests with faithfully simulated dependencies. ACKNOWLEDGEMENTS This work was supported by Fonds de recherche du Québec(Grant No.2024-NOVA-346499) [86]. R EFERENCES [1] “IoT Market with COVID-19 analysis by Component, Organization Size, Focus Area and Region - Global Forecast to 2026,” 2022. [Online]. Available: https://www.reportlinker.com/p04944762/ [2] G. Uddin, F. Sabir, Y.-G. Guéhéneuc, O. Alam, and F. Khomh, “An empirical study of iot topics in iot developer discussions on stack overflow,” ESE, vol. 26, no. 6, pp. 1–45, 2021. [3] Z. Li and R. Ranjan, “Just enough, just in time, just for “me”: Fundamental principles for engineering iot-native software systems,” in ICSE-NIER, 2022, pp. 56–60. [4] J. B. Minani, F. Sabir, N. Moha, and Y.-G. Guéhéneuc, “A systematic review of iot systems testing: Objectives, approaches, tools, and challenges,” TSE, vol. 50, no. 4, pp. 785–815, 2024. [5] A. Makhshari and A. Mesbah, “Iot bugs and development challenges,” in ICSE, 2021, pp. 460–472.

[6] N. Neshenko, E. Bou-Harb, J. Crichigno, G. Kaddoum, and N. Ghani, “Demystifying iot security: An exhaustive survey on iot vulnerabilities and a first empirical look on internet-scale iot exploitations,” IEEE Communications Surveys & Tutorials, vol. 21, no. 3, pp. 2702–2733, 2019. [7] W. Ding and H. Hu, “On the safety of iot device physical interaction control,” in CCS, 2018, pp. 832–846. [8] D. Rodriguez-Cardenas, S. A. Khan, P. Mandal, A. Nadkarni, K. Moran, and D. Poshyvanyk, “Testing practices, challenges, and developer perspectives in open-source iot platforms,” in ICST, 2025, pp. 290–301. [9] “Anonymous Characterizing-Tests-in-IoT-Software-PracticesChallenges-and-Opportunities,” 2026. [Online]. Available: https://anonymous.4open.science/r/Characterizing- Tests- in- IoT -Software-Practices-Challenges-and-Opportunities-12F7 [10] N. Pawar, T. Bourgeau, and H. Chaouchi, “Study of iot architecture and application invariant functionalities,” in IFIP/IEEE IM, 2021, pp. 667–671. [11] A. Dauda, O. Flauzac, and F. Nolot, “A survey on iot application architectures,” Sensors, vol. 24, no. 16, p. 5320, 2024. [12] D. Liu, H. Liang, X. Zeng, Q. Zhang, Z. Zhang, and M. Li, “Edge computing application, architecture, and challenges in ubiquitous power internet of things,” Frontiers in Energy Research, vol. 10, 2022. [13] V. Garousi, M. Felderer, Ç. M. Karapiçak, and U. Yılmaz, “What we know about testing embedded software,” IEEE Software, vol. 35, no. 4, pp. 62–69, 2018. [14] W. Fei, H. Ohno, and S. Sampalli, “A systematic review of iot security: Research potential, challenges, and future directions,” ACM Computing Surveys, vol. 56, no. 5, 2023. [15] M. Sulír and J. Porubän, “A quantitative study of java software buildability,” in PLATEAU@SPLASH, 2016, pp. 17–25. [16] “Home assistant,” 2024. [Online]. Available: https://github.com/home-a ssistant/core [17] “ThingsBoard,” 2024. [Online]. Available: https://github.com/thingsboa rd/thingsboard [18] “paho.mqtt.python,” 2024. [Online]. Available: https://github.com/eclip se/paho.mqtt.python [19] “kuzzle,” 2024. [Online]. Available: https://github.com/kuzzleio/kuzzle [20] “keylime,” 2024. [Online]. Available: https://github.com/keylime/keyli me [21] (2024) Getting Started. [Online]. Available: https://jestjs.io/docs/getting -started [22] (2024) The fun, simple, flexible JavaScript test framework. [Online]. Available: https://mochajs.org/ [23] (2024) Jasmine Documentation Home. [Online]. Available: https: //jasmine.github.io/pages/docs_home.html [24] (2024) pytest: helps you write better programs. [Online]. Available: https://docs.pytest.org/en/7.1.x/ [25] (2024) unittest: Unit testing framework. [Online]. Available: https: //docs.python.org/3/library/unittest.html [26] (2024) JUnit 5.9.1 API. [Online]. Available: https://junit.org/junit5/doc s/current/api/ [27] D. Gonzalez, J. C. Santos, A. Popovich, M. Mirakhorli, and M. Nagappan, “A large-scale study on the usage of testing patterns that address maintainability attributes: Patterns for ease of modification, diagnoses, and comprehension,” in MSR, 2017, pp. 391–401. [28] P. S. Kochhar, F. Thung, N. Nagappan, T. Zimmermann, and D. Lo, “Understanding the test automation culture of app developers,” in ICST, 2015, pp. 1–10. [29] V. Khorikov, Unit Testing Principles, Practices, and Patterns. Manning, 2020. [30] “istsos2,” 2024. [Online]. Available: https://github.com/istSOS/istsos2 [31] F. Pecorelli, G. Catolino, F. Ferrucci, A. De Lucia, and F. Palomba, “Testing of mobile applications in the wild: A large-scale empirical study on android apps,” in ICPC, 2020, pp. 1–12. [32] P. S. Kochhar, D. Lo, J. Lawall, and N. Nagappan, “Code coverage and postrelease defects: A large-scale study on open source projects,” IEEE TR, vol. 66, no. 4, pp. 1213–1228, 2017. [33] P. S. Kochhar, F. Thung, D. Lo, and J. Lawall, “An empirical study on the adequacy of testing in open source projects,” in APSEC, vol. 1, 2014, pp. 215–222. [34] “vandium-node,” 2024. [Online]. Available: https://github.com/vandium -io/vandium-node [35] “smart-socket,” 2024. [Online]. Available: https://github.com/smartboot /smart-socket [36] “Usbserial,” 2024. [Online]. Available: https://github.com/felHR85/Usb Serial [37] “Adafruit_io_python,” 2024. [Online]. Available: https://github.com/ada fruit/Adafruit_IO_Python

15

[38] “Mqtt: The standard for iot messaging,” 2024. [Online]. Available: https://mqtt.org/ [39] “Lightweight m2m (lwm2m),” 2024. [Online]. Available: https://omas pecworks.org/what-is-oma-specworks/iot/lightweight-m2m-lwm2m/ [40] “iotedgedev,” 2024. [Online]. Available: https://github.com/Azure/ioted gedev [41] “azure-iot-sdk-python,” 2024. [Online]. Available: https://github.com/A zure/azure-iot-sdk-python [42] “ignite-3,” 2024. [Online]. Available: https://github.com/apache/ignite-3 [43] “adaptive-lighting,” 2024. [Online]. Available: https://github.com/basni jholt/adaptive-lighting [44] G. Petrovic and M. Ivankovic, “State of mutation testing at google,” in ICSE-SEIP, 2018, pp. 163–171. [45] M. Fazzini, C. Choi, J. M. Copia, G. Lee, Y. Kakehi, A. Gorla, and A. Orso, “Use of test doubles in android testing: An in-depth investigation,” in ICSE, 2022, pp. 2266–2278. [46] H. Zhu, L. Wei, M. Wen, Y. Liu, S.-C. Cheung, Q. Sheng, and C. Zhou, “Mocksniffer: characterizing and recommending mocking decisions for unit tests,” in ASE, 2021, pp. 436–447. [47] H. Zhu, L. Wei, V. Terragni, Y. Liu, S.-C. Cheung, J. Wu, Q. Sheng, B. Zhang, and L. Song, “Stubcoder: Automated generation and repair of stub code for mock objects,” ACM TOSEM, vol. 33, no. 1, pp. 1–31, 2023. [48] L. Xiao, G. Zhao, X. Wang et al., “An empirical study on the usage of mocking frameworks in apache software foundation,” Empirical Software Engineering, vol. 29, p. 39, 2024. [49] H. Zhu, V. Terragni, L. Wei, S.-C. Cheung, J. Wu, and Y. Liu, “Understanding and characterizing mock assertions in unit tests,” in FSE, 2025. [50] “Tasty mocking framework for unit tests in java,” 2024. [Online]. Available: https://site.mockito.org/ [51] “Standalone test spies, stubs and mocks for javascript.” 2024. [Online]. Available: https://sinonjs.org/ [52] D. Tiwari, M. Monperrus, and B. Baudry, “Mimicking production behavior with generated mocks,” TSE, 2024. [53] (2024) Shelly. [Online]. Available: https://www.shelly.com [54] “openhab,” 2024. [Online]. Available: https://github.com/openhab [55] T. Nguyen, U. Koc, J. Cheng, J. S. Foster, and A. A. Porter, “igen: dynamic interaction inference for configurable software,” in FSE, 2016, pp. 655–665. [56] A. Velez-Estevez, L. Gutierrez-Madronal, and I. Medina-Bulo, “Iot-teg 4.0: A new approach 4.0 for test event generation,” IEEE Transactions on Reliability, pp. 1–13, 2021. [57] P. Zhang, Y. Wang, X. Liu, Z. Lu, Y. Yang, Y. Li, L. Chen, Z. Wang, C.-A. Sun, X. Yu, and Y. Zhou, “Assessing effectiveness of test suites: What do we know and what should we do?” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 4, 2024. [58] A. Ahmad, F. Bouquet, E. Fourneret, F. Le Gall, and B. Legeard, “Model-based testing as a service for iot platforms,” in ISoLA, 2016, pp. 727–742. [59] S. Manandhar, K. Moran, K. Kafle, R. Tang, D. Poshyvanyk, and A. Nadkarni, “Towards a natural perspective of smart homes for practical security and safety analyses,” in IEEE S&P, 2020, pp. 482–499. [60] P. Mandal, S. Manandhar, K. Kafle, K. Moran, D. Poshyvanyk, and A. Nadkarni, “Helion: Enabling natural testing of smart homes,” in ESEC/FSE, 2023, pp. 2147–2151. [61] J. a. P. Dias, F. Couto, A. C. Paiva, and H. S. Ferreira, “A brief overview of existing tools for testing the internet-of-things,” in ICSTW, 2018, pp. 104–109. [62] D. Olianas, M. Leotta, and F. Ricca, “Matter: A tool for generating end-to-end iot test scripts,” Software Quality Journal, vol. 30, no. 2, pp. 389–423, 2022. [63] L. Hu, W. E. Wong, D. R. Kuhn, R. N. Kacker, and S. Li, “Ct-iot: a combinatorial testing-based path selection framework for effective iot testing,” Empirical Software Engineering, vol. 27, no. 2, 2022.

[64] M. Bures, B. Ahmed, V. Rechtberger, M. Klima, M. Trnka, M. Jaros, X. Bellekens, D. Almog, and P. Herout, “Patriot: Iot automated interoperability and integration testing framework,” in ICST, 2021, pp. 454–459. [65] J. Hwang, A. Aziz, N. Sung, A. Ahmad, F. Le Gall, and J. Song, “Autocon-iot: Automated and scalable online conformance testing for iot applications,” IEEE Access, 2020. [66] H. Kim, A. Ahmad, J. Hwang, H. Baqa, F. Le Gall, M. A. Reina Ortega, and J. Song, “Iot-taas: Towards a prospective iot testing framework,” IEEE Access, vol. 6, pp. 15 480–15 493, 2018. [67] C. Buratti, A. Stajkic, G. Gardasevic, S. Milardo, M. D. Abrignani, S. Mijovic, G. Morabito, and R. Verdone, “Testing protocols for the internet of things on the euwin platform,” IEEE Internet of Things Journal, vol. 3, no. 1, pp. 124–133, 2016. [68] B. Ramprasad, J. Mukherjee, and M. Litoiu, “A smart testing framework for iot applications,” in UCC Companion, 2018, pp. 252–257. [69] D. Amalfitano, N. Amatucci, V. De Simone, V. Riccio, and F. A. Rita, “Towards a thing-in-the-loop approach for the verification and validation of iot systems,” in SafeThings, 2017, pp. 57–63. [70] Z. Shu and G. Yan, “Iotinfer: Automated blackbox fuzz testing of iot network protocols guided by finite state machine inference,” IEEE Internet of Things Journal, vol. 9, 2022. [71] C. Paduraru, R. Cristea, and C. Staniloiu, “Riveriot - a framework proposal for fuzzing iot applications,” in SERP4IoT, 2021. [72] C. Tsigkanos, M. M. Bersani, P. A. Frangoudis, and S. Dustdar, “Edgebased runtime verification for the internet of things,” IEEE TSC, vol. 15, no. 5, pp. 2713–2727, 2022. [73] M. Lahami and M. Krichen, “A survey on runtime testing of dynamically adaptable and distributed systems,” Software Quality Journal, vol. 29, no. 2, pp. 555–593, 2021. [74] Z. Ling, J. Luo, Y. Xu, C. Gao, K. Wu, and X. Fu, “Security vulnerabilities of internet of things: A case study of the smart plug system,” IEEE Internet of Things Journal, vol. 4, no. 6, pp. 1899–1909, 2017. [75] E. Ronen, A. Shamir, A.-O. Weingarten, and C. O’Flynn, “Iot goes nuclear: Creating a zigbee chain reaction,” in IEEE S&P, 2017, pp. 195–212. [76] M. Alhanahnah, C. Stevens, and H. Bagheri, “Scalable analysis of interaction threats in iot systems,” in ISSTA, 2020, pp. 272–285. [77] C. Rojas Castillo, M. Marra, J. Bauwens, and E. Gonzalez Boix, “Outof-things debugging: A live debugging approach for internet of things,” The Art, Science, and Engineering of Programming, vol. 7, 2022. [78] K. Fang and G. Yan, “Iotreplay: Troubleshooting cots iot devices with record and replay,” in SEC, 2020, pp. 193–205. [79] M. Norris, Z. B. Celik, P. Venkatesh, S. Zhao, P. McDaniel, A. Sivasubramaniam, and G. Tan, “Iotrepair: Flexible fault handling in diverse iot deployments,” ACM Transactions on Internet of Things, vol. 3, no. 3, 2022. [80] Y. Chi, Y. Dong, Z. J. Wang, F. R. Yu, and V. C. M. Leung, “Knowledgebased fault diagnosis in industrial internet of things: A survey,” IEEE Internet of Things Journal, vol. 9, no. 15, pp. 12 886–12 900, 2022. [81] N. Khezemi, J. Baptiste Minani, F. Sabir, N. Moha, Y.-G. Guéhéneuc, and G. El Boussaidi, “A systematic literature review of iot system architectural styles and their quality requirements,” IEEE Internet of Things Journal, vol. 11, no. 23, pp. 37 599–37 616, 2024. [82] R. S. Da Silva, R. M. De Araújo Filho, M. H. G. De Oliveira Júnior, M. L. R. P. D. Scoton, P. H. L. Brito, E. M. Dias, and M. Oliveira, “Iot: applications, potentialities and challenges in the context of quality infrastructure 4.0,” in IOTSMS, 2024, pp. 90–97. [83] J. B. Minani, F. Sabir, N. Moha, and Y.-G. Guéhéneuc, “A multimethod study of internet of things systems testing in industry,” IEEE Internet of Things Journal, vol. 11, no. 1, pp. 1662–1684, 2024. [84] J.-W. Lin, N. Salehnamadi, and S. Malek, “Test automation in opensource android apps: A large-scale empirical study,” in ASE, 2020, pp. 1078–1089. [85] R. Coppola, M. Morisio, and M. Torchiano, “Scripted gui testing of android open-source apps: Evolution of test code and fragility causes,” Empirical Software Engineering, vol. 24, pp. 3205–3248, 2019. [86] “towards reliable smart home ecosystems,” 2024.

Related documents

• A systematic literature review on IoT application layer protocol testing and future research directions.#186220
• From Automation to Intelligence: Revolutionizing Microservices and API Testing with AI#97116
• "Will This Data Break My Task?" - Interactive Synthesis of Task-Aware Data Unit Tests#441894
• DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences#381804
• International Road Tunnel Fire Detection Research Project - Phase II Task 7: Full-Scale Ventilated Fire Tests in a Laboratory Tunnel for Study of Fire Detection Technologies#1111297