1
Characterizing Tests in IoT Software: Practices, Challenges and Opportunities
arXiv:2606.12592v1 [cs.SE] 10 Jun 2026
Rufeng Chen ∗ , Hengcheng Zhu † , Wuqi Zhang † , Zixu Zhou ∗ , and Lili Wei ∗ ∗ Department of Electrical and Computer Engineering, McGill University, Montreal, Canada † The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong SAR Emails: ∗ {rufeng.chen, zixu.zhou, lili.wei}@mcgill.ca † {hzhuaq, wzhangcb}@connect.ust.hk
Abstract—The Internet of Things (IoT) is experiencing rapid growth. Smart devices are emerging in smart homes and industrial applications, performing mission-critical tasks. Bugs in IoT software can lead to severe consequences. For example, a buggy smart lock can allow unauthorized access to a private property. Testing is a primary practice to expose software bugs and ensure software quality. However, little is known about how IoT software is tested. To bridge this gap, we conducted the first empirical study on test cases in open-source IoT software. Specifically, we evaluated the effectiveness of test cases in IoT software, explored the challenges inherent in testing IoT software, and analyzed the usage of mock objects. Our results indicate that while IoT software often contains a considerable number of tests, their effectiveness remains limited. We identified the primary challenges in testing IoT software as managing complex interactions with various external dependencies, such as other network-reliant IoT components, file systems, operating systems, and databases. We also observed that the use of mock objects in IoT software closely aligns with our identified testing challenges. This alignment demonstrates the potential of mocking as a solution to enhance test coverage and address the complexities of IoT software testing. Index Terms—Internet of Things, Testing, Empirical Study
I. I NTRODUCTION HE Internet of Things (IoT) is emerging—it is estimated that the global IoT market size will grow from $300.3B in 2021 to $650.5B by 2026 [1]. IoT interconnects various smart devices equipped with sensors and actuators to perform mission-critical tasks that demand high reliability, security, and responsiveness in real-world environments. For example, IoT systems can manage smart devices in a household, including light bulbs, heaters, and door locks. Software is the key bridge that connects devices to form the Internet of Things. Bugs in IoT software can significantly compromise the reliability and security of IoT systems and cause unprecedented consequences. For instance, a bug in the software of a smart lock can allow unauthorized access to a private property. Given the importance of IoT software, an increasing number of studies have been conducted to understand the challenges in developing reliable IoT software. These studies identified interesting IoT topics [2], shared development experiences [3], reviewed testing strategies [4], investigated IoT bugs and vulnerabilities [5]–[7], and examined the perspective of developers in IoT platforms [8]. Although these studies offer valuable perspectives on what makes IoT development and testing
T
Corresponding author: Lili Wei ([email protected]).
challenging, there remains a lack of empirical knowledge on how IoT software is tested in practice. Little is known about the characteristics and effectiveness of existing test cases: how well they cover the code, and what factors contribute to test gaps. These are the questions this study seeks to answer. The heterogeneous and dynamic interdependencies of IoT systems require thorough testing to ensure reliability and security. Such systems face challenges that are rarely encountered in traditional software operating under more homogeneous and predictable conditions. For example, unlike general software that predominantly relies on HTTP, IoT systems operate using a diverse range of communication protocols, including Zigbee, MQTT, or device-specific ones. This diversity requires specialized testing approaches to accurately simulate and test each protocol’s intricate behaviors and interactions. In addition, IoT software must function in both digital and physical domains, leading to a vast input space and numerous execution scenarios. Finally, the interplay between cloud, edge, and devicelevel code creates a multi-layered execution context, where bugs may emerge only through cross-layer interactions. In practice, such interactions are particularly difficult to test, since IoT systems are often developed as separate components (e.g., device firmware, gateways, and cloud services) maintained in different repositories, and developers frequently test individual software parts without full access to the complete end-toend system. However, the unique challenges of testing in such environments remain largely underexplored by existing research, leaving a gap in enhancing IoT software testing. To bridge this gap, we conducted the first empirical study by investigating the tests in open-source IoT software to answer the following research questions. RQ1 (Test Effectiveness): How effective are existing tests in IoT software? What statement coverage, branch coverage, and mutation scores can be achieved by the existing tests? Motivation: RQ1 can establish a basic understanding of the state of testing in IoT software. By evaluating the coverage metrics, we can estimate how thoroughly IoT systems are being tested and identify specific components or functionalities where tests remain limited. Result: We used static analysis on 824 IoT projects and conducted in-depth coverage and mutation analysis on 37 projects with runnable tests. The analysis of RQ1 revealed that the tests in IoT software achieve an average statement coverage of 65.2%, branch coverage of 53.4%, and mutation score of 39.9%. Such insufficient coverage results suggest the
2
need for a more in-depth exploration to identify the barriers to testing uncovered code (RQ2). RQ2 (Factors Contributing to Uncovered Code): What factors contribute to the presence of uncovered code by IoT software tests? Motivation: RQ2 aims to discover the underlying factors that prevent certain code segments in IoT projects from being adequately tested. Understanding these factors can guide the development of tools and methods to improve the effectiveness of IoT software testing. Result: Over half of the analyzed code segments (51.4%) remain untested primarily due to overlooked interactions with external dependencies, with untested communication scenarios representing the most significant barrier. These scenarios include network operations such as sending requests to external APIs, retrieving data from remote servers, and messaging between IoT devices. This highlights the critical need for new testing strategies tailored to the challenges posed by the excessive external dependencies of IoT software. RQ3 (Mocking in IoT Testing): How are mock objects used in IoT software testing? What are their roles and what dependencies do they simulate? Motivation: IoT systems highly rely on external dependencies, and mock objects are commonly used to simulate dependencies. This question investigates the use of mock objects in IoT software testing and explores the potential to leverage mock objects to address testing challenges for IoT software. Result: The vast majority (76.04%) of the mock objects in the test cases are simulating communication APIs such as sending and handling device interactions and network messages. This is in line with our previous finding that the major hurdle to improving code coverage in IoT software is untested communication scenarios. The prevalent use of mock objects emphasizes their potential application in enhancing test coverage and addressing the challenges of IoT software testing. Meanwhile, we observe that 99.89% of the analyzed mock objects involve defining or verifying specific dependency behaviors rather than simply acting as placeholders, making it challenging to effectively generate meaningful mock objects and enhance IoT software testing. Collectively, these research questions outline the status quo of IoT testing practices, identify gaps and challenges, and suggest future research directions. Our key observations suggest a promising future research direction to migrate mock objects between IoT projects by reusing the large number of readily available mock objects in open-source IoT projects. To summarize, this paper makes the following contributions: • We conducted the first empirical study focusing on characterizing the testing practices in open-source IoT software. Our study examines how developers test IoT systems and how these practices differ from those of traditional software. • We evaluated the prevalence and effectiveness of the existing tests in open-source IoT software. Our analysis shows that while IoT projects frequently include test cases, these tests often fall short in effectively detecting faults, highlighting the need for improved test adequacy. • We conducted an in-depth analysis to summarize the key challenges in improving the code coverage for testing in IoT
Device
Edge
Cloud
App
Fig. 1. The Overview of IoT Layers
software and identified the key challenge as how to properly trigger the diverse communication scenarios between IoT software and its dependencies. • We analyzed the usage of mock objects in the test cases and proposed a promising future research direction to improve IoT software testing by migrating mock objects. Our dataset, per-project metadata, and all results for reproducing the coverage and mutation analyses are publicly available in our replication package on GitHub [9]. II. BACKGROUND Internet of Things. The Internet of Things refers to a network of interconnected physical devices that collect, exchange, and process data through the internet. These devices range from consumer products, such as smart home appliances, to industrial systems, such as manufacturing sensors. By integrating sensing, computation, and communication capabilities, IoT enables real-time monitoring, automation, and data-driven decision-making across diverse domains. IoT Software. IoT software refers to software across the device, edge (gateway), cloud, and application layers in IoT systems whose primary responsibility is to manage, coordinate, or encapsulate device communication and protocollevel interactions between physical IoT devices and networked services. IoT software may overlap with embedded firmware or traditional cloud applications. However, IoT software is distinguished by its device-facing responsibilities, such as handling protocol-specific communication, managing device identity and state, and coordinating interactions across heterogeneous physical devices and services. IoT software is commonly described using a layered architecture [5], [10] consisting of application, cloud, edge, and device layers, as illustrated in Figure 1. In the literature, these layers are sometimes referred to using alternative terminology, e.g., the edge layer is also described as the network or connectivity layer, while the cloud layer is often described as the service layer or platform layer [11], [12]. Unlike traditional software, functionality in IoT systems often spans multiple layers and relies on device communication, external services, and runtime configuration. First, IoT software must handle device-specific protocols and complex state transitions across layers, making it difficult for test suites to exercise various execution scenarios and achieve strong coverage or faultdetection effectiveness. This directly motivates our study of test effectiveness and uncovered code in IoT projects (RQ1 and RQ2). Second, IoT software frequently relies on external parties such as MQTT brokers, cloud APIs, gateways, or physical devices to fulfill its functionality. As these dependencies are often unavailable during testing, developers commonly require
3
isolation and emulation mechanisms (e.g., mocks) to enable reproducible execution. In IoT systems, mocks are used to simulate interactions with external components, such as devices, network services, and data sources, enabling controlled and reproducible testing. This motivates our investigation of mocking practices in IoT software testing (RQ3). Application Layer: Provides user services, such as mobile apps, dashboards, and user interfaces. Unlike conventional applications, IoT applications often handle heterogeneous devices, real-time monitoring, and continuous connectivity. In this layer, mocks are used to simulate external data sources or service responses so that application logic can be tested without relying on live systems. • Cloud Layer: Manages large-scale data storage, processing, and analytics. It hosts APIs, machine learning models, and integration with other systems. IoT clouds are distinct in that they need to ingest massive streams of sensor data, ensure low-latency responses for control operations, and integrate with domain-specific protocols and devices. Here, mocks are used to emulate incoming data streams or external service interactions to enable scalable and repeatable testing. • Edge Layer: Bridges devices and cloud by using lightweight protocols (e.g., MQTT, CoAP) to filter data and enable low-latency responses under limited connectivity. In this layer, mocks are used to simulate device communications or network conditions to support testing of data processing and protocol handling. • Device Layer: Consists of hardware such as sensors, actuators, and embedded controllers that collect data and execute commands. IoT devices must interact with the physical world in real time. At this layer, mocks are used to emulate device inputs or outputs when physical hardware is unavailable or impractical to use during testing. •
Our study does not exclude any of these layers. We analyze the testing artifacts available in open-source IoT repositories as they appear in practice. Since IoT systems are often decomposed into separately maintained components, individual repositories may implement only a subset of the architecture. We therefore report the layer distribution of the studied repositories in Section III to clarify the scope of our analysis. At the device layer, IoT testing may involve Software-inthe-Loop (SiL), Hardware-in-the-Loop (HiL), or Model-inthe-Loop (MiL) techniques. In the open-source IoT repositories analyzed in this study, testing practices primarily adopt Software-in-the-Loop approaches, such as protocol-level simulation and virtual device responses, and we did not observe Model-in-the-Loop infrastructure. Accordingly, our empirical results primarily reflect software-level testing. Prior secondary studies have examined software testing challenges at different IoT layers. Garousi et al. provide a comprehensive survey of embedded software testing techniques, including device-level validation and simulation-based testing [13]. More recent surveys analyze testing challenges in distributed IoT systems spanning device, edge, and cloud layers, highlighting issues such as heterogeneity, limited observability, and complex dependencies [4], [14]. These studies collectively indicate that IoT software testing faces stronger
environmental constraints and dependency complexity than general-purpose software systems. Unlike traditional software systems, IoT software often interacts with various communication protocols, operates across heterogeneous hardware platforms, and adapts to dynamic and distributed environments. These characteristics pose unique testing challenges, particularly in designing effective test inputs. Test cases must follow communication protocols across devices, networks, and cloud services, while reflecting realistic environmental conditions (e.g., temperature, motion, or location) and continuous interaction with the physical world. III. DATA C OLLECTION Figure 2 shows an overview of the workflow of our empirical study. We started by collecting a dataset of 824 GitHub projects related to IoT. Makhshari and Mesbah [5] also collected a dataset of IoT software. Although their dataset contains 91 IoT projects, many of these projects are not suitable for dynamic test execution: 29 projects had been abandoned for at least four years, and 65 do not build or run. Since answering our research questions requires executing the tests of IoT software, we collected this new dataset with more up-to-date IoT software repositories, which are more likely to build and run successfully [15]. Using static analysis, we extracted test cases from these 824 projects and summarized their distribution. Among the 317 projects that contained test cases, 88 projects with more recent updates and higher popularity were further considered for detailed analysis. Although only 37 of these projects could be successfully executed, they are representative of the larger set, accounting for 12.5% of the total lines of code and 20.4% of the test cases, and covering all four IoT layers. These 37 projects are representative of our dataset, given their substantial number of test cases and their coverage of all IoT layers. These account for 12.5% of total LOC and 20.4% of test cases observed across the 317 projects. At the 95% confidence level, this sample size corresponds to a margin of error of approximately ±5% when calculated with respect to the total number of test cases, indicating that our dataset provides a representative basis for analysis. We compared the 37 selected projects with the full set of 317 IoT repositories at an aggregate level. As shown in Table I, both groups exhibit similar project sizes, measured by the average number of source files per project. The selected projects contain more test cases and substantially lower P/T (Production-to-Test) ratios on average, reflecting the presence of active and executable test suites required to obtain meaningful coverage and mutation results. Projects with sparse or inactive tests tend to yield trivial or uninformative dynamic analysis outcomes. Such outcomes are uninformative because they primarily reflect missing or broken test activity rather than the challenges of achieving effective testing in IoT software. Therefore, we exclude them to avoid drawing misleading conclusions. We note that this selection criterion may bias the reported coverage and mutation scores toward better-tested IoT repositories. Accordingly, the effectiveness metrics reported in this study should be interpreted as characterizing test-active and runnable IoT projects, rather
4
Github
Search for IoT-Related Topic
Test Identification Static analysis to extract test cases in the projects
824 IoT Projects
RQ1: Test Effectiveness Measuring statement coverage, branch coverage and mutation score achieved by the test cases.
Selected projects: star >= 100 & executable & updated after Jan 1 2022
RQ2: Testing Challenges Analyzing the factors leading to uncovered code segments.
37 Projects
RQ3: Mocking in IoT Testing Analyzing usages of mock objects in test cases of IoT software.
Fig. 2. The Overview of Our Empirical Study
than the entire population of IoT repositories. In addition, the selected projects collectively cover all IoT layers with comparable distributions, preserving architectural diversity. To validate this observation, we performed two-proportion z-tests for each layer and found no statistically significant differences between the selected subset and the full dataset. In line with our motivation, we treat each repository as an independently developed IoT software component, reflecting the realistic setting where developers often have limited access to the full IoT system. Thus, the analyzed projects may implement only one or a few IoT layers, while their tests still need to account for cross-layer dependencies through communication, configuration, and external services. Specifically, among the 317 analyzed repositories, 7 projects implement all four layers, while 88 projects implement only a single layer, with the remaining projects covering multiple layers. The complete mapping between projects and their corresponding layers is available in our online repository. In this section, we detail the data collection process and describe our approach for each research question separately before discussing their results. TABLE I C OMPARISON B ETWEEN S ELECTED AND F ULL -S ET I OT R EPOSITORIES Metric Number of repositories Average source files per project Average test cases per project Average P/T ratio Device-layer Edge-layer Cloud-layer Application-layer
37 Selected Projects
317 IoT Repositories
37 499.4 813.3 23.0 13 (35.1%) 12 (32.4%) 20 (54.1%) 8 (21.6%)
317 518.7 466.0 356.9 112 (35.3%) 85 (26.8%) 199 (62.8%) 90 (28.4%)
Note: P/T ratio denotes the Production-to-Test ratio.
A. Project Collection To determine the selection criteria for our IoT project dataset, we initially reviewed the dataset collected by Makhshari and Mesbah [5], which includes 91 IoT projects from GitHub. We analyzed their primary languages, defined as the language with the most files in each repository. JavaScript, Python, Java, and C emerged as the top languages, accounting for 78.9% of the projects. Based on this insight, we focused on JavaScript, Python, and Java. C was excluded due to the lack of standardized testing frameworks, which prevents automatic test case identification for further analysis. We adopted a similar approach to Makhshari and Mesbah [5] in collecting IoT projects by using GitHub topic tags. We sourced our dataset from GitHub, adhering to three
selection criteria: (1) The primary language of the project is either JavaScript, Python, or Java. (2) The projects should be tagged with at least one of the following keywords: IoT, Internet of Things, IoT-platform, IoT-device, IoT-Application. These tags were the top five most popular topics on GitHub containing “IoT” at the time of the experiment. (3) The project must have more than 10 stars to exclude toy projects. We collected a dataset comprising 824 open-source GitHub projects. The dataset is characterized by a diverse representation of primary languages: 371 Python projects (45%), 182 Java projects (22%), and 271 JavaScript projects (33%). The dataset includes renowned smart home platforms such as Home Assistant (58.7k stars) [16] and ThingsBoard (13.2k stars) [17], protocol libraries like paho.mqtt.python (1.8k stars) [18], backend SDKs such as kuzzle (1.2k stars) [19], and command-line interfaces like keylime (300 stars) [20]. This variety ensures that the dataset reflects the multifaceted nature of IoT projects. Our study focuses on productionlevel IoT software repositories in which testing is part of normal development practice. Repositories primarily designed for testing frameworks or benchmarking are excluded, as they do not reflect how testing is integrated into real IoT software projects. We acknowledge such repositories as a complementary population and identify their analysis as an important direction for future work. B. Dataset Characterization The analyzed repositories are primarily software components rather than complete IoT systems. We do not distinguish whether a repository represents an entire IoT system. Instead, each repository is characterized based on the IoT layers it implements (application, cloud, edge, and device). This characterization enables us to analyze the dataset’s architectural distribution and include projects from different layers. To make this distribution explicit, Table I reports the layer coverage of both the 37 selected runnable projects and the full set of 317 IoT repositories. The results show that the selected projects preserve similar architectural diversity. Device-layer projects account for 35.1% of the selected dataset versus 35.3% in the full pool, while cloud-layer projects represent 54.1% versus 62.8%. Application-layer components are also present (21.6% of the selected projects). To further clarify what kinds of IoT software artifacts are included in our dataset, we provide examples from each IoT layer observed in our dataset. At the device layer, repositories often implement device-facing communication logic
5
that directly interacts with sensors, actuators, or embedded protocols. For example, i2c-bus provides low-level access to hardware devices over the I2C bus. At the edge layer, repositories typically provide gateway or protocol-mediation components that bridge local devices with external services. For instance, iotedgedev supports IoT Edge deployment workflows. At the cloud layer, many repositories implement backend services or coordination platforms that manage fleets of connected devices through IoT protocols. A representative example is azure-iot-sdk-python, which provides device–cloud communication support and end-to-end messaging workflows. Finally, at the application layer, some repositories include user-facing automation and management logic for configuring IoT devices and workflows. For example, home-assistant-cli provides command-line interfaces for IoT administration. C. Test Identification 1) Approach: Based on the collected dataset, we first analyzed the prevalence of test cases in IoT software via static analysis. We adopted a two-stage approach to identify the test code in our subjects. In the first stage, we identified code patterns for common testing frameworks in our subjects. In the second stage, we designed automated techniques to identify test code in these projects based on the code patterns. Stage I: Code Pattern Derivation. In the first stage, we aimed to derive code patterns of commonly used testing frameworks. For each project, we inspected the source files and directory structures to check if they contained any test cases. We then identified the testing frameworks by examining the test code and build configurations. To derive the patterns, we analyzed test cases and relevant documentation, identifying common configurations and method annotations that define test cases. The resulting code patterns are as follows: • JAVA S CRIPT : J EST [21], M OCHA JS [22], and JASMINE [23] are the commonly used frameworks in our JAVA S CRIPT subjects. We identified each function closure passed to the testing API test() or it() as a test case. Each of these closures is recognized as an individual test case. • P YTHON : P Y T EST [24] and U NIT T EST [25] are the two commonly used testing frameworks. We identified P Y T EST test cases by matching the functions whose names start with test in the source files named with the prefix test_ or postfix _test. We identified test cases using the U NIT T EST framework by detecting methods within classes that inherit unittest.TestCase. • JAVA : JU NIT [26] is the de facto testing framework for JAVA projects. We identified the methods with annotation @Test as JU NIT test cases. Stage II: Automated Test Identification. Using the derived code patterns, we implemented an automated tool to identify test cases by matching common testing frameworks patterns. The tool first locates candidate test files, those named with “test” or placed in “test” directories. Then it performs pattern matching on the AST of the source file. For example, in Java projects using JU NIT, our tool identifies methods annotated with @Test. Details of all patterns used by our tool are
TABLE II D ISTRIBUTION OF T EST C ASES BY P RIMARY L ANGUAGES
# Test Case
# Projects 317
P/T Ratio
Total
Mean
Median
Mean
Median
147,736
466.0
34
356.9
8.97
available on our project’s website [9]. In each project, we analyzed the production and test code written in its primary language. During the analysis, we ignored the directories that are commonly used to store external libraries (e.g., node_modules, platform, and third-party). 2) Results: Among the 824 projects analyzed, 317 (38.5%) contain tests. This proportion notably exceeds what is observed within the broader spectrum of GitHub projects, where only 17.2% are reported to include tests [27]. Furthermore, this rate is substantially higher than the 14.2% within Android applications as identified in previous research [28]. Such findings highlight the pronounced emphasis on testing within IoT software projects, surpassing general software development trends and specific domains like Android apps. Table II details the test case distribution across these 317 projects, with an average of 466 test cases per project and a median of 34. The absolute number of test cases may not reflect the actual test abundance since different projects can have different scales. To account for this variation, we introduce the Production-to-Test (P/T) ratio, defined as the ratio of lines of production code to lines of test code. A smaller P/T ratio corresponds to a larger proportion of test code, indicating that the underlying project is more extensively tested. P/T ratio =
Lines of Production Code Lines of Test Code
Figure 3 illustrates the distribution of the P/T ratio. In the context of projects with commendable code coverage, a P/T ratio lower than 1 is typically indicative of robust testing practices [29]. However, among our examined IoT projects, only 13 (4.1%) exhibited a P/T ratio below 1, while a substantial portion (124 out of 317, or 39.1%) have a P/T ratio exceeding 17.19, indicating that many IoT projects include very limited test code. The project with the highest P/T ratio, Istsos2 [30], an IoT server encompassing both a backend server and a GUI application, registered a P/T ratio of 68051.88, highlighting a critical lack of testing relative to its production code. These results suggest that despite the presence of test cases, testing in many IoT projects remains insufficient compared to code size. D. Test Granularity The collected test cases can have varying granularities as unit tests, integration tests, or system tests. To gain insight into this, we performed a closed coding analysis following the deductive approach by Pecorelli et al. [31], classifying 382 Python, 383 Java, and 379 JavaScript tests (95% CL, 5% ME). This classification was based on criteria such as
6
Frequency
30