Maintaining IoT Device Identification under Concept Drift via Budget-Aware Traffic Labeling

1

Maintaining IoT Device Identification under Concept Drift via Budget-Aware Traffic Labeling

arXiv:2608.15465v1 [cs.NI] 16 Aug 2026

Shayan Azizi, Norihiro Okui, Masataka Nakahara, Ayumu Kubota, Gustavo Batista, and Hassan Habibi Gharakaheili

Abstract—Identification of IoT device types from passive traffic is increasingly used for security management in enterprise and ISP networks. However, the performance of machine learningbased classifiers gradually degrades under concept drift as device behavior evolves. Therefore, maintaining classification performance requires periodic retraining with newly labeled deployment traffic, but labeling traffic at scale is operationally expensive. The practical challenge is determining how much and which deployment traffic instances to label for maintaining classification performance. We show that these two decisions should be treated separately. While retraining solely on instances selected by a drift detector is prone to systematically overlooking parts of the emerging behavioral space, uniformly sampled deployment traffic captures more representative behavioral changes. Consequently, drift detection is more effective at determining how much deployment traffic should be labeled than at selecting which deployment instances to label. We make three contributions. (1) We conduct a two-year longitudinal study of IoT traffic and characterize how behavioral evolution manifests across device classes, demonstrating how it induces concept drift from the perspective of deployed classifiers and how retraining with newly labeled traffic restores classification performance. (2) We develop a conformity-based drift detector that models class-conditional behavioral models directly from raw traffic features. The detector provides feature-level explanations of behavioral evolution and identifies deployment traffic exhibiting feature-level non-conformity. (3) We demonstrate that adjusting the traffic labeling rate according to the observed behavioral evolution, combined with uniform traffic sampling, maintains classifier performance more effectively than detector-guided sample selection and is beneficial to managing the traffic labeling effort. We further show that this strategy performs comparably to confidence-guided adaptation while providing feature-level explanations. Our evaluation uses 3.8 million IPFIX flow records collected from 21 IoT types over more than 2 years. Index Terms—IoT Device Identification, Concept Drift, Drift Detection, Model Adaptation

I. I NTRODUCTION The rapid growth of Internet-of-Things (IoT) devices in residential and enterprise networks has made visibility into connected devices essential for effective security management. Many IoT devices run lightweight firmware [1], receive limited long-term vendor support [2], and suffer from authentication weaknesses [3], [4], making them particularly susceptible to S. Azizi and H. Habibi Gharakaheili are with the School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, NSW 2052, Australia (e-mails: {s.azizi, h.habibi}@unsw.edu.au). G. Batista is with the School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia (e-mail: g.batista@unsw.edu.au). N. Okui, M. Nakahara and A. Kubota, are with KDDI Research, Japan (e-mails: {no-okui, ms-nakahara, ay-kubota}@kddi.com).

cyber compromise. These risks range from the possibility of device seizure, which can be exploited for large-scale volumetric attacks [5]–[7], to user behavior tracking and private data exfiltration [8]–[14]. These threats are particularly important for Internet Service Providers (ISPs), which provide connectivity to millions of residential and enterprise subscribers. Cyber incidents originating from vulnerable IoT devices can expose ISPs to malicious traffic, service disruption, and resource exhaustion [15]. Enterprise network operators face similar challenges and must continuously discover and monitor these assets to maintain network security. Mitigating the cyber threats posed by IoT devices requires identifying the devices so that appropriate security policies can be applied. To support this, the community has curated vulnerability inventories [16], [17] and traffic allowlists [18]– [20]. Leveraging these resources requires operators to first identify IoT devices from network traffic so that devicespecific policies can be applied. For example, ISPs are highly interested in scalable techniques that infer connected devices from passively collected traffic at their network vantage points [15], [21], enabling subscribers to be informed about vulnerable devices and appropriate mitigation strategies. Accordingly, mainstream IoT identification systems rely on Machine Learning (ML) classifiers trained on network traffic collected at different granularities [15], [22]–[26]. As these classifiers become operational components of enterprise and ISP security infrastructures, maintaining their accuracy throughout longterm deployment becomes an operational requirement. Once deployed, IoT device classifiers cannot be expected to remain accurate indefinitely. As device behaviors evolve over time because of firmware updates, software changes, and changing operating environments, the underlying traffic patterns also evolve. From the perspective of a deployed classifier, this behavioral evolution manifests as concept drift, a gradual degradation in classifier performance. Therefore, maintaining classifier accuracy becomes an operational problem instead of a one-time training problem. Existing work has reported classifier degradation under concept drift [27]–[30], investigated pseudo-labeling using foundation models [31], or proposed early mitigation strategies with limited efficacy [30]. However, none addressed the practical problem of maintaining deployed IoT traffic classifiers over long periods under limited traffic-labeling budgets. Since labeling deployment traffic is operationally expensive [32], operators can label only a limited fraction of deployment traffic. This introduces two practical questions: how much deployment traffic should be labeled, and which

2

deployment instances should be selected. Existing work has largely focused on the latter by using drift detectors to identify informative deployment instances for labeling. In this paper, we challenge this assumption. We show that the detectorselected instances are biased toward only a subset of the emerging behavioral patterns, causing retrained classifiers to overlook other aspects of behavioral evolution. In contrast, uniformly sampled deployment traffic captures a broader range of evolving behaviors and consistently provides more effective classifier maintenance. Having established that uniformly sampled deployment traffic is preferable for retraining, the remaining question is how much traffic should be labeled at each stage of deployment. Since the degree of behavioral evolution varies over time, a fixed labeling rate is unlikely to utilize a limited operational labeling budget efficiently throughout the lifetime of the classifier. This calls for a different role of drift detection. Existing approaches estimate drift either from classifier confidence scores [33]–[37] or from learned feature embeddings [38], [39]. Confidence-based methods provide limited visibility into underlying traffic changes [40], while embedding-based approaches increase computational complexity and obscure feature-level interpretation. Neither provides an interpretable behavioral signal for guiding budget-aware classifier maintenance. Accordingly, we develop a budget-aware adaptation framework that separates the two decisions introduced above: determining how much deployment traffic should be labeled and deciding which deployment instances should be selected. Our conformity-based detector measures behavioral evolution by identifying feature-level non-conformity with respect to classconditional traffic distributions. It also provides interpretable feature-level explanations for the detected behavioral changes. The resulting behavioral signal guides the traffic labeling rate, while deployment traffic itself is sampled uniformly for retraining. Newly labeled traffic is then incorporated into classifier retraining, and the detector is updated accordingly, enabling the continuous maintenance of IoT traffic classifiers as device behaviors evolve over time. Our specific contributions are as follows. (1) We conduct a longitudinal study (§IV) of IoT traffic spanning more than two years and 21 real device types, and characterize how behavioral evolution manifests across device classes and how it induces concept drift in deployed classifiers. We further demonstrate that incorporating freshly labeled deployment traffic enables effective classifier maintenance. (2) We develop an instance-level conformity-based detector (§V) that models class-conditional feature behavior and provides feature-level explanations of behavioral evolution. The resulting behavioral signal is later used to guide the utilization of a limited labeling budget throughout deployment. (3) We demonstrate that adjusting the traffic labeling rate according to the observed behavioral evolution effectively maintains classifier performance over two years of deployment (§VI). Compared with different confidence-based adaptation strategies, the proposed approach achieves comparable or better performance at a lower labeling cost, while additionally identifying the traffic features associated with behavioral evolution.

II. R ELATED W ORK Modeling network behavior of IoT devices: A large body of work has shown that IoT devices exhibit distinctive network behaviors that can be modeled from network traffic for device identification and anomaly detection [15], [22]–[27], [41]–[43]. Existing approaches use packet-level features (e.g., protocol fingerprints, domain names, TLS metadata, and TCP options), flow-level statistics (e.g., packet and byte counts, timing characteristics, and periodicity), or combinations of both to characterize IoT behaviors. These behavioral models have enabled accurate device identification and behavioral monitoring in enterprise, residential, and ISP environments. However, they generally assume that the learned behavior remains representative after deployment or that updated labeled traffic is readily available for retraining. Our work instead investigates how these behavioral models evolve over time and how they can be maintained efficiently under limited trafficlabeling resources. Concept drift and drift detection in networks and systems: Behavioral evolution in IoT traffic over time or across deployment environments has been observed in several recent studies, often through its impact on classifier performance [26]–[30], [41], [44], [45]. However, most studies report this degradation as an observed effect, offering limited insight into the underlying behavioral evolution or into how classifiers should be maintained over long-term deployment. Some works correlate performance degradation with changes in the distribution of the most important features identified by the classifier [29] and [28]. Others use accuracy degradation as a signal to trigger model updates [44]. However, such approaches often assume access to ground-truth labels, which is unrealistic in practical deployments. Similarly, [46] assumes an online learning setting where all deployment traffic is labeled, but does not address selective labeling. Alternative approaches attempt to mitigate drift without explicit detection. For example, work in [30] selects from a pool of historical models based on the distributional similarity of prediction scores to the training scores of each model. However, such methods cannot handle the previously unseen drift patterns, nor do they identify drift instances for targeted adaptations. Other studies examine variability in IoT traffic behavior, such as changes in cloud endpoints over time and across deployments [20]. Meanwhile, [47] focuses on label shift, in which class prevalence changes while per-class distributions remain stable, which is outside the scope of our work. Beyond IoT traffic classification, concept drift has also been recognized as a general problem in networking and systems applications. Some works demonstrate classifier degradation under evolving traffic patterns [48], while others explore detection or adaptation under different assumptions. For example, work in [49] detects drift in time-series forecasting (i.e., a regression problem instead of classification) by monitoring prediction error. Other works employ auxiliary models to generate pseudo-labels, enabling adaptation in the absence of ground truth [31], [50], but they introduce additional model complexity and potential error propagation. The authors of [51] propose an in-network drift detection mechanism, but

3

their method still relies on ground-truth labels, making it less suitable for real-world scenarios where labeled data is scarce or unavailable. Model selection approaches, such as [52], choose among multiple pre-trained models, but they are not designed to handle previously unseen behavioral patterns. Transfer learning [53] and domain adaptation methods [54], [55] aim to improve the model robustness to changes but do not address instancelevel drift detection. Beyond these adaptation strategies, a large body of work focuses specifically on detecting concept drift. Existing drift detection methods generally fall into three categories. The majority detect drift from classifier output confidence [33]– [37], [56], [57], which provides limited visibility into the underlying behavioral changes. Another line of work detects drift in learned feature embeddings [38], [39]. Although these methods improve robustness to complex traffic patterns, they rely on latent representations that increase computational complexity and obscure feature-level interpretation. A smaller body of work detects drift directly from raw input features [58]–[60], but these methods focus on aggregate distribution shifts rather than instance-level detection and have not been evaluated for long-term IoT traffic classification. A related line of work models feature distributions for downstream tasks, such as noisy-label correction in anomaly detection [61]. Unlike our marginal modeling approach, these methods rely on deep generative models, such as MADE [62], to model the joint feature distribution, making them computationally expensive and data-hungry. Hence, these traffic modeling approaches are less suitable for IoT traffic settings where limited samples are available per class. Collectively, these studies demonstrate the importance of detecting concept drift but offer limited insight into the behavioral evolution that drives classifier degradation and into how that information should inform long-term classifier maintenance under limited traffic-labeling budgets. These observations motivate approaches that directly model behavioral evolution from raw traffic features while providing interpretable drift detection and efficient classifier maintenance. Drift adaptation: Maintaining a classifier under concept drift requires not only detecting behavioral changes but also updating the classifier using newly labeled deployment traffic. Existing research has chiefly focused on the mechanics of model adaptation once labeled data are available rather than on how scarce labeling resources should be allocated during deployment. Numerous approaches have been proposed for model adaptation [63]. These approaches include model retraining [38], [64], [65], ensemble techniques [66], [67], gradient-based updating [39], [68] or warm-start re-training according to similarity to previous data [57]. While the majority of adaptation approaches prioritize the most recently observed data, either explicitly or implicitly, to improve responsiveness to concept drift, doing so may increase susceptibility to catastrophic forgetting [49], [69]. Throughout this paper, we adopt cumulative retraining during the evaluation to isolate the effect of traffic-labeling strategies from the choice of adaptation algorithm. A comprehensive comparison of alternative adaptation mechanisms is beyond the scope of this work.

TABLE I: The 21 IoT devices studied. Device Make-and-Model Amazon Echo Show 5 Amazon Echo Gen 2 Google Nest Mini Amazon Echo Dot with Clock Google Home Sony Smart Speaker Line Clova Wave Apple Homepod Google Chromecast Amazon Fire TV Stick 4K I-O DATA QWatch Camera Wansview Q5 Panasonic Doorphone TP-Link Kasa Smart LED Bulb TP-Link Kasa Smart Plug TP-Link Kasa Pro Nature Remo Philips Hue Bridge Panasonic Home Unit JVCKENWOOD Hub Withings Nokia Body

Category Voice assistant Voice assistant Voice assistant Voice assistant Voice assistant Voice assistant Voice assistant Voice assistant Streamer Streamer IP Camera IP Camera Doorbell Smart LED Smart plug Smart power strip Remote control Smart home hub Smart home hub Smart home hub Smart scale

III. I OT T RAFFIC DATA We use a longitudinal IoT dataset collected over more than 2 years from 21 device types (make-and-models) in a controlled lab testbed. The devices span multiple categories, including smart speakers, cameras, streaming devices, smart plugs, smart hubs, and home automation equipment. Table I summarizes the inventory of device types. In this study, we focus on TCP/443 (presumably HTTPS) traffic due to its widespread adoption across IoT devices and the rich statistical semantics captured by flow-level features. Network flows are represented as IPFIX records exported from raw PCAP files using YAF [70]. Each flow is described using 22 statistical features, including packet count, byte count, packet size, and timing statistics measured in both the forward and reverse directions. Table II summarizes the traffic features. In certain cases, late-arriving packets for an alreadyexported flow can result in additional redundant flow records with limited or noisy semantics. We filter out such redundancies based on temporal proximity among flows that share the same 5-tuple. We use traffic collected between 1 Oct 2021 and 31 Dec 2021 for training and validation, and data between 1 Jan 2022 and 31 Dec 2023 for evaluation. The resulting dataset consists of approximately 260K training flows, 113K validation flows, and 3.38M deployment-phase flows spanning two years of operation. Our data collection and analysis raise no ethical concerns. IV. B EHAVIORAL E VOLUTION OF I OT T RAFFIC UNDER L ONG -T ERM D EPLOYMENT In this section, we investigate how behavioral evolution during long-term deployment affects machine learning-based IoT traffic classification. We first establish a baseline IoT traffic classifier and evaluate its performance over a twoyear deployment period without model updates. We then analyze the observed performance changes to characterize the

4

1.00

packetTotalCount octetTotalCount smallPacketCount largePacketCount nonEmptyPacketCount dataByteCount averageInterarrivalTime

Description # of packets. # of bytes (header and payload). # of packets with less than 60 Bytes of payload. # of packets with at least 220 Bytes of payload. # packets with payload. # of payload bytes. Average time (ms) between packets.

firstNonEmptyPacketSize

# of bytes in the payload of the 1st non-empty packet.

maxPacketSize

largest payload size (bytes). std. deviation of payload size for first 10 non-empty packets. std. deviation of time (ms) between first 10 non-empty packets.

standardDeviationPayloadLength standardDeviationInterarrivalTime

Abbr. Name (reverse) pktTotCnt (rPktTotCnt) octTotCnt (rOctTotCnt) sPktCnt (rSPktCnt) lPktCnt (rLPktCnt) nonEmptPktCnt (rNonEmptPktCnt) datByteCnt (rDatByteCnt) avgIntArrTime (rAvgIntArrTime) 1stNonEmptPktSz (r1stNonEmptPktSz) maxPktSz (rMaxPktSz) stdDevPLlen (rStdDevPLlen) stdDevIntArrTime (rStdDevIntArrTime)

underlying behavioral evolution, showing that it is continuous, heterogeneous across device classes, and not necessarily monotonic. Finally, we demonstrate that classifier performance can be effectively restored by incorporating freshly labeled deployment traffic, establishing that long-term classifier maintenance is feasible and motivating the search for practical maintenance strategies under limited labeling resources. A. IoT Traffic Classification Setup To study the impact of behavioral evolution on IoT traffic classification, we first establish a baseline traffic classifier. We use a Random Forest classifier, which has demonstrated strong performance in network traffic classification tasks [26], [27], [71]. The classifier is trained on 70% of the traffic flows collected between 1 Oct 2021 and 31 Dec 2021, with the remaining 30% reserved for validation. The split is performed randomly rather than chronologically so that both partitions are drawn from the same underlying distribution. The classifier hyper-parameters are selected using 10-fold cross-validation on the training data. The resulting model consists of 100 decision trees with a maximum depth of 30 and a minimum of 5 samples per split. Evaluated on the validation set, the classifier achieves a macro-averaged F1score of 0.974. This classifier serves as the baseline model for the remainder of the paper to analyze behavioral evolution and evaluate maintenance strategies under long-term deployment. B. Behavioral Evolution under Long-Term Deployment After establishing the baseline classifier, we deploy it on previously unseen traffic collected continuously from January 2022 onward without updating the model. To assess the performance of the classifier, we divide the deployment traffic into consecutive 7-day windows from the beginning of 2022 until the end of 2023. Fig. 1 summarizes the classifier’s performance on a weekly basis over the subsequent two years. A progressive decline in the macro F1-score is observed beginning around mid-2022, indicating that the traffic patterns learned during training gradually become less representative of the deployment traffic. From the perspective of the deployed classifier, this behavioral evolution manifests as concept drift, progressively degrading classification performance. Interestingly, the degradation is not monotonic. Instead, classifier

Classifier performance

TABLE II: Traffic features in IPFIX flow records. Feature Name

macro F1-score macro recall macro precision

0.90 0.80 0.70 0.60

0.50 1 Jan'22 1 Apr'22 1 Jul'22 1 Oct'22 1 Jan'23 1 Apr'23 1 Jul'23 1 Oct'23 1 Jan'24

Time

Fig. 1: Behavioral evolution during long-term deployment progressively degrades classifier performance. The partial restoration observed in late 2023 suggests that device behaviors may evolve non-monotonically away from the training distribution. performance exhibits a partial restoration during the second half of 2023, suggesting that IoT device behaviors evolve dynamically rather than continuously drifting away from the training distribution. Behavioral evolution manifests differently across IoT device classes. To characterize this heterogeneity, we analyze the temporal recall of individual device classes. Recall measures the fraction of instances from each class that continue to be correctly identified over time and therefore provides an indirect indication of how closely deployment traffic conforms to the learned behavioral patterns. Based on the temporal evolution of recall, we observe three representative patterns of behavioral evolution across IoT devices: (i) persistence (as exemplified by Philips Hue Bridge in Fig. 2a and demonstrated by five other devices), where no significant drop in recall is observed, (ii) decay (represented by Google Nest Mini in Fig. 2b and demonstrated by ten other devices), which is characterized by an overall decrease in performance over time, and (iii) restoration (represented by JVCKENWOOD Hub in Fig. 2c and demonstrated by three other devices), in which the temporal patterns of recall show periods of restoration. To understand why these different evolutionary patterns arise, we examine the JVCKENWOOD Hub, a representative device that exhibits behavioral restoration. Fig. 2d, illustrates a t-SNE projection [72] of its traffic across different time periods: the training data pertaining to JVCKENWOOD Hub (void boxes), the collective training data of other classes (filled boxes), the test data pertaining to the high recall lasting from 1 Jan’22 to 25 Feb’23 (the plus sign markers), the low performance period of 25 Feb’23 to 5 Aug’23 (the crosses) and the restoration period of 5 Aug’23 to the end of the test phase (triangles). Note that, given the purpose of our analysis, for the two high-performing periods and the low-performing period, we have only included the correctly and incorrectly classified instances, respectively. The training data for other classes is included in deriving the t-SNE to help with contrasting and defining the patterns for JVCKENWOOD Hub against the rest of the classes. During the high-performing period, deployment traffic conforms to the behavioral patterns learned during training, even though it is not statistically identical to the training traffic. For example, no test instances fall into the data mode indicated

1.0

1.0

0.8

0.8

0.6

0.6

Recall

Recall

5

0.4 0.2 0.0 1 Jan'22

0.4 0.2

Original classifier Fully adapted classifier 1 Jul'22

Original classifier Fully adapted classifier

1 Jan'23

Time

1 Jul'23

0.0 1 Jan'22

1 Jan'24

(a) Persistent behavior in Philips Hue Bridge.

1 Jul'22

1 Jan'23

Time

1 Jul'23

1 Jan'24

(b) Behavioral decay in Google Nest Mini.

1.0

Recall

0.8 0.6 0.4 0.2 0.0 1 Jan'22

Original classifier Fully adapted classifier 1 Jul'22

1 Jan'23

Time

1 Jul'23

1 Jan'24

(c) Behavioral restoration in JVC Hub.

(d) t-SNE visualization for JVC Hub.

Fig. 2: Behavioral evolution manifests differently across IoT device classes. Representative examples show persistent behavior (a), behavioral decay (b), and behavioral restoration (c), while the corresponding t-SNE projection (d) illustrates how changes in traffic distribution explain these different evolution patterns. The black curves show the upper-bound maintenance performance obtained by retraining with all newly labeled deployment traffic.

C. Classifier Maintenance Using Fresh Traffic The previous analysis showed that IoT traffic behaviors evolve throughout long-term deployment. A natural question, therefore, arises: how can classifier performance be maintained

1.0 0.9

Macro F1-score

by the dash-dotted box on the left side of Fig. 2d. In contrast, during low performance periods, instances deviate from the training distribution and partially overlap with other classes (filled boxes). These observations suggest that the key question is not whether deployment traffic is statistically identical to the training data [58], [59], but whether it continues to conform to the behavioral patterns captured during training. During the second high-performance period (triangles), instances exhibit new yet consistent patterns that differ from those observed during training. This shows that behavioral evolution does not necessarily lead to classifier degradation. Instead, deployment traffic may evolve while remaining sufficiently consistent with the learned decision boundaries to preserve classification performance. These observations suggest that maintaining classifier performance requires more than correcting misclassified instances. As device behavior evolves, newly emerging behavioral patterns should also be incorporated into the learned representation of each device class. The following subsection demonstrates that such maintenance is feasible when freshly labeled deployment traffic is incorporated into classifier retraining.

0.8 0.7 0.6

Full adaptation No adaptation

0.5 1 Jan'22 1 Apr'22 1 Jul'22 1 Oct'22 1 Jan'23 1 Apr'23 1 Jul'23 1 Oct'23 1 Jan'24

Time

Fig. 3: Upper-bound maintenance performance established by retraining the classifier using exhaustively labeled deployment traffic. This experiment demonstrates that long-term classifier maintenance is feasible, but highlights the prohibitive labeling effort required in practice. as these behaviors evolve? To answer this question, we first consider an idealized setting in which every deploymentphase traffic instance is labeled and incorporated into model retraining. Specifically, at the end of each 7-day deployment window, all newly observed traffic is labeled and appended to a cumulative training dataset, which is then used to retrain the classifier. Then, the model is assessed on the subsequent window. Although such exhaustive labeling is operationally expensive, it establishes a practical upper bound on the performance achievable through classifier maintenance.

6

The resulting upper-bound maintenance performance is shown by the dashed curve in Fig. 3. Retraining the classifier using all newly labeled deployment traffic almost completely eliminates the degradation observed in Fig. 1. In particular, the recall profiles of all device classes become persistent after maintenance, indicating that the classifier successfully incorporates the evolving behavioral patterns observed during deployment. These results establish that long-term classifier maintenance is feasible by continuously retraining on newly labeled deployment traffic. However, achieving this performance requires labeling approximately 3.4 million traffic instances over the two-year deployment period, corresponding to about 33,000 instances per week on average. Such labeling requirements are operationally prohibitive in practice [73]. The remaining challenge is, therefore, how long-term classifier maintenance can be achieved under realistic labeling constraints. Practical network operators can only label a small fraction of deployment traffic, making exhaustive maintenance infeasible. This raises a practical question: how can evolving behaviors be identified from deployment traffic to enable classifier maintenance without relying on exhaustive labeling? V. C ONFORMITY-BASED D RIFT D ETECTION IN I OT T RAFFIC F EATURES The previous section established that behavioral evolution affects deployed classifiers and that classifier maintenance is feasible given sufficient labeled traffic. The remaining challenge is therefore to efficiently identify behavioral evolution from deployment traffic so that classifier maintenance can be achieved under limited labeling resources. To address this challenge, we develop a conformity-based detector that operates directly on raw traffic features. Unlike existing approaches that infer drift from classifier confidence scores or learned feature embeddings, our method models class-conditional behavioral patterns and detects behavioral evolution through feature-level non-conformity. The detector is

architecturally decoupled from the underlying classifier: it uses the predicted class only to select the corresponding behavioral model, while the detection itself is performed entirely in the original feature space. This design enables feature-level explanations of behavioral evolution and provides the behavioral signal required for practical classifier maintenance. A. Drift As Seen in Raw Traffic Features The previous section showed that behavioral evolution is reflected through changes in the network traffic generated by IoT devices. Now, our goal is to characterize behavioral evolution directly from raw traffic features. Existing approaches in the data mining literature [58]–[60], [74], typically quantify changes by measuring distributional divergence between training and deployment data. Although such measures provide a notion of global distributional change, they are less suited to characterizing behavioral evolution in network traffic, which is often heterogeneous and multi-modal. To illustrate this limitation, we first consider the Philips Hue Bridge during the first week of deployment, where the classifier maintains nearly perfect recall (Fig. 2a). The corresponding feature distributions are shown in Fig. 4. Although the deployment observations occupy only a subset of feature values seen during training, every observed value remains within previously learned behavioral patterns. In other words, deployment traffic conforms to the training behavior without reproducing the complete training distribution. A distributional divergence measure would nevertheless report a noticeable difference between the training and deployment data because several modes observed during training are absent during the deployment window, even though the classifier maintains nearly perfect recall. These observations suggest that behavioral evolution should be characterized through conformity to previously learned behavioral patterns rather than strict statistical equivalence of traffic distributions.

Fig. 4: Feature-level view of a correctly classified deployment window for Philips Hue Bridge. Although the observed feature values (red triangles) occupy only a subset of the training observations (gray markers), they remain within previously learned behavioral patterns, illustrating behavioral conformity despite distributional differences.

7

(a) A Google Nest Mini flow misclassified as an Apple HomePod.

(b) A misclassified Sony Speaker flow as an Apple HomePod.

Fig. 5: Drifted instances reveal feature-level non-conformity with respect to the predicted class. Even when both flows are misclassified as Apple HomePod, different features expose their deviation from the training distribution. Behavioral evolution that leads to classifier degradation exhibits a different characteristic. As a representative example, we consider a Google Nest Mini flow collected during the period of behavioral decay (Fig. 2b) that is misclassified as an Apple HomePod. As illustrated in Fig. 5a, only two feature values (i.e., rOctTotCnt and rDatByteCnt) fall outside the behavioral patterns learned for the predicted class, while the remaining features do conform. We next consider another deployment instance in which a Sony Speaker flow is misclassified as an Apple HomePod. Unlike the previous example, only a single feature, stdDevIntArrTime, exhibits non-conformity with respect to the learned behavioral patterns (Fig. 5b). The key observation here regarding the behavioral evolutions that lead to misclassification is that an instance may move into the decision region of an incorrect class while conforming to that class only partially, rather than completely. This partial conformity provides the basis for detecting behavioral evolution from raw traffic features. The provided examples show that non-conformities are both localized (i.e., for a given instance, only a subset of the traffic features typically exhibits non-conformity) and heterogeneous (i.e., different instances may evolve through different subsets of features). Furthermore, some behavioral changes may not immediately affect classifier performance. While feature-level conformity should not be viewed as a complete description of behavioral evolution, it provides an interpretable approximation that captures many practically relevant behavioral changes directly in the original feature space. The remainder of this section develops and evaluates a principled and computationally efficient way to identify these non-conforming behaviors.

B. Drift Instance Detection Building on the notion of conformity introduced in the previous subsection, we now develop an instance-level detector to identify behavioral evolution in deployment traffic. The key idea is to compare each feature value of a deployment instance against the behavioral patterns learned for its predicted device class. An instance is regarded as behaviorally non-conforming

if one or more of its feature values fall outside the corresponding class-conditional conformity regions. To realize this idea, we model the behavioral range of every feature for each device class. Using the most recent labeled traffic, we estimate for every class c and feature f a probability density function (PDF) p̂f,c together with a conformity threshold θf,c . During deployment, given a feature vector x = (x1 , x2 , ..., xn ), we first apply the traffic classifier M to obtain the predicted class M (x). The detector then evaluates each feature against the behavioral model for that predicted class. If any feature is non-conforming, the instance is Wnflagged as exhibiting behavioral evolution: D(x; M ) = i=1 1p̂fi ,M (x) <θfi ,c . In the following subsections, we explain how the behavioral models are constructed from raw traffic features and how the corresponding conformity thresholds are determined. C. Modeling Feature Distributions Having established that behavioral evolution can be characterized by feature-level conformity, we now require a model that represents the behavioral range of each traffic feature. Such a model should faithfully capture the variability observed during training while remaining sufficiently flexible to accommodate the heterogeneous and multi-modal nature of IoT traffic. We, therefore, model the behavior of each feature independently using probability density functions estimated directly from the training data. Methods for modeling data distributions are broadly categorized as parametric and nonparametric. Parametric methods assume a predefined family of distributions, which is unsuitable for complex and multimodal traffic features. Non-parametric methods, in contrast, allow the data itself to define the shape of the distribution and are therefore better suited to our setting. Among non-parametric approaches, we require a method that captures complex feature distributions while remaining computationally efficient and interpretable. We adopt Kernel Density Estimation (KDE), where the density of a random variable X given
Pmobservations x−xi 1 , x1 , x2 , ..., xm is estimated as: p̂X (x) = mh i=1 k h where k(·) is a kernel function and h is the bandwidth controlling smoothness. Applying KDE to IoT traffic features requires addressing two practical challenges: (1) choosing a valid kernel function and (2) handling zero-valued and degenerate observations. The choice of kernel k. A common 2choice of k is the t Gaussian function, where k(t) = √12π e− 2 , and the resulting method is called Gaussian Kernel Density Estimation (GKDE). A behavioral model should assign probability only to feasible traffic values. However, Gaussian kernels are symmetric and assign non-zero density to x < 0, which is invalid for nonnegative traffic features. To illustrate this limitation,Fig. 6a shows the density estimated for the averageInterarrivalTime feature of Apple Homepod flows using GKDE. The bandwidth h is chosen using Scott’s rule-of-thumb [75]. The mass of this invalid region, as estimated by the resulting distribution, is 0.24. We address this by applying a log transformation of Y = log X [76], [77], estimating the density in log-space, and transforming it back according to p̂X (x) = (log x)′ p̂Y (log x) =

8

8

p̂LNKDE(x) (£10°3)

p̂GKDE(x) (£10°4)

9

6

4

2

0

0

2000

4000

6000

8000

10000

Bandwidth = Scott’s/10 Bandwidth = Scott’s

8 7 2

1

0

0

2000

4000

6000

x : avgIntArrTime

x : avgIntArrTime

(a) GKDE.

(b) LNKDE.

8000

10000

Fig. 6: Modeling the behavioral range of non-negative traffic features. (a) GKDE assigns non-zero probability to infeasible feature values (x < 0), which is invalid for non-negative features. (b) Log-normal KDE (LNKDE) preserves non-negativity while accurately modeling the observed behavioral range of the feature. p̂Y (log x) . This yields a log-normal KDE (LNKDE) that prex

serves the non-negative support of the traffic feature while retaining its behavioral interpretation. Fig. 6b illustrates the LNKDE corresponding to Fig. 6a, which results in a density of zero for values of X ≤ 0. For the bandwidth h when constructing p̂Y (y), we scale Scott’s rule-of-thumb by a factor of 0.10 (solid curve) to avoid over-smoothing (dotted curve). In this paper, we use Scott’s rule of thumb divided by 10, since it yields quality estimates in practice. In both Fig. 6a and Fig. 6b, the scatter plot of observed samples is provided as a reference on the x-axis. Zero values and degeneracies. The log-transformation is not applicable to features that can take the value X = 0. To handle this, we model the distribution as a combination of a degenerate component (point mass) at X = 0 and a continuous component for non-zero values. The point mass captures zerovalued observations, while the non-zero component is modeled using LNKDE when sufficient variability is present. Similarly, when the non-zero observations are singular, we use a degenerate distribution at the observed value since Scott’s bandwidth relies on the variance of observations and is therefore not applicable. This leads to a total of 5 modeling scenarios depending on whether zero values and/or singularities are present, as listed in Appendix A. The vast majority of the distributions are modeled either solely by an LNKDE component or by an LNKDE component together with a degeneracy at X = 0.

D. Constructing Feature Conformity Regions Having constructed a behavioral model for each traffic feature, the next step is to determine whether an observed feature value continues to conform to the learned behavior. We therefore seek a conformity region for every class-feature pair that separates behavior consistent with the training data from behavior that is sufficiently uncommon to indicate behavioral evolution. Given the estimated density p̂f,c , we define the nonconforming region as the set of feature values with low probability density: χθf,c = {x|p̂f,c (x) ≤ θf,c } and the conforming region as its complement: χ′θf,c = {x|p̂f,c (x) > θf,c }.

Because the behavioral ranges of different traffic features vary substantially across device classes, a universal notion of “low-density” is not meaningful, making it difficult to choose a universal threshold θf,c . At the same time, manually specifying thresholds for each feature and class is impractical, especially where distribution models are continuously updated. We therefore automatically derive θf,c from the shape of the corresponding distribution. To determine an appropriate threshold, we constrain the probability mass of the non-conformity region. Specifically, we require that the probability of observing R values in χθf,c does not exceed a predefined value ϵ, i.e., χθ p̂f,c (x′ )dx′ ≤ ϵ. f,c The parameter ϵ controls the probability mass in the nonconforming region and thus the detector’s sensitivity. The trivial solution θf,c = 0 satisfies this constraint but results in all observations being classified as conforming, thereby failing to detect drift. Therefore, we seek the largest value of θf,c that satisfies the constraint, leading to the following optimization problem: maximize

θf,c Z

subject to {x|p̂f,c (x)≤θf,c }

p̂f,c (x′ )dx′ ≤ ϵ.

(1)

Note that the integral above also gives the probability of falsely flagging an instance drawn from p̂f,c (x). The parameter ϵ is the only hyperparameter in our methodology that needs to be set by the user. To compute the conformity threshold θf,c , we use a piecewise-constant approximation of the density function. We discretize the domain into bins B = {(b1 , b1 + ∆), (b2 , b2 + ∆), ..., (bBR, bB + ∆)} estimate the density in each bin as bi +∆ 1 pi = ∆ p̂f,c (x) dx. We then sort these values in bi ascending order and accumulate the probability mass until the constraint ϵ is exceeded. The threshold θf,c is selected as the largest density value that satisfies the constraint. The detailed procedure is provided in Algorithm 1. Fig. 7 illustrates this approximation for ϵ = 0.1. The piecewise constant approximation (blue dashed lines) is shown alongside the original density (the solid black curve), and

9

Algorithm 1 Conformity Threshold Calculation

𝑞!!"#$ 𝜃 = 𝑞!!"#$"# 𝜃

Fig. 7: Construction of feature-level conformity regions. The conformity threshold θ is chosen so that the probability mass of the non-conforming region (shaded) does not exceed the user-defined sensitivity parameter ϵ. the resulting θf,c separates conforming and non-conforming regions. Finally, we note that the above formulation assumes the densities are modeled using LNKDE. For degenerate distributions, the threshold is set to 0, ensuring that only values at the support points are considered conforming. In mixed cases, degenerate components are treated as conforming at their support, while the thresholding procedure is applied to the continuous component. E. Behavioral Non-conformity and Classifier Misclassification We next investigate the relationship between behavioral non-conformity and classifier misclassifications. To quantify this relationship, we evaluate the detector using the True Positive Rate (TPR), defined as the fraction of misclassified instances that are detected, and the False Positive Rate (FPR), defined as the fraction of correctly classified instances that are incorrectly flagged as exhibiting behavioral evolution. We perform the evaluation across the entire deployment phase, fitting M (x) and D(x; M ) to the training data and keeping both unchanged throughout the evaluation. We start by reporting the results for the selected sensitivity threshold of ϵ = 0.01. A discussion on the role of ϵ is provided in the following subsection. Fig. 8 summarizes the detector performance across device classes, averaged over the deployment period. We report mean TPR and specificity (1 − FPR) averaged over weekly evaluation windows, with error bars indicating the temporal standard deviation. As shown in Fig. 8a, the detector captures a substantial fraction of the misclassified traffic for most device classes, although the achieved TPR varies across classes and over time. For several devices (e.g., Line Clova Wave and Panasonic Home Unit), the TPR exhibits noticeable temporal variation. This indicates that behavioral evolution manifests differently over time across device classes. Averaged across all device classes, the detector achieves a temporal mean TPR 0.77. Depicted in Fig. 8b, specificity results show that when aggregated over the full deployment phase, the detector is prone to flagging correctly classified instances. However, the large error bars suggest that the specificity is highly variable over time. The temporal behavior is shown in Fig. 9. Specificity gradually decreases throughout deployment. During the

Input: PDF p̂(x), constant ϵ and a binning B = {(b1 , b1 + ∆), (b2 , b2 + ∆), ..., (bB , bB + ∆)}, where bi = bi−1 + ∆. Output: Threshold θ. for i = 1R bto+∆B do i p̂(x)dx pi ← bi ∆ end for q1 , q2 , ..., qB ← sort ascending(p1 , p2 , ..., pB ) θprev = 0 for k = 1 to do PB k m ← ∆ k′ =1 qk′ if m > ϵ then θ ← θprev break else θprev ← qk end if end for return θ TABLE III: Detection results for different ϵ values. ϵ 0.001 0.010 0.100

TPR 0.65 ± 0.27 0.77 ± 0.22 0.95 ± 0.08

Specificity 0.58 ± 0.31 0.49 ± 0.27 0.26 ± 0.19

Balanced accuracy 0.61 ± 0.17 0.63 ± 0.15 0.60 ± 0.10

early stages of deployment, most correctly classified instances remain behaviorally conforming (the average specificity over the first six months of deployment is 0.71, with 13 device classes exceeding 0.8). However, as device behaviors continue to evolve, an increasing number of correctly classified instances exhibit previously unseen behavioral patterns. These emerging patterns do not immediately cause misclassifications, but they are identified by the detector as behavioral nonconformity. Consequently, the detector should be updated periodically to incorporate these newly observed behaviors. The relatively stable TPR indicates that a consistent fraction of misclassifications remains detectable based on marginal feature non-conformity throughout deployment. We later show in §VI how this detection behavior affects detector-guided adaptation and how it can be addressed. F. Choice of ϵ We evaluate three values of ϵ spanning three orders of magnitude, which is the only hyper-parameter of our method: 10−3 , 10−2 and 10−1 . The TPR and specificity results, as well as balanced accuracy (defined as TPR+Specificity ), are shown in 2 Table III. By increasing ϵ, the detection rate increases, though at the cost of more FPs (especially pronounced for ϵ = 0.1). In terms of balanced accuracy, different choices meet at a similar point, while ϵ = 0.01 slightly outperforms. A more detailed sensitivity analysis around ϵ = 0.01 is provided in Appendix B. G. Feature-Level Explanations of Behavioral Evolution Operating directly on raw traffic features naturally provides feature-level explanations for every detected instance.

10

1.2

1.0

1.0

specificity: 1 - FPR

1.2

0.6 0.4 0.2

0.6 0.4 0.2 0.0

A Am mazo Am azon n Ec azo Ec ho n E ho Dot ch Ge o n Fir Sho 2 A Go pple e TV w 5 og H Sti le om ck Ch e G ro po Go oog meca d o le s I-O gle N Hom t DA est e TA M QW ini Lin a e C JVC tch lov Hu Pa na N a b Pa soni ature Wav na c D R e so o em Ph nic H orph o So ilips om one ny H e U Sm ue ni TP art Brid t TP -Link Spea ge -L TP ink Kasa ker -Li Sm P nk ar ro t Wi W Smar LED thi an t P ng sv lu s N iew g ok Q ia 5 Bo dy

0.0

0.8

Class

(a) True Positive Rate.

A Am mazo Am azon n Ec azo Ec ho n E ho Dot ch Ge o n F ir Sho 2 Go Apple e TV w 5 og H Sti le om ck Ch e G ro po Go oog meca d o le s I-O gle N Hom t DA est e TA M QW ini Lin a e C JVC tch l H Pa na Na ova W ub Pa soni ture av na c D R e so o em Ph nic orph o So ilips Hom one ny H e U Sm ue ni TP art Brid t TP -Link Spea ge -L TP ink Kasa ker -Li Sm P nk ar ro t Wi W Smar LED thi an t P ng sv lu s N iew g ok Q ia 5 Bo dy

TPR

0.8

Class

(b) 1 - False Positive Rate.

Fig. 8: Relationship between behavioral non-conformity and classifier misclassification. Mean TPR (a) and specificity (b) are reported for each device class over the deployment phase. Error bars indicate temporal standard deviation across evaluation windows.

Detection performance

1.0

The detector identifies behavioral non-conformity in: Ø firstNonEmptyPacketSize Ø maxPacketSize

0.8 0.6

(a) When predicted as Apple (b) When predicted as Amazon HomePod. Echo Dot.

0.4 0.2 0.0 1 Jan'22

The detector identifies behavioral non-conformity in: Ø reverseOctetTotalCount Ø dataByteCount Ø reverseDataByteCount

TPR specificity: 1 - FPR 1 Jul'22

1 Jan'23

Time

1 Jul'23

1 Jan'24

Fig. 9: Temporal evolution of detector performance averaged across device classes. TPR remains relatively stable throughout deployment, whereas specificity gradually decreases as behavioral evolution introduces an increasing number of correctly classified instances exhibiting previously unseen patterns.

Specifically, the subset of non-conforming features identifies which aspects of the observed traffic behavior deviate from the learned behavioral representation of the predicted device class. Such explanations provide operators with interpretable insights into how device behavior evolves over time. Each detected instance is associated with a binary indicator vector that identifies the non-conforming features. Beyond explaining individual detections, these indicator vectors can also be analyzed statistically to assess the consistency of the explanations throughout deployment. Those analyses are presented in Appendix C; here we focus on representative examples. The following example illustrates how these explanations evolve as device behavior changes over time. To illustrate the detector’s explanations, we consider the Wansview Q5 IP camera. Beginning in late 2022, the classifier starts to misclassify this device, initially as an Apple HomePod and later as an Amazon Echo Dot. The detector provides distinct feature-level explanations for these two episodes of behavioral evolution, as illustrated in Fig. 10. Among the instances misclassified as Apple HomePod, the detector consistently identifies non-conformity in the features maxPacketSize and firstNonEmptyPacketSize, as summarized in Fig. 10a. Later in deployment, the same device begins

Fig. 10: Feature-level explanations of behavioral evolution for the Wansview Q5 IP camera. Misclassifications across two deployment periods are explained by distinct subsets of nonconforming traffic features. to be misclassified not only as an Apple HomePod but also as an Amazon Echo Dot. These instances are characterized by a different explanation involving reverseOctetTotalCount, dataByteCount, and reverseDataByteCount, as illustrated in Fig. 10b. This demonstrates that the same device may undergo different forms of behavioral evolution over time. These examples demonstrate that behavioral evolution is dynamic and instance-specific. Different stages of deployment may be characterized by different subsets of non-conforming traffic features, allowing the detector to provide interpretable feature-level explanations directly in the original feature space. VI. A DAPTING I OT T RAFFIC C LASSIFIERS TO C ONCEPT D RIFT The previous sections established that classifier maintenance throughout deployment is necessary and feasible, and introduced a behavioral signal that identifies evolving traffic patterns. The remaining challenge is determining how this behavioral signal should be exploited for long-term classifier maintenance in operational deployments, where only a limited fraction of deployment traffic can realistically be labeled. The following subsections investigate this question through a series of adaptation experiments. A. Detector-Guided Adaptation We begin by considering the most natural adaptation strategy, in which the deployment instances identified by the detector are labeled and incorporated into classifier retraining. Let Wi (for i > 0) denote the (unlabeled) dataset corresponding

1.0

25%

0.9

20%

0.8 0.7 0.6

Full adaptation No adaptation Detector-guided adaptation Uniform sampling (detector-guided rate)

0.5 1 Jan'22 1 Apr'22 1 Jul'22 1 Oct'22 1 Jan'23 1 Apr'23 1 Jul'23 1 Oct'23 1 Jan'24

Time

(a) Adaptation performance.

Labeling fraction

Macro F1-score

11

Detector-guided adaptation Uniform sampling (detector-guided rate)

15% 10% 5% 0

1 Jan'22 1 Apr'22 1 Jul'22 1 Oct'22 1 Jan'23 1 Apr'23 1 Jul'23 1 Oct'23 1 Jan'24

Time

(b) Labeling cost.

Fig. 11: (a) Classification performance under different adaptation strategies. Detector-guided adaptation (solid green) substantially improves performance over no adaptation (dotted blue), but remains consistently below the upper-bound maintenance strategy (dashed gray). Replacing detector-selected retraining instances with uniformly sampled deployment traffic at the same detector-guided labeling rate (dash-dotted green) yields performance close to the upper bound. (b) Fraction of deployment traffic labeled in each evaluation window. Detector-guided adaptation labels progressively fewer instances over time as fewer deployment flows are identified as behaviorally non-conforming, whereas uniform sampling preserves the detector-guided sampling rate while providing more representative coverage of the evolving behavioral patterns. For the strategy involving random sampling, error bars indicate the standard deviation across 10 random seeds. to test window i, with i = 0 reserved for training data. We use Wi∗ to denote the labeled version of Wi (e.g., W0∗ denotes the labeled training dataset) and define the cumulative dataset ∗ ∗ up to window i as Wcum,i , initialized as Wcum,0 = W0∗ . Let M(W ∗ ) and D(W ∗ ; ϵ) denote procedures for training the classifier M and the drift detector D, respectively. The detector construction follows §V, where feature distributions and conformity thresholds are estimated using parameter ϵ. Throughout this section, we use ϵ = 0.01. The classifier is implemented as a Random Forest, with hyperparameters fixed from the training phase (see §IV-A). We define the drift instance selection function as D̃(W ; M ) = {x ∈ W |D(x; M ) = 1}. Let Mi−1 and Di−1 denote the classifier and the detector after processing test window i − 1. For window i, we first evaluate the performance of Mi−1 on Wi . Then we label D̃i−1 (Wi ; Mi−1 ) and ∗ concatenate the result to Wcum,i−1 . Finally, we update both the detector and the classifier using the newly extended cumulative dataset. This procedure is summarized in the Appendix D. The resulting adaptation performance is depicted in Fig. 11a, in terms of macro F1-score. The average of the macro F1score across all deployment windows is 0.917. This level of adaptation performance is achieved by labeling 2.66% of the deployment instances overall. Per-window labeling cost is depicted in Fig. 11b. Although detector-guided adaptation substantially improves classification performance, it does not achieve the level of recovery observed under the upper-bound maintenance strategy. This bias is attributed to the limitation of the detector in identifying all possible pattern changes (recall the TPR results from Fig. 8a), and is particularly manifested in a performance decline from early August 2022 (see Fig. 11a). To illustrate this bias, we have chosen one of the classes that experienced a performance drop during the mentioned period, namely the Google Nest Mini. We consider a slice of the deployment phase commencing at the performance drop: in the window

starting from 30 July 2022, the F1-score for this class drops to 0.446 from 0.938 in the preceding window, while the poor performance (0.389 F1-score) continues in the subsequent window. This indicates the emergence of a novel pattern from the end of July that is not adequately captured by the detector, since otherwise the degradation would not persist into the subsequent window. These observations reveal an important limitation of detector-guided adaptation. Although the detector successfully identifies a substantial fraction of behavioral evolution, the resulting retraining data do not necessarily provide representative coverage of the newly emerging traffic patterns. This raises the following question: why does detector-guided adaptation fail to fully recover classifier performance despite retraining on behaviorally non-conforming instances? B. Uniform Traffic Sampling for Model Adaptation To answer the above question, we examine the deployment window starting from 30 July 2022 for the Google Nest Mini case study. Fig. 12 compares the misclassified traffic instances in this window with those selected by the detector for retraining. The detector-selected instances concentrate on only a subset of the emerging behavioral patterns, leaving other newly formed regions unexplored. Consequently, retraining only on the detected instances fails to adequately capture the diversity of the evolving traffic behavior. This observation motivates an alternative adaptation strategy. Rather than retraining on the detector-selected instances, we instead sample deployment traffic uniformly while matching the labeling rate of the detector. Fig. 12 shows that uniformly sampled instances provide substantially broader coverage of the emerging behavioral patterns than the detectorselected samples. We next evaluate this strategy over the entire deployment period. One practical challenge is determining an appropriate sampling rate for each deployment window. To match the

12

1.0

20

0.9

Macro F1-score

t-SNE 2

10 0 10 20 30

Misclassified (novel pattern) Flagged by detector Selected by the random sampler 40

20

0

t-SNE 1

20

40

Fig. 12: Detector-selected instances (red crosses) concentrate around only a subset of the emerging behavioral patterns (orange circles). Uniformly sampled instances (black stars) provide substantially broader coverage of the evolving traffic behavior, explaining their superior adaptation performance. traffic labeling rate to the severity of the drift, we label instances at the same abundance as the detector flags at each step, and then update both the classifier and the detector using the newly labeled examples. The corresponding results are shown in Fig. 11a (the dash-dotted curve with errorbars). Note that throughout the remainder, we have repeated the evaluations involving random traffic sampling with 10 different random seeds, reporting the average ± the standard deviation of each metric over seeds. The classification performance is consistently high and close to that of the exhaustive full adaptation approach. The window-average of F1-score is 0.9647±0.0003, and is achieved by labeling 10.15±0.28% of the aggregated deployment instances. The per-window labeling effort is depicted in Fig. 11b. Comparison of labeling costs of the two approaches in Fig. 11b further explains the observed difference in adaptation performance. The number of detector-selected instances declines rapidly over time because progressively fewer instances are identified as behaviorally non-conforming. As a result, a considerable portion of the newly emerging behavioral patterns remain unlabeled and are therefore never incorporated into either the classifier or the behavioral model. Consequently, the detector gradually falls behind the evolving traffic behavior, causing the observed selection bias to persist over time. C. Adaptation Under Operational Labeling Budgets Operational deployments typically impose strict limits on the amount of traffic that can be labeled. Having shown that uniformly sampled deployment traffic provides more effective classifier maintenance than detector-selected instances, we now investigate how the fixed labeling budget should be allocated over time. While the exact budget varies across operational environments, we use a representative labeling budget of 0.1% of the deployment traffic per evaluation window, consistent with recent observations of practical traffic-labeling costs [73]. We first establish a reference operating point by uniformly sampling deployment traffic at the full budget of 0.1% in every evaluation window. The corresponding adaptation performance is shown by the dash-dotted pink curve in Fig. 13. Despite operating under a substantially tighter labeling budget, the

0.8 0.7 0.6

Full adaptation No adaptation 0.1% sampling rate Adjusted sampling rate below 0.1% Random sampling rate below 0.1%

0.5 1 Jan'22 1 Apr'22 1 Jul'22 1 Oct'22 1 Jan'23 1 Apr'23 1 Jul'23 1 Oct'23 1 Jan'24

Time

Fig. 13: Classifier maintenance under a fixed operational labeling budget. Sampling uniformly at the full budget (dashdotted pink) provides a reference operating point. Reducing the sampling rate uniformly across deployment (dotted orange) noticeably degrades adaptation performance. In contrast, adjusting the sampling rate according to the observed behavioral evolution (solid maroon) achieves comparable performance while labeling fewer deployment instances. Error bars are omitted for visual clarity.

resulting classifier maintains an average macro F1-score of 0.8970 ± 0.0042 across the deployment period. However, this performance is achieved by fully utilizing the available labeling budget, raising the question of whether similar performance can be obtained while labeling even fewer deployment instances. A natural baseline is to reduce the labeling effort uniformly throughout deployment. To this end, we independently sample the labeling rate for each evaluation window from the uniform distribution Unif[0, Cap], where Cap = 0.1%. The resulting adaptation performance is shown by the dotted orange curve in Fig. 13. Although this strategy reduces the aggregate labeling cost to 0.0501 ± 0.0022%, the average F1-score across windows decreases to 0.8797 ± 0.0067, indicating a loss in adaptation performance when the labeling rate is reduced without accounting for behavioral evolution. These results indicate that the available labeling budget should be allocated in proportion to the observed level of behavioral evolution, rather than uniformly across deployment. Specifically, for each evaluation window, we compute the fraction of deployment instances that are behaviorally nonconforming and record the maximum value observed so far. The labeling rate for the current window is then scaled linearly to this running maximum. Before moving to the next deployment window, both the classifier and detector are updated using the newly labeled instances. This procedure is outlined in Algorithm 2. Note that R(W ; Λ) denotes randomly choosing from window W at rate Λ. The resulting adaptation performance is shown by the solid maroon curve in Fig. 13. Compared with uniform rate selection, adjusting the labeling rate according to the observed level of behavioral evolution improves classifier maintenance. Moreover, its average macro F1-score (0.8934 ± 0.0049) remains close to full-budget sampling (0.8970±0.0042), while reducing the labeling cost from 0.1% to 0.0826 ± 0.0006%.

13

TABLE IV: Comparison of adaptation strategies under operational labeling constraints. Behavior-aware sampling achieves performance comparable to full-budget sampling while reducing the amount of labeled deployment traffic. Compared with confidence-based alternatives, it also provides feature-level interpretability. Metric Cap. rate Uniform below cap. Behavior-aware rate Low-confidence Low-confidence stratified Confidence-guided rate Avg. macro F1-score 0.8970 ± 0.0042 0.8797 ± 0.0067 0.8934 ± 0.0049 0.8845 0.8905 0.8942 ± 0.0039 Labeling cost 0.1% (Cap) 0.0501 ± 0.0022% 0.0826 ± 0.0006% 0.1% (Cap) 0.123% 0.0867 ± 0.0009%

Algorithm 2 Budget-Aware Labeling Rate. Input: A labeled training dataset W0∗

and unlabeled deployment windows of traffic {Wi |1 ≤ i ≤ T }, sensitivity threshold ϵ, maximum labeling budget of C. Output: Progressively evaluated performance {Πi |1 ≤ i ≤ T }, as well as detection labeling cost {Λi |1 ≤ i ≤ T } in all deployment windows. ∗ Wcum,0 ← W0∗ M0 ← M(W0∗ ) D0 ← D(W0∗ ; ϵ) ∆max ← 0 for i = 1 to T do Πi ← perf(Wi∗ ; Mi−1 ) // In iteration i, ground-truth labels for Wi are only available for the performance evaluation purpose. ∆i ← |D̃i−1 (Wi ; Mi−1 )| ∆max ← max(∆max , ∆i ) i ×C Λi ← ∆∆ max ∗ ∗ Wcum,i ← Wcum,i−1 ∥ R∗ (Wi ; Λi ) // The || operator denotes concatenation. ∗ Mi ← M(Wcum,i ) ∗ Di ← D(Wcum,i ) end for return {Πi |1 ≤ i ≤ T }, {Λi |1 ≤ i ≤ T }

D. Comparison to Confidence-based Model Adaptation Confidence-based active learning is a common strategy for adapting machine learning models under distribution shift, where deployment instances with the lowest prediction confidence are selected for labeling and retraining [78]. Since our proposed method also provides a signal for guiding classifier maintenance, it is natural to compare the behavioral signal against classifier confidence. A key limitation of confidencebased approaches is that they indicate prediction uncertainty without revealing which traffic features have changed. In contrast, the proposed behavioral signal identifies the nonconforming traffic features responsible for the detected behavioral evolution. We are therefore interested in assessing whether these interpretability benefits can be achieved without compromising adaptation performance. We evaluate three confidence-based adaptation strategies under the same experimental setting. In the first experiment, we label 0.1% of the instances with the lowest confidence in each window and use them to adapt the model. The resulting procedure yields an average macro F1-score of 0.8845 during the deployment phase. This performance is lower than both full-budget uniform sampling (0.8970) and the behavior-aware strategy (0.8934). It uses the

same labeling budget as the former (0.1%) and more labeled traffic than the latter (0.0826%). We also evaluate a stratified version in which the labeling budget is applied independently to each predicted class. Whenever the allocated budget would otherwise result in zero labeled instances for a class, we label one instance to ensure that every predicted class contributes to adaptation. This approach results in an average F1-score of 0.8905, albeit at a labeling cost of 0.123%. Despite the higher labeling cost, stratified lowconfidence sampling achieves performance comparable to, but not higher than, the behavior-aware strategy. These experiments further support the observation that the adaptation signal is more effective for determining the traffic labeling rate than for direct instance selection. We therefore construct a counterpart to the behavior-aware strategy, modifying Algorithm 2 by replacing the behavioral signal with classifier confidence while continuing to sample deployment instances uniformly. Specifically, we replace the drift severity signal (denoted by ∆i in Algorithm 2) with the number of deployment instances whose prediction confidence falls below a predefined threshold. For this experiment, the threshold is set to a representative value of 80%. The mentioned approach yields an average F1-score of 0.8942 ± 0.0039 over the deployment phase at a labeling cost of 0.0867 ± 0.0009%. The resulting macro F1-score matches that of the behavioraware strategy while requiring a slightly higher labeling cost. The performance of confidence-based methods, along with the behavior-aware strategy, over the entire deployment phase is depicted in Fig. 15 in Appendix E. The quantitative comparison of all adaptation strategies is summarized in Table IV. Overall, confidence-guided and behavior-guided sampling achieve comparable adaptation performance, while the former incurs a slightly higher labeling cost. Unlike classifier confidence, the behavioral signal identifies the traffic features responsible for behavioral evolution, providing feature-level interpretability while guiding the labeling rate. E. Goodness Properties of the Detector and its Maintenance Scheme The previous subsections demonstrated that both the classifier and the detector are updated incrementally throughout deployment. Since the detector itself evolves over time, it is important to verify that this maintenance process preserves previously learned behavioral patterns while accurately incorporating newly emerging ones. Therefore, we evaluate the proposed detector and its maintenance scheme according to three desirable properties. These properties capture three complementary aspects of detector maintenance: preservation of previously learned behavior (stability), robustness to selective

14

updates (consistency), and approximation of full-data updates (efficiency). (P1) Stability: Updating the detector with newly observed data should not introduce spurious detections (compared to the frozen detector) in subsequent windows, i.e., previously learned patterns should be preserved. (P2) Consistency: The stability property should hold when the detector is updated using only the selected drifted instances instead of all labeled data. (P3) Efficiency: Updating the detector using only selected instances should produce detection behavior similar to updating with all labeled data, indicating that the underlying marginal distributions are adequately captured with significantly fewer labeled samples. To evaluate these properties, consider the cumulative dataset ∗ Wcum,i−2 together with adjacent windows Wi−1 and Wi . Recall that, the ∗ symbol denotes the labeled version corresponding to each dataset. We define three different detector update strategies before applying them to Wi : ∗ No adaptation: DN A, i−1 = Di−2 = D(Wcum,i−2 ; ϵ), ∗ ∗ Full: DF ull, i−1 = D(Wcum,i−2 ∥ Wi−1 ; ϵ),

(2) (3)

∗ ∗ Selective: DSel, i−1 = D Wcum,i−2 ∥ D̃i−2 (Wi−1 ; M ); ϵ . (4) A visualization of the three update strategies is provided in Appendix F. Given the fixed classifier M trained on the training data, we define the following detected instance sets: SN A,i = D̃N A, i−1 (Wi ; M )

(5)

SF ull,i = D̃F ull, i−1 (Wi ; M )

(6)

SSel,i = D̃Sel, i−1 (Wi ; M )

(7)

If the three properties are satisfied, we expect: SF ull,i ⊆ SN A,i ,

SSel,i ⊆ SN A,i ,

(i)

(i)

η2 = (i)

η3 =

|SN A,i ∩ SF ull,i | |SF ull,i |

Our study has several limitations. First, we infer behavioral evolution from marginal traffic features rather than their joint distribution. While this offers computational and data efficiency by avoiding joint distribution modeling, it is possible to overlook evolutions that occur only among the feature dependencies without impacting the marginals. As discussed in §V, this limitation is shared by input-based methods that avoid explicit modeling of high-dimensional joint feature distributions, which is generally prohibitive in practice. Second, our evaluation is based on a single deployment dataset collected by an ISP using a controlled testbed. Note that, unlike the existing public datasets that are collected over relatively short time frames, this dataset captures long-term behavioral evolution, which is central to the objectives of our work. Evaluating the proposed framework across additional deployment environments would further strengthen the generality of the conclusions. Our focus in this paper has been on TCP/443 traffic (presumably HTTPS). Note that flowlevel semantics vary across network protocols, being richer for hypertext protocols [79], and IoT devices are increasingly adopting TLS for network communications [80]. Extending the proposed framework to additional protocols is an important direction for future work. Finally, our study is limited to Random Forest as the classification algorithm, which is well-established for traffic classification [71]. Although our experiments employ Random Forests, the proposed detector uses only the predicted class and the corresponding classconditional behavioral model. It therefore does not depend on the internal feature representation or confidence estimation mechanism of a particular classifier. Consequently, the proposed behavioral modeling framework can be combined with alternative traffic classifiers.

VIII. C ONCLUSION

SF ull,i = SSel,i . (8)

Accordingly, we define the following metrics: η1 =

VII. L IMITATIONS

(for stability),

(9)

|SN A,i ∩ SSel,i | |SSel,i |

(for consistency),

(10)

|SSel,i ∩ SF ull,i | |SSel,i ∪ SF ull,i |

(for efficiency).

(11)

Across all deployment windows, the three metrics achieve average values of η1 = 0.984±0.012, η2 = 0.971±0.017, and η3 = 0.953 ± 0.028. These results indicate that incremental detector updates preserve previously learned behaviors, remain consistent when updated using only selected instances, and closely approximate updates using all labeled deployment traffic. Together, these observations support the proposed detector maintenance strategy for long-term deployment. Additional temporal results are provided in Appendix F.

Accurate identification of IoT devices from network traffic is essential for security monitoring and policy enforcement. However, as device behavior evolves over time, deployed classifiers experience concept drift, requiring periodic maintenance using newly labeled deployment traffic. In this paper, we first showed that behavioral evolution manifests differently across IoT device classes and that effective long-term classifier maintenance is feasible. We then developed a conformity-based detector that models class-conditional feature behavior directly from raw traffic features, providing interpretable feature-level explanations of behavioral evolution. Finally, we demonstrated that uniformly sampling deployment traffic while adjusting the labeling rate according to the observed behavioral evolution provides more effective and label-efficient classifier maintenance than selecting detector-identified instances alone. These findings show that behavioral evolution is not merely a source of classifier degradation but also a practical signal for maintaining deployed IoT traffic classifiers under realistic operational constraints.

15

R EFERENCES [1] P. Williams et al., “A Survey on Security in Internet of Things with a Focus on the Impact of Emerging Technologies,” Internet of Things, vol. 19, p. 100564, Aug. 2022. [2] S. U. Haq et al., “A Survey on IoT & Embedded Device Firmware Security: Architecture, Extraction Techniques, and Vulnerability Analysis Frameworks,” Discover Internet of Things, vol. 3, Oct. 2023. [3] O. Alrawi et al., “SoK: Security Evaluation of Home-Based IoT Deployments,” in Proc. IEEE S&P, San Francisco, CA, USA, May 2019. [4] J. Anand et al., “PARVP: Passively Assessing Risk of Vulnerable Passwords for HTTP Authentication in Networked Cameras,” in Proc. DAI-SNAC, Virtual Event, Germany, Dec 2021. [5] P. Kumari and A. K. Jain, “A Comprehensive Study of DDoS Attacks over IoT Network and Their Countermeasures,” Computers and Security, vol. 127, p. 103096, Apr 2023. [6] M. Antonakakis et al., “Understanding the Mirai Botnet,” in Proc. USENIX Security, Vancouver, BC, Canada, Aug 2017. [7] R. Doshi et al., “Machine Learning DDoS Detection for Consumer Internet of Things Devices,” in Proc. IEEE SPW, San Francisco, CA, USA, May 2018. [8] D. J. Dubois et al., “When Speakers Are All Ears: Characterizing Misactivations of IoT Smart Speakers,” in Proc. PETS, Virtual Event, Jul 2020. [9] Z. B. Çelik et al., “Sensitive Information Tracking in Commodity IoT,” in Proc. USENIX Security, Baltimore, MD, USA, 2018. [10] U. Iqbal et al., “Tracking, Profiling, and Ad Targeting in the Alexa Echo Smart Speaker Ecosystem,” in Proc. ACM IMC, Montreal, QC, Canada, Oct 2023. [11] J. Ren et al., “Information Exposure From Consumer IoT Devices: A Multidimensional, Network-Informed Measurement Approach,” in Proc. ACM IMC, Amsterdam, Netherlands, Oct 2019. [12] N. Apthorpe et al., “Poster: A Smart Home is No Castle: Privacy Vulnerabilities of Encrypted IoT Traffic,” in Proc. NDSS, San Diego, CA, USA, Feb 2016. [13] A. Acar et al., “Peek-a-Boo: I See Your Smart Home Activities, Even Encrypted!” in Proc. ACM WiSec, Virtual Event, Jul 2020. [14] R. Li, S. Liu, H. Hu, Q. Ye, and N. Feamster, “WiFinger: Fingerprinting Noisy IoT Event Traffic Using Packet-level Sequence Matching,” in Proc. NDSS, San Diego, CA, USA, Feb 2026. [15] A. Pashamokhtari et al., “Combining Stochastic and Deterministic Modeling of IPFIX Records to Infer Connected IoT Devices in Residential ISP Networks,” IEEE IoTJ, vol. 10, no. 6, pp. 5128–5145, 2023. [16] National Institute of Standards and Technology, “National vulnerability database,” https://nvd.nist.gov/, 2026, accessed: 2026-02-23. [17] MITRE Corporation, “Common Vulnerabilities and Exposures (CVE),” https://www.cve.org/, 2026, accessed: 2026-02-23. [18] A. M. Mandalari, D. J. Dubois, R. Kolcun, M. T. Paracha, H. Haddadi, and D. Choffnes, “Blocking Without Breaking: Identification and Mitigation of Non-Essential IoT Traffic,” in Proc. PETS, Virtual Event, Jul. 2021. [19] A. Hamza et al., “Verifying and Monitoring IoTs Network Behavior Using MUD Profiles,” IEEE TDSC, vol. 19, no. 1, pp. 1–18, May 2020. [20] W. He et al., “Can Allowlists Capture the Variability of Home IoT Device Network Behavior?” in Proc. IEEE EuroS&P, Vienna, Austria, Jul 2024. [21] S. J. Saidi et al., “A Haystack Full of Needles: Scalable Detection of IoT Devices in the Wild,” in Proc. IMC, Virtual Event, USA, Oct 2020. [22] A. Sivanathan et al., “Classifying IoT Devices in Smart Environments Using Network Traffic Characteristics,” IEEE TMC, vol. 18, pp. 1745– 1759, 2019. [23] S. Marchal et al., “AuDI: Toward Autonomous IoT Device-Type Identification Using Periodic Communication,” IEEE JSAC, vol. 37, no. 6, pp. 1402–1412, 2019. [24] M. Miettinen et al., “IoT SENTINEL: Automated Device-Type Identification for Security Enforcement in IoT,” in Proc. IEEE ICDCS, Atlanta, GA, USA, Jun 2017. [25] V. Thangavelu et al., “DEFT: A Distributed IoT Fingerprinting Technique,” IEEE IoTJ, vol. 6, no. 1, pp. 940–952, 2019. [26] D. Ahmed et al., “Analyzing the Feasibility and Generalizability of Fingerprinting Internet of Things Devices,” in Proc. PETS, Sydney, Australia, July 2022. [27] R. Kolcun et al., “Revisiting IoT Device Identification,” in Proc. IFIP TMA, Virtual Event, Sep 2021. [28] S. Azizi et al., “Poster: Understanding and Managing Changes in IoT Device Behaviors for Reliable Network Traffic Inference,” in Proc. SIGCOMM Posters and Demos, Sydney, NSW, Australia, 2024.

[29] E. Maali et al., “Evaluating Machine Learning-Based IoT Device Identification Models for Security Applications,” in Proc. NDSS, San Diego, CA, USA, Feb 2025. [30] A. Pashamokhtari et al., “Dynamic Inference From IoT Traffic Flows Under Concept Drifts in Residential ISP Networks,” IEEE IoTJ, vol. 10, no. 17, pp. 15 761–15 773, 2023. [31] Q. Zhang et al., “CARAVAN: Practical Online Learning of In-network ML Models with Labeling Agents,” in Proc. USENIX OSDI, Santa Clara, CA, USA, July 2024. [32] J. L. Guerra, C. Catania, and E. Veas, “Datasets Are Not Enough: Challenges in Labeling Network Traffic,” Computers & Security, vol. 120, p. 102810, 2022. [33] R. Jordaney et al., “Transcend: Detecting concept drift in malware classification models,” in Proc. USENIX Security, Vancouver, BC, Canada, Aug 2017. [34] F. Barbero, F. Pendlebury, F. Pierazzi, and L. Cavallaro, “Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift,” in Proc. IEEE S&P, Toronto, ON, Canada, Jul 2022. [35] D. Han et al., “Anomaly Detection in the Open World: Normality Shift Detection, Explanation, and Adaptation,” in Proc. NDSS, San Diego, CA, USA, Feb 2023. [36] J. Zhang, F. Li, F. Ye, and H. Wu, “Autonomous Unknown-Application Filtering and Labeling for DL-based Traffic Classifier Update,” in Proc. IEEE INFOCOM, Toronto, ON, Canada, Jul 2020. [37] S. Jorgensen et al., “Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification,” IEEE TAI, vol. 5, no. 1, pp. 420–433, 2024. [38] L. Yang et al., “CADE: Detecting and Explaining Concept Drift Samples for Security Applications,” in Proc. USENIX Security, Virtual Event, Aug 2021. [39] Y. Chen, Z. Ding, and D. Wagner, “Continuous learning for android malware detection,” in Proc. USENIX Security, Anaheim, CA, Aug 2023. [40] Y. Ovadia et al., “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift,” in Proc. NeurIPS, Vancouver, Canada, Dec 2019. [41] R. Kolcun et al., “The Case for Retraining of ML Models for IoT Device Identification at the Edge,” 2020, https://arxiv.org/abs/2011.08605. [42] A. Pashamokhtari, G. Batista, and H. Habibi Gharakheili, “Efficient IoT Traffic Inference: From Multi-View Classification to Progressive Monitoring,” ACM Trans. Internet Things, vol. 5, no. 1, Dec 2023. [43] A. Sivanathan, D. Mishra, S. Ruj, N. Fernandes, Q. Z. Sheng, M. Tran, B. Luo, D. Coscia, G. Batista, and H. Habibi Gharakaheili, “Real-Time and Trustworthy Classification of IoT Traffic Using Lightweight Deep Learning,” IEEE Transactions on Network Science and Engineering, vol. 13, pp. 3256–3273, Nov 2025. [44] L. Yang et al., “PWPAE: An Ensemble Framework for Concept Drift Adaptation in IoT Data Streams,” in Proc. IEEE GLOBECOM, Madrid, Spain, 2021. [45] P. K. Danso et al., “Transferability of Machine Learning Algorithm for IoT Device Profiling and Identification,” IEEE IoTJ, vol. 11, no. 2, pp. 2322–2335, 2024. [46] R. M. Carnier, L. Lahesoo, and K. Fukuda, “Binary anomaly detection in streaming iot traffic under concept drift,” CoRR, vol. abs/2510.27304, 2025. [47] S. Azizi et al., “Towards Label Shift Adaptation for Robust IoT Device Identification,” in Proc. ACM CPSIoTSec, Taipei, Taiwan, Oct 2025. [48] N. Malekghaini et al., “Deep Learning for Encrypted Traffic Classification in the Face of Data Drift: An Empirical Study,” Elsevier Computer Networks, vol. 225, pp. 109 648–, 2023. [49] S. Liu et al., “LEAF: Navigating Concept Drift in Cellular Networks,” in Proc. ACM CoNEXT, Paris, France, Dec 2023. [50] G. Andresini et al., “INSOMNIA: Towards Concept-Drift Robustness in Network Intrusion Detection,” in Proc. ACM AISec, Republic of Korea: Virtual Event, Nov 2021. [51] B. M. Xavier et al., “Fast Learning Enabled by In-Network Drift Detection,” in Proc. APNet, Sydney, Australia, Aug 2024. [52] A. Mallick et al., “Matchmaker: Data Drift Mitigation in Machine Learning for Large-Scale Systems,” in Proc. MLSys, Santa Clara, CA, USA, Aug 2022. [53] N. Malekghaini et al., “FSTC: Dynamic Category Adaptation for Encrypted Network Traffic Classification,” in Proc. IFIP Networking, Barcelona, Spain, 2023. [54] A. S. Li, A. Iyengar, A. Kundu, and E. Bertino, “Revisiting Concept Drift in Windows Malware Detection: Adaptation to Real Drifted Malware with Minimal Samples,” in Proc. NDSS, San Diego, CA, USA, Feb 2025.

16

[55] X. Zhang, Y. Zhang, M. Zhong, D. Ding, Y. Cao, Y. Zhang, M. Zhang, and M. Yang, “Enhancing State-of-the-art Classifiers with API Semantics to Detect Evolved Android Malware,” in Proc. ACM CCS, Virtual Event, USA, 2020. [56] X. Wang, Y. Wang, Y. Lai, Z. Hao, and A. X. Liu, “Reliable Open-Set Network Traffic Classification,” IEEE TIFS, vol. 20, pp. 2313–2328, 2025. [57] D. Yu, X. Chen, Y. Zhang, Y. Liang, Y. Qiao, and F. Lai, “EMA: Efficient model adaptation for learning-based systems,” 2026, https://arxiv.org/abs/2605.13942. [58] A. A. Qahtan et al., “A PCA-Based Change Detection Framework for Multidimensional Data Streams: Change Detection in Multidimensional Data Streams,” in Proc. ACM SIGKDD, Sydney, NSW, Australia, Aug 2015. [59] D. M. dos Reis et al., “Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test,” in Proc. ACM SIGKDD, San Francisco, California, USA, Aug 2016. [60] X. Song, M. Wu, C. Jermaine, and S. Ranka, “Statistical change detection for multi-dimensional data,” in Proc. ACM SIGKDD, San Jose, California, USA, Aug 2007. [61] Y. Qing, Q. Yin, X. Deng, Y. Chen, Z. Liu, K. Sun, K. Xu, J. Zhang, and Q. Li, “Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic,” in Proc. NDSS, ser. NDSS Symposium, San Diego, California, USA, Feb 2024. [62] M. Germain et al., “MADE: Masked Autoencoder for Distribution Estimation,” in Proc. ICML, Lille, France, Jul 2015. [63] F. Regol, L. Schwinn, K. Sprague, M. Coates, and T. Markovich, “When to Retrain a Machine Learning Model,” in Proc. ICML, vol. 267, Jul 2025, pp. 51 369–51 404. [64] A. Bifet and R. Gavaldà, Learning from Time-Changing Data with Adaptive Windowing, April 2007, pp. 443–448. [65] V. Cerqueira, H. M. Gomes, A. Bifet, and L. Torgo, “STUDD: a Student– Teacher Method for Unsupervised Concept Drift Detection,” Machine Learning, vol. 112, pp. 4351–4378, 2023. [66] J. Z. Kolter and M. A. Maloof, “Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts,” Journal of Machine Learning Research, vol. 8, no. 91, pp. 2755–2790, 2007. [67] W. N. Street and Y. Kim, “A Streaming Ensemble Algorithm (SEA) for Large-scale Classification,” in Proc. ACM SIGKDD, ser. KDD ’01, August 2001. [68] B. Zhang, L. Li, S. Yang, S. Wang, Z.-J. Zha, and Q. Huang, “StateRelabeling Adversarial Active Learning,” in Proc. IEEE/CVF CVPR, Jun. 2020. [69] G. M. van de Ven, N. Soures, and D. Kudithipudi, Continual Learning and Catastrophic Forgetting. Elsevier, 2025, p. 153–168. [70] C. N. S. Suite, “YAF: Documentation,” https://tools.netsa.cert.org/yaf/ docs.html, [n.d.], accessed: Mar-2026. [71] Y. Wang, M. Lyu, and V. Sivaraman, “Characterizing User Platforms for Video Streaming in Broadband Networks,” in Proc. ACM IMC, Madrid, Spain, Nov 2024. [72] L. van der Maaten and G. Hinton, “Visualizing Data using t-SNE,” Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579–2605, 2008. [73] C. Fu, Q. Li, E. Bertino, and K. Xu, “Training with Only 1.0 ‰ Samples: Malicious Traffic Detection via Cross-Modality Feature Fusion,” in Proc. ACM CCS, Oct 2025. [74] S. Rabanser et al., “Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift,” in Proc. NeurIPS, Vancouver, BC, Canada, Dec 2019. [75] D. W. Scott, Multivariate Density Estimation: Theory, Practice, and Visualization. John Wiley & Sons, 1992. [76] A. Charpentier and E. Flachaire, “Log-Transform Kernel Density Estimation of Income Distribution,” Nov. 2014, https://shs.hal.science/halshs-01115988. [77] A. T. Jones et al., “Kernel Density Estimation on Positive Data via the logKDE Package for R,” 2018, https://cran.rproject.org/web/packages/logKDE/vignettes/logKDE.pdf. [78] M. Li and I. Sethi, “Confidence-based Active Learning,” IEEE PAMI, vol. 28, no. 8, pp. 1251–1261, 2006. [79] A. Pashamokhtari, N. Okui, Y. Miyake, M. Nakahara, and H. H. Gharakheili, “Inferring Connected IoT Devices from IPFIX Records in Residential ISP Networks,” in Proc. IEEE LCN, Edmonton, AB, Canada (virtual event), Oct 2021. [80] M. T. Paracha et al., “IoTLS: Understanding TLS Usage in Consumer IoT Devices,” in Proc. ACM IMC, Virtual Event, Nov 2021.

A PPENDIX A. Distribution Modeling Cases Section V-C identified five possible cases when modeling feature distributions using KDE. This appendix summarizes those cases and their prevalence across the training data. To gain a better understanding of the composition of these scenarios, each one is denoted by the number of cases among classes and features that it accounts for across all class-feature pairs when modeling the training phase data. During detection, degenerate components are treated as perfectly conforming at their support points and non-conforming elsewhere, while continuous components are evaluated using the estimated density. 1) (181 cases) Only a LNKDE component: when no training observation at X = 0 exists and the dataset is not singular. Later in the test phase when performing detection, the PDF will be manually evaluated as 0 at any realization of X = 0, since log is not defined at 0. 2) (4 cases) Only one degenerate component at x0 > 0: when the training data is singular at x0 . Later in the test phase for detections, the PDF will be evaluated as “infinity” (to meet any threshold) at X = x0 and 0, elsewhere. 3) (272 cases) One degenerate component at X = 0 together with an LNKDE component: when the non-zero observations are not singular, allowing for the application of LNKDE. The PDF will be evaluated as “infinity” at X = 0 and according to LNKDE, elsewhere. 4) (1 case) Two degenerate components at X = 0 and X = x0 > 0: when the training data has realizations at X = 0 and the rest of the datapoints are all equal to x0 . The PDF is evaluated as “infinity” at X = 0, x0 and 0, elsewhere. 5) (4 cases) One degenerate distribution at X = 0: when the training samples are all equal to zero. The PDF will be evaluated as “infinity” at X = 0 and zero, elsewhere. As expected, the vast majority of class-feature pairs are modeled using either LNKDE alone or LNKDE together with a point mass at zero, indicating that more complex cases occur only rarely in practice. B. Sensitivity Analysis for ϵ We used ϵ = 0.01 throughout our experiments and evaluations presented in this paper. Here, we assess the sensitivity of the detector to values of ϵ in the neighborhood of this operating point. Table V summarizes the TPR, specificity and balanced accuracy results for values of ϵ near 0.01. The results show relatively low sensitivity to the precise choice of ϵ. Therefore, once ϵ has been selected at the appropriate order of magnitude (e.g., 10−2 ), further fine-tuning has little impact on detector performance. C. Consistency of Feature-Level Explanations 1) Indicator Weights: We first investigate whether true and false detections differ in the number of non-conforming features contributing to their explanation. Fig. 14a compares

17

'24

7

'23 an

1J

ul'

22

1J

Time

1J

ul'

23

1J

an

True positives False positives

Class

Am Am azo 1 Am azon n Ec Jan' azo Ec ho D 22 n E ho ot ch Gen o Fir Sho 2 Ap e T w 5 Go ple V og H Stic le om k Ch ep G rom od Go oog eca o le st I-O gle N Hom DA est e TA Mi Q ni Lin J Watc e C VC h lo H Pa na Na va W ub Pa sonic ture ave na D Re so oo m Ph nic H rph o So ilips ome one ny Hu U Sm e B nit TP art S ridg TP -Link pea e -L k TP ink S Kasa er -Li m Pr nk ar o S t Wi W mar LED thi an t P ng sv lug s N iew ok Q ia 5 Bo dy

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

A Am maz Am azo on Ec azo n Ec ho n E ho Do ch Ge t o n Fir Sho 2 A p Go pl e TV w 5 og e H St le om ick Ch e G ro po Go oog mec d og le ast I-O le N Ho DA es me TA t M QW ini Lin a e C JVC tch lov Hu Pa na Na a b Pa soni ture Wav na c D R e so o e Ph nic orp mo So ilips Hom hone ny H e U Sm ue n TP art Brid it TP -Link Spe ge a TP Link Kasa ker -Li Sm P nk ar ro Wi W Sma t LED thi an rt ng sv Plu s N iew g ok Q ia 5 Bo dy

Indicator weight

0

Class

(a) Average drift indicator weights.

(b) Indicator entropy across classes and over time.

Fig. 14: Average drift indicator weights (a) show that the false positives and true positives cannot be effectively distinguished using the feature-drift indicator weights. The entropy trace (b) demonstrates a higher ambiguity in the indicators for false positives than true positives. Input: A labeled training dataset W0∗

and unlabeled deployment windows of traffic {Wi |1 ≤ i ≤ T }, sensitivity threshold ϵ. Output: Progressively evaluated performance {Πi |1 ≤ i ≤ T }, as well as detection labeling cost {Λi |1 ≤ i ≤ T } in all deployment windows. ∗ Wcum,0 ← W0∗ M0 ← M(W0∗ ) D0 ← D(W0∗ ; ϵ) for i = 1 to T do Πi ← perf(Wi∗ ; Mi−1 ) // In iteration i, ground-truth labels for Wi are only available for the performance evaluation purpose. ∗ D̃i−1 (Wi ; Mi−1 ) ← label(D̃i−1 (Wi ; Mi−1 )) Λi ← |D̃i−1 (Wi ; Mi−1 )| ∗ ∗ ∗ Wcum,i ← Wcum,i−1 ∥ D̃i−1 (Wi ; Mi−1 ) // The || operator denotes concatenation. ∗ Mi ← M(Wcum,i ) ∗ Di ← D(Wcum,i ) end for return {Πi |1 ≤ i ≤ T }, {Λi |1 ≤ i ≤ T } TABLE V: Detection results show low sensitivity to ϵ. ϵ TPR Specificity Balanced accuracy 0.0050 0.72 ± 0.23 0.53 ± 0.29 0.62 ± 0.16 0.0075 0.74 ± 0.23 0.52 ± 0.28 0.63 ± 0.16 0.0100 0.77 ± 0.22 0.49 ± 0.27 0.63 ± 0.15 0.0125 0.79 ± 0.21 0.48 ± 0.26 0.63 ± 0.14 0.0150 0.80 ± 0.20 0.46 ± 0.25 0.63 ± 0.13

the indicator weights of true and false detections across device classes. No consistent separation is observed between the two groups, and substantial temporal variation is evident in both. These results indicate that the number of non-conforming features alone is insufficient to distinguish true detections from false ones.

2) Diversity of Feature-Level Explanations: We next investigate the feature-level explanations across classes over time for both the true and false positives. To quantify the diversity of feature-level explanations, we compute the entropy of the indicator bit strings within fortnightly windows for every device class. Fig. 14b shows that true detections (left column) consistently exhibit lower entropy than false detections (right column), indicating that true behavioral evolution tends to produce more structured and repeatable explanation patterns. Overall, the indicator-weight and entropy analyses show that feature-level explanations remain structured across deployment, while true behavioral evolution exhibits considerably lower explanation diversity than false detections. These observations further support the detector’s ability to provide meaningful and interpretable explanations of behavioral evolution. 1.0 0.9

Macro F1-score

Algorithm 3 Adaptation Using the Detected Instances.

0.8 0.7 0.6

Full adaptation No adaptation Adjusted sampling rate below 0.1% The 0.1% with lowest confidence The 0.1% with lowest confidence (stratified) Confidence-adjusted random labeling

0.5 1 Jan'22 1 Apr'22 1 Jul'22 1 Oct'22 1 Jan'23 1 Apr'23 1 Jul'23 1 Oct'23 1 Jan'24

Time

Fig. 15: Temporal comparison of confidence-based and behavior-aware adaptation strategies. D. Formal Description of Detector-guided Adaptation The formal description of the detector-guided adaptation scheme (as discussed in §VI-A) is presented in Algorithm 3. E. Performance of the Confidence-based Methods Over Time

18

∗ 𝑊!"#,%&'

𝑊%&)

𝑊%

1.0 0.9

𝒟

"!"# 𝐷

0.8

∗(label)

"$%&, !") 𝐷

𝑆!"#,%

∗(label) 𝒟 "*+&&, !") 𝐷

𝑆&'##,% "!"# 𝐷

𝑆,-,!

(a) Detector update strategies for evaluating stability, consistency, and efficiency.

0.7

Goodness metrics

𝒟

0.6 0.5 0.4 0.3 0.2

η1 (Stability)

0.1

η2 (Consistency)

0 1 Jan’22

1 Apr’22

1 Jul’22

1 Oct’22

η3 (Efficiency)

1 Jan’23

1 Apr’23

1 Jul’23

1 Oct’23

1 Jan’24

Time

(b) Evaluation metrics (η1 , η2 , and η3 ) over the deployment period.

Fig. 16: Evaluation of the behavioral signal for adaptation. (a) Detector update strategies used to assess stability, consistency, and efficiency. (b) Corresponding evaluation metrics over the deployment period. Values close to one indicate that detector updates using only selected instances closely preserve the behavior of updates performed using all labeled deployment traffic. Fig. 15 presents the temporal classification performance of the behavior-aware and confidence-guided sampling strategies, together with two confidence-based instance-selection baselines discussed in §VI-D. The behavior-aware and confidenceguided strategies achieve comparable performance throughout deployment and generally outperform the two confidencebased instance-selection approaches. As reported in Table IV, the behavior-aware strategy achieves this performance at a slightly lower labeling cost while additionally offering featurelevel explanations of behavioral evolution.

F. Measures of Detector Goodness for Adaptation This appendix provides the temporal results supporting the summary statistics reported in §VI-E. As discussed in §VI, we evaluate the detector according to three desirable properties: stability (η1 ), consistency (η2 ), and efficiency (η3 ). Computation of the goodness metrics relies on detection results performed continuously based on three different strategies as demonstrated in Fig. 16a. The classifier model M is fitted on the training data and kept fixed throughout the process. The goodness metrics η1 , η2 and η3 are depicted over time in Fig. 16b. The results show that all three properties are consistently satisfied throughout deployment.

Related documents

• Binary Anomaly Detection in Streaming IoT Traffic under Concept Drift#974252
• Concept Drift Detection and Adaptive Retraining of Malware Classification Models#451217
• Learner-based Concept Drift Detection: Analysis and Evaluation#290603
• ADAWU-IDS: a dynamic adaptive weight update mechanism for concept drift-aware network intrusion detection systems#306104
• Navigating the Latent Manifold: Proactive Concept Drift Adaptation for Resilient NIDS#985958