arXiv CSopen access

Concept Drift Adaptation Using Self-Supervised and Reinforcement Learning In Android Malware Detection

Concept Drift Adaptation Using Self-Supervised and Reinforcement Learning In Android Malware Detection

arXiv:2605.24294v1 [cs.CR] 22 May 2026

Ahmed Sabbah† , Mohammed Kharma† , Mohammad Alkhanafseh† , Samer Zain† , Radi Jarrar† , David Mohaisen‡ † Birzeit University ‡ University of Central Florida Abstract—Android malware detectors often degrade after deployment because of concept drift, while full retraining at each maintenance step is costly. We propose a chronological adaptive maintenance framework that models deployment-time maintenance as a sequential decision problem. The framework learns a stable latent representation through self-supervised learning during initialization, freezes the encoder, measures latent drift in the fixed representation space, and performs lightweight downstream adaptation using a trainable adapter and classification head. A proximal policy optimization controller selects low-cost maintenance actions based on the detector state, including current utility, retention on a fixed memory set, latent drift indicators, and update cost. We evaluate the framework under a causal deployment-style protocol on emulator and real Android malware datasets with static and dynamic features. Results show that the RL controller provides a strong costaware adaptation strategy, consistently remaining among the topperforming policies while achieving a favorable balance between temporal performance, memory retention, and maintenance cost under non-stationary deployment conditions. Index Terms—Android Malware; Self-Supervised Learning; Reinforcement Learning; Malware Detection; Concept Drift.

I. I NTRODUCTION Android is the dominant mobile operating system worldwide, accounting for 67.46% of the global market in March 2026 [35]. Its scale and prevalence make it a persistent target for mobile threats. According to Kaspersky, attacks on Android smartphone users in the first half of 2025 were 29% higher than in the first half of 2024 and 48% higher than in the second half of 2024 [22]. As a result, machine learning has become a widely adopted approach for Android malware detection using static, dynamic, or hybrid features. However, the effectiveness of these detectors depends not only on their initial accuracy but also on their ability to remain reliable as the Android ecosystem evolves. In supervised learning, concept drift refers to changes in the relationship between input variables and target labels over time [10]. In Android malware detection, this challenge is particularly severe because malware behavior, benign application behavior, platform restrictions, API usage, and feature distributions continuously evolve [1]. Consequently, detectors trained on historical data may perform well during development but degrade substantially after deployment. A growing body of work has shown that this degradation is persistent and driven by multiple factors. Early studies highlighted the challenge of concept drift in Android malware detection and explored ensembles and online adaptation to sustain performance over time [20], [39]. Later work examined the causes of temporal degradation more closely, showing that timestamping choices, cross-device behavior, malware family

evolution, benign application changes, and the distinction between feature-space and data-space drift all affect long-term detector reliability [7], [8], [14]–[16]. More recent studies reinforced these findings through large longitudinal benchmarks, showing that concept drift remains widespread across datasets, feature types, and detector families [17], [32]. Collectively, these results suggest that Android malware detection should be treated not as a one-time training problem, but as a longterm maintenance problem under temporal distribution shift. To address this issue, prior work has explored several adaptation strategies. Ensemble and online learning methods incrementally update detectors as new data arrive [20], [21], [39]. Transfer learning approaches reuse prior knowledge to improve adaptation to newer samples [9], [12]. Active learning and pseudo-labeling methods reduce annotation costs while supporting continuous retraining [2], [3], [6], [27], [28]. Rejection-based and optimization-oriented methods identify uncertain or drifted samples, improve training, and enhance robustness under evolving data distributions [5], [18], [24], [29]. More recently, self-supervised learning (SSL) and reinforcement learning (RL) have also emerged in this space. MADCAT showed that self-supervised masked autoencoding with testtime adaptation can improve robustness under concept drift in Android malware detection [31], while DRMD demonstrated that reinforcement learning can support drift-aware decision making in time-aware malware detection settings [25]. Existing approaches do not jointly address adaptation, retention, and cost-aware maintenance under temporal drift. Despite this progress, an important gap remains. Most prior studies either analyze the causes of drift or adapt detectors through repeated retraining, continual relabeling, or full-model updates. Few approaches jointly address adaptation and retention within a deployment-oriented framework, and recent SSL and RL methods are typically applied in isolation. This creates a need for a unified chronological maintenance framework that can adapt to new drifts, preserve previously learned behaviors, and control update costs. In this study, we address this gap by proposing a chronological adaptive maintenance framework for Android malware detection under a temporal distribution shift. The framework is designed to reflect a deploymentstyle setting in which a detector is initialized from early time windows and then maintained as later windows arrive. Instead of retraining the entire model repeatedly, we learn a stable latent representation during initialization using SSL. An RL controller then selects cost-aware maintenance actions based on drift signals, current detection behavior, retention performance, and action history. It also adopts a realistic

evaluation protocol by separating adaptation and evaluation within each deployment window and measuring not only the current window performance, but also retention on a fixed memory set and the cumulative cost of maintenance actions. Contributions. This work makes several contributions. First, we formulate Android malware detection under temporal distribution shift as a chronological adaptive maintenance problem, where detectors are maintained over deployment windows instead of being repeatedly retrained from scratch. Second, we propose a unified framework combining selfsupervised learning and reinforcement learning: SSL learns a stable latent representation during initialization, after which deployment-time updates are limited to a lightweight adapter and classification head. Third, we introduce latent drift monitoring using previous-window and initialization-reference drift in the frozen representation space. Fourth, we develop a maintenance policy that selects among multiple cost-aware actions, including keep, head-tune, adapter-tune, joint-tune, and reset-adapter. Finally, we adopt a deployment-oriented evaluation protocol that jointly measures adaptation, retention, and maintenance cost under chronological evaluation. II. R ELATED W ORK Concept drift is a well-established challenge in supervised learning and refers to temporal changes in the relationship between features and labels [7]–[10], [12], [15]–[17], [21], [27], [32]. In Android malware detection, the problem is particularly severe because malware behavior, benign application behavior, platform APIs, and data collection conditions continuously evolve over time. Early studies showed that detectors trained on historical data degrade under chronological evaluation and therefore require explicit drift-aware maintenance rather than one-time training [5], [6], [9], [20], [24], [25], [29], [31], [39]. In the following, we review the main directions of prior work and position our framework relative to them. Drift characterization and empirical evidence. A first line of work studies why Android malware detectors degrade over time. Prior studies showed that timestamping choices, featurespace versus data-space drift, malware family evolution, benign application changes, feature type, and collection settings all affect chronological robustness [7], [8], [14]–[16]. More recent longitudinal benchmarks and empirical studies, including LAMDA [17] and the work of Sabbah et al. [32], further showed that concept drift persists across datasets, models, and feature representations and must therefore be treated as a deployment-time problem, and not only at design time. Adaptation through model updating. A second line of work addresses drift through detector updates as new data arrive. Hu et al. proposed NBCS, which combines feature selection, sliding windows, and multiple sub-classifiers to sustain performance over time [20]. DroidEvolver and DroidEvolver++ extended this direction with model pools, weighted voting, pseudo-label updates, and mechanisms for retiring outdated models [21], [39]. Transfer learning approaches reuse prior knowledge to adapt to emerging malware samples. Fu et al. fine-tuned an LSTM detector using augmented malware

data, while Garcia et al. studied transfer learning under class imbalance across conventional classifiers [9], [12]. Other work explored anomaly-detection-based online adaptation for settings with limited malicious labels [11] and continual-learning regularization to reduce regression after updates [13]. Although these approaches delay degradation, most still depend on repeated retraining, additional labels, or broad parameter updates that may be costly in long-term deployment. Label-efficient and optimization-oriented maintenance. Because continuous relabeling is expensive, several studies aim to reduce annotation cost while maintaining adaptation quality. Prior work explored active learning, pseudo-labeling, selective querying, and streaming-style retraining to support labelefficient adaptation under drift [2], [3], [6], [28]. Other studies examined periodic versus drift-triggered retraining [27], conformal prediction for rejecting uncertain or out-of-distribution samples [5], [29], and drift-resilient feature representations and training objectives [18], [24], [30]. While these approaches improve specific aspects of robustness, they do not formulate maintenance as a sequential decision problem that jointly considers adaptation benefit, retention, and update cost. Closest directions to our framework. Most closely related to ours are self-supervised adaptation and reinforcement learning. MADCAT used self-supervised masked autoencoding with test-time adaptation to improve robustness under Android malware drift [31], while DRMD formulated time-aware malware detection as a deep reinforcement learning problem for driftaware classification and rejection [25]. However, MADCAT focuses on representation adaptation and DRMD focuses on decision optimization. Neither jointly addresses stable representation, lightweight maintenance, memory retention, and cumulative update cost within a unified framework. Our work. Prior work established that Android malware concept drift is persistent, multi-causal, and difficult to address through static training alone. Existing approaches have contributed important advances through drift analysis, online learning, transfer learning, active learning, rejection mechanisms, and recent SSL- or RL-based methods [5], [6], [9], [20], [24], [25], [29], [31], [39]. However, most methods still depend on frequent relabeling, full-model retraining, or objectives focused only on current adaptation performance. Our work instead targets chronological maintenance under deployment constraints by learning a stable latent space during initialization, restricting deployment-time updates to lightweight components, and using RL to select cost-aware actions based on drift, current performance, and retention. III. BACKGROUND A. Reinforcement Learning Reinforcement learning (RL) addresses sequential decisionmaking problems in which an agent interacts with an environment to maximize cumulative rewards over time [36]. RL has been successfully applied in domains such as robotics [19], gaming [26], [37], and increasingly in cybersecurity applications, including malicious botnet detection in IoT environ-

ments [4]. RL problems are commonly formulated as Markov Decision Processes (MDPs), represented as: M DP = (S, A, T, R, γ),

(1)

where S denotes the set of states, A is the action space, T defines the state transition probability distribution, R is the reward function, and γ ∈ [0, 1] is the discount factor controlling the trade-off between immediate and future rewards. The agent follows a policy π that maps states to actions: π(a|s) = P (at = a | st = s).

(2)

Under policy π, the state-value function is defined as [36]: "∞ # X π t−1 v (s) = Eπ γ Rt | S1 = s , (3) t=1

while the state-action value function is: "∞ # X π t−1 q (s, a) = Eπ γ Rt | S1 = s, A1 = a .

(4)

t=1

Using the Bellman equations [36], the recursive value formulation becomes: v π (s) = Eπ [Rt+1 + γv π (St+1 ) | St = s] .

(5)

RL methods are categorized into model-based and modelfree approaches where the first rely on explicit transition dynamics and the latter methods learn directly through interaction with the environment. Because cybersecurity environments are dynamic and evolve over time, model-free approaches are often more suitable. In this work, we use proximal policy optimization (PPO), a model-free RL algorithm. B. Proximal Policy Optimization (PPO) PPO is a policy-based reinforcement learning algorithm designed to achieve stable policy updates while maintaining sample efficiency [34]. Unlike value-based methods such as Q-learning, PPO directly optimizes the policy through gradient ascent on the expected cumulative reward. PPO uses a clipped surrogate objective: h  i LCLIP (θ) = Et min rt (θ)Ât , clip(rt (θ), 1 − ϵ, 1 + ϵ)Ât , (6)

where rt (θ) = πθπθ (a(at |st |st )t ) is the policy probability ratio, Ât is old the advantage estimate at t, and ϵ controls the clipping range. PPO performs multiple mini-batch updates from sampled experiences while constraining harmful policy shifts through the clipped objective, combining the stability of trust-region methods with the simplicity of policy-gradient optimization.

malware detection, concept drift arises because malware behavior, benign application behavior, API usage, and platform restrictions evolve over time, causing detectors trained on historical data to degrade after deployment. IV. M ETHODOLOGY We propose a chronological adaptive maintenance framework for Android malware detection under temporal distribution shift. The framework simulates deployment operation, where a detector is initialized on early time windows and then maintained sequentially as new windows arrive. Instead of repeatedly retraining the full model, the framework learns a stable latent representation once, freezes it, and performs lightweight downstream maintenance using a trainable adapter and classification head controlled by reinforcement learning. As shown in Figure 1, the framework consists of three stages: Stage 1: Initial SSL pretraining. A feature scaler and masked self-supervised encoder are trained using the first K chronological windows. After pretraining, the decoder is discarded and the encoder is frozen. • Stage 2: Latent drift quantification. For each newly arrived window, drift is measured in the frozen latent space relative to both the previous window and a fixed initialization reference pool. • Stage 3: RL-controlled chronological maintenance. Starting from the first post-initialization window, a PPO policy selects among several low-cost maintenance actions that update only the adapter and/or classification head while keeping the encoder frozen. •

A. Problem definition Let {Dt }Tt=1 denote a chronological sequence of labeled t time windows, where Dt = {(xi , yi )}ni=1 , xi ∈ Rd is the feature vector, and yi ∈ {0, 1} indicates benign or malware class membership. The first K windows, {D1 , . . . , DK }, define the initialization period used for feature scaling, SSL encoder pretraining, construction of a fixed memory set, and construction of a fixed initialization-reference latent pool. Sequential maintenance begins from DK+1 . At each deployment step, the objective is to maximize current-window detection performance, preserve previously learned behavior, and minimize maintenance cost.

C. Concept Drift

B. Pre-processing and normalization

Concept drift refers to changes in the statistical properties of data over time in non-stationary environments [33]. Let Pt (X, Y ) denote the joint distribution of features X and labels Y at time t. Concept drift occurs when:

After feature selection and chronological partitioning into time windows {Dt }Tt=1 , samples with missing values are removed to preserve a consistent feature space across windows. Each feature vector is normalized using z-score scaling: x̃ = x−µ σ , where µ and σ denote the feature mean and standard deviation. To preserve temporal S causality, µ and σ K are estimated from the initialization period t=1 Dt . The same scaler is then reused unchanged for all subsequent windows.

Pt1 (X, Y ) ̸= Pt2 (X, Y ),

t1 ̸= t2 .

Prior work categorizes concept drift into several forms, including sudden, gradual, and recurring drift [38]. In Android

Stage 2: Latent Drift Quantification (Distributional Profiling)

Stage 1: Initial SSL Pretraining (Representation Stabilization)

Latent Z ෡) Reconstruction Loss (X′, 𝑿 Target X′

෡ Reconstructed 𝑿

Init Ref. Pool {𝐷1 … 𝐷𝑘 }

Back propagation

Current Window 𝐷𝑡

Masking

Scaler

input 𝑥

𝑚

Training

Encoder 𝐸

Decoder 𝐷

Warm start 𝐷𝑘

Current Window 𝐷𝑡 (𝐷𝑘+1 )

🔒 Frozen Encoder

RL Controller (PPO Agent) State 𝑆𝑡

𝐸

Window 𝐷𝑡 − 1

𝐷𝑡𝑒𝑣 𝐷𝑡𝑡𝑟 Detector Model Frozen 𝐸 Encoder Trainable 𝐴 Adapter

Latent 𝑍𝑡 Drift Calculation (KS & WD Metrics)

🔒 Frozen Encoder 𝐸 Initialization Period {𝐷1 … 𝐷𝑘 }

Stage 3: RL-Controlled Chronological Adaptation (Adaptive Maintenance)

Reuse Frozen Encoder

Actions : 0.Do Nothing 1.Update H 2.Update A 3.Update A+H 4.Reset A, Apply 3

Trainable 𝐻 Head

Prediction 𝑦ො

Reward 𝑅𝑡

Performance Evaluation

Fixed Memory Set 𝓜

Current & Memory

Figure 1: Proposed stages of maintenance pipeline. Stage (1) fits the scaler and pretrains the encoder using only the initialization windows {D1 , . . . , DK }. Stage (2) quantifies latent drift in the frozen representation space using previous-window and initialization-reference comparisons. Stage (3) starts from DK+1 after a warm start on DK , splits each deployment window into disjoint adaptation and evaluation subsets, and lets a PPO controller choose a maintenance action over the trainable adapter and head while retention is monitored through a fixed memory set M. C. Detector architecture The detector consists of three components: ŷ = H(A(E(x̃))), where E(·) is a frozen encoder that maps normalized input to a latent representation z ∈ Rdz , A(·) is a lightweight trainable adapter, and H(·) is a trainable classification head. The encoder is implemented as a multilayer perceptron mapping d → 256 → 128 → dz . The decoder used during SSL pretraining mirrors this architecture in reverse. The adapter is implemented as a lightweight residual bottleneck transformation: A(z) = z + Wup (ReLU(Wdown z)) ,

(7)

where Wdown : Rdz → Rb projects the latent representation into a lower-dimensional bottleneck space, Wup : Rb → Rdz projects it back to the original latent dimension, and b ≪ dz . The residual connection preserves the original representation while allowing the adapter to learn a compact correction. Freezing the encoder stabilizes the latent space across time, making drift estimates comparable and restricting sequential maintenance to a small number of trainable parameters. D. Stage 1: Initial SSL pre-training The encoder is pretrained during the initialization period using masked feature reconstruction [23]. For each normalized input x̃, a binary masking vector m ∈ {0, 1}d is sampled with masking probability p, where mj = 1 indicates that feature j is masked. The corrupted input is defined as: x̃(m) = x̃ ⊙ (1 − m), where ⊙ denotes element-wise multiplication. The masked input is passed through the encoder to obtain

the latent representation: z = E(x̃(m) ), and the decoder reconstructs the input: x̂ = D(z). The SSL objective is a masked mean-squared reconstruction loss computed only over masked feature positions:
2 Pd j=1 mj x̃j − x̂j LSSL = , (8) Pd j=1 mj + ϵ where ϵ > 0 avoids division by zero. After pretraining, the decoder is discarded and the encoder is frozen for the remainder of the pipeline. SSL learns stable feature dependencies directly from raw feature vectors without relying on a specific label distribution, reducing the need for repeated representation re-learning under concept drift. E. Stage 2: Chronological latent drift quantification To monitor temporal distribution shift, drift is quantified in the frozen latent space produced by the encoder. For each chronological window Dt , normalized samples are mapped t to latent representations: Zt = {E(x̃i )}ni=1 , where nt is the number of samples in Dt . Because the encoder remains fixed after initialization, latent distributions from different windows remain directly comparable over time. At each deployment step, two drift indicators are computed. The first is the previous-window drift: dprev (t) = Drift(Zt , Zt−1 ), which measures the change relative to the immediately preceding window. The second is the initialization-reference drift: dinit (t) = Drift(Zt , Zinit ), which measuresSdeviation from a K fixed initialization reference pool: Zinit = i=1 Zi .

The drift operator combines two complementary distributional distances computed independently for each latent dimension and averaged across dimensions. The first is the mean Kolmogorov-Smirnov (KS) distance: d

mean

KS

z 1 X sup |Fa,k (u) − Fb,k (u)| , (Za , Zb ) = dz u

k=1

where Fa,k and Fb,k are the empirical cumulative distribution functions of the k-th latent dimension. The second is the mean Wasserstein distance (WD): WDmean (Za , Zb ) =

dz 1 X

dz

st = [Bt , Ft , Mt , KStp , W Dtp , KSti , W Dti , at−1 , aget , ct−1 ]



(k)

W1 Za(k) , Zb



.

k=1

The KS distance captures localized distributional differences, while WD captures broader distributional displacement. Using both provides complementary views of latent drift. To keep drift computation tractable and comparable across windows with different sample sizes, drift estimation is performed on a fixed-size subsample of latent vectors from each window. F. Stage 3: RL-controlled chronological maintenance Warm start. Stage 3 performs chronological maintenance while keeping the encoder frozen. It begins with a supervised warm start on the last initialization window DK , where the adapter and classification head are jointly updated. Starting from DK+1 , the RL controller observes the current state, selects a maintenance action, and updates the detector. Deployment timing. The framework follows a deploymentstyle, window-based maintenance setting rather than online sample-by-sample prediction. At step t, a newly labeled window Dt becomes available. The controller constructs a chronological state using only information available up to that window, including detector performance, the fixed memory set, and latent drift indicators. A maintenance action is then selected and applied while keeping the encoder frozen. Because the state includes supervised performance on Dt , the setting represents periodic post-deployment maintenance. Disjoint adaptation and evaluation splits. At each deployment step t ≥ K + 1, the labeled window Dt is partitioned into two disjoint subsets: Dt = Dttr ∪ Dtev , where Dttr is used exclusively for supervised maintenance updates and Dtev is used exclusively for state construction and reward evaluation. This separation prevents optimistic bias from evaluating on samples used for the current update. Maintenance actions. At each step, the controller selects one action: at ∈ {A0, A1, A2, A3, A4}, where A0 performs no update, A1 updates only the classification head, A2 updates only the adapter, A3 jointly updates the adapter and classification head, and A4 reinitializes the adapter before jointly updating the adapter and classification head. All supervised updates minimize the cross-entropy loss: X LCE = − yc log p̂c , (9) c∈{0,1}

where yc is the target indicator for class c and p̂c is the predicted probability. If a training budget B is imposed, updates are performed using at most B samples from Dttr . Fixed memory set for retention. To measure retention without storing all previous windows, a fixed class-balanced memory set M is sampled from the initialization period and kept unchanged during deployment. Retention at time t is measured using balanced accuracy on M, making the retention estimate less sensitive to class imbalance. State representation. At deployment step t, the PPO controller observes the following 10-dimensional state vector:

where Bt and Ft are the pre-update balanced accuracy and macro-F1 on Dtev , respectively, and Mt is the pre-update balanced accuracy on M. The terms KStp and W Dtp denote drift relative to the previous window, while KSti and W Dti denote drift relative to the initialization reference pool. The remaining variables correspond to the previous action at−1 , the number of deployment windows since the last strong refresh action (A3 or A4), and the previous action cost ct−1 . Reward function. After applying action at using Dttr , the detector is re-evaluated on Dtev and M. Let Btpost , Ftpost , and Mtpost denote the post-update balanced accuracy, macro-F1, and memory-set balanced accuracy. The reward is: Rt = [αFtpost + (1 − α)Btpost + βMtpost + Gt − λc Cost(at )]

where α controls the trade-off between macro-F1 and balanced accuracy, β controls the contribution of memory retention, and λc controls the maintenance cost penalty. The within-step gain term is: Gt = η1 (Btpost − Btpre ) + η2 (Mtpost − Mtpre ), where Btpre and Mtpre are the pre-update balanced accuracy values. In this formulation, drift is not directly penalized; instead, it is provided as part of the state representation. Policy learning. The maintenance policy πθ (a | s) is learned using PPO. At each step, the controller constructs the current state, selects a maintenance action, updates the detector using Dttr , evaluates the updated detector on Dtev and M, computes the reward, and stores the transition for PPO optimization. Deterministic baselines. To evaluate the benefit of the RL controller, we compare it against several fixed maintenance strategies. Frozen-init always selects A0, head-tune selects A1, adapter-tune selects A2, and joint-tune selects A3. Periodicjoint(k) selects A3 every k deployment windows and A0 otherwise. Drift-Rule selects A3 whenever any drift indicator exceeds a predefined threshold and selects A1 otherwise. G. Evaluation Protocol All policies are evaluated under the same chronological deployment protocol. They use the same initialization period, frozen encoder, feature configuration, fixed memory set, initialization reference pool, adaptation and evaluation split, training budget, and evaluation budget. The proposed PPO controller is compared against fixed maintenance baselines with predefined actions from the same maintenance action set.

Table I: Consolidated Experimental Parameters and Settings. Category

Item

Setting / Value

Data

Setup

2008–2020; K = 3 (Init); Seeds: {0 . . . 5}

State (st )

Vector (d = 10) Curr/Mem Perf, Causal Drift (KS, WD vs. t − 1 & Init), Prev. Action, Age, Prev. Cost

Actions (at ) Operators Update Epochs Costs (C)

A0 : Keep, A1 : Head, A2 : Adapter, A3 : Joint, A4 : Reset+Joint Head: 2; Adapter/Joint: 3; Warm start: 4 0.0, 0.5, 1.0, 1.5, 2.5

Gain Weights Multipliers

R = (αF1 + (1 − α)BA) + βBAmem + Gain − λc C η1 (∆Curr) = 1.0, η2 (∆Mem) = 0.5 α, β = 0.5; λc = 0.02

PPO

Architecture Hyperparams

MLP Policy; n steps = 64; Batch=64 lr = 3 × 10−4 ; γ = 0.95; ent coef = 0.01

Buffers

Budgets

Train/Eval/Mem: 2000 samples (Class Stratified)

Reward

Formulation

Unless otherwise stated, performance is measured after the selected maintenance action has been applied at each deployment step. We use four primary evaluation criteria: 1) Current-window balanced accuracy. 2) Current-window macro-F1. 3) Memory-set balanced accuracy. 4) Accumulated maintenance cost. Balanced accuracy is reported to account for potential class imbalance as it weighs both classes equally through class-wise recall. Macro-F1 is also reported since it captures the class-wise precision-recall tradeoff and is therefore more informative than plain accuracy under possible imbalance. To summarize temporal performance, we compute the Area Under Time (AUT), following the chronological evaluation protocol used in TESSERACT [29]. We report AUT for accuracy, macro-F1, and memory-set accuracy. Additionally, the accumulated maintenance cost is computed as TotalCost = PT Cost(A t ), where At denotes the maintenance action t=K+1 selected at deployment step t. V. R ESULTS AND D ISCUSSION Figure 2 shows that temporal performance depends on both the data source and feature type. Static features produce more stable trajectories across emulator and real data, whereas dynamic features yield lower and more variable performance, particularly for current-window macro-F1. FrozenInit is consistently among the weakest policies, confirming that initialization-only training is insufficient under drift. In contrast, adaptive maintenance policies improve deployment stability. The proposed RL controller is particularly effective because it selects maintenance actions according to the current deployment state, drift indicators, memory behavior, and update cost rather than following a fixed update schedule. Takeaway. Maintenance during deployment is necessary to address temporal drift. Static features are easier to maintain than dynamic features and the RL controller offers a costaware alternative to fixed maintenance schedules by adapting its update decisions to observed deployment states.

A comparison of the adaptive methods shows that no deterministic rule is uniformly optimal across all settings. HeadTune performs strongly in emulator-dynamic settings, whereas Joint-Tune, Adapter-Tune, and Periodic-Joint(2) remain competitive across several static windows. This is expected because deterministic baselines follow fixed update rules that perform well only when they match the local maintenance needs. In contrast, the proposed RL controller remains competitive across all four settings by selecting maintenance actions according to the current detector state, drift indicators, memory behavior, and update cost. This distinction is most visible in the more challenging dynamic feature settings. In the realdynamic setting, RL maintains comparatively strong currentwindow balanced accuracy while keeping memory accuracy relatively stable, whereas several aggressive deterministic rules show sharper late-stage degradation, particularly in macro-F1. More broadly, RL does not need to dominate every year or metric to be useful. Its contribution is replacing handcrafted maintenance rules with a learned, state-dependent controller operating over the same action space. These results support the central claim of this work: under temporal distribution shift, the maintenance policy itself should be adaptive. Takeaway. RL does not enforce a single preferred update type but learns when different maintenance actions are appropriate. Deterministic baselines are constrained by the limited of their predefined action-selection rules. Table II summarizes AUT-based current-window balanced accuracy, macro-F1, and accumulated maintenance cost across all settings. No deterministic baseline is uniformly optimal. In emulator-dynamic, Head-Tune achieves the strongest average performance, with approximately 0.551 accuracy and 0.404 macro-F1. In real-dynamic, RL achieves the highest average accuracy, approximately 0.628, whereas Periodic-Joint(2) achieves the highest macro-F1, approximately 0.473. The main advantage of RL is not that it dominates every metric, but that it provides a learned, state-dependent maintenance policy over the same action space. This is most visible in real-dynamic, where RL achieves the highest average accuracy while using a substantially lower cost, approximately 3.75, compared with 7.50 for Periodic-Joint(2), 10.00 for AdapterTune, and 15.00 for Joint-Tune and Drift-Rule. In real-static, RL achieves the highest average macro-F1, approximately 0.752, whereas Adapter-Tune achieves slightly higher accuracy at lower cost. Similarly, in emulator-static, RL remains competitive, with approximately 0.774 accuracy and 0.719 macroF1, while the slightly higher F1 values of Joint-Tune and DriftRule require the maximum cost of 15.00. These results support the central claim of this study: under temporal distribution shift, the maintenance policy itself should be adaptive. Deterministic baselines can perform well when their predefined update rules match local maintenance needs, but they remain constrained by fixed behaviors. In contrast, RL learns when to keep the detector unchanged, apply lightweight updates, or perform stronger adaptation based on the observed deployment state and cost. Another observation is the consistency of RL across seeds.

Static

Dynamic

0.8

Current accuracy

Static RL Frozen Head Adapter Joint Periodic(2) Drift-Rule

0.7 0.6 0.5

0.8 0.7 0.6

0.85

0.8

0.80

0.7

Memory accuracy

Memory accuracy

RL Frozen Head Adapter Joint Periodic(2) Drift-Rule

0.5

0.4

0.6 0.5 0.4 0.3 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Year

0.75 0.70 0.65 0.60 0.55 0.50 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Year

Year

(a) Current-window and memory accuracy (Emulator). Static

Current macro-F1

0.6

RL Frozen Head Adapter Joint Periodic(2) Drift-Rule

0.4 0.2

0.0 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

Year

Year

(c) Current-window macro-F1 (Emulator).

Year

(b) Current-window and memory accuracy (Real-data).

Dynamic

0.8

Current macro-F1

Dynamic

0.9

Current accuracy

0.9

Static

Dynamic

Year

Year

0.9 0.8 0.7 0.6 RL 0.5 Frozen Head 0.4 Adapter 0.3 Joint Periodic(2) 0.2 Drift-Rule 0.1 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020

(d) Current-window macro-F1 (Real-data).

Figure 2: Temporal performance across emulator and real datasets with static and dynamic features. Accuracy panels report current-window and memory-set performance, while F1 panels report current-window macro-F1. Although RL is not always the top method, its AUT values remain within relatively narrow ranges. For example, in emulator-static, RL maintains accuracy between 0.769 and 0.777 and macro-F1 between 0.707 and 0.732 across the six seeds. In real-dynamic, RL maintains accuracy between 0.618 and 0.648 while using lower cost than the more aggressive Joint-Tune and Drift-Rule baselines. This suggests that the learned controller is not only competitive on average, but also less sensitive to random seed variation than fixed update rules. Takeaway. The AUT results show that RL should be interpreted as a cost-aware and seed-consistent maintenance controller rather than as a method that must win every isolated metric. Across the evaluation settings, the RL remained competitive while avoiding the rigid behavior of the deterministic rules. This supports the use of learned, statedependent maintenance under temporal distribution shift. A. Threats to Validity Our findings are bounded by several factors: (1) the deployment protocol uses yearly chronological windows from 2008–2020 with initialization on the first K = 3 windows, so the results support yearly deployment-time maintenance under this chronology rather than arbitrary temporal granularities or deployment schedules; (2) the study assumes a label-available maintenance setting, where each newly labeled window is split into disjoint adaptation and evaluation subsets for state construction, maintenance, and reward computation, rather than fully online adaptation without immediate labels; (3) the comparison uses a fixed maintenance action space and deterministic baselines, with the RL controller trained for

4096 PPO steps, so policy rankings should be interpreted within this controlled design; and (4) the evaluation focuses on current-window balanced accuracy, current-window macroF1, memory-set balanced accuracy, accumulated maintenance cost, and AUT, and therefore does not assess properties such as calibration, adversarial robustness, or delayed feedback. VI. C ONCLUSION This paper presented a chronological adaptive maintenance framework for Android malware detection under temporal distribution shift. The framework learns a stable latent representation during initialization, freezes the encoder, quantifies latent drift in the fixed representation space, and performs lightweight downstream maintenance through an RLcontrolled adapter-head update mechanism. Rather than relying on fixed maintenance rules, the proposed RL controller learns state-dependent maintenance decisions based on drift, detector behavior, memory retention, and update cost. Experiments on emulator and real datasets showed that static features are easier to maintain than dynamic features, that no-maintenance deployment is insufficient under temporal drift, and that no deterministic maintenance rule is uniformly optimal across settings. In contrast, the RL controller remained competitive while using more adaptive and cost-aware maintenance behavior. Future work includes evaluating the framework under additional datasets, temporal granularities, and weaker-feedback deployment settings, as well as expanding the maintenance action space and evaluation criteria.

Table II: AUT summary across seeds and policies. Policy abbreviations are defined as follows: RL: reinforcement learning controller, Fr: Frozen-Init, Hd: Head-Tune, Ad: Adapter-Tune, Jo: Joint-Tune, P2: Periodic-Joint(2), and Dr: Drift-Rule, Acc: Accuracy, C: Cost. S0–S5 are the seeds. Color-coding: ■: worst, ■: second, and ■: best. Emulator Policy

Static Acc

S0 RL Fr Hd Ad Jo P2 Dr S1 RL Fr Hd Ad Jo P2 Dr S2 RL Fr Hd Ad Jo P2 Dr S3 RL Fr Hd Ad Jo P2 Dr S4 RL Fr Hd Ad Jo P2 Dr S5 RL Fr Hd Ad Jo P2 Dr

F1

Real Dynamic

C

Acc

F1

Static C

Acc

F1

Dynamic C

Acc

F1

C

0.775 0.718 10.500 0.508 0.363 12.000 0.811 0.742 8.500 0.620 0.401 0.000 0.686 0.546 0.000 0.500 0.234 0.000 0.710 0.552 0.000 0.620 0.401 0.000 0.793 0.689 5.000 0.564 0.432 5.000 0.783 0.662 5.000 0.634 0.449 5.000 0.769 0.715 10.000 0.436 0.298 10.000 0.812 0.743 10.000 0.577 0.453 10.000 0.777 0.724 15.000 0.512 0.370 15.000 0.794 0.741 15.000 0.536 0.426 15.000 0.727 0.616 7.500 0.505 0.396 7.500 0.796 0.689 7.500 0.619 0.482 7.500 0.776 0.724 15.000 0.513 0.381 15.000 0.797 0.743 15.000 0.533 0.422 15.000 0.777 0.707 9.500 0.543 0.451 15.000 0.821 0.760 10.000 0.618 0.450 6.500 0.641 0.479 0.000 0.500 0.236 0.000 0.713 0.548 0.000 0.606 0.380 0.000 0.753 0.632 5.000 0.570 0.441 5.000 0.820 0.707 5.000 0.618 0.461 5.000 0.756 0.664 10.000 0.542 0.419 10.000 0.821 0.758 10.000 0.555 0.435 10.000 0.787 0.738 15.000 0.541 0.449 15.000 0.818 0.761 15.000 0.547 0.432 15.000 0.742 0.626 7.500 0.538 0.429 7.500 0.799 0.687 7.500 0.612 0.468 7.500 0.781 0.731 15.000 0.532 0.434 15.000 0.814 0.757 15.000 0.548 0.431 15.000 0.769 0.707 9.500 0.516 0.391 10.500 0.807 0.755 15.000 0.648 0.456 4.000 0.639 0.478 0.000 0.500 0.234 0.000 0.713 0.551 0.000 0.586 0.349 0.000 0.751 0.645 5.000 0.548 0.393 5.000 0.811 0.712 5.000 0.595 0.423 5.000 0.775 0.716 10.000 0.523 0.369 10.000 0.830 0.760 10.000 0.585 0.452 10.000 0.761 0.709 15.000 0.507 0.386 15.000 0.807 0.754 15.000 0.540 0.431 15.000 0.716 0.614 7.500 0.511 0.302 7.500 0.803 0.705 7.500 0.607 0.466 7.500 0.767 0.718 15.000 0.498 0.370 15.000 0.809 0.759 15.000 0.537 0.430 15.000 0.777 0.732 10.000 0.533 0.371 10.500 0.819 0.752 10.000 0.619 0.485 4.000 0.682 0.541 0.000 0.500 0.234 0.000 0.667 0.480 0.000 0.615 0.395 0.000 0.774 0.680 5.000 0.540 0.374 5.000 0.795 0.685 5.000 0.595 0.437 5.000 0.768 0.726 10.000 0.572 0.381 10.000 0.820 0.754 10.000 0.575 0.435 10.000 0.768 0.739 15.000 0.510 0.372 15.000 0.796 0.737 15.000 0.531 0.423 15.000 0.747 0.645 7.500 0.514 0.296 7.500 0.801 0.695 7.500 0.612 0.483 7.500 0.760 0.730 15.000 0.510 0.373 15.000 0.792 0.730 15.000 0.530 0.420 15.000 0.774 0.718 10.000 0.521 0.411 9.000 0.790 0.733 12.500 0.622 0.474 5.000 0.682 0.542 0.000 0.500 0.234 0.000 0.689 0.515 0.000 0.603 0.375 0.000 0.768 0.665 5.000 0.529 0.361 5.000 0.789 0.674 5.000 0.628 0.479 5.000 0.772 0.713 10.000 0.506 0.342 10.000 0.796 0.728 10.000 0.576 0.450 10.000 0.774 0.720 15.000 0.510 0.397 15.000 0.776 0.738 15.000 0.533 0.424 15.000 0.713 0.609 7.500 0.503 0.357 7.500 0.772 0.669 7.500 0.608 0.464 7.500 0.764 0.709 15.000 0.513 0.398 15.000 0.774 0.737 15.000 0.534 0.424 15.000 0.774 0.732 11.500 0.558 0.400 10.000 0.802 0.770 16.000 0.638 0.509 3.000 0.709 0.580 0.000 0.500 0.234 0.000 0.670 0.482 0.000 0.611 0.388 0.000 0.775 0.669 5.000 0.556 0.425 5.000 0.820 0.725 5.000 0.598 0.461 5.000 0.777 0.732 10.000 0.508 0.358 10.000 0.834 0.762 10.000 0.556 0.441 10.000 0.768 0.731 15.000 0.516 0.388 15.000 0.811 0.762 15.000 0.535 0.430 15.000 0.728 0.627 7.500 0.521 0.333 7.500 0.762 0.649 7.500 0.588 0.477 7.500 0.770 0.733 15.000 0.518 0.382 15.000 0.811 0.764 15.000 0.535 0.431 15.000

R EFERENCES [1] A. Abusnaina, A. Anwar, M. Saad, A. Alabduljabbar, R. Jang, S. Salem, and D. Mohaisen. One step forward, two steps back: Ml-based malware detection under concept drift. Computing, 107(11):207, 2025. [2] M. T. Alam, R. Fieblinger, A. Mahara, and N. Rastogi. MORPH: Towards Automated Concept Drift Adaptation for Malware Detection. arXiv, 2024. [3] M. T. Alam, A. Piplai, and N. Rastogi. ADAPT: A Pseudo-Labeling Approach to Combat Concept Drift in Malware Detection. arXiv, 2025. [4] S. Bakhshad, V. Ponnusamy, R. Annur, M. Waqas, H. Alasmary, and S. Tu. Deep Reinforcement Learning-Based Intrusion Detection System with Feature Selection Method and Optimal Hyper-Parameter in IoT Environment. In IEEE CITS, 2022. [5] F. Barbero, F. Pendlebury, F. Pierazzi, and L. Cavallaro. Transcending TRANSCEND: Revisiting Malware Classification in the Presence of Concept Drift. In IEEE SP, 2022. [6] Y. Chen, Z. Ding, and D. A. Wagner. Continuous Learning for Android Malware Detection. In USENIX Security, 2023. [7] Z. Chen, Z. Zhang, Z. Kan, L. Yang, J. Cortellazzi, F. Pendlebury, F. Pierazzi, L. Cavallaro, and G. Wang. Is It Overkill? Analyzing Feature-Space Concept Drift in Malware Detectors. In IEEE SPW, 2023. [8] T. Chow, Z. Kan, L. Linhardt, L. Cavallaro, D. Arp, and F. Pierazzi. Drift Forensics of Malware Classifiers. In ACM AISec, 2023.

[9] Z. Fu, Y. Ding, and M. Godfrey. An LSTM-Based Malware Detection Using Transfer Learning. J. Cybersecur., 2021. [10] J. Gama, I. Zliobaite, A. Bifet, M. Pechenizkiy, and A. Bouchachia. A Survey on Concept Drift Adaptation. ACM Comput. Surv., 2014. [11] D. E. Garcı́a and N. DeCastro-Garcı́a. Application of Anomaly Detection Models to Malware Detection in the Presence of Concept Drift. In HAIS, 2023. [12] D. E. Garcı́a, N. DeCastro-Garcı́a, and Ángel Luis Muñoz Castañeda. An Effectiveness Analysis of Transfer Learning for the Concept Drift Problem in Malware Detection. Expert Syst. Appl., 2023. [13] D. Ghiani, D. Angioni, G. Piras, A. Sotgiu, L. Minnei, S. Gupta, M. Pintor, F. Roli, and B. Biggio. Regression-Aware Continual Learning for Android Malware Detection. arXiv, 2025. [14] A. Guerra-Manzanares and H. Bahsi. On the Relativity of Time: Implications and Challenges of Data Drift on Long-Term Effective Android Malware Detection. Comput. Secur., 2022. [15] A. Guerra-Manzanares, M. Luckner, and H. Bahsi. Android Malware Concept Drift Using System Calls: Detection, Characterization, and Challenges. Expert Syst. Appl., 2022. [16] A. Guerra-Manzanares, M. Luckner, and H. Bahsi. Concept Drift and Cross-Device Behavior: Challenges and Implications for Effective Android Malware Detection. Comput. Secur., 2022. [17] M. A. Haque, I. Hossain, M. M. Kamol, M. J. Alam, S. K. Amalapuram, S. Talukder, and M. S. Rahman. LAMDA: A Longitudinal Android Malware Benchmark for Concept Drift Analysis. arXiv, 2025. [18] Y. He, J. Lei, Z. Qin, K. Ren, and C. Chen. Combating Concept Drift with Explanatory Detection and Adaptation for Android Malware Classification. In ACM CCS, 2025. [19] R. V. Hoa, T. D. Chuyen, N. T. Lam, T. N. Son, N. D. Dien, and V. T. T. Linh. Reinforcement Learning-Based Method for Autonomous Navigation of Mobile Robots in Unknown Environments. In IEEE ICAMechS, 2020. [20] D. Hu, Z. Ma, X. Zhang, P. Li, D. Ye, and B. Ling. The Concept Drift Problem in Android Malware Detection and Its Solution. Secur. Commun. Netw., 2017. [21] Z. Kan, F. Pendlebury, F. Pierazzi, and L. Cavallaro. Investigating Labelless Drift Adaptation for Malware Detection. In ACM AISec, 2021. [22] Kaspersky. Kaspersky Report: Attacks on Smartphones Increased in the First Half of 2025. Press Release, 2025. [23] X. Lu, Y. Liu, F. Feng, Y. Liu, and Z. Liu. Mask-Based Self-Supervised Network Intrusion Detection System. Appl. Soft Comput., 2025. [24] W. Maillet and B. Marais. Neural Networks Optimizations Against Concept and Data Drift in Malware Detection. arXiv, 2023. [25] S. McFadden, M. Foley, M. D’Onghia, C. Hicks, V. Mavroudis, N. Paoletti, and F. Pierazzi. DRMD: Deep Reinforcement Learning for Malware Detection Under Concept Drift. arXiv, 2025. [26] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-Level Control Through Deep Reinforcement Learning. Nat., 2015. [27] B. Molina-Coronado, U. Mori, A. Mendiburu, and J. Miguel-Alonso. Efficient Concept Drift Handling for Batch Android Malware Detection Models. Pervasive Mob. Comput., 2023. [28] A. Muzaffar, H. R. Hassen, H. Zantout, and M. A. Lones. ActDroid: An Active Learning Framework for Android Malware Detection. Comput. Secur., 2026. [29] F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro. TESSERACT: Eliminating Experimental Bias in Malware Classification Across Space and Time. In USENIX Security, 2019. [30] R. Rocha, S. de Rosa, P. Castagno, I. Drago, and L. A. P. Junior. Does Asm2Vec Reduce Drift on Malware Classification? In SBSeg, 2023. [31] E. Roh, Y. Kaya, C. Kruegel, G. Vigna, and S. Hong. MADCAT: Combating Malware Detection Under Concept Drift with Test-Time Adaptation. arXiv, 2025. [32] A. Sabbah, R. Jarrar, S. Zein, and D. Mohaisen. Empirical Evaluation of Concept Drift in ML-Based Android Malware Detection. arXiv, 2025. [33] J. C. Schlimmer and R. H. Granger. Incremental Learning from Noisy Data. Mach. Learn., 1986. [34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. arXiv, 2017. [35] S. G. Stats. Mobile Operating System Market Share Worldwide. Website, 2026.

[36] R. S. Sutton. Reinforcement Learning: An Introduction. Bradford Book, 2018. [37] S. N. T. Vu, M. Stege, P. I. El-Habr, J. Bang, and N. Dragoni. A Survey on Botnets: Incentives, Evolution, Detection, and Current Trends. Future Internet, 2021. [38] Q. Xiang, L. Zi, X. Cong, and Y. Wang. Concept Drift Adaptation Methods Under the Deep Learning Framework: A Literature Review. Appl. Sci., 2023. [39] K. Xu, Y. Li, R. H. Deng, K. Chen, and J. Xu. DroidEvolver: SelfEvolving Android Malware Detection System. In IEEE EuroS&P, 2019.