Towards Automated Recommendation of Steady-State Detection Algorithms for Software Performance Benchmarks

Highlights Towards Automated Recommendation of Steady-State Detection Algorithms for Software Performance Benchmarks Martin Beseda, Vittorio Cortellessa, Federico Di Menna, Daniele Di Pompeo, Luca Traini • We formulate steady-state detection as an automatic algorithm-recommendation task.

arXiv:2611.16566v1 [cs.PF] 12 Nov 2026

• On 407 steady benchmark time series with judgment-aggregated ground truth, our TSFresh-plus-cluster-signature recommender achieves 72.97% accuracy, compared with 62.65% when always selecting Kernel-based Kelly’s SteadyState Detection. • Exact McNemar testing and bootstrap confidence intervals show that the gain over always selecting the majoritywinning detector is statistically significant and robust to project-level resampling. • Recommendations are highly stable when clustering is learned either from the 407 labelled steady series or from all 5,860 available time series, with 98.03% agreement.

Towards Automated Recommendation of Steady-State Detection Algorithms for Software Performance Benchmarks Martin Beseda∗ , Vittorio Cortellessa, Federico Di Menna, Daniele Di Pompeo and Luca Traini University of L’Aquila, via Vetoio (Coppito), 1, L’Aquila, 67100, Italy

ARTICLE INFO

ABSTRACT

Keywords: steady-state detection software performance benchmarking artificial intelligence clustering algorithm recommendation

Steady-state detection is a central step in software performance benchmarking, yet different detection methods rely on different assumptions and may perform best on different execution profiles. This paper formulates steady-state detection as an algorithm-recommendation problem: given a benchmark time series, predict which of two complementary detectors is more appropriate. The first, CP-SSD, identifies steady state through changepoint-based segmentation of the execution trace, whereas the second, KB-KSSD, uses a kernel-based approach to identify stable behaviour. We propose a datadriven recommender that characterises benchmark time series using automatically extracted features and combines supervised prediction with information derived from groups of time series exhibiting similar behaviour. The approach is evaluated on 407 labelled benchmark time series for which one of the two detectors can be identified as preferable with respect to a reference steady-state index. The proposed recommender selects the better detector correctly for 72.97% of the series, compared with 62.65% when always selecting the detector that performs best most often, corresponding to an absolute improvement of 10.32 percentage points. Paired statistical testing and bootstrap confidence intervals support this improvement. A comparison with a recommender using time-series features alone shows that the cluster-derived information mainly improves interpretability rather than predictive accuracy. Recommendations also remain highly stable when the unsupervised representation is learned from either the labelled subset or the full collection of 5,860 time series. These results demonstrate the feasibility of per-series steady-state detector recommendation while highlighting the need for broader labelled data, particularly for execution profiles favouring the changepoint-based detector.

1. Introduction Reliable performance benchmarking requires distinguishing measurements affected by unstable execution behaviour from those that are sufficiently stable to be used for performance inference. This distinction is particularly important for managed runtimes, where just-in-time compilation, garbage collection, resource allocation, and other runtime mechanisms can cause execution times to change substantially during the initial part of a benchmark run (Georges, Buytaert and Eeckhout, 2007; Kalibera and Jones, 2013). In many executions, this behaviour can be interpreted as a transition from an initial warm-up region towards a more stable steady-state region. However, such a transition is not necessarily sharp or even guaranteed to occur: benchmark traces may contain temporary plateaus, repeated transitions, outliers, or long-term non-stationarity (Barrett, BolzTereick, Killick, Mount and Tratt, 2017; Traini, Cortellessa, Di Pompeo and Tucci, 2023). Identifying the beginning of a steady state is therefore an important and non-trivial part of performance analysis. If the transition is detected too early, unstable observations may be included in the final performance estimate and bias the reported result. Conversely, an overly conservative martin.beseda@univaq.it (M. Beseda); vittorio.cortellessa@univaq.it (V. Cortellessa); federico.dimenna@graduate.univaq.it (F.D. Menna); daniele.dipompeo@univaq.it (D.D. Pompeo); luca.traini@univaq.it (L.

Traini)

ORCID (s): 0000-0001-5792-2872 (M. Beseda); 0000-0002-4507-464X (V. Cortellessa); 0009-0003-5834-6389 (F.D. Menna); 0000-0003-2041-7375 (D.D. Pompeo); 0000-0003-3676-0645 (L. Traini)

Beseda et al.: Preprint submitted to Elsevier

detector may discard useful observations, unnecessarily extend benchmark execution, or reduce the amount of steadystate data available for subsequent analysis. Consequently, steady-state detection (SSD) has received increasing attention in software performance benchmarking, where several approaches have been proposed, ranging from changepointbased segmentation to statistical and probabilistic tests (Beseda, Cortellessa, Di Pompeo, Traini and Tucci, 2026; Barrett et al., 2017; Traini et al., 2023; Bethea and Rhinehart, 2019; Boesler and Weber, 2017; Cao, 1996; Wu, Chen and Zhou, 2016). An issue that comparatively receives less attention is that SSD methods embody different assumptions about what constitutes the transition to steady behaviour. Changepointbased approaches identify structural changes in the observed execution trace and infer a final stable region from the resulting segmentation. In contrast, other methods may evaluate local smoothness, probabilistic steadiness, or specific transition patterns. These differences are not merely methodological: they can lead different detectors to be preferable on different execution profiles. This is particularly visible when comparing two recent SSD approaches used in this study. The first is a changepointbased approach introduced by Traini et al., which we denote as CP-SSD (Traini et al., 2023). The second is KB-KSSD, introduced in our previous work (Beseda et al., 2026), which combines robust preprocessing, kernel-based transition detection, and a probabilistic steady-state criterion adapted from process monitoring. Indeed, on the benchmark collection considered in that study, KB-KSSD achieved lower total detection error with Page 1 of 20

Automatic SSD Recommendation

respect to judgment-aggregated ground truth (JAGT), which provides the human-selected reference steady-state indices against which automatic detections are evaluated, and fewer false-negative detections than CP-SSD. Nevertheless, the total result does not imply that KB-KSSD is the better choice for every individual execution. In the 407 steady JAGT time series used in the present work, KB-KSSD provides the better steady-state estimate for 255 of them, whereas CP-SSD works better for the remaining 152. Thus, always applying a globally stronger detector is not always the best choice, as it can be applied to a substantial number of benchmark executions where another method could produce more accurate results. This observation motivates a different formulation of the SSD problem. Instead of asking which detector is best on average, we ask whether the characteristics of an individual benchmark time series can be used to predict which detector would work better on that particular execution. The problem is therefore naturally related to per-instance algorithm selection, where features describing an instance are used to choose the algorithm expected to perform best on that instance (Rice, 1976; Kerschke, Hoos, Neumann and Trautmann, 2019). In the present context: an “instance” is a benchmark execution trace, the algorithm portfolio contains CP-SSD and KB-KSSD, and the target is the detector producing the smaller absolute error relative to the reference JAGT steady-state indices. More broadly, this formulation represents an application of artificial intelligence to software performance analysis, using machine learning to adapt the choice of SSD method to the characteristics of each execution trace. The central challenge is therefore to construct a representation of an execution trace that contains information about the relative performance of the two SSD methods. We begin by extracting a broad set of time-series descriptors with TSFresh (Christ, Braun, Neuffer and Kempa-Liehr, 2018), covering distributional, entropy-based, frequency-domain, temporal, and other characteristics. These descriptors are used in two complementary branches of the recommendation pipeline. The first branch searches for structure among execution traces. We reduce the TSFresh representation using Principal Component Analysis (PCA) (Abdi and Williams, 2010) and Uniform Manifold Approximation and Projection (UMAP) (McInnes, Healy, Saul and Grossberger, 2018), and subsequently identify groups of similar traces using HDBSCAN (McInnes, Healy, Astels et al., 2017). Clusters containing labelled series are then characterised by whether their members tend to favour CP-SSD or KB-KSSD. This information is converted into cluster-signature similarity features that describe how closely a new execution resembles regions associated with either detector. The second branch uses TSFresh descriptors directly as supervised predictive features. We combine the selected direct descriptors with the cluster-signature features and train a Random Forest classifier (Kulkarni and Sinha, 2013) to recommend one of the two SSD methods. We refer to

Beseda et al.: Preprint submitted to Elsevier

this as the hybrid recommender because its feature representation combines directly predictive time-series descriptors with meta-features derived from the unsupervised cluster structure. Figure 1 summarises this pipeline. This design lets us study two related questions. First, we ask whether per-series recommendation can improve upon simple reference strategies, including random selection and always selecting KB-KSSD, which is the method that wins most often in the labelled dataset. Second, we examine whether the cluster-derived representation contributes useful information beyond the directly predictive TSFresh descriptors. We therefore evaluate both the hybrid recommender and an ablation that uses only the selected TSFresh features. The purpose of the clustering component is not to assume that clustering must improve predictive accuracy, but to determine whether method-specialised regions exist and whether they provide an interpretable description of when the two SSD approaches tend to succeed. Based on these objectives, we investigate the following research questions: RQ1

How often does each SSD method provide the better estimate, and does method preference vary systematically across clusters of similar time series? This question establishes whether the recommendation problem is meaningful in the first place. If one detector dominates almost every series, perinstance selection would offer little practical value. We therefore analyse both the global distribution of CP-SSD and KB-KSSD wins and their distribution within the clusters.

RQ2

Can the better SSD method be recommended automatically, and does recommendation outperform random choice and/or majority-winning detector? We evaluate the predictive performance of our recommender against equiprobable random selection, prevalence-matched random selection, and the strong majority strategy of always selecting KB-KSSD. We additionally compare the hybrid model with a direct TSFresh classifier to isolate the incremental contribution of the cluster-signature representation.

RQ3

What kinds of recommendation errors remain, particularly for the minority CP-SSD class? Overall accuracy alone can obscure asymmetric behaviour between the two classes. We therefore analyse the confusion matrix, balanced accuracy, precision, recall, and F1-score, with particular attention to the ability to recognise executions for which CP-SSD is preferable.

RQ4

How sensitive are the recommendations to the population used for unsupervised clustering?

Page 2 of 20

Automatic SSD Recommendation Sec. 3.1: Dataset and recommendation target JAGT reference + CPSSD/KB-KSSD outputs → binary recommendation target 𝑦𝑖

Benchmark time series

TSFresh feature extraction Sec. 3.2: TSFresh representation and preprocessing

Cleaning, imputation, and standardisation

Mutual-information feature ranking

PCA projection

UMAP embedding

Sec. 3.2: Direct predictive features

Selected TSFresh descriptors Sec. 3.3: Clustering and cluster signatures

HDBSCAN clustering

Training-set cluster characterisation

Cluster-signature similarity features Sec. 3.4: Hybrid SSD recommendation

Hybrid feature vector Selected TSFresh descriptors + cluster-signature features

Random Forest recommender

Recommend CP-SSD

Recommend KB-KSSD

Figure 1: Workflow of the hybrid SSD recommender and its correspondence to the methodological stages described in Section 3. Section 3.1 defines the benchmark input and binary recommendation target. Section 3.2 defines the shared TSFresh representation and preprocessing as well as the direct predictive-feature branch. Section 3.3 describes the PCA–UMAP–HDBSCAN structure-discovery branch and the construction of cluster-signature features. Section 3.4 combines the direct and cluster-derived representations in the hybrid Random Forest recommender. The dashed containers indicate these subsection boundaries. Labeldependent operations, feature selection, and model fitting are performed within the training portion of each evaluation split.

The primary clustering analysis is learned from the 407 labelled JAGT series, whereas a secondary analysis derives the clustering structure from all 5,860 cached time series. We compare the resulting recommendations to determine whether the downstream classifier depends strongly on the population used for unsupervised structure discovery.

• We formulate SSD-method choice as a per-instance algorithm recommendation problem, rather than treating one detector as universally preferable for all benchmark executions. • We construct a recommendation dataset from 407 decisive JAGT-labelled time series, containing 152 CP-SSD

The main contributions of this paper are therefore: Beseda et al.: Preprint submitted to Elsevier

Page 3 of 20

Automatic SSD Recommendation

winners and 255 KB-KSSD winners, using explicit rules for detector failures and absolute-error comparisons. • We introduce a hybrid recommendation pipeline combining TSFresh descriptors with similarity features derived from clusters of behaviourally similar time series, providing both predictive and interpretable information about detector preference. • We provide a repeated-holdout evaluation and a complementary unique-series inferential analysis using exact paired hypothesis testing, effect-size estimation, and bootstrap uncertainty quantification. • We show that automatic SSD recommendation improves substantially over random and majority strategies, while also demonstrating that the present cluster-signature representation does not produce a statistically supported accuracy improvement over direct TSFresh features. • We show that the recommendations are highly robust to the population used for clustering: clustering learned from the labelled JAGT subset and clustering learned from all 5,860 cached time series produce 98.03% identical final recommendations. The data, code, archived intermediate results, and replication instructions accompanying this study are described in Section 8.

2. Background and Related Work This section provides the background necessary to position the proposed SSD recommendation framework. We first discuss steady-state behaviour and its importance for reliable performance benchmarking, followed by the automatic SSD methods that constitute the recommendation portfolio considered in this work. We then relate the proposed formulation to the broader problem of per-instance algorithm selection and finally review the time-series representation, dimensionality-reduction, clustering, and classification techniques used to construct the recommender.

2.1. Managed-runtime benchmarking and steady-state behaviour Reliable microbenchmarking requires more than repeated execution: measurements must be collected under a protocol that controls sources of variability and separates transient behaviour from the region used for performance inference. Georges et al. established a statistically rigorous methodology for Java performance evaluation, while Kalibera and Jones subsequently showed how careful experimental design can provide rigorous conclusions without prohibitively long experiments (Georges et al., 2007; Kalibera and Jones, 2013). These concerns are particularly important for managed runtimes, where just-in-time compilation, garbage collection, adaptive optimisation, and other runtime mechanisms can produce non-stationary execution profiles. The conventional assumption that a virtual machine warms up and then remains at a stable peak-performance Beseda et al.: Preprint submitted to Elsevier

state is not universally valid. Barrett et al. used an automated changepoint-based analysis and showed that many benchmark–VM pairs do not consistently exhibit the expected warm-up-to-steady-state pattern (Barrett et al., 2017). Traini et al. later investigated steady-state assessment in Java microbenchmarking at a larger scale and showed that benchmarks may fail to reach a steady state and that developerspecified warm-up durations can be inaccurate (Traini et al., 2023).

2.2. Automatic steady-state detection The first candidate considered in this study is the changepoint-based procedure used by Traini et al.; throughout this paper we refer to this procedure as CP-SSD. It builds on the changepoint formulation of Barrett et al. and uses multiplechange detection to partition an execution trace into statistically homogeneous regions (Barrett et al., 2017; Traini et al., 2023). The underlying PELT algorithm finds an exact optimum for a penalised multiple-changepoint objective while achieving linear computational cost under suitable conditions (Killick, Fearnhead and Eckley, 2012). Post-processing then removes negligible changes and the final segments are inspected to determine whether a sufficiently stable region has been reached. The second candidate, KB-KSSD, follows a different design. It combines robust preprocessing, kernel-based transition detection, and a probabilistic steady-state test adapted from Kelly and Hedengren’s process-monitoring formulation (Kelly and Hedengren, 2013). The complete method was introduced for performance time series by Beseda et al. and evaluated against CP-SSD and other reference methods on a JAGT (Beseda et al., 2026). That study reported a 14.5% reduction in total absolute steady-state-index error relative to CP-SSD and a lower false-negative rate. Importantly for the present work, however, collection-level superiority does not imply that one method is best for every benchmark execution. The JAGT-labelled series contain substantial numbers of instances won by each method, which makes per-instance selection meaningful. The studies above evaluate SSD procedures as competing methods over benchmark collections. The present work asks a different question: given the observed characteristics of one execution trace, can the method that will produce the smaller steady-state-index error be selected automatically? This shifts the unit of decision from the benchmark collection to the individual time series.

2.3. Per-instance algorithm selection and meta-learning This formulation is closely related to the classical algorithmselection problem of Rice, in which instance characteristics are mapped to the algorithm expected to perform best on that instance (Rice, 1976). Modern algorithm selection systems operationalise this idea through machine learning over instance-level meta-features, and surveys by Kotthoff and by Kerschke et al. show that such per-instance selection can exploit complementary strengths of algorithms that are Page 4 of 20

Automatic SSD Recommendation

obscured by aggregate rankings (Kotthoff, 2014; Kerschke et al., 2019). A central requirement is therefore a feature representation that captures properties relevant to relative algorithm performance. Automated algorithm recommendation has also received considerable attention in time series analysis (Prudêncio and Ludermir, 2004; Talagala, Hyndman and Athanasopoulos, 2023). For example, Prudêncio and Ludermir (2004) employed meta-learning to identify the best-performing algorithm for TSF, whereas Shahoud, Khalloof, Duepmeier and Hagenmeyer (2020) developed DSTMS to support algorithm selection in energy forecasting tasks. Our recommendation target follows this paradigm directly. Each labelled benchmark time series is an instance; the portfolio contains CP-SSD and KB-KSSD; and the target label identifies which method has the smaller absolute error relative to the reference steady-state index. Unlike conventional algorithm-selection studies, where instance descriptors often encode a static problem definition, here the descriptors are extracted from the observed execution-time trajectory itself.

2.4. Time-series representation and structure-aware recommendation TSFresh provides a large, automatically generated set of time-series descriptors spanning distributional statistics, autocorrelation, entropy, frequency-domain coefficients, peaks, quantiles, and change-related properties (Pincus, 1991; Christ et al., 2018). This makes it suitable for converting variablelength benchmark traces into a common fixed-dimensional representation without hand-designing a small domainspecific feature set. The proposed pipeline combines this direct representation with unsupervised structure discovery. PCA is used to obtain lower-dimensional linear representations (Jolliffe, 2002); UMAP provides a nonlinear manifold embedding (McInnes et al., 2018); and HDBSCAN identifies density-based clusters while retaining an explicit noise class (Campello, Moulavi, Zimek and Sander, 2015). The discovered clusters are not treated as prediction targets. Instead, training labels are used to characterise whether particular clusters exhibit CP-SSD or KB-KSSD specialisation, and these specialised regions are transformed into clustersignature similarity features. The final classifier is a Random Forest (Breiman, 2001). We evaluate both a direct TSFresh model and a hybrid model that augments selected TSFresh features with the cluster-signature representation. This ablation is important because it separates two questions: whether the original time-series descriptors are sufficient for SSD recommendation, and whether the unsupervised structure adds incremental predictive information. As shown later, the signatures expose interpretable method-specialised structure, but the present representation does not yield a statistically supported accuracy improvement over the direct TSFresh classifier.

Beseda et al.: Preprint submitted to Elsevier

3. Methodology This section formalises the complete analysis pipeline, from the benchmark time series and the construction of the recommendation labels to the clustering configuration score, cluster-signature representation, hybrid feature vector, and final supervised recommender. All feature-selection and label-dependent signature operations used for prediction are performed using the training portion of each repeated holdout split. Figure 1 provides a visual roadmap of the methodology. The dashed regions in the figure correspond directly to the main stages described in this section. Section 3.1 defines the benchmark input and recommendation target; Section 3.2 describes the TSFresh representation, preprocessing, and direct predictive-feature branch; Section 3.3 describes the PCA–UMAP–HDBSCAN structure-discovery branch and the construction of cluster signatures; and Section 3.4 combines the two branches into the final hybrid Random Forest recommender. Sections 3.5 and 3.6 then define the experimental and statistical evaluation of the resulting recommendations.

3.1. Benchmark Dataset and Recommendation Target The available benchmark collection contains 𝑁all = 5860 cached time series. We denote series 𝑖 by 𝐱𝑖 , its length by 𝐿𝑖 , and its measurement at time index 𝑡 by 𝑥𝑖𝑡 . The labelled recommendation subset contains 𝑁 = 407 JAGT time series. For each labelled series 𝑖, let 𝑎ref denote the 𝑖 CP KB reference steady-state index, while 𝑎𝑖 and 𝑎𝑖 denote the indices returned by CP-SSD and KB-KSSD, respectively. An index equal to −1 means that the corresponding method did not identify a steady state. For a method 𝐴 ∈ {CP, KB} that returns a valid index, its absolute detection error is | 𝐴 ref | 𝑒𝐴 𝑖 = ||𝑎𝑖 − 𝑎𝑖 || .

(1)

The separate-wins formulation used in the implementation KB assigns binary win indicators 𝑤CP 𝑖 and 𝑤𝑖 . If exactly one method returns a valid index, that method receives one win. If both methods return valid indices, the method with the smaller absolute error receives one win. If both fail or the valid errors are equal, neither method receives a win. The supervised target is defined only for decisive observations KB = 1: satisfying 𝑤CP 𝑖 + 𝑤𝑖 𝑦𝑖 = 𝑤CP 𝑖 ∈ {0, 1},

(2)

where 𝑦𝑖 = 1 denotes a CP-SSD win and 𝑦𝑖 = 0 denotes a KB-KSSD win. The resulting labelled dataset therefore consists of the 𝑁 = 407 pairs (𝐱𝑖 , 𝑦𝑖 ). In the analysed dataset, all 407 JAGT time series were decisive under this formulation. KB-KSSD wins on 255 series, while CP-SSD wins on 152 series. The benchmark dataset composition is summarized in Table 1, while the outcome reasons are presented in Table 2. Page 5 of 20

Automatic SSD Recommendation Table 1 Summary of the benchmark dataset. Property

Value

Total number of time series Number of JAGT time series CP-SSD wins KB-KSSD wins Non-decisive cases

5860 407 152 255 0

Table 2 Outcome reasons under the separate-wins formulation. Outcome reason

Count

KB lower absolute error CP lower absolute error CP valid, KB failed KB valid, CP failed Both failed

251 147 5 4 0

3.2. TSFresh Representation and Preprocessing After removing the first observation and normalising each time series, TSFresh maps the variable-length input to a fixed-dimensional descriptor vector 𝐳𝑖 = (𝑧𝑖1 , … , 𝑧𝑖𝑑 )𝖳 , where 𝑑 is the number of cached Efficient TSFresh descriptors. For any analysed collection of 𝑛 series, stacking these vectors yields the feature matrix 𝐙 ∈ ℝ𝑛×𝑑 . The descriptors include distributional statistics, entropy and complexity measures, autocorrelation, peak and strike statistics, quantile-based change features, and frequency-domain coefficients. Before dimensionality reduction, each TSFresh feature is standardised using the mean and standard deviation estimated from the data on which the corresponding preprocessing operation is fitted. Columns that are constant in a training split are removed before supervised feature selection. The exact standardisation transformation is given in Appendix A.1. For the supervised branch, candidate TSFresh features are ranked by their estimated mutual information with the binary recommendation target. Within every training split, the top 𝑝 = 100 descriptors are retained for the best-performing model. The standard mutual-information definition used for feature ranking is given in Appendix A.4. Denoting their index set by 𝑝 , the selected direct TSFresh vector is 𝑝

𝐛𝑖 = (𝑧𝑖𝑗 )𝑗∈𝑝 ∈ ℝ .

(3)

3.3. PCA–UMAP–HDBSCAN Clustering The clustering sweep evaluates both PCA–HDBSCAN and PCA–UMAP–HDBSCAN configurations. The primary analysis fits the dimensionality-reduction and clustering models to the 407 labelled JAGT time series so that the clustering population and recommendation population coincide. A secondary robustness analysis repeats the downstream recommendation using a clustering configuration learned from the complete collection of 5,860 cached time series. Beseda et al.: Preprint submitted to Elsevier

3.3.1. Principal component analysis PCA is applied to the standardised TSFresh feature matrix using the standard covariance-eigenvector formulation (Jolliffe, 2002). The retained principal components provide the representation used either directly by HDBSCAN or as input to UMAP. For the PCA-only sweep, 𝑝PCA ∈ {10, 20, 30, 50, 75, 100}, whereas configurations followed by UMAP use 𝑝PCA ∈ {20, 50, 100}. The standard PCA formulation used in the analysis is given in Appendix A.2. 3.3.2. UMAP embedding and HDBSCAN partition For the nonlinear branch, UMAP (McInnes et al., 2018) is applied to the PCA representation with embedding dimension 𝑝U ∈ {5, 10, 20}, neighbourhood size 𝑛nbr ∈ {10, 15, 30}, and minimum-distance parameter 𝛿 ∈ {0, 0.05, 0.1}. Euclidean distance is used by UMAP. Because UMAP employs stochastic optimisation, its random seed is fixed to 42 to make the embeddings and clustering sweep reproducible; the seed is not treated as a tunable hyperparameter. HDBSCAN (Campello et al., 2015) is then applied either to the PCA representation or to the UMAP embedding using Euclidean distance and minimum cluster size 𝑚cs ∈ {10, 20, 30, 50}. Let 𝑐𝑖 denote the HDBSCAN cluster label assigned to observation 𝑖. The label 𝑐𝑖 = −1 denotes an outlier, while the remaining labels identify the 𝐾 non-outlier clusters. Because HDBSCAN does not minimise a 𝑘-meanstype sum of squared distances, each complete clustering configuration is evaluated using the scoring function introduced in the next section. 3.3.3. Internal clustering quality Let 𝑁clust denote the number of observations in the population used to fit a given clustering configuration,  = {𝑖 ∶ 𝑐𝑖 ≠ −1} the set of non-outlier observations, and 𝑛out the number of outliers. We define clustering coverage as 𝐶cov = 1 − 𝑛out ∕𝑁clust . Geometric clustering quality is evaluated on the nonoutlier observations using the standard Silhouette coefficient 𝑆sil (Řezanková, 2018) and the Davies–Bouldin index (DBI) (Anfossi and Chicco, 2026). Their standard definitions are given in Appendix A.3. The implementation in this work transforms the metrics into 𝑆+ = max{𝑆sil , 0}, 1 , 𝐷+ = 1 + DBI min{𝐾, 20} 𝐾+ = , 20

(4) (5) (6)

and defines the internal clustering score as 𝑄internal = 0.45𝑆+ + 0.25𝐶cov + 0.20𝐷+ + 0.10𝐾+ . (7) By construction, 𝑆+ , 𝐶cov , and 𝐾+ lie in [0, 1], while 𝐷+ ∈ (0, 1]. Because the coefficients in Eq. 7 are nonnegative and sum to one, 𝑄internal is consequently bounded to [0, 1]. Page 6 of 20

Automatic SSD Recommendation

The weights in Eq. 7 are heuristic design weights rather than parameters estimated from the classification results. Their purpose is to encode the relative importance of the four properties considered desirable for the exploratory clustering. Silhouette score receives the largest weight 0.45, because separation from neighbouring clusters and cohesion within a cluster constitute the primary geometric criterion. Coverage receives weight 0.25 to prevent configurations obtaining a high geometric score by assigning a large fraction of difficult observations to the HDBSCAN noise class. The transformed Davies–Bouldin index contributes a further 0.20 as a complementary measure of within-cluster compactness relative to between-cluster separation. Finally, the number-of-clusters term receives only 0.10. It therefore acts as a weak regularising preference against degenerate solutions with very few clusters rather than rewarding fragmentation of the data. Capping this term at 20 clusters further prevents an arbitrarily large number of clusters from increasing the score.

3.3.4. Method-specialisation signal and configuration score For every usable non-outlier cluster containing at least five JAGT series, the numbers of non-tied KB-KSSD and CP-SSD wins are compared with a two-sided binomial test under equal win probability. Benjamini–Hochberg adjustment is applied across cluster-level tests, and a cluster is considered significant when its adjusted value is below 0.05 and it contains at least five non-tied wins. Let 𝐹best and 𝐹orig denote the fractions of usable JAGT series contained in significant clusters for, respectively, the best and original CP-SSD configurations. Let 𝑛best and 𝑛orig be the corresponding numbers of significant clusters. For usable cluster 𝑘, let 𝑛KB and 𝑛CP denote the non-tied 𝑘,cfg 𝑘,cfg KB-KSSD and CP-SSD win counts used to evaluate the clustering configuration. We define the corresponding winrate margin as 𝑚𝑘 = (𝑛KB − 𝑛CP )∕(𝑛KB + 𝑛CP ). The 𝑘,cfg 𝑘,cfg 𝑘,cfg 𝑘,cf g maximum absolute margin across usable clusters is 𝑀max = max𝑘 |𝑚𝑘 |. The method-specialisation signal used by the clustering sweep is 𝑄method = 0.45𝐹best + 0.25𝐹orig + 0.15 + 0.10

min{𝑛orig , 5} 5

min{𝑛best , 5} 5

+ 0.05𝑀max .

(8)

Finally, the explicit clustering-configuration quality function is 𝑄cluster = 0.45𝑄internal + 0.55𝑄method .

(9)

Equation (9) is the metric used to evaluate the complete PCA–UMAP–HDBSCAN configurations in the initial sweep. While its coefficients are fixed heuristic design weights, we assign slightly greater weight to the method-specialisation component because the purpose of the clustering stage is not only to identify geometrically coherent groups, but also Beseda et al.: Preprint submitted to Elsevier

to expose regions in which the relative performance of the SSD methods differs systematically. At the same time, the internal-quality component retains substantial weight to discourage partitions that exhibit apparent method specialisation but poor geometric structure. The coefficients sum to one so that the combined score remains on the same normalized scale as its two components. For the subsequent signature-based recommendation experiment, candidate clusterings are additionally ranked lexicographically by their specialisation. Let 𝑁CP and 𝑁KB be the numbers of JAGT series in significant CP-favouring and KB-favouring clusters. We define 𝐵 = min{𝑁CP , 𝑁KB } as the balanced-specialisation criterion, 𝑃 = 𝑁CP 𝑁KB as its product criterion, and 𝑇 = 𝑁CP + 𝑁KB as total specialised coverage. The implementation first favours configurations containing at least one significant cluster of each type and then orders candidates by 𝐵, 𝑃 , 𝑇 , the numbers of CP- and KB-specialised clusters, and their respective series counts, all in descending order. For the primary steady-JAGT-only analysis, the selected configuration is pca50_umap10_nn30_md0.0_mcs20. It contains three non-outlier clusters: two KB-favouring clusters of 144 and 84 labelled series and one CP-favouring cluster of 86 labelled series. The all-timeseries robustness analysis uses pca50_umap5_nn30_md0.0_mcs50.

3.3.5. Cluster signatures For cluster 𝑘, let 𝑘tr denote the set of labelled training observations assigned to that cluster. For original TSFresh ∑ feature 𝑗, let 𝑧𝑘𝑗 = |𝑘tr |−1 𝑖∈ tr 𝑧𝑖𝑗 denote its mean within 𝑘 the training members of cluster 𝑘. The descriptive cluster z-score is 𝑔𝑘𝑗 =

𝑧𝑘𝑗 − 𝜇𝑗 𝜎𝑗

(10)

.

Features are ranked by |𝑔𝑘𝑗 |. For a requested signature size 𝑞,

let 𝑘(𝑞) contain the 𝑞 highest-ranked features available in the training split. The hybrid search evaluates 𝑞 ∈ {5, 8, 12, 20}, and only clusters containing at least five training observations are eligible to form signatures. Within a training split, let 𝑛CP and 𝑛KB be the numbers 𝑘 𝑘 of CP- and KB-winning training observations assigned to cluster 𝑘. Clusters tied in these counts are discarded. Each remaining signature is labelled CP when 𝑛CP > 𝑛KB and KB 𝑘 𝑘 CP KB when 𝑛𝑘 > 𝑛𝑘 . Its dominance strength is 𝜔𝑘 =

|𝑛CP − 𝑛KB | 𝑘 𝑘 𝑛CP + 𝑛KB 𝑘 𝑘

(11)

.

The centroid of signature 𝑘 in its selected raw-feature subspace is 1 ∑ 𝜸𝑘 = 𝐳𝑖, (𝑞) , (12) 𝑛𝑘 𝑘 tr 𝑖∈𝑘

where 𝑛𝑘 = |𝑘tr | = 𝑛CP +𝑛KB and 𝐳𝑖, (𝑞) denotes observation 𝑘 𝑘 𝑘

𝑖 restricted to the signature feature set. Page 7 of 20

Automatic SSD Recommendation

3.4. Hybrid SSD Recommendation The recommendation task is a binary classification: given a time series, the model estimates whether CP-SSD (𝑦𝑖 = 1) or KB-KSSD (𝑦𝑖 = 0) is expected to produce the better steady-state estimate. The evaluated strategies include a majority baseline, a direct supervised TSFresh model, a cluster-signature rule, and the hybrid recommender. The reported best model is the hybrid Random Forest.

3.4.1. Signature similarity scores For observation 𝑖 and signature 𝑘, the Euclidean distance in the signature-specific feature subspace is 𝑑𝑖𝑘 = ‖𝐳𝑖, (𝑞) − 𝑘 𝜸 𝑘 ‖2 . Because signatures may contain different numbers of features, this distance is normalised by the square root of the signature dimension and converted to a bounded similarity. The exact score implemented is ⎞ ⎛ 𝑑𝑖𝑘 ⎟ ⎜ 𝑟𝑖𝑘 = 𝜔𝑘 exp⎜− √ ⎟. (𝑞) ⎟ ⎜ | | 𝑘 ⎠ ⎝

(13)

Let CP and KB be the sets of CP- and KB-labelled signatures. With the configured weighted_max aggregation, KB = the class-specific scores are 𝑟CP 𝑖 = max𝑘∈CP 𝑟𝑖𝑘 and 𝑟𝑖 max𝑘∈KB 𝑟𝑖𝑘 . If a class has no available signature, its score is set to zero. The IDs of the clusters attaining these maxima KB are denoted by 𝑘CP 𝑖 and 𝑘𝑖 .

define each cluster signature; it is distinct from the 13 derived meta-features in Equation 14. The final supervised learner is a Random Forest (Breiman, 2001). We fix the number of trees to 𝐵RF = 300 for all Random Forest experiments; this value is an implementation choice and is not selected using the evaluation data. The final class recommendation is obtained by standard majority voting across the 300 trees. The implemented Random Forest uses unrestricted tree depth, minimum leaf size 2, balanced_subsample class weights, random seed 42, and parallel tree construction. Thus, class weighting is recomputed on the bootstrap sample used for each tree.

3.4.4. Recommendation workflow Figure 1 summarises the complete recommendation pipeline. The layout separates the shared preprocessing stage from two feature-construction branches. The left branch discovers unsupervised structure and converts training-label cluster specialisation into signature-similarity features; the right branch retains the TSFresh descriptors selected directly for prediction. The two representations are concatenated only after the branch-specific processing has been completed.

3.5. Experimental Setup and Evaluation Protocol

The predictive evaluation uses repeated stratified holdout validation with 𝑅 = 20 repetitions and a test fraction of te 𝜏 = 0.30. For repetition 𝑟, let tr 𝑟 and 𝑟 denote the training and test sets. Feature selection, training-label cluster char3.4.2. Signature meta-feature vector acterisation, signature construction, and model fitting are The implementation converts the signature results into performed using the training portion of the corresponding the following 13-dimensional meta-feature vector: split. The test portion is used only for prediction and metric computation. ( CP KB 𝑟𝑖 𝑟𝑖 KB CP KB KB CP Performance is evaluated using accuracy, balanced ac|, − 𝑟 , |𝑟 − 𝑟 , 𝑟 , 𝑟 𝐬𝑖 = 𝑟CP , , 𝑖 𝑖 𝑖 𝑖 𝑖 𝑖 CP 𝑟KB 𝑖 + 𝜀 𝑟𝑖 + 𝜀 curacy, CP-SSD precision, CP-SSD recall, and CP-SSD F1)𝖳 score, with CP-SSD treated as the positive class. The stanCP KB 𝑘𝑖 , 𝑘𝑖 , 𝐾CP , 𝐾KB , 𝑊CP , 𝑊KB , 𝑊CP − 𝑊KB , dard definitions of these metrics are given in Appendix A.5. Repeated-holdout performance is summarised descrip(14) tively by the mean and standard deviation of each metric across the 𝑅 = 20 test splits. Because the same benchmark where 𝜀 = 10−12 prevents division by zero, 𝐾CP = |CP | series can appear in several test sets, split-level results are and 𝐾KB = |KB | are the numbers of signatures, and ∑ ∑ treated descriptively rather than as independent statistical 𝑊CP = 𝑘∈CP 𝜔𝑘 and 𝑊KB = 𝑘∈KB 𝜔𝑘 are their total observations. For the primary inferential analysis, repeated dominance strengths. The two cluster identifiers are treated held-out predictions are aggregated into one decision per as numeric meta-features, exactly as in the implementation. unique time series. If 𝑝𝑖 denotes the fraction of held-out appearances in which series 𝑖 is predicted as CP-SSD, the 3.4.3. Hybrid feature vector and Random Forest input aggregated prediction is CP-SSD when 𝑝𝑖 ≥ 0.5 and KBFor each split, mutual-information selection is applied KSSD otherwise. This yields one prediction for each of the only to the direct TSFresh features, producing 𝐛𝑖 ∈ ℝ𝑝 . 407 labelled JAGT series. The signature meta-features are then concatenated without further selection. The complete input to the classifier is therefore ( ) 𝐛𝑖 𝐡𝑖 = ∈ ℝ𝑝+13 . (15) 𝐬𝑖

For the best configuration, 𝑝 = 100, so the Random Forest receives 113 columns per observation. The signature size 𝑞 = 20 controls how many prominent TSFresh descriptors Beseda et al.: Preprint submitted to Elsevier

3.6. Statistical Analysis Protocol The statistical analysis distinguishes between descriptive repeated-holdout performance and inferential uniqueseries performance. The 20 repeated 70/30 holdout splits overlap, namely a benchmark series can occur in more than one test set. Consequently, the 20 split-level accuracies are not treated as independent experimental units and are used only to characterise the variability of the training/evaluation Page 8 of 20

Automatic SSD Recommendation

procedure. Confirmatory statistical tests are instead based on one aggregated held-out decision for each of the 407 unique JAGT series.

Reference strategies. The natural majority strategy al-

ways recommends KB-KSSD and is therefore correct for 255 of 407 series, corresponding to 62.65% accuracy. Two stochastic reference strategies are also considered. Equiprobable random selection has expected accuracy 50.00%, while prevalence-matched random selection, based on the observed class proportions of 152 CP-SSD and 255 KB-KSSD winners, has expected accuracy 53.20%.

Binomial inference. Let 𝑋 denote the number of correct

recommendations among the 𝑁 = 407 aggregated uniqueseries predictions, so that the observed accuracy is 𝑝̂ = 𝑋∕𝑁. Uncertainty in this proportion is quantified using the exact Clopper–Pearson 95% confidence interval (Clopper and Pearson, 1934). Its standard definition is given in Appendix A.6. To test whether the recommender performs better than a random reference with success probability 𝑝0 , we use an exact one-sided binomial test. The test is applied to the equiprobable baseline 𝑝0 = 0.5 and the prevalence-matched baseline 𝑝0 = 0.5320. Its standard formulation is given in Appendix A.7. These tests concern stochastic reference strategies and therefore do not exploit the observation-wise pairing available for the deterministic always-KB-KSSD baseline.

Paired comparison with always-KB-KSSD. The hybrid

recommender and the always-KB-KSSD strategy are evaluated on the same 𝑁 benchmark series and their correctness indicators are therefore paired. Let 𝑏 denote the number of series on which only the hybrid is correct and 𝑐 the number on which only always-KB-KSSD is correct. With 𝐷 = 𝑏 + 𝑐 discordant pairs and 𝑚 = min{𝑏, 𝑐}, the exact two-sided McNemar 𝑝-value is { } 𝑚 ( ) ∑ 𝐷 −𝐷 𝑝McN = min 1, 2 2 . (16) 𝑗 𝑗=0 This exact conditional test is used as the primary inferential comparison with the always-KB-KSSD baseline (McNemar, 1947). The magnitude and direction of the paired effect are summarised by the matched-pairs odds ratio 𝑏 ORmatched = . 𝑐

(17)

The asymptotic McNemar statistics and the construction of the exact odds-ratio confidence interval are given in Appendix A.8.

Bootstrap uncertainty. Bootstrap uncertainty is evaluated

for accuracy, balanced accuracy, CP precision, CP recall, CP F1-score, and the paired accuracy gain over always-KBKSSD using 𝑅boot = 50,000 replications. For the primary series-level bootstrap, resampling is stratified by the true Beseda et al.: Preprint submitted to Elsevier

SSD winner: 152 CP-SSD-winning and 255 KB-KSSDwinning series are sampled independently with replacement within their respective classes, thereby preserving the observed class composition in every replicate. We report percentile confidence intervals and bootstrap standard deviations. For the paired accuracy gain over always-KB-KSSD, the same resampled series are used for both strategies in every replicate, preserving the paired structure of the comparison (Efron and Tibshirani, 1993). The corresponding bootstrap definitions are given in Appendix A.9.

Project-level dependence sensitivity. Individual bench-

mark series originating from the same software project may not be statistically independent. As a sensitivity analysis, the 407 labelled series are partitioned into 𝐽 = 30 project groups inferred from the benchmark identifiers. Each project-bootstrap replicate samples 30 projects with replacement; whenever a project is selected, all of its constituent series enter the replicate, with corresponding multiplicity if the same project is selected more than once. Metrics and paired accuracy gains are then recomputed on the resulting replicate, and percentile intervals are obtained using the bootstrap procedure described in Appendix A.9. Because project membership is inferred from the benchmark naming convention rather than from an independently curated identifier, this analysis is treated as a dependence-sensitivity check rather than as the principal inferential procedure.

Clustering-population robustness. Let 𝑦̂(𝐴) ̂(𝐵) 𝑖 and 𝑦 𝑖 de-

note the final unique-series recommendations obtained from the steady-JAGT-only and all-timeseries clustering regimes, respectively. Their agreement is quantified using Cohen’s 𝜅, which corrects the observed agreement for agreement expected from the predictors’ marginal recommendation frequencies (Cohen, 1960): 𝜅=

𝑝𝑜 − 𝑝 𝑒 . 1 − 𝑝𝑒

(18)

The definitions of 𝑝𝑜 and 𝑝𝑒 are given in Appendix A.10. Predictive correctness under the two regimes is additionally compared using the same exact McNemar formulation as Eq. 16, with 𝑏 and 𝑐 redefined as the numbers of series on which only one of the two regimes is correct. A nonsignificant result in this comparison is interpreted as absence of evidence for a directional difference, rather than as evidence of formal statistical equivalence.

4. Results This section presents the empirical results with respect to the four research questions introduced in Section 1. We first analyse the distribution of CP-SSD and KB-KSSD wins and the extent to which these preferences are reflected in the discovered cluster structure. We then evaluate the predictive performance of the SSD recommender against random and majority-based reference strategies, followed by an analysis of the remaining classification errors and the contribution Page 9 of 20

Automatic SSD Recommendation 144

Total labelled JAGT series KB-KSSD wins

140

Ties CP-SSD wins

Number of labelled JAGT series

120 100

95

86

84

80 60 53

40

55

41 30

20 0

20 8

3

9

Cluster 2

Cluster 1

Cluster 0

Figure 2: CP-SSD and KB-KSSD outcomes in the three non-outlier clusters of the selected steady-JAGT-only configuration. Ties in the cluster-level method comparison are shown separately and are not used to define the supervised winner label.

of the cluster-signature features. Finally, we assess the robustness of the recommendations to the population used for unsupervised clustering.

4.1. RQ1: How often does each SSD method provide the better estimate, and does method preference vary systematically across clusters of similar time series? All 407 labelled JAGT time series are decisive under the separate-wins definition. KB-KSSD is preferred on 255 series (62.65%), while CP-SSD is preferred on 152 (37.35%). Most decisions arise from smaller absolute error when both methods return valid indices: KB-KSSD has the smaller error in 251 cases and CP-SSD in 147. In nine additional cases, exactly one method returns a valid steady-state index, producing five CP-SSD wins and four KB-KSSD wins. The selected steady-JAGT-only clustering also exposes method specialisation rather than merely reproducing the global class imbalance. Two non-outlier clusters are KBfavouring: one contains 95 KB-KSSD wins, 41 CP-SSD wins, and 8 ties in the cluster-level comparison, while a second contains 55 KB-KSSD wins, 20 CP-SSD wins, and 9 ties. The third non-outlier cluster is CP-favouring, with 53 CP-SSD wins, 30 KB-KSSD wins, and 3 ties. Figure 2 visualises these distributions. Answer to RQ1. Both SSD methods are preferred for substantial subsets of the labelled series, and the selected clustering contains both KB-favouring and CP-favouring regions, demonstrating systematic cluster-level method specialisation.

Beseda et al.: Preprint submitted to Elsevier

4.2. RQ2: Can the better SSD method be recommended automatically, and does recommendation outperform random choice and/or majority-winning detector? The primary hybrid recommender correctly selects the better method for 297 of the 407 unique series, corresponding to 72.97% accuracy (exact 95% CI 68.38–77.23%). This is 10.32 percentage points above the 62.65% accuracy obtained by always recommending KB-KSSD, which is correct for 255 of 407 series (exact 95% CI 57.75–67.37%). The paired comparison with always-KB-KSSD is more informative because both strategies make a decision on the same series. Of the 407 observations, both strategies are correct on 228 and both are wrong on 83. The hybrid alone is correct on 69 series, whereas always-KB-KSSD alone is correct on only 27. The exact two-sided McNemar test therefore rejects equality of the paired error rates (𝑝 = 2.148 × 10−5 ). The corresponding uncorrected statistic is 𝜒 2 = 18.375 (𝑑𝑓 = 1, 𝑝 = 1.814 × 10−5 ), and the continuity-corrected statistic is 𝜒 2 = 17.510 (𝑑𝑓 = 1, 𝑝 = 2.857 × 10−5 ). The matchedpairs odds ratio is 2.56 with an exact 95% CI of 1.62–4.15, i.e., among discordant cases the hybrid is approximately 2.6 times as likely to be the strategy that is correct. The recommender also exceeds both random reference strategies. Against equiprobable random selection, the exact one-sided binomial test gives 𝑝 = 2.95 × 10−21 ; against the 53.20% prevalence-matched random expectation, 𝑝 = 2.24×10−16 . These results show that the recommender is not merely exploiting the empirical winner prevalence. Figure 3 summarizes the unique-series accuracy of the recommender relative to the reference strategies. Page 10 of 20

Automatic SSD Recommendation 80

73.0%

70

73.0%

62.7%

73.0%

Accuracy

67.4%

Balanced accuracy

60

Accuracy (%)

53.2% 50.0%

50

71.9%

CP precision

40 30

45.4%

CP recall

20 10 Random 50/50

Random prevalence

Always KB-KSSD

Hybrid JAGT-only

The 50,000-replicate stratified paired bootstrap yields a 95% interval of 69.29–76.66% for accuracy and 6.63– 14.00 percentage points for the gain over always-KB-KSSD. The corresponding bootstrap SD is 1.93 percentage points for both accuracy and the paired accuracy gain. Thus, the entire series-level bootstrap distribution of the paired effect remains well above zero. The project-cluster sensitivity analysis is wider, as expected, but remains positive: accuracy spans 67.80–78.20%, while the corresponding gain interval is 5.25–15.97 percentage points. The project-bootstrap SD is 2.66 percentage points for accuracy and 2.74 percentage points for the paired gain. The positive gain interval under this more dependence-aware resampling scheme provides evidence that the improvement is not driven only by treating series from the same project as independent observations. The paired bootstrap distribution of the accuracy gain and the uncertainty intervals for the principal performance metrics are visualized in Figures 4 and 5. 95% CI: 6.63 to 14.00 pp

Observed: 10.32 pp

4000

No improvement

5000

3000

2000

1000

0

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Accuracy gain over always-KB-KSSD (percentage points)

17.5

Figure 4: Stratified paired-bootstrap distribution of the accuracy gain of the primary hybrid recommender over always recommending KB-KSSD. The distribution is based on 50,000 bootstrap replications; the observed gain is 10.32 percentage points and the 95% percentile interval is 6.63–14.00 percentage points.

Across the 20 repeated 70/30 holdouts, the hybrid recommender has mean accuracy 71.83% (SD 2.57 percentage Beseda et al.: Preprint submitted to Elsevier

40

Hybrid all-series

Figure 3: Statistical comparison of the primary recommender with reference strategies. The plot presents unique-series accuracy relative to random and always-KB-KSSD baselines, with exact 95% intervals where applicable.

Bootstrap replicates

55.6%

CP F1

0

50

60 Performance (%)

70

80

90

Figure 5: Primary hybrid-recommender performance with 95% percentile intervals obtained from 50,000 class-stratified bootstrap replications. Intervals are shown for accuracy, balanced accuracy, CP-SSD precision, recall, and F1-score.

Table 3 Configuration and repeated-holdout performance of the selected hybrid recommender. Values are means over 20 stratified 70/30 holdouts. Property Model Feature selection Selected TSFresh features Signature size 𝑞 Mean holdout accuracy Mean balanced accuracy Mean CP precision Mean CP recall Mean CP F1-score

Value Random Forest Mutual information 100 20 71.83% 66.23% 69.62% 44.02% 53.76%

points), median 71.54%, and range 66.67–78.05%, compared with 62.60% for the always-KB-KSSD strategy on the same held-out class proportions. The repeated holdouts are not used as independent observations for hypothesis testing, but they provide a useful stability diagnostic. Random reference strategies fluctuate around their expected levels and remain well below the hybrid distribution. The selected hybrid configuration and its repeated-holdout performance are summarised in Table 3, while Table 4 reports the corresponding unique-series performance. The accuracy distribution is visualized in Figure 6.

4.2.1. Direct TSFresh ablation and feature-level interpretation To isolate the contribution of the cluster-signature metafeatures, we compare the hybrid recommender with an otherwise comparable model trained only on the selected direct TSFresh descriptors. The clustering-derived signatures do not produce a consistent accuracy improvement over a direct TSFresh classifier. The best direct Random Forest using mutualinformation selection and 100 TSFresh features obtains 71.38% mean holdout accuracy, compared with 71.83% for the hybrid model, a difference of only 0.45 percentage Page 11 of 20

Automatic SSD Recommendation

75

Holdout accuracy (%)

70 65 60 55 50 45 Random 50/50

Random prevalence

Always KB-KSSD

Hybrid recommender

Figure 6: Distribution of accuracy over the 20 repeated 70/30 holdout splits for the primary steady-JAGT-only analysis. Points show individual split values; the boxplot is descriptive because the test sets overlap.

Table 4 Unique-series performance of the primary steady-JAGT-only hybrid recommender. Metric Correct recommendations Accuracy Balanced accuracy CP-SSD precision CP-SSD recall CP-SSD F1-score Predicted CP-SSD cases Predicted KB-KSSD cases

Value 297 / 407 72.97% 67.40% 71.88% 45.39% 55.65% 96 311

points. When repeated held-out predictions are aggregated by the same majority rule used for the hybrid model, the direct classifier correctly predicts 299 of 407 series (73.46%), compared with 297 (72.97%) for the hybrid. The two unique-series predictors agree on 397 of 407 series; among the 10 disagreements, the direct model is correct in six cases and the hybrid in four (exact McNemar 𝑝 = 0.754). These results are summarized in Table 5. Consequently, the results do not support a claim that cluster signatures improve predictive accuracy. Their present value is instead in exposing interpretable cluster-level method specialisation and in providing a structured basis for future meta-feature designs. Answer to RQ2. Yes. The hybrid recommender reaches 72.97% unique-series accuracy and significantly outperforms both random reference strategies and the always-KBKSSD baseline, although it does not outperform the direct TSFresh classifier.

Beseda et al.: Preprint submitted to Elsevier

Table 5 Primary recommender and direct-TSFresh ablation. Repeatedholdout values are means over 20 splits; unique-series values aggregate repeated held-out predictions per benchmark. Metric Mean holdout accuracy Mean balanced accuracy Unique-series accuracy Unique balanced accuracy Unique CP precision Unique CP recall Unique CP F1-score

Direct TSFresh

Hybrid

71.38% 65.96% 73.46% 68.46% 71.15% 48.68% 57.81%

71.83% 66.23% 72.97% 67.40% 71.88% 45.39% 55.65%

The most influential hybrid-model variables are overwhelmingly direct TSFresh descriptors. Prominent descriptors include change-quantile statistics, peak counts, permutation entropy, index-mass quantiles, and Fourier coefficients. In the aggregated feature-importance summary, signature-derived variables account for only about 1.31% of the reported importance mass. This is consistent with the ablation result: the cluster signatures encode meaningful structure, but the current similarity-based representation contributes little additional predictive signal beyond the direct time-series descriptors. The most influential variables of the primary hybrid recommender are listed in Figure 7, while Table 6 reports the ten highest-ranked direct TSFresh features and their mean importances.

Page 12 of 20

Automatic SSD Recommendation

change_quantiles | f_agg_"mean" | isabs_True | qh_0.8 | ql_0.0

0.022

number_peaks | n_1

0.022

permutation_entropy | dimension_7 | tau_1

0.022

permutation_entropy | dimension_5 | tau_1

0.021

permutation_entropy | dimension_6 | tau_1

0.021

change_quantiles | f_agg_"mean" | isabs_True | qh_0.2 | ql_0.0

0.021 0.020

change_quantiles | f_agg_"mean" | isabs_True | qh_0.4 | ql_0.0 index_mass_quantile | q_0.4

0.019

change_quantiles | f_agg_"mean" | isabs_True | qh_0.6 | ql_0.0

0.019

fft_coefficient | attr_"abs" | coeff_0

0.017

permutation_entropy | dimension_3 | tau_1

0.017

permutation_entropy | dimension_4 | tau_1

0.017

number_crossing_m | m_0

0.017

fft_coefficient | attr_"real" | coeff_5

0.017

cwt_coefficients | coeff_10 | w_20 | widths_(2, 5, 10, 20)

0.017

sig_kb_over_cp_ratio [signature; overall #48]

0.013

sig_cp_over_kb_ratio [signature; overall #55]

0.012 0.010

sig_margin_cp_minus_kb [signature; overall #143]

0.009

sig_cp_score [signature; overall #182]

0.000

TSFresh feature Cluster-signature feature (overall rank shown)

0.006

sig_abs_margin [signature; overall #428]

0.005

0.010

0.015 0.020 0.025 Mean feature importance

0.030

0.035

Figure 7: Highest-ranked direct TSFresh and cluster-signature features of the primary hybrid recommendation model. For clustersignature features, labels also report their rank in the complete feature-importance ordering.

Table 6 Ten highest-ranked TSFresh features in the aggregated feature-importance summary of the primary hybrid model. Rank

TSFresh feature

1 2 3 4 5 6 7 8 9 10

Change quantiles (𝑞𝑙 = 0.0, 𝑞ℎ = 0.8) Number of peaks (𝑛 = 1) Permutation entropy (𝑑 = 7, 𝜏 = 1) Permutation entropy (𝑑 = 5, 𝜏 = 1) Permutation entropy (𝑑 = 6, 𝜏 = 1) Change quantiles (𝑞𝑙 = 0.0, 𝑞ℎ = 0.2) Change quantiles (𝑞𝑙 = 0.0, 𝑞ℎ = 0.4) Index mass quantile (𝑞 = 0.4) Change quantiles (𝑞𝑙 = 0.0, 𝑞ℎ = 0.6) FFT coefficient, absolute part (𝑘 = 0)

Mean importance

Selected in splits

0.0222 0.0220 0.0217 0.0212 0.0212 0.0211 0.0201 0.0191 0.0187 0.0175

1 7 7 14 6 2 3 2 7 7

4.3. RQ3: What kinds of recommendation errors remain, particularly for the minority CP-SSD class? The hybrid model is conservative in recommending the minority CP-SSD class. It predicts CP-SSD for 96 series although CP-SSD is the reference winner for 152. The uniqueseries confusion matrix contains 69 true positives, 27 false Beseda et al.: Preprint submitted to Elsevier

Page 13 of 20

Automatic SSD Recommendation

Actual KB-KSSD

Actual CP-SSD

Predicted KB-KSSD

Predicted CP-SSD

228

27

89.4%

10.6%

83

69

54.6%

45.4%

Figure 8: Confusion matrix of the steady-JAGT-only hybrid recommender.

The bootstrap intervals reflect the same imbalance. Under the stratified series bootstrap, balanced accuracy has a 95% interval of 63.01–71.75%, CP precision 63.92–80.00%, CP recall 37.50–53.29%, and CP F1-score 48.07–62.75%. The project-cluster sensitivity analysis widens the recall interval further to 32.75–56.93%, indicating that the principal remaining limitation is not an inability to identify KB-KSSD cases but insufficient sensitivity to CP-SSD-favourable executions. Answer to RQ3. The main remaining error mode is insufficient sensitivity to CP-SSD-favourable series: CP-SSD recall is 45.39%, whereas most KB-KSSD-favourable cases are correctly retained.

4.4. RQ4: How sensitive are the recommendations to the population used for unsupervised clustering? The primary analysis learns the clustering representation only from the 407 labelled JAGT series. As a robustness check, the pipeline was also evaluated with the selected clustering configuration learned from all 5,860 cached time series. The all-timeseries variant again obtains 297 correct unique-series recommendations (72.97%), with balanced accuracy 67.67%, CP precision 71.00%, and CP recall 46.71%. Because the overall accuracy is unchanged, the alltimeseries robustness check also yields the same binomial results against the random references. Its paired comparison with always-KB-KSSD contains 71 hybrid-only correct cases and 29 always-KB-only correct cases, giving exact McNemar 𝑝 = 3.216 × 10−5 and matched-pairs OR = 2.45 Beseda et al.: Preprint submitted to Elsevier

(95% CI 1.57–3.91). The complete unique-series statistical comparison for the primary and all-timeseries analyses is provided in Table 7. More importantly, the two clustering regimes produce nearly indistinguishable final decisions. They agree on 399 of 407 predictions (98.03%), with Cohen’s 𝜅 = 0.946. Only eight series receive different recommendations. Among these, the steady-JAGT-only regime is correct while the alltimeseries regime is wrong in four cases, and the reverse occurs in four cases. The exact two-sided McNemar comparison of correctness is therefore 𝑝 = 1.000. This value arises because the only discordant correctness outcomes are split perfectly evenly (4 vs. 4); it does not mean that the two predictors are literally identical. Together with the identical 72.97% overall accuracy and the highly overlapping bootstrap intervals, the result indicates strong empirical stability to the clustering population, while not constituting a formal equivalence test. The all-timeseries robustness check exhibits almost identical uncertainty. Its stratified 95% gain interval is +6.39– +14.25 percentage points and its project-cluster interval is +5.19–+16.11 percentage points; the corresponding gain SDs are 1.96 percentage points and 2.79 percentage points, respectively. The consistency of these intervals with the primary analysis supports the robustness of the effect to the population used for unsupervised structure discovery. The complete bootstrap intervals for both clusteringpopulation analyses are presented in Table 8. The paired comparison between the two clusteringpopulation regimes is summarized in Table 9, while the comparison of the principal performance metrics is visualized in Figure 9. Answer to RQ4. The recommendations are highly robust to the clustering population: the two regimes have identical overall accuracy and agree on 98.03% of the individual recommendations. 85 80

JAGT-only clustering

All-series clustering

73.0

75

Performance (%)

positives, 83 false negatives, and 228 true negatives. Thus, CP-SSD precision is 71.88%, but recall is only 45.39%; the corresponding F1-score is 55.65%. In contrast, 228 of the 255 KB-KSSD-winning series are retained as KB-KSSD recommendations. Figure 8 shows the complete confusion matrix.

67.4

70 65

73.0

55.6 67.7

60 55 50

56.3

45 40

Accuracy

Balanced accuracy

CP F1

Figure 9: Robustness of the final recommendation to the population used for clustering. Both regimes reach the same unique-series accuracy and agree on 98.03% of individual recommendations.

Across the same 20 repeated 70/30 holdouts, the alltimeseries analysis has mean accuracy 71.50% (SD 3.49 percentage points), compared with 71.83% (SD 2.57 percentage Page 14 of 20

Automatic SSD Recommendation Table 7 Unique-series statistical validation for the primary steady-JAGT-only regime and the all-timeseries robustness regime. Exact McNemar 𝑝-values are two-sided; random-baseline binomial tests are one-sided. Quantity Class counts (CP / KB) Correct / total Accuracy Exact 95% CI for accuracy Balanced accuracy CP precision CP recall CP F1-score Always-KB-KSSD accuracy Exact 95% CI for always-KB 50/50 random expectation Prevalence-random expectation Gain over always-KB-KSSD Both strategies correct Hybrid-only correct (𝑏) Always-KB-only correct (𝑐) Both strategies wrong Discordant pairs (𝑏 + 𝑐) Exact McNemar 𝑝 (two-sided) Exact McNemar 𝑝 (one-sided) McNemar 𝜒 2 , uncorrected McNemar 𝜒 2 , corrected Matched-pairs OR Binomial 𝑝 vs 50/50 random Binomial 𝑝 vs prevalence random

Steady-JAGT-only

All-timeseries

152 / 255 297 / 407 72.97% [68.38, 77.23]% 67.40% 71.88% 45.39% 55.65% 62.65% [57.75, 67.37]% 50.00% 53.20% +10.32 pp 228 69 27 83 96 2.148 × 10−5 1.074 × 10−5 18.375 (𝑝 = 1.814 × 10−5 ) 17.510 (𝑝 = 2.857 × 10−5 ) 2.56 [1.62, 4.15] 2.954 × 10−21 2.243 × 10−16

152 / 255 297 / 407 72.97% [68.38, 77.23]% 67.67% 71.00% 46.71% 56.35% 62.65% [57.75, 67.37]% 50.00% 53.20% +10.32 pp 226 71 29 81 100 3.216 × 10−5 1.608 × 10−5 17.640 (𝑝 = 2.669 × 10−5 ) 16.810 (𝑝 = 4.132 × 10−5 ) 2.45 [1.57, 3.91] 2.954 × 10−21 2.243 × 10−16

Table 8 Bootstrap uncertainty for the hybrid recommender. Cells report the point estimate followed by the 95% percentile interval. “Series” denotes the class-stratified paired bootstrap; “Project” denotes the project-cluster sensitivity analysis. Metric Accuracy Balanced accuracy CP precision CP recall CP F1-score Gain vs always-KB

JAGT Series

JAGT Project

All Series

All Project

72.97 [69.29, 76.66]% 67.40 [63.01, 71.75]% 71.88 [63.92, 80.00]% 45.39 [37.50, 53.29]% 55.65 [48.07, 62.75]% +10.32 [+6.63, +14.00] pp

72.97 [67.80, 78.20]% 67.40 [62.15, 72.13]% 71.88 [63.95, 79.22]% 45.39 [32.75, 56.93]% 55.65 [44.24, 64.15]% +10.32 [+5.25, +15.97] pp

72.97 [69.04, 76.90]% 67.67 [63.22, 72.08]% 71.00 [63.21, 78.95]% 46.71 [38.82, 54.61]% 56.35 [48.80, 63.36]% +10.32 [+6.39, +14.25] pp

72.97 [68.19, 77.92]% 67.67 [62.72, 72.16]% 71.00 [62.77, 78.95]% 46.71 [34.59, 57.82]% 56.35 [45.74, 64.34]% +10.32 [+5.19, +16.11] pp

Table 9 Direct comparison of the two clustering-population regimes. Quantity Identical predictions Prediction disagreements Cohen’s 𝜅 JAGT-only correct / all-series wrong All-series correct / JAGT-only wrong Exact McNemar 𝑝

Value 399 / 407 (98.03%) 8 0.946 4 4 1.000

points) for the primary analysis. Its accuracy ranges from 65.04% to 77.24%, indicating similar average performance but somewhat greater split-to-split dispersion. The corresponding statistics for both clustering-population analyses are provided in Table 10. Beseda et al.: Preprint submitted to Elsevier

5. Discussion The main result is that SSD selection contains learnable signal. Always choosing KB-KSSD is already a relatively strong strategy because KB-KSSD wins 62.65% of the labelled series, yet the learned recommender raises uniqueseries accuracy to approximately 73%. The paired analysis is important here: the hybrid model recovers 69 CP-SSD cases that the majority strategy necessarily misses while sacrificing only 27 KB-KSSD cases. The corresponding odds ratio and bootstrap intervals indicate that this improvement is not explained by a small number of marginal observations. At the same time, the results refine the role of clustering in the framework. The selected clusters clearly reveal SSD specialisation, including a CP-favouring cluster despite the

Page 15 of 20

Automatic SSD Recommendation Table 10 Descriptive accuracy over the 20 repeated 70/30 holdout splits. The splits overlap and therefore these values are not used as independent samples in inferential tests. Method Random 50/50 Random prevalence Always KB-KSSD Hybrid recommender

JAGT mean ± SD

JAGT median [range]

All mean ± SD

All median [range]

50.41 ± 3.75 53.94 ± 4.19 62.60 ± 0.00 71.83 ± 2.57

50.41 [42.28, 56.10] 54.47 [46.34, 60.16] 62.60 [62.60, 62.60] 71.54 [66.67, 78.05]

50.41 ± 3.75 53.94 ± 4.19 62.60 ± 0.00 71.50 ± 3.49

50.41 [42.28, 56.10] 54.47 [46.34, 60.16] 62.60 [62.60, 62.60] 71.14 [65.04, 77.24]

global KB-KSSD majority. However, the current centroidsimilarity signature representation does not consistently improve prediction over direct TSFresh features. The hybrid model is slightly better on mean holdout accuracy, whereas the direct model is slightly better after unique-series aggregation, and the difference between their unique predictions is not significant. Cluster signatures should therefore be viewed as an interpretable structural layer and a starting point for richer meta-features rather than as an established source of predictive improvement. The robustness experiment strengthens this interpretation. Learning the clustering structure from all 5,860 cached traces changes the selected PCA/UMAP/HDBSCAN configuration, yet 98.03% of final recommendations remain unchanged. Thus, the supervised decision boundary is not strongly dependent on one particular clustering population. This stability is encouraging for replication, although it also suggests that most of the predictive signal currently comes from the direct TSFresh branch. The principal modelling weakness is CP-SSD recall. The hybrid model identifies fewer than half of the CPfavourable series. Improving this quantity is more important than marginal gains in aggregate accuracy because an accuracy-only objective is naturally biased toward the KBKSSD majority. Future models could therefore optimise balanced accuracy or a cost-sensitive objective, explicitly calibrate the CP decision threshold, and evaluate whether the resulting increase in CP recall provides a better practical trade-off.

6. Threats to Validity 6.1. Internal validity The reported model family, feature-selection strategy, signature size, and clustering configuration were selected from evaluated alternatives. Although all label-dependent feature selection and signature construction are fitted only on the training portion of each holdout, model/configuration selection and performance estimation use the same repeatedholdout experiment. The reported mean holdout performance may therefore be optimistic relative to a fully nested model-selection protocol. The unique-series aggregation reduces dependence among repeated test appearances for the final statistical comparison, but it does not remove this model-selection issue. Future work should use nested resampling or an external test collection for confirmatory evaluation. Beseda et al.: Preprint submitted to Elsevier

A second threat concerns data alignment. The replication pipeline explicitly aligns extracted TSFresh rows with the canonical JAGT series keys before any label merge. This is necessary because filesystem enumeration order is not a reliable identifier. The replication package linked in Section 8 therefore uses key-based alignment rather than positional matching.

6.2. Construct validity The recommendation target is defined by the smaller absolute error relative to the JAGT reference steady-state index, with explicit rules for cases in which only one detector returns a valid index. This operationalisation captures location error but does not cover every property that may matter in practice, such as computational cost, stability under perturbation, or asymmetric consequences of early and late detection. Different utility functions could therefore induce different recommendation labels. The project-cluster bootstrap uses the repository/project prefix encoded in the benchmark key and yields 30 project groups. This provides a useful sensitivity analysis for withinproject dependence, but the grouping is derived from the naming convention rather than an independently curated project identifier.

6.3. External validity The labelled evaluation comprises 407 JAGT Java microbenchmark time series and focuses on two SSD methods. Although the full cache contains 5,860 series, the additional unlabelled observations do not contribute directly to supervised model training. The extent to which the observed results transfer to other benchmark suites, languages, runtime environments, hardware platforms, or SSD portfolios should therefore be investigated further. CP-SSD is also the minority winner in the present dataset, which may contribute to the lower recall observed for CP-SSD-favourable cases. Evaluating the recommender on broader labelled datasets, including more CP-favourable and borderline cases, would help clarify the generality of this behaviour.

6.4. Conclusion validity The 20 repeated holdout splits overlap and are not treated as independent replicates for hypothesis testing. Inferential comparisons instead use one aggregated prediction per unique series. The exact McNemar test is appropriate for

Page 16 of 20

Automatic SSD Recommendation

the paired comparison with always-KB-KSSD, while bootstrap intervals quantify uncertainty in effect sizes. The alltimeseries versus steady-JAGT-only comparison has only eight discordant predictions; the resulting McNemar 𝑝 = 1.0 indicates no detectable directional difference but must not be interpreted as proof of statistical equivalence.

7. Conclusions This paper introduced a data-driven framework for recommending steady-state detection methods for software performance benchmark time series. Using 407 decisive JAGTlabelled series, the primary hybrid recommender selects the lower-error method in 297 cases (72.97%), improving by 10.32 percentage points over always choosing the majority KB-KSSD method. Exact paired testing and two complementary bootstrap analyses show that this improvement is statistically robust. The experiments also identify the limits of the current design. Direct TSFresh features already contain most of the predictive signal, and the present cluster-signature representation does not yield a consistent accuracy advantage. Nevertheless, clustering exposes interpretable SSDspecialisation patterns and the downstream recommendations remain highly stable when the clustering population is expanded from the labelled JAGT subset to all 5,860 cached time series. The framework therefore supports the broader premise of automatic SSD recommendation while providing a clear direction for refinement. Future work should: (i) expand the labelled benchmark collection, especially with CP-SSD-favourable and difficult boundary cases; (ii) evaluate cost-sensitive and threshold-calibrated recommenders; (iii) replace centroid signatures with richer distributional or soft-membership representations; (iv) exploit the unlabelled series through semi-supervised learning; (v) extend the portfolio beyond two SSD algorithms. A final confirmatory study should use nested model selection or an external benchmark collection to obtain an unbiased estimate of generalisation performance.

8. Data and code availability The data and analysis materials required to reproduce the results reported in this study are available in the accompanying versioned replication package: https://github.com/ MartinBeseda/steady-state/releases/tag/v1.2.0. The package contains the archived detector-comparison results, cached TSFresh representation, analysis and statistical-validation scripts, generated result tables, and figure-generation code. Upstream CP-SSD/KB-KSSD execution scripts are also provided for provenance, although their specialised external research dependencies are not distributed as a self-contained software environment.

Beseda et al.: Preprint submitted to Elsevier

Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process During the preparation of this work, the authors used OpenAI ChatGPT to assist with manuscript organisation, language and readability, and consistency checking. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

References Abdi, H., Williams, L.J., 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics 2, 433–459. Anfossi, A., Chicco, D., 2026. An easy guide to the davies-bouldin index for unsupervised internal clustering evaluation. Discover Computing 29, 212. Barrett, E., Bolz-Tereick, C.F., Killick, R., Mount, S., Tratt, L., 2017. Virtual machine warmup blows hot and cold. Proceedings of the ACM on Programming Languages 1, 52:1–52:27. doi:10.1145/3133876. Beseda, M., Cortellessa, V., Di Pompeo, D., Traini, L., Tucci, M., 2026. A kernel-based approach for accurate steady-state detection in performance time series. Future Generation Computer Systems 177, 108233. doi:10.1016/j.future.2025.108233. Bethea, R.M., Rhinehart, R.R., 2019. Applied engineering statistics. Routledge. Boesler, M., Weber, N., 2017. Steady state detection for computational fluid dynamics. arXiv preprint arXiv:1709.03158 . Breiman, L., 2001. Random forests. Machine Learning 45, 5–32. doi:10. 1023/A:1010933404324. Campello, R.J.G.B., Moulavi, D., Zimek, A., Sander, J., 2015. Hierarchical density estimates for data clustering, visualization, and outlier detection. ACM Transactions on Knowledge Discovery from Data 10, 5:1–5:51. doi:10.1145/2733381. Cao, S., 1996. Statistically-based techniques for process control applications. Texas Tech University. Christ, M., Braun, N., Neuffer, J., Kempa-Liehr, A.W., 2018. Time series feature extraction on basis of scalable hypothesis tests (tsfresh – a python package). Neurocomputing 307, 72–77. doi:10.1016/j.neucom.2018.03. 067. Clopper, C.J., Pearson, E.S., 1934. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404–413. doi:10.1093/biomet/26.4.404. Cohen, J., 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46. doi:10.1177/ 001316446002000104. Efron, B., Tibshirani, R.J., 1993. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, Chapman and Hall, New York. Georges, A., Buytaert, D., Eeckhout, L., 2007. Statistically rigorous java performance evaluation, in: Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications, ACM. pp. 57–76. doi:10.1145/1297027.1297033. Jolliffe, I.T., 2002. Principal Component Analysis. Springer Series in Statistics. 2 ed., Springer, New York. doi:10.1007/b98835. Kalibera, T., Jones, R.E., 2013. Rigorous benchmarking in reasonable time, in: Proceedings of the 2013 International Symposium on Memory Management, ACM. doi:10.1145/2464157.2464160.

Page 17 of 20

Automatic SSD Recommendation Kelly, J.D., Hedengren, J.D., 2013. A steady-state detection (SSD) algorithm to detect non-stationary drifts in processes. Journal of Process Control 23, 326–331. doi:10.1016/j.jprocont.2012.12.001. Kerschke, P., Hoos, H.H., Neumann, F., Trautmann, H., 2019. Automated algorithm selection: Survey and perspectives. Evolutionary Computation 27, 3–45. doi:10.1162/evco_a_00242. Killick, R., Fearnhead, P., Eckley, I.A., 2012. Optimal detection of changepoints with a linear computational cost. Journal of the American Statistical Association 107, 1590–1598. doi:10.1080/01621459.2012. 737745. Kotthoff, L., 2014. Algorithm selection for combinatorial search problems: A survey. AI Magazine 35, 48–60. doi:10.1609/aimag.v35i3.2460. Kulkarni, V.Y., Sinha, P.K., 2013. Random forest classifiers: a survey and future research directions. Int. J. Adv. Comput 36, 1144–1153. McInnes, L., Healy, J., Astels, S., et al., 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw. 2, 205. McInnes, L., Healy, J., Saul, N., Grossberger, L., 2018. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software 3, 861. doi:10.21105/joss.00861. McNemar, Q., 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 153–157. doi:10.1007/BF02295996. Pincus, S.M., 1991. Approximate entropy as a measure of system complexity. Proceedings of the National Academy of Sciences 88, 2297–2301. doi:10.1073/pnas.88.6.2297. Prudêncio, R.B., Ludermir, T.B., 2004. Meta-learning approaches to selecting time series models. Neurocomputing 61, 121–137. doi:10. 1016/j.neucom.2004.03.008. Řezanková, H., 2018. Different approaches to the silhouette coefficient calculation in cluster evaluation, in: 21st international scientific conference AMSE applications of mathematics and statistics in economics, pp. 1– 10. Rice, J.R., 1976. The algorithm selection problem, in: Advances in Computers. Elsevier. volume 15, pp. 65–118. doi:10.1016/S0065-2458(08) 60520-3. Shahoud, S., Khalloof, H., Duepmeier, C., Hagenmeyer, V., 2020. Descriptive statistics time-based meta features (dstmf): Constructing a better set of meta features for model selection in energy time series forecasting, in: Proceedings of the 3rd International Conference on Applications of Intelligent Systems, ACM. doi:10.1145/3378184.3378221. Talagala, T.S., Hyndman, R.J., Athanasopoulos, G., 2023. Meta-learning how to forecast time series. Journal of Forecasting . Traini, L., Cortellessa, V., Di Pompeo, D., Tucci, M., 2023. Towards effective assessment of steady state performance in java software: Are we there yet? Empirical Software Engineering 28, 13. doi:10.1007/ s10664-022-10247-x. Wu, J., Chen, Y., Zhou, S., 2016. Online detection of steady-state operation using a multiple-change-point model and exact bayesian inference. IIE transactions 48, 599–613.

A. Supplementary Methodological Definitions A.1. Feature Standardisation Let f it denote the observations on which a preprocessing operation is fitted and let 𝑛f it = |f it |. For feature 𝑗, define 1 ∑ 𝑧 , 𝑛f it 𝑟∈ 𝑟𝑗 f it √ 1 ∑ 𝜎𝑗 = (𝑧𝑟𝑗 − 𝜇𝑗 )2 . 𝑛f it 𝑟∈

𝜇𝑗 =

𝑧𝑖𝑗 − 𝜇𝑗

𝑧̃𝑖𝑗 =

𝜎𝑗

A.2. Principal Component Analysis ̃ ∈ ℝ𝑁clust ×𝑑 be the standardised feature matrix for Let 𝐙 the population used to fit the clustering representation. Its empirical covariance matrix is 𝐂=

1 ̃ 𝖳 𝐙. ̃ 𝐙 𝑁clust − 1

Beseda et al.: Preprint submitted to Elsevier

(22)

PCA solves (23)

𝐂𝐯𝓁 = 𝜆𝓁 𝐯𝓁 ,

(24)

𝜆1 ≥ 𝜆2 ≥ ⋯ ≥ 𝜆𝑑 ,

where 𝜆𝓁 and 𝐯𝓁 are the 𝓁-th eigenvalue and eigenvector. For a requested dimension 𝑝PCA , the projection matrix is (25)

𝐕𝑝PCA = (𝐯1 , … , 𝐯𝑝PCA ), and the PCA representation is

(26)

̃ 𝑝 𝐏 = 𝐙𝐕 ∈ ℝ𝑁clust ×𝑝PCA . PCA

A.3. Silhouette and Davies–Bouldin Indices Let  = {𝑖 ∶ 𝑐𝑖 ≠ −1} denote the set of non-outlier observations. For 𝑖 ∈ , let 𝑎(𝑖) be the mean distance from observation 𝑖 to the other members of its cluster and 𝑏(𝑖) the smallest mean distance from 𝑖 to another cluster. The Silhouette coefficient (Řezanková, 2018) is 𝑆sil =

1 ∑ 𝑠(𝑖), || 𝑖∈

(27)

𝑏(𝑖) − 𝑎(𝑖) . max{𝑎(𝑖), 𝑏(𝑖)}

(28)

where 𝑠(𝑖) =

Let 𝝃 𝑖 denote the representation of observation 𝑖 used for clustering, i.e., its PCA representation for PCA–HDBSCAN configurations or its UMAP representation for PCA–UMAP– HDBSCAN configurations. For cluster 𝑘, let 𝑘 = {𝑖 ∈  ∶ 𝑐𝑖 = 𝑘} denote its set of non-outlier members and let 𝝁𝑘 be its centroid in this representation. Its mean within-cluster dispersion is 𝑆𝑘 =

1 ∑ ‖𝝃 𝑖 − 𝝁𝑘 ‖2 . |𝑘 | 𝑖∈

(29)

𝑘

(19) (20)

Let 𝑀𝑘𝓁 = ‖𝝁𝑘 − 𝝁𝓁 ‖2 denote the distance between clusters 𝑘 and 𝓁. The Davies–Bouldin index (Anfossi and Chicco, 2026) is

f it

An observation 𝑖 is then standardised as

(21)

.

𝑆 + 𝑆𝓁 1 ∑ max 𝑘 . 𝐾 𝑘=1 𝓁≠𝑘 𝑀𝑘𝓁 𝐾

DBI =

(30) Page 18 of 20

Automatic SSD Recommendation

A.4. Mutual Information

A.8. Additional McNemar Statistics

For a continuous feature 𝑍𝑗 and the binary target 𝑌 , mutual information is

For 𝐷 = 𝑏+𝑐 discordant pairs, under the null hypothesis of equal discordant success probability,

(

∑

𝐼(𝑍𝑗 ; 𝑌 ) =

𝑦∈{0,1}

∫

𝑝𝑍𝑗 ,𝑌 (𝑧, 𝑦) log

𝑝𝑍𝑗 ,𝑌 (𝑧, 𝑦)

)

𝑝𝑍𝑗 (𝑧)𝑝𝑌 (𝑦) (31)

where 𝑝𝑍𝑗 ,𝑌 is the joint density/mass function and 𝑝𝑍𝑗 and 𝑝𝑌 are the corresponding marginals.

A.5. Classification Metrics

d𝑧,

( ) 1 𝐵 ∣ 𝐷 ∼ Binomial 𝐷, . 2

For completeness, the uncorrected and continuity-corrected asymptotic McNemar statistics are (𝑏 − 𝑐)2 , 𝑏+𝑐 (|𝑏 − 𝑐| − 1)2 2 𝜒corr = , 𝑏+𝑐

1 ∑ 𝕀(𝑦 = 𝑦̂𝑖 ). 𝑛te 𝑖∈te 𝑖

(32)

With CP-SSD treated as the positive class and TP, TN, FP, and FN denoting the confusion-matrix counts, TP , TP + FP TP RecallCP = , TP + FN 2 PrecisionCP RecallCP 𝐹1,CP = , PrecisionCP + RecallCP ( ) 1 TP TN BalancedAcc = + . 2 TP + FN TN + FP

(33)

(36)

ORmatched =

{ 𝑝U =

𝜃 . 1−𝜃

(45)

If [𝜃L , 𝜃U ] is the exact binomial confidence interval for 𝜃, the corresponding odds-ratio interval is

For 𝑋 successes among 𝑁 observations and nominal confidence level 1 − 𝛼, the exact Clopper–Pearson limits are

𝑝L =

(44)

so that

[

A.6. Clopper–Pearson Confidence Interval

{

𝑏 , 𝑏+𝑐

(34) (35)

(43)

respectively, and are compared with a 𝜒 2 distribution with one degree of freedom. For the matched-pairs odds ratio, define 𝜃=

PrecisionCP =

(42)

2 𝜒uncorr =

For a test set te of size 𝑛te = |te |, with labels 𝑦𝑖 and predictions 𝑦̂𝑖 , accuracy is Acc =

(41)

] 𝜃U 𝜃L , . 1 − 𝜃L 1 − 𝜃U

(46)

A.9. Bootstrap Uncertainty 0, ( ) 𝐵 −1 𝛼2 ; 𝑋, 𝑁 − 𝑋 + 1 ,

𝑋 = 0, 𝑋 > 0,

Let (37) (47)

𝑇 = 𝑔(𝐲, ̂ 𝐲)

1, ( ) 𝑋 = 𝑁, 𝛼 −1 𝐵 1 − 2 ; 𝑋 + 1, 𝑁 − 𝑋 , 𝑋 < 𝑁, (38)

where 𝐵 −1 (𝑞; 𝑎, 𝑏) denotes the 𝑞-quantile of a Beta distribution. In this study, 𝛼 = 0.05.

A.7. Exact Binomial Test For a random-reference success probability 𝑝0 , the onesided hypotheses are

denote a scalar statistic computed from the observed labels and predictions. For bootstrap replicate 𝑟 with resampled index set  ∗(𝑟) , ( ) 𝑇 ∗(𝑟) = 𝑔 𝐲 ∗(𝑟) , ̂ 𝐲 ∗(𝑟) ,

𝑟 = 1, … , 𝑅boot .

(48)

The percentile bootstrap confidence interval is [ ( ) ( )] ∗(𝑟) 𝑅boot ∗(𝑟) 𝑅boot CIboot = 𝑄 {𝑇 } , 𝑄 {𝑇 } , 𝛼∕2 1−𝛼∕2 1−𝛼 𝑟=1 𝑟=1

(39)

(49)

If 𝑋 = 𝑥 correct recommendations are observed among 𝑁 cases, the exact one-sided binomial 𝑝-value is

where 𝑄𝑞 (⋅) denotes the empirical 𝑞-quantile. The bootstrap standard deviation is

𝐻0 ∶ 𝑝 ≤ 𝑝0 ,

𝐻1 ∶ 𝑝 > 𝑝0 .

𝑁 ( ) ∑ 𝑁 𝑗 𝑝bin = 𝑃𝐻0 (𝑋 ≥ 𝑥) = 𝑝0 (1 − 𝑝0 )𝑁−𝑗 . (40) 𝑗 𝑗=𝑥

Beseda et al.: Preprint submitted to Elsevier

𝑠boot =

√ √ √ √

1

𝑅boot (

∑

𝑅boot − 1 𝑟=1

𝑇 ∗(𝑟) − 𝑇

) ∗ 2

,

(50)

Page 19 of 20

Automatic SSD Recommendation

with

1

∗

𝑇 =

𝑅boot

∑

𝑅boot 𝑟=1

(51)

𝑇 ∗(𝑟) .

For the paired accuracy gain, Δ∗(𝑟) = Acc∗(𝑟) − Acc∗(𝑟) . KB hybrid

(52)

A.10. Cohen’s Kappa For predictors 𝐴 and 𝐵, the observed agreement is

𝑝𝑜 =

𝑁 ( ) 1 ∑ (𝐵) 𝕀 𝑦̂(𝐴) = 𝑦 ̂ . 𝑖 𝑖 𝑁 𝑖=1

(53)

1 ∑ (𝐴) 𝕀(̂ 𝑦 = 𝑐), 𝑁 𝑖=1 𝑖

(54)

Let 𝑁

𝑞𝐴,𝑐 =

1 ∑ (𝐵) 𝑞𝐵,𝑐 = 𝕀(̂ 𝑦 = 𝑐), 𝑁 𝑖=1 𝑖 𝑁

𝑐 ∈ {0, 1},

(55)

be their marginal class frequencies. The chance agreement is 𝑝𝑒 = 𝑞𝐴,0 𝑞𝐵,0 + 𝑞𝐴,1 𝑞𝐵,1 .

Beseda et al.: Preprint submitted to Elsevier

(56)

Page 20 of 20

Related documents

• Towards Automated Recommendation of Steady-State Detection Algorithms for Software Performance Benchmarks#1660448
• CAROL: Context-Aware Online Learning for Fuzzer Scheduling#974861
• Harnessing MPI mutations for AI error detection#1355605
• Fail-Aware and Explainable Test Oracle Prediction#363322
• Regression-Based Prediction for Task-Based Program Performance#1027257