This article has been accepted for publication in IEEE International Conference on Network Softwarization (NetSoft 2026)
RANalyzer: Automated Continuous RAN Software Evaluation and Regression Analysis
Abstract—Software-driven O-RAN architectures enable rapid innovation through frequent, independent updates to virtualized components. However, attributing performance variations to specific software changes is challenging due to the stochastic nature of wireless systems, where channel conditions, interference, and hardware variability confound analysis. Traditional thresholdbased monitoring and manual troubleshooting do not scale with modern software evolution. This paper presents RANalyzer, an automated test analysis framework that quantifies the performance impact of software updates beyond what can be explained by wireless channel conditions. RANalyzer combines LLM-assisted semantic extraction with residuals analysis. The first categorizes code changes by affected protocol layers and functional components, while the second provides insights on the effect of load, channel, or code changes on the test performance. We contribute an extensive dataset collected over more than two years of continuous overthe-air testing on an experimental O-RAN testbed, comprising over 8,600 automated tests across 69 releases of the OpenAirInterface (OAI) stack. By modeling expected performance and interpreting deviations as software-induced effects, we identify degraded instances attributable to code changes and correlate them with specific change categories. The framework can be integrated into CI/CD/CT pipelines for automated, continuous evaluation of software updates at scale. Index Terms—Open RAN, CI/CD/CT, Performance Analysis, 5G, 6G.
I. Introduction Radio Access Network (RAN) architectures are increasingly software-driven, programmable, and disaggregated, driven by 3GPP NR and O-RAN ALLIANCE specifications [1]–[3]. A growing portion of the network is implemented in software, including virtualized Central Units (CUs) and Distributed Units (DUs), RAN Intelligent Controllers (RICs), and their hosted control and optimization applications, such as rApps, xApps, and dApps. This softwarization enables flexible deployment and creates opportunities for rapid innovation and fine-grained network control. At the same time, tight coupling to a complex software ecosystem introduces new challenges for network design, implementation, operation, and performance. In particular, attributing observed performance variations to individual software updates becomes difficult as updates occur frequently and span independently developed components. This paper is based on material supported in part by the U.S. National Telecommunications and Information Administration (NTIA)’s Public Wireless Supply Chain Innovation Fund (PWSCIF) under Award No. 25-60-IF054 and NSF TI-2449452. ∗ The authors contributed equally.
In response to this software complexity, automated frameworks for Continuous Integration (CI), Continuous Deployment (CD), and Continuous Testing (CT) have been introduced to integrate updates, deploy, test, and validate cellular network software at scale [4], [5]. Such frameworks also enable continuous Over-The-Air (OTA) testing of RAN components under realistic channel conditions using experimental testbeds, and detailed collection of test results over time, e.g., from network logs and traffic-generation tools [6]. The research and prototyping on these frameworks, however, has primarily focused on data collection and performance validation, while providing limited support for systematic performance analysis at scale and for the interpretation of performance variations across tests. Performance analysis is commonly performed using threshold-based monitoring of Key Performance Indicators (KPIs), comparisons against historical KPIs, and postdeployment manual troubleshooting when regressions are suspected [7]. While these approaches are effective for detecting service-level objective failures, they rely on coarse-grained metrics and assume infrequent and limited software updates. However, in O-RAN deployments, multiple software components are updated independently and at a much faster pace (e.g., as fast as on a weekly basis). As a result, manual inspection of test results and simple statistical comparisons across software versions do not scale and provide limited insight into how individual code changes affect network performance. Furthermore, unit tests alone cannot capture the complex cross-component interactions that arise in this disaggregated architecture. RSRP Moving Average Normalized Value
arXiv:2604.23153v1 [cs.NI] 25 Apr 2026
Ravis Shirkhani∗ , Reshma Prasad∗ , Leonardo Bonati, Tommaso Melodia, Michele Polese Institute for Intelligent Networked Systems, Northeastern University, Boston, MA, USA Email: {shirkhani.r, re.prasad, l.bonati, melodia, m.polese}@northeastern.edu
Throughput Efficiency Moving Average
1
0.5
0 0
1,000
2,000
3,000
4,000 5,000 Test Index
6,000
7,000
8,000
Fig. 1: Moving average of downlink RSRP and throughput efficiency across all tests considered. High signal strength does not consistently correspond to high throughput efficiency.
This challenge is further exacerbated by the stochastic nature of wireless systems, where performance is inherently affected by channel conditions, interference, and hardware variability. As a result, tests executed with identical software configurations may yield different performance outcomes and can mask the true impact of software changes.
Analysis and Prediction
Dataset Processing CI/CD/CT Output Data
& KPMs extraction KPIsKPIs & KPMs extraction Application layer metrics
1
Hybrid code change feature extraction Prompt with instructions
Variance Decomposition
Commit Tag Extraction
gNB logs
Commit metadata acquisition
Keyword based categorization
LLM refinement
Merge Request Extraction
Rule-based Pattern Matching
Commit Text Semantic Inference
Change Statistics Extraction
Confidence Scores Computation
Justification Provision
Environmental Model Residuals Analysis Performance Risk Estimation
2 3 4 5
Fig. 2: High-level overview of RANalyzer.
To illustrate this, Figure 1 reports the moving average of throughput efficiency, defined as the ratio of actual to target throughput, and downlink Reference Signal Received Power (RSRP) of daily tests for the open-source OpenAirInterface (OAI) protocol stack over more than 24 months. While higher RSRP generally indicates favorable channel conditions, the figure reveals groups of tests where throughput efficiency remains low despite consistently strong signal strength. This mismatch suggests contribution of other factors on performance variations beyond channel conditions. The growing mismatch between software evolution pace and analysis capabilities, combined with the challenge of distinguishing software-induced performance changes from environmental factors, necessitates automated and systematic performance analysis methodologies that can account for multiple confounding factors. Contributions. To address this gap, in this paper we design, prototype, and evaluate RANalyzer, an automated test analysis solution for software-driven cellular networks. RANalyzer quantifies the performance impact of software updates in ORAN systems, beyond what can be explained by wireless channel conditions. Figure 2 shows a high-level view of RANalyzer. It combines (i) a Large Language Model (LLM)assisted method to automatically categorize software updates by extracting semantic information from commit messages (ii) stochastic analysis that accounts for confounding elements and test conditions. To design RANalyzer, we leverage an extensive continuous OTA testing dataset, collected over more than two years on an experimental O-RAN testbed [6]. It includes end-to-end (E2E) performance metrics and detailed Next Generation Node Base (gNB) logs across 69 software releases of OAI. We show the effectiveness of reasoning capabilities of LLMs in understanding semantics over unstructured text (e.g., commit messages). We then model the expected network performance under given environmental and load conditions, and interpret deviations from this expected behavior as indicators of software-induced performance improvements or degradations. By combining this residual-based analysis with semantic features extracted from software commits, we provide systematic insights into how different classes of code changes affect mobile network performance. Once trained, the proposed framework can be integrated into any CI/CD/CT pipeline to enable automated testing and evaluation of software updates at scale, allowing new software releases to be assessed continuously as part of the workflow. The contributions of the paper are as follows:
We collect and analyze a dataset obtained through continuous OTA testing over more than two years, comprising more than 8600 automated tests with E2E performance metrics and detailed gNB logs across a wide range of OAI protocol stack updates and traffic loads.1 • We introduce a hybrid code-change characterization approach that combines keyword-based pattern matching with LLM-assisted refinement to automatically extract and classify semantic features from software commits, including affected protocol layers and functional components. • We propose an automated methodology to isolate codeinduced performance effects from other sources of variability by explicitly accounting for traffic load and wireless channel conditions. • We demonstrate RANalyzer on real-world 5G RAN tests, identifying degraded instances attributed to code-related issues. We validate the practical utility of our approach through case study analysis correlating degradations with code change categories. The remainder of this paper is organized as follows. In Section II, we review related literature works. In Section III, we provide a high-level description of the workflows on the 5G infrastructure. In Section IV, we provide details on the collected dataset. In Section V, we propose RANalyzer and analyze codeinduced performance changes. Finally, we discuss our results in Section VI, and draw our conclusions in Section VII. •
II. Related Work Detection of performance degradation in software changes has been extensively studied, but existing approaches face significant challenges when applied to wireless systems where performance depends on both code quality and environmental factors. Previous statistical methods, designed for stable environments like data centers, include controlled comparison approaches. The authors of [8] employ Difference-in-Differences (DiD) analysis by comparing performance in servers with old and new software versions simultaneously. Authors of [9] extend this with two-layer DiD to account for seasonality, comparing post-change instances against both pre-change instances and historical baselines. DiD methods [8], [9] require identical environmental conditions across versions, an assumption that fails in wireless cellular testing [10]. Indeed, channel conditions may vary, causing performance differences that would be incorrectly attributed to code changes under DiD analysis. 1 The collected dataset is available at https://github.com/wineslab/RANalyzerDataset.
[11] uses ensemble voting and temporal-spatial correlation to analyze failures over specific components among hundreds of concurrent rollouts in Azure infrastructure, as well as historical human-labeled deployment data to decide whether to proceed with the rollout or not. Beyond statistical methods, machine learning approaches have been explored. The authors of [12] treat defect identification as an anomaly detection task. This approach monitors performance data from multiple source and utilizes a multivariate LSTM to capture defective changes. [13] uses distance-based comparison on KPI timeseries, comparing post-change measurements against historical and pre-change baselines via Siamese LSTM. However, this assumes environmental stability, which is valid for data centers but fails in wireless networks where channel conditions can vary over time. Performance degradation analysis in cellular networks has been explored in related contexts. [10] employs spatial regression to assess configuration changes by comparing study group against control group (changed and unchanged deployments). This approach learns pre-change relationships via linear regression, then forecasts expected post-change performance from control group behavior. This requires similar unchanged deployments as predictors, whereas we use environmental features directly, eliminating control group requirements. The authors of [14], [15] propose the use of Conditional Dependence Coefficient (CODEC), a variance-based statistical method, to identify which configuration parameters affect performance. They build a two-stage model that predicts performance impact, identifying parameters using CODEC scores first, then building predictive models for continuous parameter optimization. Their approach requires continuous, directional variables for predicting change direction, but code changes are categorical. In our framework, we use a similar variance decomposition approach to quantify the contribution of various factors to performance. Network change analysis has been addressed through pre/post statistical comparison methods including Kruskal-Wallis testing [16] and rank-based CUSUM for upgrade detection [17]. These approaches detect performance shifts but cannot distinguish code-induced changes from environmental variations. III. System Description This section provides a high-level description of the system and workflows used for this work. We introduce the automated private 5G infrastructure and the CI/CD/CT workflows used to continuously deploy and test RAN software. We then describe the system design of RANalyzer. A. Automated 5G Infrastructure We leverage a heterogeneous private 5G network testbed designed to support automated, repeatable, and large-scale experimentation. The testbed is deployed on an OpenShift cluster and integrates open-source and commercial components (e.g., core networks, multiple Radio Units (RUs) and Softwaredefined Radios (SDRs)) for a complete E2E 5G network. Network functions are deployed as containerized workloads on general-purpose compute nodes (e.g., the RAN uses Mi-
croway servers with AMD EPYC 7262). OpenShift provides native support for application lifecycle management, container orchestration, and automation primitives that are essential for continuous experimentation. It integrates CI/CD capabilities through cloud-native tools such as Tekton and GitOps-based configuration management via ArgoCD. The RAN workloads deployed on the cluster interface with a grid of SDRs (USRP X410) to enable OTA tests in a scatteringrich indoor space. The User Equipments (UEs) used in the testbed are commercial Sierra Wireless 5G modems connected to a small host compute node, enabling automated control of attachment procedures and traffic generation. Open5Gs is used as core network. Together, the containerized network functions, radio grid, and commercial UEs form the physical and virtual foundation on which the automated CI/CD/CT workflows described in the following subsection are implemented. B. CI/CD/CT Workflow on the Testbed Through the CI/CD/CT capabilities available on the Openshift cluster, we collect our dataset with the 5G-CT framework described in [6]. In this framework, automated pipelines build, deploy, and test an E2E network through series of tasks. Figure 3 shows the high-level components of 5G-CT. The sequence of tasks includes periodic triggers, retrieving the latest OAI software tag from GitLab, checking previously built images in a registry, building a new image when a new tag is detected, deploying the workload on the mentioned Microway, running continuous tests OTA with constant bandwidth of 60MHz over carrier frequency 3.629 GHz in band n78, and finally storing the collected artifacts in a database. [18] provides the means and configurations to have this deployment and testing on the Openshift cluster. This design allows the gNB software to be continuously updated while keeping test execution repeatable across software versions. Using this methodology, we collected a dataset that spans 69 releases of OAI from July 2023 to December 2025, with more than 8600 automated tests. Cron job trigger
Build Image
Get image
Query image registry
New Git Commit
Deploy gNB Pod
Build latest tag code
Data Collection & Analysis Store CI/CD/CT
Get available resources (e.g., node, radio, spectrum)
Microway Worker Nodes
Automated Over-the-Air Test
output data
RANalyzer
Dell Control Nodes
Data Processing Analysis and Prediction
x410 USRPs
Sierra 5G Modem
Fig. 3: High-level overview of CI/CD and RANalyzer workflow on the 5G infrastructure.
Each E2E test runs downlink traffic using iPerf at multiple target data rates (e.g., 10, 20, and 30 Mbps, as well as higher rates such as 80 Mbps). At the end of each test, gNB logs, iPerf results, and UE metadata (e.g., serial number, hostname) are stored in a database for further analysis. C. Introduction to RANalyzer RANalyzer is an analysis layer designed to operate alongside the CI/CD/CT workflow. As illustrated in Figure 3, RANalyzer
interfaces directly with the automated pipeline and consumes the artifacts produced and stored by the tests. The framework augments the existing testing infrastructure without modifying deployment or execution procedures, but by providing key insights into the stack performance in the tests. At a high level, RANalyzer aggregates results across software versions and test conditions, processes raw artifacts (test results, commit messages, stack logs) into structured inputs, and supports automated analysis of performance behavior under continuous updates by series of steps spanning through mathematical variance analysis and training ML models. Once trained, RANalyzer can be integrated into CI/CD/CT pipelines to enable automated analysis of test results, e.g., anytime there is a software update. Next, we describe the dataset built from these OTA tests, as its structure is essential for understanding the subsequent data processing and analysis steps. IV. RANalyzer Dataset This section describes the RANalyzer dataset, comprising two years of results of continuous and automated OTA tests. We discuss how we transform the raw output of the OTA tests into an analysis-ready dataset and analyze exemplary metrics. A. CI/CD/CT Output Data Organization Data is organized into folders based on the date and time the automated pipeline ran each test (triggered via cron jobs on OpenShift). At the top level, each day corresponds to a single directory with name in yyyymmdd format (e.g., 20250913/). Within each day, multiple tests are executed, each one stored in a separate subdirectory based on its start time in the hhmmss format (e.g., 040124/). Each test directory contains: (i) iPerf3 results saved as csv files with E2E performance metrics (throughput, packet loss, and jitter); and (ii) OAI gNB runtime logs from which Key Performance Measurements (KPMs) and protocollevel events can be extracted. B. KPI/KPM and Dataset Consolidation To facilitate the parsing of the dataset for further analysis, we convert the heterogeneous result files from the OTA tests (e.g., log files and iPerf results) into a unified set of numerical features aligned at the test level. E2E KPIs from iPerf. We parse the iPerf output to extract application-level metrics including throughput, packet loss, jitter, and total bytes and packets. Because tests are executed with different target rates (ranging from 10 Mbps to 80 Mbps), we also compute normalized metrics that enable comparison across loads, such as throughput efficiency: 𝜂test = 𝑇test /𝑇target
(1)
where 𝑇test denotes the average measured throughput and 𝑇target the requested data rate. Low-level KPM and event metrics from gNB logs. In addition to E2E KPIs, we extract radio and protocol indicators from the gNB logs. These include numerical KPMs reported across multiple layers of the RAN stack, such as physicallayer measurements (e.g., RSRP, SINR), link-layer reliability
indicators (e.g., BLER and HARQ retransmission statistics), and scheduling- and control-related metrics from higher layers (e.g., CQI reports). In parallel, we extract event-based features corresponding to protocol and system events across the PHY, MAC, and RRC layers, including connection setup and release markers, scheduling anomalies, warnings, and error patterns (e.g., number of active Packet Data Unit (PDU) sessions, failed msg2 in the Random Access (RA) procedure). Together, these numerical and event-based log features provide a comprehensive characterization of the radio environment and protocol behavior during each test. Dataset assembly and identifiers. For the unified dataset construction, the extracted KPIs/KPMs from each test are aggregated into a single vector, indexed by a unique test identifier (timestamp-based folder name) and linked to the software revision identifier (commit hash). Over 80 features are extracted in this step. These features are combined with code-related features, extracted from the corresponding commit hash, as described in Section V-A. While this Section focuses on offline post-processing of all the collected CI/CD/CT test outputs, the same aggregation and feature-extraction can be executed automatically at the end of an individual test run. Specifically, a dedicated Python dataprocessing component runs inside an OpenShift pod upon test completion to collect the generated logs and measurement artifacts at the last step before RANalyzer (Figure 3). The feature-extraction pipeline and the rest of the RANalyzer workflow can be executed as subsequent tasks within the same pod, enabling a complete E2E execution and analysis workflow. C. Dataset Analysis We perform an analysis on different KPIs and KPMs of our dataset to show how different factors influence system performance. In particular, we show that performance is strongly affected by factors such as channel conditions and traffic load. To understand the impact of traffic load, we separately analyze tests with high load (≥50 Mbps) and low load (<50 Mbps).2 Figure 4a shows the throughput efficiency across different Git tags (i.e., a new release) for low-load (blue curve) and high-load (red curve) tests. The marker size is proportional to the number of tests executed for each Git tag. While performance fluctuates across Git tags, the overall trend shows that higher-load tests tend to achieve lower throughput efficiency. This observation further highlights the importance of accounting for traffic load in the performance analysis. Figure 4b shows the average uplink BLER, extracted from the gNB logs. For this metric, we observe an overall increase toward the more recent commits compared to earlier ones. The values are similar for both load levels, which suggests that the observed changes are likely driven by environmental factors, such as channel conditions, which affect all tests in a similar way. Another interesting feature is the number of first round Hybrid Automatic Repeat reQuest (HARQ) retransmissions, shown in Figure 4c. We observe that the values increase over time but remain consistently higher 2 High or low load depend on the capacity of the infrastructure on which the tests are executed and test bandwidth.
Git commit date
6 3 1.5 8 0/ 09 25
/1
9
/0 /0 9 25
0 /1
/0 6 25
/0
11 3/ /0
/0 4 25
24
8
16
/2
0/ /1 24
05 2/
/0 7 24
7/
/0
/0
1/ /1 23
Git hash date
Git commit date
(a) Throughput efficiency under low and high load
28
0
9
08
/0 /1 0
25
9 /0
9/ /0 25
10 4/
/0 6
/0 25
25
6
11 3/ /0
24
8
/1 24 /1 0
/0 5
/2
24 /0 7
/0 2
/2
24
/0 7
18 1/ /1 23
8
0
High Load
4.5
24
0.05
Low Load
·104
7.5
18
0.1
09
08
0/ /1 25
6/
09
25
/0
9/
10
/0 25
11
4/ /0
25
16
24
/0
3/
28
0/ /1 24
05
24 /0
7/
28
2/ /0
7/ /0 24
23
24
18
40
0.15
24
Average UL BLER
60
High Load
DLSCH Rounds 1
Low Load
24
High Load
80
/1 1/
Throughput Efficiency (%)
Low Load
100
(b) UL BLER under low and high load
(c) DL-SCH transport block retransmissions
Fig. 4: Metrics across Git commits for low- and high-load test cases.
under high load, as expected from aggressive scheduling and higher modulation schemes under traffic pressure. V. RANalyzer Automated Workflow Analysis in the previous sections showed that performance variations in wireless systems arise from multiple interacting factors, including environmental conditions (channel quality and interference), workload characteristics (traffic load), and code modifications (protocol-layer changes). We develop a multistage framework to identify and attribute performance changes to specific causes. As shown at a high-level in Figure 2, the framework consists of five stages: (i) automated hybrid code change categorization, (ii) variance decomposition, (iii) environment modeling, (iv) residual-based performance analysis, and (v) performance risk estimation. In this section, we discuss each of these stages in detail. A. Automated Hybrid Code Change Categorization In the first step of RANalyzer, we automatically extract a structured feature vector from each OAI commit to relate performance variations to software evolution. For a given commit 𝑐, we retrieve its message from GitLab, along with basic code-churn (e.g., files changed and lines added/deleted), and aggregate this information into a single text record 𝑚 𝑐 . We then infer protocol layers and functional components that are impacted by the change and assign a high-level change type (e.g., bug fix, optimization, feature, refactoring). We propose a hybrid feature extraction pipeline that combines fast keyword-based categorization with selective LLM-based semantic refinement, enabling scalable and accurate processing of large commit histories with ambiguous terminology. Let L denote the set of protocol layers (PHY, MAC, RLC, PDCP, RRC, NAS, NGAP, F1AP, E1AP) and C the set of functional components (memory, threading, radio, scheduler, timer, queue). We define Γ = L ∪ C, with 𝑘 = |Γ|. For a commit 𝑐, the affected categories Γ𝑐 ⊆ Γ are encoded as a binary feature vector f𝑐 ∈ {0, 1} 𝑘 , where f𝑐 [𝑖] = 1 if 𝑐 affects category 𝑖, and 0 otherwise (e.g., for Γ𝑐 = {PHY, MAC}, f𝑐 = [1, 1, 0, . . . , 0]). This vector is populated via the following automated steps: 1) Keyword-Based Categorization: The first stage of the hybrid pipeline performs fast, rule-based categorization using domain-specific keyword sets organized by protocol layer and functional component. The keyword sets were constructed through expert-guided curation based on RAN protocol specifications and the terminology of the OAI codebase. The input
to this stage is the aggregated commit text 𝑚 𝑐 , constructed from the commit title, description, and merge-request references extracted from the GitLab commit page. For each commit, we apply case-insensitive substring matching against predefined keyword sets associated with each protocol layer and component. Each set contains characteristic terms commonly used in that part of the RAN stack (e.g., L1 and NR for PHY, MAC and LogicalChannelConfig for MAC, and RRC for the RRC layer). Keywords are labeled as strong, medium, or weak, reflecting their specificity (e.g., the term NR_MAC is strong, while scheduler is weak as it may appear across multiple layers), and contribute weighted evidence to the corresponding category. The weighted keyword score for a given category 𝑗 is computed as ∑︁ 𝑤 𝑖 · 1match (𝑖), (2) 𝑖∈K𝑗
where K 𝑗 denotes the keyword set for category 𝑗 ∈ Γ, 𝑤 𝑖 ∈ {2, 1, 0.5} corresponding to strong, medium, and weak keywords, and 1match (𝑖) indicates whether keyword 𝑖 appears in 𝑚 𝑐 . A category is marked as affected if its score exceeds a category-specific threshold 𝜃 𝑗 . This procedure yields a binary feature vector indicating the set of affected layers and components, as well as aggregate statistics such as the total number of detected layers 𝐿 and components 𝐶. In addition, the keyword-based stage assigns an extraction confidence label using a deterministic, rule-based logic. We denote this dependency abstractly as Confidence ≜ R (𝐿, 𝐶, 𝐾, 𝑆),
(3)
where R (·) represents a threshold-based decision function. The confidence depends on four quantities: the number of detected protocol layers 𝐿 and functional components 𝐶, the total amount of keyword evidence 𝐾, and the number of matched strong keywords 𝑆. Commits with large 𝐾 and 𝑆 concentrated within a small 𝐿 and 𝐶 are labeled as high confidence, while commits exhibiting weak or diffuse evidence across many categories are labeled as medium or low confidence. This confidence label is rule-based rather than probabilistic and is used to gate subsequent semantic LLM-refinement. Figure 5 illustrates representative commit messages and their matched keyword patterns, highlighting how layer-specific terminology enables rapid identification of affected parts of the RAN stack; for example, the presence of the strong MAC
2557: fix duplicate call of RCconfig_NR_L1 2556: Support RC SM aperiodic subscription for " UE RRC State Change " 2550: use pointer to structure instead of module_id inside MAC 2548: NR UE MSG3 buffer 2495: Sidelink configuration passed from RRC->MAC 2490: reworking configuration of LogicalChannelConfig at MAC UE 2220: L1 tx thread 2557 2556 2550 2548 2495 2490 2220
-> [ PHY ] (" L1 " ," NR ") -> [ RRC ] (" RRC ") -> [ MAC ] (" MAC ") -> [ MAC ] (" MAC " ," MSG3 " ," buffer ") -> [ RRC , MAC ] (" RRC " ," MAC ") -> [ MAC ] (" MAC " ," LogicalChannelConfig ") -> [ PHY ] (" L1 " ," tx " ," thread ")
Fig. 5: Keyword-based categorization examples.
2) LLM-Based Refinement: The LLM is provided with a fixed, human-designed instruction prompt together with the commit text and the results of the keyword-based classification. It is instructed to perform structured validation by (i) confirming or rejecting candidate layers and components to eliminate false positives from keyword matching, (ii) enforcing an upper bound of four affected layers to prevent over-classification, and (iii) identifying a single primary change type. The model returns a structured output containing the refined layer set, component set, primary change category, and a brief justification. The output of this refinement stage is the finalized set of affected protocol layers and functional components, Γ𝑐 , along with an associated confidence indicator. Invoking the LLM only for commits with ambiguous or low-confidence keyword evidence preserves scalability while improving semantic accuracy. The final vector f𝑐 produced with 34 features for each commit includes binary layer and component indicators, normalized change-type scores, aggregate counts (number of affected layers and components), code-churn metrics, and a composite change-complexity score that summarizes the overall scope and structural impact of the modification. B. Variance Decomposition In the second step of RANalyzer, we quantify dependencies of extracted features from Section IV-B alongside code change features from Section V-A using variance decomposition. We want to determine the relative contributions of environmental factors versus code modifications to performance variation. This analysis enables us to quantify the marginal effect of each feature on a given performance metric and to track how these contributions evolve over time across successive software updates and in relation to one another. Variance decomposition quantifies how much additional variance in a target metric 𝑌 is explained by a set of predictors 𝑃, while controlling for another set of variables 𝑄. This approach follows principles commonly used in ANOVA and regression
analysis, and answers the question: “how much more of the performance variation can be explained by adding a specific factor, after accounting for other confounding variables?” In our case, we separately evaluate the contribution of channel conditions, traffic load, and code changes while controlling for the remaining factors. To enable statistical grouping and conditional expectation computation, continuous KPI/KPM metrics are discretized into ordered categories. The variance decomposition score is computed using Equation (4). By the law of total variance, 𝑉 𝑎𝑟 (𝐸 [𝑌 |𝑋]) represents the portion of variance in 𝑌 that can be explained by 𝑋. The proposed metric captures the additional variance explained by predictors 𝑃 beyond what is explained by the variables 𝑄, normalized by the total variance of 𝑌 . Var(E[𝑌 | 𝑃, 𝑄]) − Var(E[𝑌 | 𝑄]) (4) Var(𝑌 ) Figure 6 presents variance decomposition results for throughput efficiency, packet loss, and jitter, showing the proportion of variance explained by each factor while controlling for the others. As expected, channel-related features and traffic load explain a significant portion of the performance variation. The impact of channel conditions is most pronounced for packet loss, accounting for approximately 40% of the variance, which is more than double its effect on throughput efficiency. In contrast, traffic load has a stronger influence on jitter, explaining about 27% of its variance, compared to a much smaller contribution for packet loss. We also observe that code changes exhibit significant and consistent contribution to performance variation across all metrics. We proceed to residual-based analysis after confirming the contribution of code-change on performance. Periodic variance decomposition checks can track temporal shifts in the balance between environmental and code-induced effects as the code evolves. 𝐶𝑣𝑎𝑟 (𝑌 , 𝑃|𝑄) =
Channel Score 𝐶𝑣𝑎𝑟
keyword MSG3 and MAC yields a weighted score of 2.0, exceeding the MAC detection threshold and triggering layer identification. Keyword-based categorization relies on lexical matching and does not capture semantic context, which can lead to misclassification. For example, while terms such as tx and rx are ambiguous in isolation, they can be disambiguated when combined with contextual keywords like L1 or MAC. To address this limitation, commits that produce low or medium confidence scores are passed to an LLM for semantic refinement. We use Claude 4.5 Sonnet as the LLM for this stage.
0.6
Load
Code
54.8%
52.6%
47.1% 40.8%
0.4
28.6% 27.9% 21.8%
0.2
13.1%
17.3%
0 Jitter
Packet Loss Throughput Efficiency Performance Metric
Fig. 6: Variance decomposition for affecting factors.
C. Environment Model To isolate code-induced performance variations, we first establish an environmental baseline predicting expected performance under given environmental conditions. This model effectively answers the question: “what throughput efficiency should we ideally achieve given the network conditions and traffic load?” To predict baseline throughput from environmental conditions, we evaluate several machine learning models including Random Forest, XGBoost, Decision Tree, Linear Regression, and K-Nearest Neighbors. The input features to the model include channel quality indicators like Signal-to-Noise-Ratio
specific protocol layers by mapping the residual value 𝜌 with the set of modified layers, establishing a direct link between code changes and performance impact.
100 Predicted 𝜂𝑒𝑥 𝑝 (%)
Predicted 𝜂𝑒𝑥 𝑝 (%)
100 80 60 40 20 0
80 60 40
E. Performance Risk Estimation
20
Having labeled training instances based on residual analysis, we train a classifier to predict the probability of code-induced degradation for new commits. This can enable proactive risk assessment in continuous integration workflows. We leverage a gradient boosting classifier (LightGBM) that uses both environmental features (SNR, BLER, target rate) and code changes (affected categories f𝑐 ) as features. The classifier is trained with 400 estimators, maximum depth 4, and learning rate 0.1 to predict performance degradation. To address class imbalance in our performance dataset, we employ Synthetic Minority Over-sampling Technique (SMOTE) [19]. This generates synthetic samples for the degraded performance class, which is the minority to balance the class distribution. SMOTE creates new minority class instances by interpolating between existing samples in the feature space, which helps prevent overfitting. We also employ balanced class weighting to assign higher penalties to the minority class during training. The model outputs a degradation probability score for each test instance, enabling proactive identification of high-risk commits prior to deployment.
0 0
20 40 60 80 100 Actual 𝜂𝑡𝑒𝑠𝑡 (%) (a) Random Forest
0
20 40 60 80 100 Actual 𝜂𝑡𝑒𝑠𝑡 (%) (b) XGBoost
Fig. 7: Predicted vs. actual throughput efficiency for the best learning models.
(SNR), RSRP, link layer performance metrics like HARQ retransmission statistics, traffic and load characteristics including target data rate. We show the results of two best environment models and their quantitative performance metrics in Figure 7 and Table I. The figure reports how much the predicted values (y axis) deviate from the actual values (x axis). Random Forest achieves the best overall performance with 𝑅 2 = 0.767, Root Mean Squared Error (RMSE) = 5.626 Mbps, and Mean Absolute Error (MAE) = 2.17 Mbps. We configure the Random Forest regressor with an ensemble of 100 decision trees, with maximum depth of 8 levels to balance expressiveness with generalization. TABLE I: Environmental model performance for the best learning models. Model
Test R2
MAE
RMSE
Random Forest XGBoost
0.767 0.725
2.170 2.418
5.623 6.105
D. Residual-Based Performance Analysis Once we have the baseline estimation of the throughput performance that can be obtained based on the current conditions, we can isolate the anomalies in performance by calculating residuals. Formally, the residual of a test instance is defined as the performance ratio: 𝜌 = 𝜂𝑡𝑒𝑠𝑡 /𝜂𝑒𝑥 𝑝
VI. Evaluation and Results In this section, we discuss RANalyzer results related to LLMrefinement on a commit, degradation classification, the analysis on the code change impact, and the results on performance risk estimation.
(5)
where 𝜂𝑡𝑒𝑠𝑡 is the throughput efficiency observed in the test instance and 𝜂𝑒𝑥 𝑝 is the expected throughput efficiency given the environment. With residual values, we can label the observed performance as an anomaly or not. A test instance is labeled as degraded if two conditions are met: (i) the residual falls below a degradation threshold 𝜏𝜌 (𝜌 < 𝜏𝜌 ), indicating underperformance, while (ii) the expected throughput exceeds a minimum threshold 𝜏𝑒𝑥 𝑝 (𝜂𝑒𝑥 𝑝 ≥ 𝜏𝑒𝑥 𝑝 ), confirming that environment conditions were sufficiently favorable to support normal operations. The second criterion is essential because wireless networks exhibit inherent variability. In conditions where 𝜂𝑒𝑥 𝑝 < 𝜏𝑒𝑥 𝑝 , environmental factors dominate performance variation, making it unreliable to attribute degradations to code changes. We select 𝜏𝑒𝑥 𝑝 by analyzing the trade-off between test coverage and detection reliability, ensuring we capture degradations in realistic operating conditions. For each instance labeled as degraded, we attribute the performance deviation to
A. LLM-Refinement This section illustrates the necessity of LLM-based refinement using a representative example. Figure 8 shows commit 2024.w16, where the keyword-based categorization stage failed to identify any protocol layers due to insufficient keywords, detecting only the memory component based on the term buffer and resulting in a low confidence score. The LLM refinement was therefore triggered and was able to infer the affected layers by interpreting the semantic context of the commit messages, associating “UE UL payload” modifications with the MAC layer and the “remove hardcoding of 5G-S-TMSI on nrUE” with the NAS layer. B. Residual Analysis and Degradation Classification We first examine the residual distribution to validate our environmental model. Figure 9 shows the distribution of residuals 𝜌 across all test instances. The majority of residuals cluster around 𝜌 ≈ 1.0 with mean = 0.998 and median = 1.00, indicating performance aligned with environmental * * * *
!2680 !2681 !2685 !2690
CI : Modification of log collection in UndeployObject () remove a useless copy and specific buffer for all UE UL payload Clang : make executable run , fix clang warnings , fix memsan warnings Remove hardcoding of 5G -S - TMSI on nrUE
Fig. 8: Commit example which triggered LLM refinement.
Frequency
Threshold = 0.90 (4.2% below)
1,500
20
1,000
10
500
0 0.2
0.4
0.6
Expected = 1.00 (27.7% below)
0.8 0.9
0 0.2
0.4
0.6
0.8 0.9
1
1.2
1.4
Throughput Efficiency Residuals
Fig. 9: Throughput efficiency residuals distribution.
Actual Throughput
Expected (Environment)
Residual Efficiency
100
2
50
1
1.5 0.5
0
0 206c766
52d58ef
69d1d5f
Residual Efficiency
Throughput Efficiency (%)
Next, we select the expected throughput threshold 𝜏𝑒𝑥 𝑝 that determines which test instances have adequate environmental conditions for reliable code attribution. Higher thresholds (e.g., 80%) would miss degradations in moderate channel conditions, while lower ones (e.g., 40%) would produce false positives from environmental variability. We analyze the trade-off between test coverage and detection reliability across thresholds from 40% to 90%. Based on this analysis, we select 𝜏𝑒𝑥 𝑝 = 60%, which provides optimal balance, retaining 99.5% of test instances with standard deviation of 0.068 enabling reliable analysis. To demonstrate how the residual-based framework distinguishes code-based degradations from environmental effects, in Figure 10, we present three representative evaluation cases.
a3b9db2
Commit Hash
Fig. 10: Example cases.
Case 1 - Normal operation: In this case, the test achieves similar performance as the baseline prediction. The first two commits of Figure 10 are examples of this case, where the residual 𝜌 is 1 and 0.95, respectively, and thus falls within normal environmental fluctuation. This is treated as normal operation. Case 2 - Environmental limitation: In this case, baseline and test throughput efficiency results are much lower than 60%, indicating degraded environmental conditions. Under such poor conditions, 𝜌 loses reliability and we cannot confidently correlate under-performance to code changes because the environment itself fundamentally limits the achievable throughput efficiency. Thus, our framework prevents degradation flagging in these cases. This case is observed for commit 69d1d5f. We also observe that when the commit is tested under normal environmental conditions, no performance degradation occurs.
Case 3 - Code-induced degradation: In this case, the baseline throughput efficiency is higher but the test result is much lower. A low residual value 𝜌, combined with favorable baseline, triggers degradation flagging. This is observed for commit a3b9db2, which upgraded NVIDIA Aerial (PHY acceleration) from version 24-3 to 25-1. A commit branching from a3b9db2 fixed PHY bugs, where the SINR was wrongly set to 0 for SISO and the CQI bit length was incorrectly calculated. These bugs were likely regressions from the Aerial 25-1 version upgrade. We compare our residual-based degradation detection against a temporal correlation baseline adapted from [11]. This approach uses multi-window aggregation with exponential weights to prioritize faults occurring shortly after code deployment, based on the intuition that temporal proximity indicates causal relationships. Figure 11 shows flagging patterns using this approach across a sample set of test instances for target throughput rate of 30 Mbps. Our proposed method detects degradation when there is a series of degradation with expected throughput > 60%. The temporal-correlation-based method, Throughput Efficiency (%)
expectations. We choose 𝜏𝜌 = 0.9 as threshold for underperformance and 4.2% of instances exhibit 𝜌 < 0.9. A small Welch’s 𝑡-test 𝑝-value (< 0.001) and Cohen’s 𝑑 of 2.45 suggest that the two groups are distinct and validate the choice of the 0.9 threshold. The degraded group, with mean around 0.7 and standard deviation 0.159, has a three times higher variability than the group of residuals above the threshold with standard deviation 0.053.
Expected
Actual RANalyzer
Temporal Correlation [11]
100 80 60 40 20 300
350
400
450
500
Commit Index
Fig. 11: Baseline comparison.
however, misses certain degradations that happens with a longer delay after the code change, as their temporal distance reduces their contribution under the exponential weighting. Our residual analysis explicitly models environmental baselines, achieving better precision by only flagging under-performance when conditions were favorable. C. Residual-Based Analysis of Code Change Impact We analyze the relationship between code changes and performance degradation leveraging residuals derived from the environment model. Table II presents performance degradation statistics for each protocol layer, showing the distribution of residual values for commits modifying each layer. We observe that PDCP modifications impact degradation the most, with mean residual of 0.70, followed by RRC and NGAP. PDCP shows particularly high variance (standard deviation of 0.28), indicating inconsistent impact on performance. The large median-mean gap for this layer shows that the distribution is skewed by a subset of commits with extreme degradation. Meanwhile, we note that MAC changes have the highest number of degraded cases. One reason may be the high frequency of MAC modifications in the development process. NAS modifications, instead, show less severe degradation with lower variation. This layer-specific analysis enables risk-based code reviews. For example, commits modifying PDCP, RRC or NGAP may require enhanced scrutiny due to their larger average impact, while MAC changes require attention primarily due to high frequency despite average degradation. PHY changes, despite
TABLE II: Performance impact by modified protocol layer. Layer
Degraded Cases
Mean 𝜌
Median 𝜌
Std Dev
PDCP RRC NGAP MAC PHY NAS
14 87 87 101 87 40
0.70 0.72 0.72 0.74 0.74 0.78
0.83 0.73 0.73 0.75 0.74 0.84
0.28 0.15 0.15 0.15 0.12 0.19
moderate severity, are more predictable and may be easier to validate through targeted testing. D. Results on Performance Risk Estimation We now evaluate the result of the performance risk estimation. The classifier that predicts code-induced degradation achieves excellent performance on majority class with precision = 0.98, recall = 0.94, F1 = 0.96. This indicates reliable identification of commits that do not degrade performance. With only 18 instances in the test set, the classifier achieves recall = 0.61 (detecting 11 out of 18 true degradations) but precision = 0.26 (26% of degradation predictions are correct). This is expected given the challenges of severe class imbalance, high variability in performance outcomes, and prediction from commit metadata and environmental context without using actual test performance measurements. The low precision indicates false alarms, classifying benign commits as degrading, though the moderate recall demonstrates the ability to identify genuine degradation cases. The recall-precision trade-off is acceptable for CI/CD where missing a degradation (false negative) is more costly than investigating a false alarm. VII. Conclusions In this paper, we proposed RANalyzer to address the challenge of attributing performance variations in software-defined cellular networks. Here, observed metrics reflect the joint impact of software changes and environmental conditions. Our approach combined (i) hybrid software release categorization, using keyword pattern matching with selective LLM refinement to construct a code-change feature vector, with (ii) environmentcontrolled residual analysis, which models baseline performance from channel and traffic conditions and isolates code-induced performance deviations. Evaluation on more than two years of OAI 5G CI/CD/CT data comprising 69 software releases and more than 8600 tests demonstrated the RANalyzer effectiveness. Our approach identified code-induced degradations, distinguishing between tests that underperform due to channel conditions or software regression. Case studies validated practical utility across bug fixes, optimizations, and refactoring, with layer-specific attribution accelerating root cause identification. In future extensions, RANalyzer can evolve through continuous data accumulation enabling model refinement, fine-grained event metrics extraction, and multi-level severity classification for nuanced risk assessment.
References [1] A. S. Abdalla, P. S. Upadhyaya, V. K. Shah, and V. Marojevic, “Toward Next Generation Open Radio Access Networks: What O-RAN Can and Cannot Do!” IEEE Network, vol. 36, no. 6, pp. 206–213, 2022. [2] M. Polese, L. Bonati, S. D’Oro, S. Basagni, and T. Melodia, “Understanding O-RAN: Architecture, Interfaces, Algorithms, Security, and Research Challenges,” IEEE Communications Surveys & Tutorials, vol. 25, pp. 1376–1411, 2023. [3] B. Agarwal, R. Irmer, D. Lister, and G.-M. Muntean, “Open RAN for 6G Networks: Architecture, use cases, and open issues,” IEEE Communications Surveys & Tutorials, 2025. [4] J. Luis Herrera, S. Montebugnoli, D. Scotece, L. Foschini, and P. Bellavista, “A Tutorial on O-RAN Deployment Solutions for 5G: From Simulation to Emulated and Real Testbeds,” IEEE Communications Surveys & Tutorials, vol. 28, pp. 1709–1748, 2026. [5] S. Motamary, “A Deep Dive into CI/CD Pipelines Tailored for Telecom,” American Journal of Analytics and Artificial Intelligence, vol. 1, no. 1, 2023. [6] L. Bonati, M. Polese, S. D’Oro, P. B. del Prever, and T. Melodia, “5G-CT: Automated Deployment and Over-the-Air Testing of End-to-End Open Radio Access Networks,” IEEE Communications Magazine, 2024. [7] Y. Zelalem Jembre, W.-y. Jung, M. Attique, R. Paul, and B. Kim, “Mobile Broadband Performance Evaluation: Analysis of National Reports,” Electronics, vol. 11, no. 3, p. 485, 2022. [8] S. Zhang, Y. Liu, D. Pei, Y. Chen, X. Qu, S. Tao, and Z. Zang, “Rapid and Robust Impact Assessment of Software Changes,” in Proc. ACM CoNEXT, 2015, pp. 1–13. [9] G. Yu, P. Chen, Z. He, Q. Yan, Y. Luo, F. Li, and Z. Zheng, “ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems,” Proc. ACM Softw. Eng., vol. 1, no. FSE, pp. 24–46, 2024. [10] A. Mahimkar, Z. Ge, J. Yates, C. Hristov, V. Cordaro, S. Smith, J. Xu, and M. Stockert, “Robust Assessment of Changes in Cellular Networks,” in Proc. ACM CoNEXT, 2013, pp. 175–186. [11] Z. Li, Q. Cheng, K. Hsieh, Y. Dang, P. Huang, P. Singh, X. Yang, Q. Lin, Y. Wu, and S. Levy, “Gandalf: An Intelligent End-to-End Analytics Service for Safe Deployment,” in Proc. USENIX NSDI, 2020, pp. 389–402. [12] N. Zhao, J. Chen, Z. Yu, H. Wang, J. Li, B. Qiu, H. Xu, W. Zhang, K. Sui, and D. Pei, “Identifying Bad Software Changes via Multimodal Anomaly Detection,” in Proc. ACM ESEC/FSE, 2021, pp. 527–539. [13] X. Wang, K. Yin, Q. Ouyang, X. Wen, S. Zhang, W. Zhang, L. Cao, J. Han, X. Jin, and D. Pei, “Identifying Erroneous Software Changes through SelfSupervised Contrastive Learning,” in IEEE ISSRE, 2022, pp. 366–377. [14] K. Patel, C. Ge, A. Mahimkar, S. Shakkottai, and Y. Shaqalle, “CIPAT: Latent-Resilient Toolkit for Performance Impact Prediction due to Configuration Tuning,” in Proc. ACM MobiCom, 2024, pp. 2377–2382. [15] ——, “Predicting the Performance of Cellular Networks: A LatentResilient Approach,” in Proc. ACM MobiCom, 2024, pp. 1581–1583. [16] A. Mahimkar, Z. Ge, X. Liu, Y. Shaqalle, Y. Xiang, J. Yates, S. Pathak, and R. Reichel, “Aurora: Conformity-Based Configuration Recommendation to Improve LTE/5G Service,” in Proc. ACM IMC, 2022, pp. 83–97. [17] A. A. Mahimkar, H. H. Song, Z. Ge, A. Shaikh, J. Wang, J. Yates, Y. Zhang, and J. Emmons, “Detecting the Performance Impact of Upgrades in Large Operational Networks,” in Proc. ACM SIGCOMM, 2010, pp. 303–314. [18] S. Maxenti, R. Shirkhani, M. Elkael, L. Bonati, S. D’Oro, T. Melodia, and M. Polese, “AutoRAN: Automated and Zero-Touch Open RAN Systems,” IEEE Trans. on Mobile Comput. (to appear), 2026. [Online]. Available: arxiv.org/abs/2504.11233 [19] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-Sampling Technique,” J. Artif. Intell. Res., vol. 16, pp. 321–357, 2002.