arXiv:2604.13462v1 [cs.SE] 15 Apr 2026
Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment Eileen Kapel
Jan Lennartz
Luis Cruz
[email protected] ING Bank Amsterdam, The Netherlands
ING Bank Brussels, Belgium [email protected]
Delft University of Technology Delft, The Netherlands [email protected]
Diomidis Spinellis
Arie van Deursen
Delft University of Technology Delft, The Netherlands [email protected]
Delft University of Technology Delft, The Netherlands [email protected]
Abstract Effective IT change management is important for businesses that depend on software and services, particularly in highly regulated sectors such as finance, where operational reliability, auditability, and explainability are essential. A significant portion of IT incidents are caused by changes, making it important to identify high-risk changes before deployment. This study presents a predictive incident risk scoring approach at a large international bank. The approach supports engineers during the assessment and planning phases of change deployments by predicting the potential of inducing incidents. To satisfy regulatory constraints, we built the model with auditability and explainability in mind, applying SHAP values to provide feature-level insights and ensure decisions are traceable and transparent. Using a one-year real-world dataset, we compare the existing rule-based process with three machine learning models: HGBC, LightGBM, and XGBoost. LightGBM achieved the best performance, particularly when enriched with aggregated team metrics that capture organisational context. Our results show that data-driven, interpretable models can outperform rule-based approaches while meeting compliance needs, enabling proactive risk mitigation and more reliable IT operations.
CCS Concepts • Software and its engineering → Software reliability; Software post-development issues; • Computing methodologies → Natural language processing; Supervised learning by classification; Classification and regression trees; • Social and professional topics → Governmental regulations.
Keywords change management, incident management, reliability, predictive models, classification
This work is licensed under a Creative Commons Attribution 4.0 International License. ICSE-SEIP ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2426-8/2026/04 https://doi.org/10.1145/3786583.3786876
ACM Reference Format: Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, and Arie van Deursen. 2026. Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment. In 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEIP ’26), April 12–18, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 12 pages. https: //doi.org/10.1145/3786583.3786876
1
Introduction
At present, major industries such as finance, healthcare, and retail are increasingly reliant on software and related services. However, the reliability and availability of these services are often compromised by incidents: unplanned interruptions to a service or reductions in service quality [19]. A significant portion of incidents stem from IT changes, which involve additions, modifications, or deletions to existing IT applications [19]. As reported by Google [7], roughly 70% of outages in its live systems are due to such changes. To monitor these change-induced incidents, we often refer to the change failure rate, which measures the percentage of deployments causing a failure in production [14]. It is important to ensure service reliability and availability, because ineffective incident management can lead to customer dissatisfaction, financial losses, and reputational damage. For example, infrastructure failures may cost $100,000 per hour, while critical application failures can reach $1 million per hour [13]. To mitigate these risks and prevent severe business disruptions, financial institutions in Europe are required to follow rigorous change and incident management processes, ensuring that all changes are carefully assessed and auditable [1, 4]. Identifying potential incident-inducing changes before deployment can help prevent incidents and ensure service reliability [15]. Engineers prefer proactive measures [21], such as enhancing testing and issue detection early in the development process, a practice known as “Shift Left”. This approach improves service reliability by detecting incident-causing bugs before they reach production [15]. Incident prevention entails actively identifying potential failures and forecasting severe outages by using historical data and analytics to predict and mitigate risks [36]. Predictive incident management is enabled by artificial intelligence for IT Operations (AIOps), which uses machine learning (ML) and big data mining to forecast potential system malfunctions by analysing historical patterns [36]. Ensuring trust, involving humans in the decision-making process, and providing interpretability and explainability of AIOps
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, and Arie van Deursen
solutions is essential for gaining confidence from industry practitioners [36]. Interpretable models are often favoured over black-box models, even when slightly less accurate, as they offer great transparency and support informed decision-making. Deploying predictive incident management in a highly regulated financial environment introduces unique challenges. Change and incident management processes in financial institutions are tightly governed by regulatory standards that demand compliance, auditability, and traceability [1, 4, 10]. These constraints limit the use of black-box models like deep neural networks or large language models [38], which often lack the transparency needed for both regulatory approval and practitioner trust. In this work, we address these challenges by developing an approach that predicts the probability that an IT change will cause incidents. We focus on boosted tree-based classifiers (HGBC, LightGBM, and XGBoost), which are proven in practice [32], well-suited for tabular IT management data and support post-hoc interpretability through established methods such as SHapley Additive exPlanations (SHAP). Our approach generates an incident prediction score for each planned change, enabling engineers to assess deployment readiness. SHAP-based explanations provide feature-level transparency that supports user trust, informed decision-making and meets financial-sector audit requirements. Furthermore, we enrich the dataset with aggregated metrics that reflect team performance, such as change success rates, incident counts, and release outcomes, to evaluate their impact on model performance. A key design consideration is the trade-off between accuracy and explainability. Regulatory constraints make high-performing but opaque models unsuitable, as change-approval decisions must remain traceable and auditable. The resulting prediction scores produced by the models are intended as risk signals, helping teams prioritise which changes merit closer review. We evaluate our approach using a one-year dataset from our case company ING (International Netherlands Group), a multinational banking and financial services corporation. Building on insights from our prior work [22], which analysed the characteristics of incident-inducing changes, we tailor our solution to the organisational and regulatory requirements of financial institutions. This ensures the model is not only accurate, but also usable, trustworthy, and compliant for engineers making deployment decisions – aspects that are essential in our context but have received little attention in previous work. Our main goal is to equip engineers with a predictive and explainable model that complies with regulatory standards, allowing them to identify high-risk changes during the assessment & planning phase and take preventive action to ensure reliable deployments. In particular, we seek to improve upon methods based on business rules and human effort, as currently employed at ING, by training ML models on historical data. We also investigate how team performance metrics can contribute to assessing incident prediction. We address the following research questions:
• RQ2: How can we use data-driven ML models for incident prediction scoring, and how does their performance compare to the rule-based approach? • RQ3: What is the effect of including aggregated team metric data on the accuracy of ML-based models for incident prediction scoring?
• RQ1: What is, in practice, the performance of a rule-based approach for obtaining the incident prediction score for deployment changes?
In our study, we demonstrate how data-driven ML methods can be employed within a tightly regulated change management process, providing more effective risk assessment than the rule-based approaches currently used in practice. Among the evaluated models, LightGBM delivers the best performance, particularly when enriched with aggregated team metrics. The model’s predictions remain explainable, with influential factors including team metadata, the machine involved, and other product-specific risk indicators. By combining predictive performance with transparency, this approach has the potential to strengthen IT system reliability while reducing the time and resources spent on incident management. To our knowledge, this is the first study to emphasise both featurelevel explainability and the integration of aggregated team metrics for assessing change deployment risk in a financial context. To sum up, this work contributes the following. (1) A comparative evaluation of a rule-based approach and three ML classifiers (HGBC, LightGBM, and XGBoost) for predicting incident risk of IT changes, using a one-year real-world dataset from a large bank. LightGBM achieves the highest weighted recall and F2-measure. (2) A method for achieving enhanced model transparency by applying SHAP values to interpret predictions. This shows that textual change descriptions and team-related metadata are among the most influential features, providing actionable insights that support compliance and informed decisionmaking. (3) An analysis of model performance stability over time, demonstrating consistent weighted F2-measure over time. (4) Evidence that incorporating aggregated team metrics yields modest but meaningful performance improvements, especially in AUC. This underscores the potential of enriching predictive models with other data sources to improve accuracy.
2
Related Literature
Predictive models face several significant challenges, such as the lack of ground truth labels, the necessity for manual effort to obtain high-quality data, highly imbalanced datasets, and complex dependencies among components and services [36]. Despite this, substantial progress has been made in developing predictive AIOps models, particularly for incident detection and prediction. This paper examines the impact of changes on incidents, categorised into two main areas: 1) pre-change risk analysis and incident prevention [5, 16]; and 2) post-change identification of failed changes [24, 42, 44–46]. Our focus is on pre-change incident identification. Pre-change incident identification involves predicting whether a change will result in an incident before it is deployed, based on similar historical changes that induced incidents. Changes are a common cause of incidents in live systems, responsible for up to
Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment
70% of outages [7]. Large-scale software companies have many services and resources consisting of numerous components [8], which obscure a complete view of the entire system and its relationships. These dependencies make it difficult to predict when a change will induce an incident, especially since seemingly successful changes often lead to incidents [16, 45]. Previous work on this topic is limited due to the challenges in systematically collecting data on changes inducing incidents [17]. More focus and time pressure are placed on resolving incidents than on the procedural guidelines that require proper administration [21]. Previous work at ING [22] indicates the complexity of determining links between changes and incidents, emphasising the necessity of handling data imbalance issues when utilising ML on this particular data. On the research side, Batta et al. proposed a risk management system based on supporting evidence from past bad changes [5]. They trained multiple classification models, including logistic regression, random forest, passive aggressive, support vector machine, and LSTMs (Long Short-Term Memory models), to distinguish problematic changes from successful ones, evaluated on IBM data. Similarly, Güven and Murthy analysed IBM data to identify change-incident linkages and conducted predictive analytics to reduce change-related incidents [16]. They applied various machine learning algorithms to the properties of a change, finding that classification and regression trees achieved the highest recall. Ahmed et al. [3] demonstrated that XGBoost is highly effective for predicting IT incident severity, often achieving performance comparable to advanced deep learning models such as BERT, RoBERTa, or ERNIE 2.0. However, their study focused exclusively on major incidents and analysed incidents after they had occurred. In contrast, our work targets high-priority incidents, including both priority 1 and priority 2 events, and predicts the risk of a change before an incident occurs. Furthermore, we extend the feature space with aggregated team metrics, providing richer context. This enables a predictive and interpretable model that leverages the proven effectiveness of tree-based classifier models while aligning with regulatory requirements for pre-deployment risk assessment in financial IT operations. Looking at industry solutions, ServiceNow provides a cloudbased platform that aids in digitising any business process, with offerings that include IT service and IT operations management [39]. One of its features provides data-driven insight into the risk during the deployment of a change. However, its purpose is to speed up the change approval process by automatically evaluating and applying approval decisions based on business requirements [29]. While ServiceNow looks at historical change data for insights into team performance [40], we analyse historical change, incident, and aggregated metric data for insights into team performance. However, due to the proprietary nature of ServiceNow’s risk assessment feature and the lack of publicly available technical details, we were unable to directly compare our model with theirs. A key distinction of our approach is its focus on explainability. It offers engineers transparency by providing insights into which features influence the scoring and informing users about what determines their deployment risk prediction. Since all approval decisions at ING require a human-in-the-loop, this additional insight into the scoring aids the approval step in the change deployment
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
process, reinforcing both operational effectiveness and regulatory compliance.
3
Background
This section provides an overview of the case company, its change management process, and the risk-related factors influencing it.
3.1
Case Company
Our research examines ING, a global financial services company offering financial products and services to millions of customers. With over 15,000 engineers deploying thousands of monthly changes, ING has evolved from a traditional bank into a digital platform offering online and mobile banking services. The increasing role of ICT in finance necessitates its integration into daily operations [1]. Consequently, ING can be categorised as a financial software-defined business [21]. Banks play a systemic role in the global economy and are subject to regulatory oversight to ensure they comply with risk guidelines and policies. Regulatory agencies, particularly the European Banking Authority (EBA), significantly influence the management of ING’s processes, given its European base [22]. The EBA aims to ensure effective and consistent prudential regulation and supervision across Europe’s banking sector [33]. Regulatory milestones such as the revised Payment Services Directive introduced in 2017 [11] and the 2019 Guidelines on ICT and security risk management [34] have significantly influenced IT operations in the financial sector. In 2023, the European Parliament adopted the Digital Operational Resilience Act (DORA) [1], establishing a regulatory framework for digital operational resilience, ensuring technological safety, proper functioning, and quick recovery from ICT breaches and incidents, which took effect in 2025. DORA aims to enable the effective and smooth provision of financial services while preserving consumer and market trust and confidence. It specifically addresses ICT risks through rules on ICT risk-management capabilities, incident reporting, operational resilience testing, and monitoring of ICT third-party risks.
3.2
Change Management Process
Change management is the process responsible for handling change requests and managing associated risks [12]. At ING, this process follows the Information Technology Infrastructure Library (ITIL) [12] guidelines, combined with agile principles [6], enabling multi-disciplinary teams to be responsible for entire processes and value chains, end-to-end [21]. The focus is on IT changes affecting services such as hardware, networks, middleware, and software. Each change is assigned to a single team, affects at least one configuration item (CI), and has a specific implementation moment. The configuration management database (CMDB) tracks all IT assets, processes, and changes to their attributes and relationships [12]. This process is closely tied to the incident management process [12], which manages the life cycle of all incidents. When a change is required to resolve an incident, it must be logged and processed through the change management process. Conversely, incident management is responsible for the detection and resolution of incidents that may arise from unsuccessful changes.
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
The change management process involves five stages: 1) Logging: Registering IT change activities as change tickets in the service management tool, with detailed descriptions to aid approvals and information sharing. 2) Assessment & Planning: Evaluating the risk of a change, focusing on the probability of failure and potential damage if the change fails. This stage involves noting dependencies with other teams and changes, and automating deployment impact analysis and risk calculation. 3) Approval: Approval groups assess the readiness of a change for deployment. The product owner must approve any change released to production, accepting the delivered quality and risks. Unapproved changes are cancelled. 4) Coordinate Implementation: Deploying the change within the agreed time window. 5) Evaluation & Closure: Evaluating the change postimplementation to determine if it functions as expected. If not, the rollback or remediation plan should be executed. A closure code is registered to indicate if the change was successful, successful with problems, failed, or cancelled. Our study focuses on the Assessment & Planning and Approval stages by introducing incident prediction scores to help engineers determine whether a planned change is ready for deployment. Accurate prediction at this stage can reduce the likelihood of incidentprone changes reaching production.
3.3
Risk-related Influences to the Process
As a financial services provider, ING must ensure reliability and business continuity [22]. Failed changes can lead to service disruptions, which negatively impact the business. Therefore, robust change deployment management is needed to ensure compliance with governance, legal, contractual, and regulatory requirements. Under article 9 of DORA [1], financial entities need to continuously monitor and control the security and functioning of ICT systems and tools [1]. They must minimise the impact of ICT risks by designing, procuring, and implementing security policies, procedures, protocols, and tools to ensure ICT system resilience, continuity, and availability. Specifically, for production, documented policies, procedures, and controls based on a risk assessment approach are required [34]. These should be integral to the financial entity’s overall change management process to ensure all changes are recorded, tested, assessed, approved, implemented, and verified in a controlled manner [1]. The risk assessment should consider potential impacts on the continuity and quality of financial services [34]. Furthermore, post-change follow-ups should be conducted to verify the successful implementation without unexpected impacts or the need for remediation [1]. Our research supports the risk assessment approach by predicting if a change is predicted to cause an incident, thus aiding the decision on whether the change should be deployed in its current state. Additionally, the EU’s AI Act [10], introduced in early 2024, establishes a legal framework for the development, market placement, putting into service, and use of AI systems within the EU. The goal is to promote the adoption of human-centric and trustworthy AI. To align with this, we emphasise human-in-the-loop decision-making and explainability in our predictions to build user trust and improve actionability.
Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, and Arie van Deursen
4
Data
We analyse 175k closed change tickets deployed in the company’s production environment over one year (November 2022 to October 2023), linked to incident tickets from the same period filtered by priority levels 1 and 2 (out of 5). Priority 1 incidents are characterised by high urgency, requiring to be solved as soon as possible, and high impact, indicating critical effects company-wide or across multiple business domains. Major incidents are identified as special cases of priority 1 that have a major impact on critical business processes or services [12]. Priority 2 incidents have a medium impact on a single business domain and medium urgency. A change is considered to have triggered an incident if its identifier appears in the incident’s Caused by Change field or is mentioned in its Solution field as the cause. This results in about 2.4% (around 4k) of changes being identified as incident-inducing. We further enrich the data with aggregated team metrics, specifically for the team assigned to implement each change. These metrics are derived from incidents, changes, and releases associated with that team. Release data is only available for 23.1% of changes, providing an overview of all production releases executed through pipelines and their compliance status. For each release, there is information on team metadata, start and end times, release outcomes (success, partial success, or failure), and control columns such as product owner approvals, peer code reviews, and related changes. Aggregated team metrics are linked to change tickets via the IT Product, which represents the deliverable owned by a single engineering team. To protect confidentiality, we report only aggregated dataset characteristics rather than detailed feature statistics.
5
Methods
To enhance change deployment reliability, we propose generating a predictive incident prediction score during the planning phase. We start by reviewing the company’s current risk assessment approach, then introduce and evaluate three classification-based ML models. We identify the best-performing model and further test its accuracy by incorporating additional features.
5.1
Baseline: Rule-Based Scoring
The company’s change deployment risk assessment is supported by an automated calculation within its IT Service Management (ITSM) tool. This calculation is configured by the company and is based on data available in the tool and the CMDB. The resulting score provides an indication of the probability of failure for the planned change and the potential impact. The calculation result depends on the quality of the information in the change tickets and the CMDB, making it reliant on human effort and expertise to ensure detailed and accurate tickets. The ITSM tool uses specific criteria, or business rules, to derive a risk category, considering factors of the probability of failure (e.g., the scope of impacted IT services, deployment complexity, and incident history) and of potential damage (e.g., Confidentiality, Integrity, and Availability ratings, SOx criticality [41], and recoverability). The output is a score from 0 to 100, mapped to risk levels: low (0-33), medium (34-59), and high (60-100).
Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment
This score is generated during the Assessment & Planning stage and informs the Approval stage, where the approval group uses it to evaluate deployment readiness [21]. This supports the four-eye principle, a widely adopted practice in financial institutions and other safety-critical domains, which requires at least two independent reviewers to approve each change before deployment. Due to confidentiality constraints, we cannot provide a deeper description of the exact business rules, weightings, or configuration logic used in the current baseline.
5.2
Scoring using ML models
To improve the current rule-based assessment, we apply data-driven ML methods to historical change and incident data from ING’s ITSM system. Unlike the baseline, which depends on manually entered data and expert judgment, ML models learn patterns from past deployments that could negatively impact stability and cause incidents, reducing subjectivity and improving efficiency. This approach helps engineers and change managers maintain high service quality and fosters continuous improvement in a fast-moving environment. Also, it helps them by ensuring that significant changes receive closer scrutiny, particularly in areas such as the execution plan, testing, and rollback, while allowing lower-risk changes to proceed with less oversight. This allows for streamlining approvals by IT leads. This is particularly beneficial for Deployment Change Advisory Boards (DCABs), also known as Change Advisory Boards (CABs). In addition to predictive scoring, we offer explainability of the models by employing SHAP, a unified framework for interpreting predictions [25]. SHAP values enable us to provide users with insights into a single prediction of a planned change. They show how each feature contributes positively or negatively to a given prediction, and identify the most important features that drive the ML models’ predictions. SHAP was chosen because it is particularly well-suited for boosted tree models [27], provides individualised explanations for each prediction, and is widely used in industry and research [30]. We report the top 15 most important features influencing the predictions of the best-performing ML model. This approach aligns with other ongoing research at ING, where SHAP is similarly used to provide explanations of an ML model’s decision to end users [37]. By explaining predictions, we increase user understanding and acceptance [2, 36]. From a regulatory perspective, explainability also ensures that model-driven decisions remain auditable and accountable, supporting financial sector requirements for transparency, traceability, and human oversight. 5.2.1 Feature Preparation. Before modeling, we perform extensive data processing and feature engineering to ensure the quality and relevance of our data. For processing the incident data, we begin by removing duplicates and filtering out irrelevant incidents based on the company’s closure codes, such as ‘Invalid event’ or ‘Withdrawn by Customer’. Major incidents are a sub-category of Priority 1 incidents, but we introduce them separately as Priority 0. To enrich the data, we check if a change identifier is mentioned in the incident ticket as a cause and add this identifier to the Caused by Change field if not already present.
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
For processing the change data, we focus on closed changes. In preparing the features, we start by removing stopwords from the Short Description and Description columns of the changes. We then generate Natural Language Processing (NLP) features using scikitlearn’s [35] CountVectorizer, which are fed into a TruncatedSVD for latent semantic analysis. Date features are generated from the change start timestamp, including the starting hour, day of the week, quarter, month, day of the year, day of the month, week of the year, and whether it is a weekend. We merge both datasets, incorporating all categorical, numerical, date, and NLP features. For categorical features, we include metadata of a change, such as attributes of the CI, security ratings (confidentiality, integrity, and availability), change management information (change category, change state, CAB approval group), incident and support management information (e.g., support offering or assignment group), compliance and criticality information, and deployment and architecture information (automated deployment, fallback options, and redundant architecture). The data is merged with incident data to determine which changes caused incidents by linking the Caused by Change incident fields with change data. Connections where the change occurred after the incident are removed to ensure only causal links are retained. As described in Section 4, we incorporate aggregated team metrics as features, derived from incident, change, and release data. These features consist of median weekly and monthly aggregates of team-level performance metrics, computed using the IT Product as a proxy for team identity. To capture team performance, we compute aggregate metrics such as the number of changes, percentage of successful changes, number of changes causing incidents, number of high-priority incidents, percentage of successful releases, and number of releases. However, since the IT Product field currently has only 50% coverage (with ongoing efforts to improve this), we exclude tickets lacking this information in our analysis for RQ3. 5.2.2 Approach. The Caused by Change field labels whether a given change results in an incident. A change is labelled as causing an incident if it has at least one link to a high-priority incident (i.e., major, priority 1, priority 2), independent of how many incidents are linked. For each change, we also record the highest incident priority, which is later used for sample weighting. This labelling process results in a binary classification, indicating that a change has either induced at least one high-priority incident (1) or not (0). Based on these labels and the prepared features, we train multiple classification ML models on a training set. We split the data temporally, using the first eight months for training (120K changes), then two months for validation (26K changes), and the last two months as the test set (30K changes). We implemented three gradient boosting algorithms in Python: HistGradientBoostingClassifier from scikit-learn [35] (referred to as HGBC), LightGBM [23], and XGBoost [9]. We focused on classification models that can be explained post-hoc to support explainability and auditability, which is a critical requirement in our context. Simpler models were unable to handle the Not a Numbers (NANs) inherent in the data, which provide meaningful information regarding unfilled fields. To preserve this information, we opted for models capable of handling such values, unlike other models like linear regression, support vector machine or random forest classifiers.
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, and Arie van Deursen
Through iterative training, the model learns which features are useful indicators for predicting the labelled outcome. The target variable is an incident prediction score between 0 and 100, indicating the probability that the planned change will trigger an incident.
6
Evaluation
Each model serves as a classifier that produces a numerical score between 0 and 100, indicating the probability that a change will trigger a high-priority incident. To convert these into binary predictions, we identify an optimal threshold using a temporal split: eight months for training and two months for validation. The threshold that maximizes the weighted F2-measure (defined below) on the validation set is selected and applied to a held-out two-month test set. To simulate real-world deployment, we evaluate the best-performing model using a sliding window setup. In practice, new changes and incident data are generated continuously, and risk assessments must remain aligned with the latest information. Therefore, the model is retrained daily, incorporating the most recent data, and produces predictions on a weekly cycle to support the organisation’s change planning and approval processes. From each prediction window, we derived confusion matrices and computed evaluation metrics to monitor performance and stability over time. 𝑃 Precision, defined as 𝑇 𝑃𝑇+𝐹 𝑃 , captures how many positive predictions are correct. Given the significant class imbalance due to the rarity of high-priority incidents triggered by a change, we report weighted variants of recall and F-measure to more fairly assess model performance. Weighted recall or wR (as defined in Equation 1) averages recall per class, weighted by the number of true instances in each class: wR =
∑︁ 𝑛𝑐
·
𝑁 𝑐 ∈𝐶
𝑇 𝑃𝑐 𝑇 𝑃𝑐 + 𝐹 𝑁𝑐
w𝐹 𝛽 =
(1 + 𝛽 2 ) · 𝑃𝑐 · 𝑅𝑐 𝑛𝑐 · 𝑛 + 𝑛 𝛽 2 · 𝑃𝑐 + 𝑅𝑐 1 𝑐 ∈𝐶 0 ∑︁
7
Results
This section compares the effectiveness of the company’s existing rule-based risk assessment against the ML methods. We first evaluate the baseline to establish a performance benchmark, then assess how the ML models improve predictive accuracy. We also examine the stability of the best performing model its performance over time, and examine the impact of incorporating aggregated team metric data.
7.1
RQ1 - Performance of Rule-Based Approach
At ING, changes with a risk score of 60 or higher are considered a high-risk level during the Assessment & Planning stage of the change deployment management process. In this paper, we define a high-risk level to indicate that the case company considers a given change as potentially resulting in an incident. Using this threshold of 60 on the two-month test set, the baseline performance for all metrics is presented in Table 1. Table 1: Performance of Baseline, ML Models, and LightGBM with and without Additional Features
(1)
Here, 𝐶 is the set of classes (high-priority incident/ no highpriority incident), 𝑛𝑐 the number of true instances of class 𝑐, and 𝑁 the total number of instances. To reflect the importance of minimising false negatives so that we do not miss incident-inducing changes when they are being planned, we compute the weighted F2-measure or w𝐹 2 (as defined in Equation 2). Greater emphasis is placed on recall using 𝛽=2, which is inspired by prior work [5].
Therefore, a high recall (even at moderate precision) supports riskaware decision-making. Finally, to account for the practical significance of varying incident severities, we introduce priority-based weighting in our evaluation. Changes that lead to major or priority 1 incidents receive a weight of 5, those resulting in priority 2 incidents receive a weight of 3, and non-incident changes are weighted as 1. This ensures that the evaluation process gives proportionally more importance to correctly identifying severe incidents.
(2)
Where 𝑃𝑐 and 𝑅𝑐 are the class-specific precision and recall scores. Our evaluation aligns with the Assessment & Planning stage of the change management process, where the objective is to proactively flag changes that trigger incidents before deployment. A high recall is crucial to ensure that most incident-inducing changes are identified, even at the cost of some false positives. These predictions are accompanied by model explanations to assist change managers and engineers in making informed decisions on whether to proceed and if additional actions are needed. By reviewing the prediction explanations, we ask for more due diligence from the engineer to avoid incidents similar to those in historic changes.
LightGBM Metric
Base
HGBC
XGB
Without
With
Threshold Precision wR w𝐹 2 AUC
60 0.04 0.56 0.88 0.55
99 0.02 0.86 0.88 0.61
98 0.05 0.89 0.90 0.71
17 0.02 0.91 0.92 0.67
29 0.04 0.93 0.93 0.60
The baseline approach achieves very low precision, indicating that most changes flagged as high risk do not actually result in incidents. This is largely due to the highly imbalanced nature of the dataset, where incident-inducing changes are rare, making it difficult for static rules to distinguish incident-inducing changes from safe ones. However, the wR and w𝐹 2 are better, since the recall identifies 56% of the relevant cases within the test set, and the w𝐹 2 shows the weighted precision and recall, giving a good indication for our business purposes. When examining the area under the receiver operating curve (AUC), the score of 0.56 is close to 0.5, suggesting that the model has almost no discriminating ability [18] and thus can be interpreted as failing in classifying this task [31]. Finding 1. The baseline risk assessment fails to effectively identify highrisk changes, as evidenced by low precision, a moderate wR of 56%, and an AUC of 0.56, indicating poor discriminative performance.
Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment
RQ2 - Comparison of ML Models vs. Rule-Based Approach
7.2.1 Performance. We evaluated three ML models, HGBC, LightGBM, and XGboost, against the rule-based baseline, optimising each model’s threshold on a validation set and assessing final performance on a two-month test set (see Table 1). For each performance metric, the best-performing value is highlighted in bold. The baseline achieves a precision of 0.04, but is outperformed by XGBoost, which achieves the highest precision (0.05). Both HGBC and LightGBM record lower precision values of 0.02. However, our primary objective is to maximise correct identifications, acting more as a guard dog rather than prioritising the minimisation of false positives. This approach favours recall over precision, making the small loss in precision acceptable. In terms of wR, all ML models substantially outperform the baseline. LightGBM achieves the highest value (0.91), followed by XGBoost (0.89) and HGBC (0.86), whereas the baseline reaches 0.56. This performance gap is also apparent in the w𝐹 2 , which weighs recall more heavily. LightGBM again leads with a w𝐹 2 of 0.92, followed by XGBoost(0.90), and HGBC (0.88), with the baseline also scoring 0.88. Despite matching HGBC on w𝐹 2 numerically, this value of the baseline is less meaningful given its substantially lower recall, meaning it identifies fewer relevant cases overall. The AUC scores provide further insight into each model its overall discriminative ability. XGBoost achieves the highest AUC (0.71), followed by LightGBM (0.67) and HGBC (0.61), while the baseline records the lowest score (0.55). As an AUC score falling between 0.5 and 0.6 means it fails in classifying this task [31] and has near-random performance, the ML models demonstrate moderate improvements in classification ability over the baseline. There is also considerable variation in decision thresholds across models. HGBC and XGBoost both use very high thresholds (99 and 98, respectively), indicating a low decision boundary likely aimed at maximising recall. In contrast, LightGBM uses a much lower threshold (17), suggesting that its scoring scale differs and may necessitate model-specific calibration. Overall, LightGBM offers the best performance in terms of wR and w𝐹 2 , aligning well with our goal of maximising correct identifications. Despite relatively modest AUC values, all ML models outperform the rule-based baseline and are better suited for highrecall tasks. 7.2.2 Important Features. Next, we examine the best-performing model, LightGBM. Using SHAP, we analyse the top 15 most important features for prediction outcomes, as shown in Fig. 1. Description values have been concatenated due to the abundance of NLP features. Interestingly, the textual features Full Description and Short Description have the most influence, highlighting that the content of the change significantly impacts the outcome. This is followed by the CI Name, which represents the configuration item that is being changed. Additionally, the Assignment Group, which refers to the team assigned to work on the change, and the Support Offerings, which are the team supporting the CI, are also highly influential. Other features, such as the change approval board group, the CI configuration group, and the owner of the CI, are less important but still influence the predictions.
Full_Description Short_Description CI_Name Assignment_Group Support_Offerings CAB_Approval_Group
Feature
7.2
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
CI_Config_Group CI_Owner CI_Asset_Owner Outage_Total_Duration CI_Business_Unit datetime_quarter CI_Sys_Class datetime_weekend CI_Crucial 0
50
100
150
200
Importance
Figure 1: Feature Importance of LightGBM. 7.2.3 Stability Over Time. To further assess the stability of LightGBM, we used a sliding window evaluation to simulate its performance over time, comparing it with the rule-based baseline method (see Fig. 2). This evaluation shows that LightGBM consistently achieves higher wR values compared to the baseline over time, indicating a stronger ability to correctly identify positive cases. Both wR values show some fluctuations, with a peak around early September for LightGBM. This inconsistency is likely due to a period of minimal changes at the company, during which modifications were infrequent and strictly supervised. In contrast, the w𝐹 2 for both models remains relatively stable over time. LightGBM again outperforms the baseline, highlighting its ability to maintain a good balance between precision and recall, which is important for our goal since we want to maximise the identification of correct cases. These results suggest that LightGBM not only delivers performance improvements over the baseline but also maintains its performance in a production-like environment. Finding 2. Among the evaluated models, LightGBM performs best in wR and w𝐹 2 , making it the most suitable for high-risk change detection. Key features are primarily textual and team-related. Despite some fluctuations in wR during periods of low change activity, LightGBM shows stable w𝐹 2 performance over time.
7.3
RQ3 - Inclusion of Aggregated Team Metric Data
7.3.1 Performance. The best-performing model, LightGBM, is used to evaluate whether the inclusion of additional features from metric data aggregated by IT Product improves its performance. The performance is compared with and without these additional features (See Table 1). The addition of team aggregated features results in a higher classification threshold and a noticeable improvement in both precision and wR, increasing from 0.02 to 0.04 and from 0.91 to 0.93, respectively. The w𝐹 2 also improves slightly from 0.92 to 0.93, reflecting that the model is better at balancing between precision and
ICSE-SEIP ’26, April 12–18, 2026, Rio de Janeiro, Brazil
Eileen Kapel, Jan Lennartz, Luis Cruz, Diomidis Spinellis, and Arie van Deursen
1
Baseline LightGBM
0.6
Short_Description Assignment_Group
0.4
CI_Name
0.2
Support_Offerings
0 Aug 27 2023
Sep 3
Sep 10
Sep 17
Sep 24
Oct 1
CI_Config_Group
Oct 8
Feature
Value
Full_Description
Model
0.8
Date
(a) wR 1
CAB_Approval_Group datetime_quarter
Model
0.8
Value
CI_Owner IT_Product
Baseline LightGBM
0.6
CI_Asset_Owner Availability_Rating
0.4
Outage_Total_Duration
0.2
Releases_Percentage_Successful_per_Week
0 Aug 27 2023
Sep 3
Sep 10
Sep 17
Sep 24
Oct 1
Changes_Median_Count_per_Month
Oct 8
0
100
recall. However, the AUC score decreases slightly from 0.67 to 0.60, suggesting that the model’s performance based on the optimised threshold improves, but its overall scoring consistency across all instances slightly worsens. Despite this drop in AUC, the gain in precision and recall indicates a more confident identification of positive instances. Integrating data from other sources, with an emphasis on ease of merging, proves to be a worthwhile approach, given the coverage issues discussed in Section 5). 7.3.2 Important Features. Upon closer inspection of the top 15 SHAP ranking of the model with the added features (see Fig. 3), we observe the inclusion of these new features. Notably, the IT Product ranks 8th, while Release Percentage Successful per Week and Changes Median Count Per Month are also included, appearing near the bottom of the ranking. 7.3.3 Examples. Fig. 4 shows two SHAP plots explaining individual predictions. To improve interpretability, we display the ten most impactful features. For multi-component features such as Full Description and Short Description, we use the maximum signed SHAP value to highlight the most influential element. Features with positive SHAP values (right side) contribute to a higher predicted risk, while negative values (left side) reduce it, offering users a transparent view into the model’s decision-making process by quantifying the contribution of each feature to the output. Figure 4a shows a high-scoring change, specifically a monthly infrastructure patch that resulted in a user-facing incident. In this example, Impacted Services is the most influential positive feature, pushing the score toward a high-risk classification. Other top contributors, which were also in the top 5 in Figure 3, include Assignment Group, Full Description, Short Description, CI Name, and Support Offerings, which appear on both sides of the SHAP scale, suggesting that they have different effects depending on context.