Conceptio › Archive › arXiv CS

arXiv CSopen access

From Prediction to Practice: A Task-Aware Evaluation Framework for Blood Glucose Forecasting

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neural-networks

machine learning, deep learning, neural networks

Preprint: Under Review [VOLUME # TBD]:1–25, 2025

Machine Learning for Healthcare

From Prediction to Practice: A Task-Aware Evaluation Framework for Blood Glucose Forecasting Alireza Namazi

[email protected]

Department of Computer Science University of Virginia Charlottesville, VA, USA

arXiv:2605.00645v1 [cs.LG] 1 May 2026

Heman Shakeri

[email protected]

School of Data Science University of Virginia Charlottesville, VA, USA

Abstract Clinical time-series forecasting is increasingly studied for decision support, yet standard aggregate metrics can obscure whether a model is actually useful for the task it is meant to serve. In safety-critical settings, low average error can coexist with dangerous failures in exactly the high-risk regimes that matter most. We present a task-aware evaluation framework for blood glucose forecasting built around two downstream uses: hypoglycemia early warning and insulin dosing decision support. For early warning, we evaluate on real data from three clinical cohorts using event-level recall and false alarms per patient-day, metrics that reflect operational alarm burden rather than aggregate accuracy. We show that models appearing acceptable overall, with recall above 0.9 on the full test set, can fail badly in the post-bolus slice, where insulin-on-board is elevated and missed warnings carry the greatest clinical consequences. Standard forecasting evaluation, however, does not test whether a model can reason about the effects of actions, a requirement for supporting insulin dosing decisions. We therefore add a second, interventional arm using the FDA-accepted UVA/Padova simulator, where we evaluate whether forecasters can predict glucose responses to altered insulin plans in paired factual/counterfactual scenarios. We show that models that look strong on real-data forecasting often fail to predict the direction, magnitude, or ranking of intervention effects, and choose poor insulin doses when evaluated under a clinically motivated cost. Taken together, the two arms reveal a consistent gap between forecasting accuracy and task-relevant usefulness. We release the benchmark, the standardized preprocessing pipeline for public cohorts, and the simulator-based interventional dataset as a reproducible toolkit.

1. Introduction Accurate short-horizon blood glucose forecasting is clinically important and is enabled by the widespread adoption of continuous glucose monitors (CGMs), which measure interstitial glucose at frequent intervals and provide near real-time feedback (Facchinetti, 2016; Shah et al., 2018; Welsh et al., 2019). Forecasts can strengthen safety layers in diabetes management, including hypoglycemia early warning and decision-support prompts (Buckingham et al., 2010; Puhr et al., 2019). They are also relevant to the broader vision of closed-loop

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Arm 1 — Real data (observational) Overall Slice Hypo detected ✓

ground truth

forecast

Post-bolus Slice ▼

Hypo missed ×

hypo threshold

▼ bolus

Forecaster CGM sensor

Arm 2 — Counterfactual simulation (interventional) perturbation onset

ground truth

forecast

■ increased bolus forecast

■ factual

■ reduced bolus forecast

Figure 1: From prediction to practice. Aggregate-accurate forecasters can still fail where it matters clinically, e.g. missing hypoglycemic events in the post-bolus slice (Arm 1) and reversing the predicted response to changes in insulin dose (Arm 2).

insulin delivery and artificial pancreas systems, where prediction models can support dosing decisions (Moon et al., 2021; Lee et al., 2024; Fischer, 2025). Realizing this potential, however, requires careful evaluation of the forecasters. The clinical utility of a glucose forecaster depends on the downstream use case. For safety gating, the key question is whether impending hypoglycemia is detected with acceptable alarm burden. For controller support, the forecaster must provide reasonable trajectory fidelity, avoid clinically dangerous forecast errors, and predict the consequences of altered control actions. Evaluation design choices can change the conclusions (Castle and Beck, 2024) yet current blood glucose forecasting studies use inconsistent protocols. They differ in datasets, split strategy, prediction horizons, input definitions, and reported clinical metrics, making performance claims hard to interpret (Karagoz et al., 2025; Huang et al., 2025; Fox et al., 2018; Sergazinov et al., 2023). Even when “clinical” evaluation is reported, the choice of risk metrics and reporting conventions varies substantially, including different error-grid frameworks (Clarke et al., 1987; Parkes et al., 2000; Klonoff et al., 2014) and scalar summaries (Del Favero et al., 2012; Wolff et al., 2025). Moreover, because well-managed patients spend most time in safe range, global metrics such as RMSE can underweight critical rare events (Wolff et al., 2025). This work presents a task-aware evaluation framework for blood glucose forecasting. The framework has two complementary arms. The first evaluates performance on real observational data, including both overall performance and clinically meaningful slices. The second uses the FDA-accepted UVA/Padova simulator to generate matched counterfactual insulin-action scenarios and tests whether a forecaster can predict the glucose consequences 2

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

of altered control inputs when insulin injection policy changes. This isolates action sensitivity without confounding from a downstream MPC or RL controller (Lee et al., 2024; Herrero et al., 2026). Across these two arms, we find that models that appear acceptable under standard forecasting evaluation can still fail badly in safety-critical slices such as post-bolus periods and under altered insulin actions. We release the benchmark as a reproducible toolkit containing a standardized preprocessing pipeline for the public cohorts and a simulator-based interventional dataset. Contributions. 1. We frame blood glucose forecasting evaluation as a task-aware problem covering safety gating and controller support in artificial pancreas systems. 2. We standardize evaluation on real data across public cohorts with patient-level splits, clinically meaningful slices, and a unified metric set. 3. We introduce an interventional evaluation arm using the FDA-accepted UVA/Padova simulator to test action-conditional prediction quality without confounding from a downstream controller. 4. We release a reproducible benchmark toolkit including the preprocessing pipeline, evaluation metrics, and simulator-based interventional dataset. Generalizable Insights about Machine Learning in the Context of Healthcare. Usefulness is task-dependent: the metrics that matter for early warning and controller support are not interchangeable. In safety-critical forecasting, population-averaged summaries can overstate usefulness by obscuring failures in minority high-risk regimes. More broadly, benchmark design in healthcare should standardize use-case-aligned evaluation and clinically meaningful stress tests.

2. Related Work Clinical-risk metrics for glucose measurement and prediction. The clinical consequences of glucose errors are asymmetric and depend strongly on the operating range, so the diabetes community has developed evaluation tools that go beyond symmetric point errors. Error-grid frameworks map each reference–estimate pair to clinically meaningful risk regions, including the Clarke Error Grid (CEG) (Clarke et al., 1987) and the Parkes (Consensus) Error Grid (PEG) (Parkes et al., 2000). More recently, the Surveillance Error Grid (SEG) provides a continuous risk surface intended to reflect contemporary treatment practice (Klonoff et al., 2014). Many studies also report scalar summaries such as gMSE/gRMSE-style measures motivated by error-grid penalties (Del Favero et al., 2012) and device-oriented measures such as mean absolute relative difference (MARD) (Freckmann et al., 2019). However, these metrics do not play the same role in every downstream application: early warning and controller support emphasize different aspects of forecast quality. In practice, papers often mix grids, scalar summaries, and reporting conventions, which makes comparisons across methods and cohorts difficult.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Forecasting models, benchmark efforts, and reporting pitfalls. Blood-glucose forecasting methods span physiological and statistical models, classical machine-learning regressors, and deep sequence models. Recent surveys review recurrent and convolutional architectures commonly used in this literature (Alshehri et al., 2024), and more recent comparison work has adapted state-of-the-art Transformer-based time-series forecasters to glucose prediction (Karagoz et al., 2025). Several methods have also been designed specifically for blood glucose forecasting rather than adapted directly from the general time-series literature (Khamesian et al., 2025; Alshehri et al., 2024). Prior comparison work has emphasized that conclusions can depend strongly on dataset choice and setting (Hameed and Kleinberg, 2020). More recently, GlucoBench curated CGM datasets and provided prediction benchmarks, reducing dataset fragmentation and improving access to shared baselines (Sergazinov et al., 2024). Yet data standardization alone does not solve the evaluation problem: summary choices across horizons, cohorts, and metrics can still hide clinically important differences. These issues motivate benchmark designs that standardize evaluation across clinically meaningful slices and task-relevant metrics. From forecasting accuracy to decision-support evaluation. Standard forecasting metrics on real data do not directly tell us whether a forecaster is useful for downstream decision making. In glucose forecasting, the relevant actions may lie outside the empirical action distribution seen during training, so a forecaster that performs well under the behavior policy may still fail to predict the consequences of out-of-distribution control inputs. Simple attribution-based checks such as SHAP (Lundberg and Lee, 2017) can show whether a model appears to use insulin or meal channels, but they do not establish that the model predicts the magnitude or ranking of action effects correctly. Prior work has shown that lower forecasting RMSE under the behavior policy does not necessarily translate into better closed-loop glycemic control when the forecaster is used inside a controller (Lee et al., 2024). Digital-twin studies suggest that improved glucose prediction can improve glycemic outcomes when the predictor is embedded in a control loop (Herrero et al., 2026). However, controller-in-the-loop evaluation makes it difficult to isolate forecasting quality from controller design and tuning (Lee et al., 2024).

3. Benchmark Design and Task-Aware Evaluation 3.1. Overview We study blood glucose forecasting in two downstream settings: safety gating, such as hypoglycemia early warning, and controller support, where forecasts are used to compare candidate insulin actions. We therefore use two evaluation arms: an evaluation on real data and a counterfactual simulator-based evaluation. In both cases, we evaluate the forecaster itself rather than the full controller–forecaster stack, so that forecasting errors are not mixed with controller design or tuning choices. Each example consists of a history window xt−H+1:t of length H and a prediction target gt+1:t+L over a horizon of length L, where gt denotes glucose (mg/dL) at time t. Glucose is evaluated on a uniform 5-minute grid after cohort-specific harmonization. We focus on point forecasts. Inputs include past CGM glucose together with available exogenous signals such

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

as basal insulin rate, bolus insulin events, and meals when present. Cohort construction and data harmonization are described in Section 4 and Appendix A. 3.2. Evaluation on real data In the real-data arm, the model predicts future glucose from observed history alone. This arm covers both overall forecasting quality and coarse-grained safety gating on real-world cohorts. Forecasting metrics. For overall forecasting quality, we report RMSE (mg/dL) and the percentage of predictions falling in dangerous Parkes Error Grid zones C–E, which we refer to as the unsafe fraction. RMSE captures average pointwise error, whereas PEG unsafe isolates clinically dangerous predictions. In the main paper, these metrics are reported at 30 and 60 minutes. Safety-gating metrics. For safety gating, we evaluate alarms over the full prediction window rather than at a single forecasted time point. A hypoglycemic event is defined as CGM glucose < 70 mg/dL for at least 15 continuous minutes, i.e., 3 consecutive 5-minute samples below threshold, with onset assigned to the first sample in that run; the event ends when glucose is ≥ 70 mg/dL for at least 15 continuous minutes (Danne et al., 2017). We use 70 mg/dL as the primary alarm threshold because the goal is early warning rather than only detection of more severe hypoglycemia. We exclude precision and F1 as primary safety-gating metrics. Due to the infrequency of hypoglycemia, precision is highly sensitive to event prevalence. More importantly, these metrics treat alarms as generic classification counts rather than operational episodes, failing to reflect the actual patient burden. Instead, we use false alarms per patient-day to quantify nuisance burden, targeting ≤ 3 daily as clinically acceptable (Harvey et al., 2012), while relying on recall and warning lead time to evaluate the effectiveness and timeliness of event detection. We therefore report three operational metrics: • Event-Level Recall (Sensitivity): the fraction of true hypoglycemic events that are detected by the forecaster; • Median Warning Lead Time (minutes): the median time between the first matched alarm and event onset, among detected events; • False Alarms per Patient-Day: the average number of unmatched alarms per patient per day. For an alarm horizon PH, an event with onset t0 is counted as detected if at least one alarm occurs in the interval [t0 − PH, t0 − ∆]. We generate alarms using the full PH-minute forecast window and enforce a refractory rule of length PH: after one alarm fires, no new alarm can be triggered for the next PH minutes. This prevents repeated alarms during the same event. In the main benchmark, these gating metrics are reported for PH = 30 minutes.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Safety-relevant slices. To test whether overall performance hides important failures, we also evaluate on subsets of the data. The overall slice is the full test set. The post-bolus slice restricts to samples where a bolus was delivered within the preceding 60 minutes of the prediction window. The nocturnal slice restricts to samples between midnight and 06:00. The main paper reports the overall and post-bolus slices; nocturnal results are reported in the appendix. 3.3. Counterfactual simulator-based evaluation The second arm is designed for controller-supporting use cases. Here the forecaster receives the same historical input window up to time t together with a planned future insulin sequence over the prediction horizon, and predicts the corresponding future glucose trajectory. The main question is whether the forecaster can predict the consequences of altered insulin actions rather than merely extrapolate trajectories generated by the behavior policy. We use the FDA-accepted UVA/Padova Type 1 diabetes simulator with the default model predictive controller (MPC) (Garcia-Tirado et al., 2021). Standard simulator trajectories generated under the default behavior policy are used for model training. Counterfactual evaluation is performed on paired factual/counterfactual episodes from held-out virtual subjects that share the same subject, seed, pre-onset history, and future meals, but differ in the future insulin plan. Future meals are held fixed across paired rollouts and are not treated as control actions. Full simulator and perturbation details are given in Appendix B. Counterfactual response prediction. The first simulator experiment asks whether the fact and g cf denote the factual forecaster predicts the effect of an insulin perturbation. Let gt+k t+k fact and ĝ cf denote and counterfactual simulator glucose values at lead time k∆, and let ĝt+k t+k the corresponding model predictions. We define the true and predicted intervention effects as fact cf fact cf (1) ∆pred ∆true t+k = gt+k − gt+k , t+k = ĝt+k − ĝt+k . We report three metrics. First, effect RMSE measures whether the model predicts the magnitude of the intervention effect: v u N 2 u1 X true RMSEeff (k) = t − ∆ . (2) ∆pred n,t+k n,t+k N n=1

Second, sign agreement measures whether the model predicts the direction of the effect: SA(k) =

N i 1 X h true 1 sign ∆pred = sign ∆ . n,t+k n,t+k N

(3)

n=1

Third, we evaluate whether the forecaster preserves the relative ordering of candidate actions within a perturbation family using Kendall’s τb : (k) (a) (a) τb = τb {gt+k }a∈A , {ĝt+k }a∈A , (4) where A is the set of valid candidate actions for that episode. These metrics are reported at 30, 60, and 120 minutes. We also report ordinary RMSE on factual and counterfactual trajectories as a secondary diagnostic. 6

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Bolus action selection. The second simulator experiment asks whether the forecaster can choose the best action from a discrete bolus menu. For each episode, we simulate a set of candidate bolus scales and compare the model-chosen action against the simulator-optimal action under a clinically motivated cost. We report action match rate, the fraction of episodes in which the model selects the same action as the simulator oracle, and policy regret, the increase in simulator-evaluated cost incurred by following the model’s chosen action instead of the simulator-optimal action. Let ˆ (5) a⋆ = arg min J(a), â⋆ = arg min J(a), a∈A

a∈A

ˆ is the corresponding where J(a) is the simulator-evaluated clinical cost of action a and J(a) cost evaluated on the model-predicted trajectory. We define regret as Regret = J(â⋆ ) − J(a⋆ ).

(6)

Because the clinical consequences of glucose excursions are asymmetric, we define the cost in the Blood Glucose Risk Index (BGRI) space (Kovatchev et al., 1997): T

f (g) = 1.509 (ln g)1.084 − 5.381 ,

J(a) =

2 1X 10 · f gt+k (a) . T

(7)

k=1

This risk-based objective penalizes hypoglycemic trajectories more heavily than mild hyperglycemia and therefore reflects the asymmetric clinical stakes of model-guided insulin decisions. The exact bolus action menu and implementation details are given in Appendix B. 3.4. Data splits and model selection All primary real-data results use patient-level splits to evaluate generalization to unseen individuals. For each cohort, we partition subjects into disjoint train/validation/test sets and construct all sliding windows within each split independently. Unless otherwise specified, split ratios are fixed across cohorts (e.g., 70/10/20). For the simulator-based evaluation, we apply the same principle at the level of virtual subjects. Virtual patients are partitioned into disjoint train/validation/test sets. Standard simulator trajectories generated under the default MPC (Garcia-Tirado et al., 2021) behavior policy are used for model training and model selection. In this arm, the planned future exogenous insulin variables are provided to the model by shifting the basal and bolus channels left by the prediction horizon, so that the future insulin plan over the forecast window is available as input. Paired factual/counterfactual episodes are generated only for the test virtual subjects and are not used for training. We compare a representative suite of forecasting methods spanning a classical statistical baseline, deep sequence models, modern time-series forecasters, and glucose-specific forecasting methods. To ensure fair comparison across heterogeneous model families, we standardize both data access and hyperparameter tuning. For each method, we start from author-recommended hyperparameters when available and search in a bounded neighborhood around those defaults under a fixed compute budget. Validation performance at the 30-minute horizon is used for hyperparameter selection, and the resulting hyperparameters are reused across the prediction horizons. 7

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Cohort DCLP3 DCLP5 PEDAP UVA/Padova simulator

Population

Duration

Key signals

T1D (14+) T1D (children 6–13y) T1D (children 2–6y) virtual T1D adults

112 101 102 100

6 months 16 weeks 13 weeks 15 days

CGM, basal/bolus CGM, basal/bolus CGM, basal/bolus, carbs CGM, basal/bolus, carbs

Table 1: Cohorts used in this study. DCLP3 (Brown et al., 2019), DCLP5 (Breton et al., 2020), and PEDAP (Wadwa et al., 2023) are randomized T1D trials with pump/CGM logs. The counterfactual evaluation uses the adult virtual cohort of the FDA-accepted UVA/Padova in-silico T1D simulator (Kovatchev et al., 2009; Visentin et al., 2014).

For the simulator-based evaluation, models are retrained from scratch on the standard simulator training set using the same architecture family and the same tuning policy. Model selection for the interventional arm is performed on the standard simulator validation set, not on perturbed validation episodes, so that the counterfactual benchmark remains a heldout stress test.

4. Cohorts We evaluate the main real-data results on three cohorts of insulin-treated Type 1 diabetes (T1D) with synchronized insulin delivery logs. In addition, we use the FDA-accepted UVA/Padova in-silico T1D simulator for the counterfactual evaluation arm. Table 1 summarizes the cohorts. Rationale. DCLP3 provides high-quality CGM trajectories paired with pump-recorded basal and bolus insulin in older participants, while DCLP5 and PEDAP provide pediatric cohorts with different age ranges and more challenging glycemic dynamics. PEDAP additionally includes meal information. The UVA/Padova simulator serves a different role: it provides controlled paired insulin-action scenarios for the counterfactual benchmark. We use the adult virtual cohort only (N = 100) to reduce physiological heterogeneity. Why not healthy or Type 2? We exclude healthy cohorts because glucose excursions are limited, reducing the relevance of short-horizon forecasting. We also exclude Type 2 cohorts because many lack the fine-grained insulin intervention records needed for the controller-supporting questions studied here. Standardized harmonization. To ensure cross-cohort compatibility and prevent leakage, all real-world cohorts are converted to a common multivariate time-series representation. We resample glucose to a uniform 5-minute grid, split sequences on prolonged missingness, interpolate short gaps within contiguous segments, and align available auxiliary channels such as basal insulin, bolus events, meals, and weight to the glucose timeline using rules consistent with their semantics. Basal insulin is treated as a piecewise-constant rate signal, whereas bolus and meal variables are represented as sparse event channels. We

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 2: Overall standard forecasting performance on real-data cohorts. Each cell is reported as mean [95% CI]. Unsafe% refers to forecasts falling into C–E zones in Parkes Error Grid.2 Cohort H Metric

DCLP3

PMLP

Glim

RMSE 15.1 [13.8, 16.5] 13.5 [12.3, 15.1] 13.4 [12.5, 14.3] 13.5 [12.6, 14.3] 30 Unsafe% 0.10 [0.07, 0.14] 0.08 [0.05, 0.12] 0.11 [0.08, 0.16] 0.08 [0.07, 0.10] RMSE 31.3 [29.7, 32.7] 23.8 [22.0, 25.4] 24.2 [22.3, 26.1] 23.5 [22.0, 24.9] 60 Unsafe% 0.82 [0.63, 1.06] 0.76 [0.58, 1.05] 0.94 [0.72, 1.21] 0.76 [0.66, 0.87]

ARX

14.6 [13.7, 15.3] 0.09 [0.07, 0.12] 25.2 [23.7, 26.6] 0.51 [0.42, 0.60]

14.4 [13.6, 15.1] 0.06 [0.05, 0.08] 25.7 [24.3, 27.0] 0.45 [0.37, 0.53]

RMSE 25.4 [22.7, 28.2] 24.2 [21.5, 27.2] Unsafe% 0.75 [0.57, 0.94] 0.81 [0.62, 1.03] RMSE 37.7 [35.2, 40.3] 35.1 [32.4, 37.8] 60 Unsafe% 2.30 [1.95, 2.68] 2.41 [2.05, 2.80]

GRU

23.4 [20.7, 26.3] 0.92 [0.67, 1.23] 33.3 [30.8, 36.1] 2.50 [2.04, 3.01]

22.2 [19.4, 25.3] 0.73 [0.51, 0.98] 32.9 [30.1, 35.9] 2.31 [1.91, 2.74]

21.6 [19.1, 24.4] 0.72 [0.51, 0.97] 32.5 [30.0, 35.3] 2.22 [1.79, 2.67]

RMSE 24.6 [22.5, 26.9] 22.7 [20.7, 24.9] 20.1 [18.4, 21.9] Unsafe% 0.87 [0.76, 1.00] 0.63 [0.54, 0.74] 0.63 [0.53, 0.74] RMSE 37.9 [35.0, 40.9] 34.2 [31.6, 37.1] 31.7 [29.4, 34.1] 60 Unsafe% 3.23 [2.86, 3.61] 2.69 [2.35, 3.06] 2.80 [2.44, 3.17]

20.3 [18.6, 22.3] 0.66 [0.56, 0.77] 31.5 [28.9, 34.2] 2.21 [1.93, 2.52]

20.3 [18.6, 22.2] 21.6 [19.9, 23.6] 0.55 [0.46, 0.65] 0.64 [0.54, 0.74] 33.5 [31.1, 36.2] 31.2 [28.7, 33.9] 1.93 [1.67, 2.23] 2.03 [1.77, 2.31]

30 PEDAP

iTr

21.8 [19.5, 24.3] 0.85 [0.62, 1.13] 32.6 [30.2, 35.2] 2.65 [2.17, 3.17]

30 DCLP5

DLin

retain only segments long enough to support the benchmark history window and forecasting task. The harmonized output uses a common schema based on subject identifier, sequence identifier, timestamp, glucose, and available auxiliary channels. This standardization allows the same forecasting interface and evaluation code to be applied across cohorts despite differences in the original source files. Full preprocessing details, parameter values, and cohort-specific output schemas are provided in Appendix A. Simulator data construction. For the simulator arm, standard trajectories are generated under the default MPC behavior policy and used for training and model selection; separate held-out counterfactual episodes are then constructed for response-prediction and bolus-selection evaluation. Full simulator details are provided in Appendix B.

5. Results We organize the results around the two evaluation arms introduced in Section 3. We first examine performance on real data and ask whether standard forecasting metrics are sufficient for clinically meaningful hypoglycemia early warning. We then turn to the counterfactual simulator-based evaluation and ask whether the same models can predict the effects of altered insulin actions and support insulin-action selection. 5.1. Experimental setup Unless otherwise stated, all real-data results are reported separately for each cohort, and we do not pool them into a single cross-cohort aggregate. For each cohort and metric, we use strict macro-averaging at the patient level. Main-text tables report means with 95% bootstrap confidence intervals. Additional slice-specific results, warning lead-time summaries, and extended counterfactual results are reported in Appendix C. 2. ZOH (zero-order hold) is included in the safety-gating evaluation (Table 3) as a naive baseline but omitted from Table 2 for space; its forecasting results appear in Appendix C.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 3: Hypoglycemia safety-gating performance at the 30-minute alarm horizon. Each cell is Recall/FA-day. Higher recall and lower false alarms per patientday are better. DLin

GRU

iTr

PMLP

Glim

ZOH

Overall

ARX

DCLP3 0.97/0.81 0.98/1.01 0.71/0.78 0.90/1.42 0.91/0.70 0.99/1.71 0.01/1.02 DCLP5 0.61/1.52 0.37/1.52 0.28/1.43 0.82/1.87 0.84/1.69 0.91/2.11 0.04/1.71 PEDAP 0.05/1.35 0.54/1.71 0.52/1.42 0.68/1.64 0.93/2.38 0.87/1.86 0.03/1.83

Post bolus

Slice Cohort

DCLP3 0.50/0.43 0.27/0.63 0.18/0.58 0.22/0.58 0.18/0.38 0.38/0.74 0.00/0.60 DCLP5 0.44/1.37 0.12/1.39 0.08/1.21 0.31/1.53 0.31/1.53 0.33/1.51 0.01/1.40 PEDAP 0.06/1.07 0.19/1.49 0.06/0.83 0.20/1.22 0.32/1.67 0.24/1.28 0.00/1.25

For the real-data arm, we compare ZOH (Karagoz et al., 2025), ARIMAX (Box et al., 2015), DLinear (Zeng et al., 2023), GRU-based forecasting (Alshehri et al., 2024), iTransformer (Liu et al., 2023), PatchMLP (Tang and Zhang, 2025), and Glimmer*3 . We report standard forecasting metrics at 30 and 60 minutes and use a 30-minute alarm horizon for hypoglycemia safety gating. For the counterfactual simulator-based arm, we evaluate ARIMAX, GRU, iTransformer, and Glimmer* on the adult UVA/Padova simulator using paired factual/counterfactual insulin-action episodes. The main text reports family-level summaries and the actionselection benchmark; per-perturbation and expanded-model results are reported in Appendix C.2. 5.2. First evaluation arm: performance on real data 5.2.1. Standard forecasting on real data Table 2 shows that no single model dominates across cohorts and horizons under standard forecasting metrics. At 30 minutes, GRU is best in RMSE on DCLP3 and PEDAP, whereas PatchMLP is best on DCLP5. At 60 minutes, iTransformer is best on DCLP3, PatchMLP remains best on DCLP5, and Glimmer* is best on PEDAP. ARIMAX is competitive at 30 minutes but degrades substantially at 60 minutes across all three cohorts. The more important pattern is that RMSE and PEG unsafe do not induce the same ordering. On DCLP3, Glimmer* achieves the lowest PEG unsafe fraction at both horizons without achieving the lowest RMSE. On PEDAP, PatchMLP has the lowest PEG unsafe fraction at both horizons even though GRU and Glimmer* are the best RMSE models at 30 and 60 minutes, respectively. By contrast, DCLP5 is the only cohort where the same model, PatchMLP, is strongest under both RMSE and PEG unsafe. Thus, even within standard forecasting, lower average error does not necessarily imply fewer clinically dangerous errors. A second point is that the confidence intervals are often overlapping among the leading models, especially on the harder cohorts, so the real-data standard forecasting differences are 3. Glimmer* denotes our benchmark-compatible implementation of the Glimmer loss (Khamesian et al., 2025) on top of PatchMLP (Tang and Zhang, 2025). The original Glimmer paper presents the loss as architecture-agnostic; we use PatchMLP as the base architecture to obtain a stable patient-level benchmark implementation.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

ARIMAX

GRU

iTransformer

(a) DCLP3

1.0

Event-level recall

DLinear

Glimmer ∗

PatchMLP

(b) DCLP5

ZOH

0.8 0.6 0.4 0.2 0.0 0.6

0.8

1.0

1.2

1.4

1.6

False alarms per day

1.8

1.4

1.6

1.8

2.0

False alarms per day

1.4

1.6

1.8

2.0

2.2

False alarms per day

2.4

Figure 2: Recall–false-alarm tradeoff for hypoglycemia safety gating on the overall slice. Each point is a model. The dashed line traces the Pareto frontier for event-level recall and false alarms per patient-day. The frontier makes the tradeoff between sensitivity and alarm burden explicit and shows that dominated models differ across cohorts.

modest relative to the sharper failures that appear in the safety-gating and counterfactual evaluations. Appendix C reports the extended slice-specific real-data results. 5.2.2. Safety gating on real data We next ask whether the same models are useful for coarse-grained safety gating. Table 3 reports event-level hypoglycemia recall and false alarms per patient-day at the 30-minute alarm horizon for both the overall test set and the post-bolus slice. Each cell is reported as Recall/FA-day. Median warning lead time and additional slices are reported in Appendix C. Figure 2 visualizes the recall–false-alarm tradeoff on the overall slice. Models on the Pareto frontier are not dominated under these two criteria, whereas dominated models are worse in both recall and alarm burden than at least one alternative. On DCLP3, GRU, iTransformer, and ZOH are dominated; on DCLP5, DLinear, iTransformer, and ZOH are dominated; and on PEDAP, DLinear and ZOH are dominated. The frontier makes clear that the best operating point is cohort-specific rather than universal. The overall slice suggests that several models are plausible early-warning candidates, but the preferred model depends on the tradeoff between recall and nuisance alarms. On DCLP3, PatchMLP offers the lowest false-alarm burden among the high-recall models, while Glimmer* achieves the highest recall at a substantially higher alarm cost. On DCLP5, Glimmer* has the highest overall recall, but GRU has the lowest false-alarm rate. On PEDAP, PatchMLP attains the highest overall recall, but at the cost of the largest falsealarm burden. ARIMAX is an extreme case on PEDAP: its forecasts rarely cross the 70 mg/dL alarm threshold, yielding the lowest false-alarm rate but near-zero recall (0.05), placing it on the Pareto frontier as a maximally conservative operating point. This already shows that overall safety gating cannot be reduced to a single scalar notion of “best model.”

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 4: Counterfactual forecasting performance across the perturbation menu at 120 minutes. eRMSE = effect RMSE, SA = sign agreement, and τb = Kendall rank correlation over the action menu. Each cell is reported as mean [95% CI]. Lower is better for eRMSE; higher is better for SA and τb . Best values in each column are bolded.

Bolus pert. Basal pert.

Model

eRMSE

τb

ARIMAX GRU iTransformer Glimmer*

9.83 [7.51, 12.24] 0.43 [0.28, 0.57] -0.17 [-0.46, 0.15] 48.72 [38.75, 57.90] 0.02 [0.00, 0.04] -1.00 [-1.00, -1.00] 54.91 [43.99, 65.06] 0.02 [0.00, 0.04] -0.99 [-1.00, -0.97] 49.87 [40.36, 58.32] 0.02 [0.00, 0.04] -0.98 [-1.00, -0.96]

ARIMAX GRU iTransformer Glimmer*

7.69 [5.54, 10.31] 0.78 [0.72, 0.85] -0.03 [-0.44, 0.33] 18.50 [14.36, 22.68] 0.14 [0.09, 0.19] -0.95 [-1.00, -0.85] 15.40 [11.66, 19.45] 0.15 [0.10, 0.20] -1.00 [-1.00, -1.00] 18.26 [13.43, 23.30] 0.31 [0.19, 0.43] -0.91 [-1.00, -0.79]

The post-bolus slice points to a potentially dangerous failure mode. Recall falls substantially for every cohort, and no single model dominates across all three datasets. ARIMAX has the highest post-bolus recall on DCLP3 and DCLP5, whereas PatchMLP is best on PEDAP. Even these best-case recalls remain limited: 0.50 on DCLP3, 0.44 on DCLP5, and 0.32 on PEDAP. This is the main safety finding of the first evaluation arm. A model can look acceptable on the overall slice and still miss a large fraction of events after bolus delivery, precisely when insulin-on-board is elevated and missed warnings are most concerning. Appendix C shows that the same conclusion persists in the complementary slice-specific results. 5.3. Second evaluation arm: counterfactual simulator-based evaluation We now turn to the second evaluation arm and ask whether the same models remain useful when the future insulin plan is explicitly given as an input and then perturbed away from the observational policy. This is the setting relevant to fine-grained decision support: the forecaster must not only extrapolate glucose trajectories, but also predict how glucose would change under alternative control actions. 5.3.1. Counterfactual response prediction Table 4 summarizes family-level performance on the paired factual/counterfactual benchmark at the 120-minute horizon. For each perturbation family, we report effect RMSE (eRMSE), sign agreement (SA), and Kendall’s rank correlation coefficient τb over the action menu. Detailed per-perturbation results and expanded-model comparisons are reported in Appendix C.2. The main result is not simply that performance worsens under altered insulin actions, but that the errors become qualitatively decision-damaging. For GRU, iTransformer, and Glimmer* , sign agreement is near zero under basal perturbations and the rank correlation is close to −1 for both basal and bolus menus. This indicates not merely noisy effect 12

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 5: Policy-regret benchmark AMR = action match rate and Regret = mean policy regret. Each cell is reported as mean [95% CI]. Higher is better for AMR; lower is better for Regret. Best values in each column are bolded. Model ARIMAX GRU iTransformer Glimmer*

AMR

Regret

0.68 [0.57, 0.79] 0.31 [0.23, 0.41] 0.09 [0.03, 0.16] 0.11 [0.05, 0.19]

0.18 [0.10, 0.26] 0.47 [0.33, 0.61] 0.69 [0.50, 0.89] 0.67 [0.48, 0.85]

estimation, but near-complete reversal of the action ordering. In other words, these models appear to treat insulin changes in the wrong direction under counterfactual evaluation. ARIMAX is the least-bad model in this experiment, but it does not solve the problem. It has the lowest eRMSE in both perturbation families, yet its basal sign agreement remains below 0.5 and its Kendall τb is still negative for both basal and bolus perturbations. Thus, even the strongest model under these metrics does not reliably preserve the direction or ranking of action effects. The central conclusion of this experiment is therefore that standard forecasting competence does not translate into reliable action-conditional prediction. 5.3.2. Bolus action selection and policy regret We next evaluate a more decision-oriented use of the counterfactual benchmark. Instead of asking only if the forecaster predicts the effect of a single perturbation, we ask whether it selects the best insulin dose from a discrete bolus menu. For each episode, we evaluate the nine candidate bolus values defined in Section 3.3, simulate all nine actions, and compare the model-chosen action against the simulator-optimal action under the BGRI-based cost. Table 5 reports action match rate and mean policy regret on the overall slice. ARIMAX is clearly strongest by these summary metrics, with the highest action match rate and the lowest regret. GRU is the only neural baseline that remains partially competitive, whereas iTransformer and Glimmer* rarely match the simulator-optimal action and incur substantially larger regret. However, this result should be interpreted together with Table 4. ARIMAX’s comparatively high action-match rate does not imply reliable mechanistic use of the control signal, because in the counterfactual response-prediction experiment it still shows below-chance sign agreement for basal perturbations and negative rank correlation for both perturbation families. This suggests that its strong action-selection result may partly reflect the structure of the discrete action menu and cost functional rather than consistently correct effect modeling. The broader point remains unchanged: even when one model is less bad than the others, action-conditional usefulness is a much stricter requirement than standard forecasting accuracy. Appendix C.2 provides the expanded counterfactual menu results, including additional model families and full confidence intervals.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

6. Discussion Main finding. The main finding of this paper is that usefulness in blood glucose forecasting is strongly task-dependent. Models that look competitive under standard forecasting metrics on real data can still fail under the two use cases that matter most here: hypoglycemia early warning and action-conditional decision support. The real-data arm shows that aggregate performance can hide failures in clinically important slices. The counterfactual arm shows that observational forecasting skill does not imply reliable prediction under altered insulin actions. Together, these results argue against judging glucose forecasters by average accuracy alone. Real-data findings and safety implications. In the real-data arm, the key result is the gap between overall safety-gating performance and post-bolus performance. On the overall slice, several models occupy plausible positions on the recall–false-alarm frontier, and the preferred model depends on the acceptable tradeoff between missed events and nuisance alarms. But this picture changes after bolus delivery, where recall drops markedly across all cohorts. This matters clinically because false alarms are not harmless. Repeated nuisance alarms can reduce quality of life, contribute to alarm fatigue, and make patients or caregivers less willing to respond to subsequent alarms (Shivers et al., 2013; Howsmon and Bequette, 2015). In diabetes care they may also trigger unnecessary rescue carbohydrate intake, which can worsen glycemic variability and erode trust in the system. For that reason, the first evaluation arm should be read not as a search for a single best early-warning model, but as evidence that overall summaries can overstate practical safety usefulness. Counterfactual findings, OOD prediction, and shortcut learning. The counterfactual arm presents a different and stricter challenge: out-of-distribution prediction under altered actions. In machine-learning terms, the model is asked to generalize beyond the action patterns it mostly sees under the observational policy. In decision-making terms, this is exactly the regime required for policy improvement. The connection to reinforcement learning is direct: choosing among actions requires reliable predictions in parts of the state–action space that are only weakly covered by past experience, which is the same challenge that makes exploration necessary in RL. Our results suggest that the neural forecasters largely fail in this regime. Their near-zero sign agreement and near-−1 rank correlations indicate that the learned representation is not merely imprecise, but often qualitatively wrong about how insulin changes future glucose. A plausible explanation is shortcut learning under observational data: high insulin often cooccurs with high glucose because the controller gives insulin in response to elevated glucose. A model that relies on such correlations can perform reasonably under the behavior policy while still treating insulin as a marker of rising glucose rather than as a driver that should reduce future glucose. This would explain why the deep models appear to reverse the action ordering under counterfactual evaluation. Overcoming this failure mode may require either mechanistic inductive biases grounded in the known physiology of insulin–glucose dynamics or training procedures informed by causal inference that can distinguish the effect of insulin from the confounded associations present in observational data. ARIMAX performs best in the counterfactual arm, but only in a relative sense. It is the least bad model, not a satisfactory one. Its negative rank correlations and below-chance

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

basal sign agreement show that it, too, does not use the control signal reliably enough for robust action-conditional prediction. One possible reason for its relative advantage is that its explicit autoregressive structure with exogenous inputs imposes a stronger low-capacity dynamical bias than the more flexible deep models. That bias may help with short-horizon extrapolation under limited training data, but our results do not support the stronger claim that ARIMAX is mechanistically accurate in any meaningful physiological sense. Scope and limitations. This paper is intentionally scoped as an evaluation study rather than a controller paper. We isolate the forecaster instead of embedding it in a closed-loop controller because controller-in-the-loop experiments can blur together forecasting quality, controller design, and simulator artifacts. A strong controller might compensate for a weak forecaster, or exploit quirks of the simulator, making it harder to diagnose what the forecaster itself has learned. For the same reason, the simulator arm should be interpreted as a controlled stress test of action-conditional prediction, not as a claim of deployment readiness. In addition, the present study focuses on point forecasting, so it does not address how predictive uncertainty should be incorporated into alarm design or decision support. Implications and future work. The clearest implication is that glucose forecasting models intended for decision support should be evaluated against the downstream task they are meant to serve, not only against aggregate forecasting error. If the goal is early warning, slice-aware alarm evaluation is necessary. If the goal is action selection, then out-of-distribution action-response prediction becomes central. The present results suggest that progress on this second problem may require stronger physiological inductive bias, hybrid mechanistic–learning approaches, or training objectives that explicitly target action-conditional robustness rather than observational fit alone. More broadly, they argue for moving from evaluating glucose forecasters only as predictors of future trajectories to evaluating them as components of decision-making systems with distinct operational requirements.

Data and Code Availability The benchmark code, including the standardized preprocessing pipeline, evaluation metrics, and model training scripts, is available at https://anonymous.4open.science/r/ bg-benchmark-anonymous-ADFA/. The real-data cohorts (DCLP3, DCLP5, and PEDAP) can be obtained through the Jaeb Center for Health Research at https://public.jaeb. org/datasets/diabetes under their standard data use agreement. The simulator-based counterfactual dataset will be released upon acceptance.

References Osama S Alshehri, Osama M Alshehri, and Hussein Samma. Blood glucose prediction using rnn, lstm, and gru: a comparative study. In 2024 IEEE International Conference on Advanced Systems and Emergent Technologies (IC ASET), pages 1–5. IEEE, 2024. George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Marc D. Breton, Lauren G. Kanapka, Roy W. Beck, et al. A randomized trial of closed-loop control in children with type 1 diabetes. The New England Journal of Medicine, 2020. doi: 10.1056/NEJMoa2004736. PMCID: PMC7920146. Steven A. Brown, Boris P. Kovatchev, Dario Raghinaru, et al. Six-month randomized, multicenter trial of closed-loop control in type 1 diabetes. The New England Journal of Medicine, 2019. doi: 10.1056/NEJMoa1907863. Bruce Buckingham, H Peter Chase, Eyal Dassau, Erin Cobry, Paula Clinton, Victoria Gage, Kimberly Caswell, John Wilkinson, Fraser Cameron, Hyunjin Lee, et al. Prevention of nocturnal hypoglycemia using predictive alarm algorithms and insulin pump suspension. Diabetes care, 33(5):1013–1017, 2010. Jessica R Castle and Roy W Beck. The importance of trial design in evaluating the performance of continuous glucose monitoring systems: Details matter. Journal of Diabetes Science and Technology, 18(3):741–742, 2024. William L Clarke, Daniel Cox, Linda A Gonder-Frederick, William Carter, and Stephen L Pohl. Evaluating clinical accuracy of systems for self-monitoring of blood glucose. Diabetes care, 10(5):622–628, 1987. Thomas Danne, Revital Nimri, Tadej Battelino, Richard M Bergenstal, Kelly L Close, J Hans DeVries, Satish Garg, Lutz Heinemann, Irl Hirsch, Stephanie A Amiel, et al. International consensus on use of continuous glucose monitoring. Diabetes care, 40(12): 1631–1640, 2017. Simone Del Favero, Andrea Facchinetti, and Claudio Cobelli. A glucose-specific metric to assess predictors and identify models. IEEE transactions on biomedical engineering, 59 (5):1281–1290, 2012. Andrea Facchinetti. Continuous glucose monitoring sensors: past, present and future algorithmic challenges. Sensors, 16(12):2093, 2016. Courtney Fischer. Open ecosystem through secure plug and play interoperability: An overview. Journal of Diabetes Science and Technology, 2025. doi: 10.1177/ 19322968241310253. Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, and Jenna Wiens. Deep multioutput forecasting: Learning to accurately predict blood glucose trajectories. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1387–1395, 2018. Guido Freckmann, Stefan Pleus, Mike Grady, Steven Setford, and Brian Levy. Measures of accuracy for continuous glucose monitoring and blood glucose monitoring devices. Journal of Diabetes Science and Technology, 13(3):575–583, 2019. Jose Garcia-Tirado, Dayu Lv, John P Corbett, Patricio Colmegna, and Marc D Breton. Advanced hybrid artificial pancreas system improves on unannounced meal response-in silico comparison to currently available system. Computer Methods and Programs in Biomedicine, 211:106401, 2021. 16

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Hadia Hameed and Samantha Kleinberg. Comparing machine learning techniques for blood glucose forecasting using free-living and patient generated data. In Proceedings of the 5th Machine Learning for Healthcare Conference, volume 126 of Proceedings of Machine Learning Research, pages 871–894. PMLR, 2020. URL https://proceedings. mlr.press/v126/hameed20a.html. Rebecca A Harvey, Eyal Dassau, Howard C Zisser, Wendy Bevier, Dale E Seborg, Lois Jovanovič, and Francis J Doyle III. Clinically relevant hypoglycemia prediction metrics for event mitigation. Diabetes Technology & Therapeutics, 14(8):719–727, 2012. Pau Herrero, Magı́ Andorrà, Marc D Breton, Ajandek Peak, Matthias Koehler, Yannick Klopfenstein, Eemeli Leppäaho, Mattia Zanon, Christian Ringemann, Patrick Lustenberger, et al. Glucose predictions improve glycemic control: A digital twin evaluation. Diabetes Technology & Therapeutics, page 15209156261432144, 2026. Daniel Howsmon and B Wayne Bequette. Hypo-and hyperglycemic alarms: devices and algorithms. Journal of diabetes science and technology, 9(5):1126–1137, 2015. Wei Huang, Ni Fan, Weiping Wang, Jinqiang Wang, Xiaojuan Qi, and Shiming Zhang. Improving long-term glucose prediction accuracy with uncertainty-estimated probsparsetransformer. Advanced Intelligent Systems, 7(12):e202500235, 2025. Meryem Altin Karagoz, Marc D Breton, and Anas El Fathi. A comparative study of transformer-based models for multi-horizon blood glucose prediction. IFACPapersOnLine, 59(2):155–160, 2025. Saman Khamesian, Asiful Arefeen, Maria Adela Grando, Bithika M Thompson, and Hassan Ghasemzadeh. Type 1 diabetes management using glimmer: Glucose level indicator model with modified error rate. arXiv preprint arXiv:2502.14183, 2025. David C Klonoff, Courtney Lias, Robert Vigersky, William Clarke, Joan Lee Parkes, David B Sacks, M Sue Kirkman, Boris Kovatchev, and Error Grid Panel. The surveillance error grid. Journal of diabetes science and technology, 8(4):658–672, 2014. Boris P Kovatchev, Daniel J Cox, Linda A Gonder-Frederick, and William Clarke. Symmetrization of the blood glucose measurement scale and its applications. Diabetes care, 20(11):1655–1658, 1997. Boris P. Kovatchev, Marc Breton, Chiara Dalla Man, and Claudio Cobelli. In silico preclinical trials: A proof of concept in closed-loop control of type 1 diabetes. Journal of Diabetes Science and Technology, 2009. PMCID: PMC2681269. Jung Min Lee, Rodica Pop-Busui, Joyce M Lee, Jesper Fleischer, and Jenna Wiens. Shortcomings in the evaluation of blood glucose forecasting. IEEE Transactions on Biomedical Engineering, 71(12):3424–3431, 2024. Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023. 17

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. Sun Joon Moon, Inha Jung, and Cheol-Young Park. Current advances of artificial pancreas systems: a comprehensive review of the clinical evidence. Diabetes & metabolism journal, 45(6):813–839, 2021. Joan L Parkes, Stephen L Slatin, Scott Pardo, and Barry H Ginsberg. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes care, 23(8):1143–1148, 2000. Sarah Puhr, Mark Derdzinski, John B Welsh, Andrew Scott Parker, Tomas Walker, and David A Price. Real-world hypoglycemia avoidance with a continuous glucose monitoring system’s predictive low glucose alert. Diabetes technology & therapeutics, 21(4):155–158, 2019. Renat Sergazinov, Mohammadreza Armandpour, and Irina Gaynanova. Gluformer: Transformer-based personalized glucose forecasting with uncertainty quantification. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. Renat Sergazinov, Edwin Chun, and Vyacheslav Rogovchenko. Glucobench: Curated list of continuous glucose monitoring datasets with prediction benchmarks. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/ forum?id=cUSNs8nGaV. Viral N Shah, Lori M Laffel, R Paul Wadwa, and Satish K Garg. Performance of a factorycalibrated real-time continuous glucose monitoring system utilizing an automated sensor applicator. Diabetes Technology & Therapeutics, 20(6):428–433, 2018. Joseph P Shivers, Linda Mackowiak, Henry Anhalt, and Howard Zisser. “turn it off!”: diabetes device alarm fatigue considerations for the present and the future. Journal of diabetes science and technology, 7(3):789–794, 2013. Peiwang Tang and Weitai Zhang. Unlocking the power of patch: Patch-based mlp for long-term time series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 39, pages 12640–12648, 2025. Roberto Visentin, Chiara Dalla Man, Boris Kovatchev, and Claudio Cobelli. The university of virginia/padova type 1 diabetes simulator matches the glucose traces of a clinical trial. Diabetes Technology & Therapeutics, 2014. PMCID: PMC4074748. R. Paul Wadwa et al. Trial of hybrid closed-loop control in young children with type 1 diabetes. The New England Journal of Medicine, 2023. PMCID: PMC10082994. J B Welsh, P Gao, M Derdzinski, et al. Accuracy, utilization, and effectiveness comparisons of different continuous glucose monitoring systems. Diabetes Technology & Therapeutics, 21(3):128–132, 2019.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Miriam K Wolff, Hans Georg Schaathun, Sebastien Gros, Rune Volden, Martin Steinert, and Anders L Fougner. Blood glucose prediction algorithms require clinically relevant performance criteria beyond accuracy. Diabetes Technology & Therapeutics, 27(10):858– 870, 2025. Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121–11128, 2023.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Appendix A. Preprocessing and Harmonization Details This appendix specifies the harmonization procedures used to convert each cohort into a common multivariate time-series format suitable for patient-level evaluation and crosscohort comparability. Our goal is to (i) enforce a uniform sampling grid, (ii) standardize handling of missingness, and (iii) align auxiliary channels (insulin, meals, anthropometrics) to glucose timestamps using rules consistent with their semantics (continuous rates vs. sparse events). All parameter values used in the main real-data experiments are summarized in Table 6. A.1 Unified representation For each cohort, we produce a set of contiguous sequences indexed by subject (pat id) and sequence identifier (seq id). Each sequence is represented on a regular time grid with step size ∆ = 5 minutes and includes at minimum glucose gt and, when available, basal , and optional , meal/carbohydrate events umeal , bolus insulin events ubolus insulin rate ubasal t t t slowly varying covariates (e.g., weight). Timestamps are normalized to a cohort-consistent, monotone time axis within each subject. A.2 Harmonization for pump/CGM clinical-trial cohorts (DCLP3, DCLP5, PEDAP) DCLP3, DCLP5, and PEDAP share a common harmonization pipeline with cohort-specific parsers for vendor exports. The following steps are applied independently per subject. CGM validation and within-subject deduplication. We retain numeric glucose observations with valid timestamps and sort records by time. To reduce duplicate exports and near-duplicate records, we collapse clusters of glucose records within a short tolerance window, retaining the most recent record in each cluster. Segmentation on prolonged missingness. We split each subject’s glucose stream into contiguous segments whenever the gap between consecutive valid glucose measurements exceeds a fixed threshold. Segmentation prevents interpolation across prolonged periods where the sensor was absent or disconnected. Resampling to a uniform grid and interpolation within segments. Each segment is reindexed to a regular grid at ∆ = 5 minutes. Glucose values are linearly interpolated in time within each segment (i.e., between observed values that lie inside the same segment). No interpolation is performed across segment boundaries. Basal insulin alignment. Basal insulin is treated as a piecewise-constant rate signal. For each grid time t, we assign the basal rate as the most recent recorded basal value in a lookback window relative to t. After alignment, basal is forward-filled within each segment; if basal is unavailable at the beginning of a segment, the initial value is set to zero.4 4. This convention reflects missing pump records rather than true physiology; sensitivity to this choice can be explored by alternative initializations.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Bolus insulin alignment and representation. Bolus insulin is treated as a sparse event channel. For each grid time t, we aggregate bolus delivery records that fall within a short alignment window around t and assign the total delivered amount to the corresponding time step; when no events occur in the window, the bolus value is set to zero. When pump metadata distinguishes delivery types (e.g., standard vs. extended bolus), we store them in separate channels. Long “no-bolus” spans. To reduce the impact of extended stretches that are inconsistent with typical pump usage or that reflect missing pump logs, we optionally remove time spans that occur beyond a maximum duration since the last observed nonzero bolus event. Weight alignment (optional). When available, weight is treated as a slowly varying covariate. We align weight records to the glucose timeline by nearest-neighbor matching in time and interpolate linearly between measurements to obtain a value at each grid time. Meal channel. When meal or carbohydrate information is available in cohort records, we construct a sparse meal event channel aligned to the grid (grams when available; otherwise an event indicator). Missing meal values are represented as zero. Minimum-length filtering and sequence identifiers. We retain only segments long enough to support the benchmark history window and the maximum lead time used in evaluation. Each retained segment is assigned a cohort-global seq id. Output schema. For DCLP3/DCLP5/PEDAP the harmonized output includes: pat id, seq id, date, cgm, basal, bolus standard, bolus extended, weight kg, meal. Sequence assembly and time normalization. Each subject record is mapped to a unique pat id. We construct a monotone time axis, assign a unique seq id per subject sequence, and standardize units where needed. Insulin and meal channels. Insulin-related entries are parsed into event channels distinguishing, when possible, short-acting vs. long-acting and subcutaneous vs. intravenous delivery. Dietary intake is represented as a sparse event indicator (binary) unless quantitative carbohydrate values are available. All event channels are aligned to the resampled grid by assigning events to the corresponding time step and setting absent events to zero. Resampling. Sequences are reindexed to the common 5-minute grid. Glucose is interpolated linearly, continuous therapy-like streams are forward-filled, and event channels default to zero outside event times. Output schema. The harmonized output includes: pat id, seq id, date, cgm, basal, bolus short, bolus long, bolus iv short, bolus iv long, iv glucose g, weight kg, meal. A.4 Standard UVA/Padova simulator trajectories For the interventional evaluation, we first generate standard simulator trajectories under the default MPC behavior policy. Standard trajectories are exported onto the same 5-minute grid and represented using the same basic schema as the real-data cohorts, with glucose and 21

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Parameter

Value

Deduplication tolerance (within-subject) Segment split gap (CGM missingness) Resampling grid Basal alignment lookback/lookahead window Bolus alignment window Maximum “no-bolus” span filter (optional) Minimum segment length (DCLP3/DCLP5/PEDAP) Standard simulator duration per virtual subject Virtual-subject split (simulator)

15 seconds 30 minutes 5 minutes 3 hours before, 15 seconds after 285 seconds before, 15 seconds after 12 hours 312 steps (24h history + 2h horizon) 15 days 70/10/20

Table 6: Key preprocessing and standard simulator parameters used in the main experiments.

insulin channels aligned to the benchmark time axis. These standard trajectories are used for model training and model selection in the simulator arm. Paired factual/counterfactual perturbation episodes and the bolus action-selection benchmark are generated separately for held-out virtual subjects and are described in Appendix B.

Appendix B. Interventional Simulator Benchmark Details This appendix records the construction of the simulator-based interventional benchmark and the implementation details of the interventional metrics defined in Sections 3.3. We use the FDA-accepted UVA/Padova Type 1 diabetes simulator under the default MPC behavior policy (Garcia-Tirado et al., 2021). B.1 Standard simulator data We use the adult virtual cohort only (N = 100). Virtual subjects are split into disjoint train/validation/test sets with a 70/10/20 ratio. For each subject, we generate 15 days of standard simulator data under the default behavior policy. Models used in the interventional arm are trained from scratch on the standard simulator training set and selected using the standard simulator validation set only. B.2 Paired episode construction For each virtual subject, we sample four perturbation onset times on day 15, subject to a minimum spacing constraint, and construct one episode per onset. Each episode spans two hours after perturbation onset. The factual rollout follows the default MPC insulin plan. Counterfactual rollouts share the same subject, seed, pre-onset history, and future meals, but modify the future insulin plan according to the perturbation menu below. This yields paired factual/counterfactual episodes aligned at the same onset time.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

B.3 Action-conditional inputs In the interventional task, the forecaster receives the same historical multivariate window used in the real-data benchmark together with the planned future insulin sequence over the prediction horizon. Future meals are held fixed across factual and counterfactual rollouts and are not treated as control actions. The benchmark therefore isolates insulin-response prediction under policy shift rather than full behavior forecasting. B.4 Experiment 1: perturbation-response benchmark The first interventional experiment evaluates whether the forecaster predicts the response to altered insulin actions. We use two perturbation families: basal and bolus. Basal perturbations are always valid; bolus perturbations are applied when the corresponding target bolus exists and the modified action remains inside the two-hour episode window. Invalid scenarios are excluded from metric computation. Basal perturbations. Basal perturbations begin immediately at perturbation onset and persist until the end of the two-hour episode: Type

Operation

Value

Basal suspension Basal reduction Basal increase Strong basal increase

Multiply basal rate Multiply basal rate Multiply basal rate Multiply basal rate

0× 0.5× 1.5× 2×

Table 7: Basal perturbation menu used in Experiment 1. Bolus perturbations. perturbations:

When a target bolus is available, we apply the following discrete

Type

Operation

Value

Remove bolus Scale down Scale up Advance bolus Delay bolus Add bolus

Set target bolus to zero Multiply target bolus Multiply target bolus Shift target bolus earlier Shift target bolus later Add extra bolus at onset

0× 0.5× 2× −30 min +30 min patient mean bolus

Table 8: Bolus perturbation menu used in Experiment 1.

B.5 Experiment 2: bolus action-selection benchmark The second interventional experiment evaluates whether the forecaster can select the best bolus action from a discrete menu. For each episode, we evaluate the following bolus scales applied at the start of the prediction window: {0, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2} × avg bolus. 23

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

For each candidate action, we run the simulator to obtain the factual glucose response and compare the model-preferred action against the simulator-optimal action under the BGRI-based cost in Eq. 7. B.6 Evaluation horizons Although each episode spans two hours after perturbation onset, the effect-prediction metrics in Experiment 1 are evaluated at 30, 60, and 120 minutes. Policy regret in Experiment 2 is evaluated over the full two-hour episode window. B.7 Metric implementation details Sections 3.3 define the interventional metrics. Here we record only the implementation details needed for reproduction. Effect metrics. Effect RMSE and sign agreement are computed on factual/counterfactual pairs at each evaluation horizon. Ordinary RMSE on factual and counterfactual trajectories is also recorded as a secondary diagnostic. Rank correlation. For Kendall’s τb in Experiment 1, the action set A consists of the valid members of the perturbation menu for a given episode and perturbation family. Rank correlation is computed separately for basal and bolus perturbation families at each evaluation horizon. Action match rate. For Experiment 2, action match rate is the fraction of episodes in which the model-selected bolus scale matches the simulator-optimal bolus scale under the BGRI-based objective. Policy regret. Policy regret is computed over the full two-hour episode window using the BGRI-based cost defined in Eqs. 6–7. To avoid numerical issues, predicted glucose values are clipped to a minimum of 20 mg/dL before applying the BGRI transform. When multiple actions achieve the same minimum predicted cost, we break ties by choosing the action closest to the factual plan. Aggregation. All interventional metrics are first computed at the episode level and then macro-averaged at the subject level. Reported confidence intervals are obtained by subjectlevel bootstrapping.

Appendix C. Additional Results C.1. Extended Real-Data Forecasting and Safety-Gating Results This subsection extends Tables 2 and 3 with slice-specific safety-gating results at the same 30- and 60-minute horizons used in the main text. For each slice, we report RMSE, PEG Unsafe%, event-level hypoglycemia recall, false alarms per patient-day, and median warning lead time, all as patient-level macro means with 95% bootstrap confidence intervals. These tables provide the complementary slice-specific detail referenced in Sections 5.2.1 and 5.2.2. ‘NA’ indicates that median warning lead time is undefined because no events were detected in that slice.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 9: Extended real-data results on the overall test set. Unsafe% refers to forecasts falling in Parkes C–E zones. Recall is event-level hypoglycemia recall, FA/day is false alarms per patient-day, and Median Lead is the median warning lead time in minutes. Each cell is reported as mean [95% CI]. Lower is better for RMSE, Unsafe%, and FA/day; higher is better for Recall and Median Lead. Cohort H Metric

ARX

DLin

GRU

iTr

PMLP

Glim

ZOH

RMSE 15.1 [13.8, 16.5] 13.5 [12.3, 15.1] 13.4 [12.5, 14.3] 13.5 [12.6, 14.3] 14.6 [13.7, 15.3] 14.4 [13.6, 15.1] 17.7 [16.5, 18.9] Unsafe% 0.10 [0.07, 0.14] 0.08 [0.05, 0.12] 0.11 [0.08, 0.16] 0.08 [0.07, 0.10] 0.09 [0.07, 0.12] 0.06 [0.05, 0.08] 0.19 [0.14, 0.26] 30 Recall 0.97 [0.96, 0.98] 0.98 [0.97, 0.99] 0.70 [0.64, 0.76] 0.90 [0.85, 0.94] 0.91 [0.88, 0.94] 0.99 [0.98, 1.00] 0.01 [0.01, 0.02] FA/day 0.81 [0.60, 1.08] 1.01 [0.77, 1.27] 0.78 [0.53, 1.08] 1.42 [0.90, 2.24] 0.70 [0.50, 0.94] 1.71 [1.26, 2.30] 1.02 [0.72, 1.40] Median Lead 5.8 [5.2, 6.7] 9.7 [9.2, 10.0] 5.3 [5.0, 5.8] 9.8 [8.3, 11.7] 5.9 [5.2, 6.7] 14.2 [12.8, 15.8] 23.2 [19.4, 26.4] DCLP3 RMSE 31.3 [29.7, 32.7] 23.8 [22.0, 25.4] 24.2 [22.3, 26.1] 23.5 [22.0, 24.9] 25.2 [23.7, 26.6] 25.7 [24.3, 27.0] 28.6 [26.6, 30.5] Unsafe% 0.82 [0.63, 1.06] 0.76 [0.58, 1.05] 0.94 [0.72, 1.21] 0.76 [0.66, 0.87] 0.51 [0.42, 0.60] 0.45 [0.37, 0.53] 1.18 [0.92, 1.53] 60 Recall 0.99 [0.98, 0.99] 0.86 [0.83, 0.89] 0.03 [0.01, 0.08] 0.65 [0.59, 0.72] 0.98 [0.97, 0.99] 0.99 [0.98, 1.00] 0.04 [0.02, 0.05] FA/day 1.98 [1.58, 2.44] 0.52 [0.41, 0.64] 0.11 [0.06, 0.18] 0.90 [0.52, 1.53] 1.29 [0.99, 1.68] 1.82 [1.30, 2.64] 0.69 [0.51, 0.92] Median Lead 22.5 [19.8, 25.4] 8.0 [7.1, 9.0] 11.4 [5.0, 18.8] 12.3 [10.2, 15.1] 19.6 [17.4, 22.5] 23.3 [20.8, 26.3] 36.7 [31.1, 42.1] RMSE Unsafe% 30 Recall FA/day Median Lead DCLP5 RMSE Unsafe% 60 Recall FA/day Median Lead

25.4 [22.7, 28.2] 0.75 [0.57, 0.94] 0.61 [0.58, 0.64] 1.52 [1.22, 1.84] 5.0 [5.0, 5.0] 37.7 [35.2, 40.3] 2.30 [1.95, 2.68] 0.76 [0.70, 0.82] 1.28 [1.05, 1.59] 6.8 [5.5, 8.0]

24.2 [21.5, 27.2] 21.8 [19.5, 24.3] 0.81 [0.62, 1.03] 0.85 [0.62, 1.13] 0.37 [0.34, 0.41] 0.28 [0.24, 0.32] 1.52 [1.22, 1.84] 1.43 [1.13, 1.72] 5.0 [5.0, 5.0] 6.0 [5.0, 7.5] 35.1 [32.4, 37.8] 32.6 [30.2, 35.2] 2.41 [2.05, 2.80] 2.65 [2.17, 3.17] 0.41 [0.38, 0.44] 0.00 [0.00, 0.00] 1.10 [0.92, 1.29] 0.06 [0.02, 0.10] 5.6 [5.0, 6.5] 46.7 [35.0, 55.0]

22.2 [19.4, 25.3] 0.73 [0.51, 0.98] 0.82 [0.77, 0.87] 1.87 [1.53, 2.22] 9.4 [8.6, 10.0] 32.9 [30.1, 35.9] 2.31 [1.91, 2.74] 0.73 [0.69, 0.78] 1.34 [1.15, 1.55] 8.8 [7.5, 9.9]

21.6 [19.1, 24.4] 23.4 [20.7, 26.3] 26.1 [23.9, 28.5] 0.72 [0.51, 0.97] 0.92 [0.67, 1.23] 0.88 [0.70, 1.08] 0.84 [0.80, 0.88] 0.91 [0.87, 0.95] 0.04 [0.02, 0.05] 1.69 [1.37, 2.02] 2.11 [1.78, 2.46] 1.71 [1.38, 2.07] 7.9 [6.8, 8.9] 10.2 [10.0, 10.8] 20.8 [18.5, 22.9] 32.5 [30.0, 35.3] 33.3 [30.8, 36.1] 38.2 [35.8, 40.7] 2.22 [1.79, 2.67] 2.50 [2.04, 3.01] 2.70 [2.32, 3.11] 0.87 [0.82, 0.90] 0.86 [0.81, 0.91] 0.08 [0.06, 0.10] 1.74 [1.55, 1.94] 1.88 [1.66, 2.12] 1.29 [1.07, 1.51] 12.6 [11.2, 14.1] 14.2 [13.0, 15.4] 39.6 [34.8, 43.2]

RMSE 24.6 [22.5, 26.9] 22.7 [20.7, 24.9] 20.1 [18.4, 21.9] Unsafe% 0.87 [0.76, 1.00] 0.63 [0.54, 0.74] 0.63 [0.53, 0.74] 30 Recall 0.05 [0.03, 0.06] 0.54 [0.50, 0.58] 0.52 [0.48, 0.56] FA/day 1.35 [1.13, 1.61] 1.71 [1.48, 1.97] 1.42 [1.23, 1.66] Median Lead 8.2 [5.6, 11.4] 5.0 [5.0, 5.0] 5.0 [5.0, 5.0] PEDAP RMSE 37.9 [35.0, 40.9] 34.2 [31.6, 37.1] 31.7 [29.4, 34.1] Unsafe% 3.23 [2.86, 3.61] 2.69 [2.35, 3.06] 2.80 [2.44, 3.17] 60 Recall 0.04 [0.03, 0.06] 0.48 [0.44, 0.51] 0.15 [0.13, 0.18] FA/day 0.87 [0.72, 1.05] 1.23 [1.07, 1.39] 0.43 [0.35, 0.55] Median Lead 34.2 [26.7, 41.3] 5.1 [5.0, 5.4] 5.4 [5.0, 6.0]

20.3 [18.6, 22.3] 0.66 [0.56, 0.77] 0.68 [0.64, 0.72] 1.64 [1.40, 1.89] 5.2 [5.0, 5.7] 31.5 [28.9, 34.2] 2.21 [1.93, 2.52] 0.74 [0.69, 0.79] 1.42 [1.23, 1.62] 9.3 [8.3, 10.2]

20.3 [18.6, 22.2] 21.6 [19.9, 23.6] 25.1 [23.1, 27.3] 0.55 [0.46, 0.65] 0.64 [0.54, 0.74] 0.85 [0.73, 1.00] 0.93 [0.91, 0.94] 0.87 [0.84, 0.89] 0.03 [0.02, 0.03] 2.38 [2.04, 2.72] 1.86 [1.60, 2.14] 1.83 [1.57, 2.13] 10.0 [10.0, 10.0] 8.9 [8.1, 9.8] 21.4 [19.6, 23.0] 33.5 [31.1, 36.2] 31.2 [28.7, 33.9] 37.9 [35.0, 40.9] 1.93 [1.67, 2.23] 2.03 [1.77, 2.31] 3.01 [2.65, 3.42] 0.96 [0.95, 0.97] 0.94 [0.92, 0.95] 0.06 [0.05, 0.07] 2.51 [2.22, 2.80] 1.80 [1.58, 2.04] 1.44 [1.26, 1.65] 14.6 [13.8, 15.6] 10.7 [10.0, 11.7] 40.4 [35.4, 45.2]

C.2. Expanded Counterfactual Menu Results This subsection extends Table 4 by adding DLinear and PatchMLP to the 120-minute counterfactual menu benchmark.

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 10: Extended real-data results on the post-bolus slice. Unsafe% refers to forecasts falling in Parkes C–E zones. Recall is event-level hypoglycemia recall, FA/day is false alarms per patient-day, and Median Lead is the median warning lead time in minutes. Each cell is reported as mean [95% CI]. Lower is better for RMSE, Unsafe%, and FA/day; higher is better for Recall and Median Lead. Cohort H Metric

ARX

DLin

GRU

iTr

PMLP

Glim

ZOH

RMSE 18.2 [16.6, 20.3] 16.0 [14.3, 18.6] 15.6 [14.5, 16.6] 15.7 [14.7, 16.6] 17.5 [16.5, 18.3] 17.5 [16.5, 18.3] 21.3 [19.9, 22.5] Unsafe% 0.13 [0.10, 0.16] 0.08 [0.06, 0.11] 0.10 [0.07, 0.14] 0.10 [0.07, 0.13] 0.10 [0.07, 0.13] 0.09 [0.06, 0.13] 0.26 [0.21, 0.32] 30 Recall 0.50 [0.41, 0.59] 0.58 [0.50, 0.65] 0.44 [0.36, 0.54] 0.48 [0.39, 0.58] 0.46 [0.37, 0.56] 0.64 [0.54, 0.73] 0.01 [0.00, 0.02] FA/day 0.43 [0.27, 0.63] 0.60 [0.42, 0.83] 0.39 [0.19, 0.70] 0.46 [0.29, 0.69] 0.30 [0.17, 0.47] 0.81 [0.60, 1.08] 0.46 [0.27, 0.69] Median Lead 7.6 [6.6, 8.6] 13.0 [11.4, 14.5] 7.0 [5.9, 8.3] 9.1 [7.9, 10.3] 6.9 [5.8, 7.9] 14.4 [12.6, 16.0] 20.0 [10.0, 30.0] DCLP3 RMSE 32.8 [31.0, 34.8] 27.6 [25.5, 29.8] 28.0 [26.2, 29.8] 27.6 [25.9, 29.1] 28.3 [26.7, 29.9] 28.4 [26.9, 29.8] 33.7 [31.7, 35.6] Unsafe% 0.91 [0.76, 1.07] 1.04 [0.85, 1.26] 1.12 [0.95, 1.32] 1.09 [0.89, 1.30] 0.62 [0.53, 0.72] 0.71 [0.52, 0.97] 1.74 [1.46, 2.07] 60 Recall 0.52 [0.43, 0.60] 0.36 [0.29, 0.44] 0.03 [0.00, 0.07] 0.22 [0.16, 0.30] 0.44 [0.37, 0.51] 0.51 [0.44, 0.59] 0.01 [0.00, 0.02] FA/day 1.14 [0.84, 1.51] 0.42 [0.30, 0.57] 0.09 [0.03, 0.17] 0.34 [0.22, 0.49] 0.65 [0.47, 0.90] 0.90 [0.68, 1.19] 0.40 [0.25, 0.58] Median Lead 21.9 [19.2, 24.8] 12.9 [10.8, 15.5] 14.2 [5.0, 32.5] 11.9 [9.5, 14.7] 18.3 [15.7, 21.1] 22.9 [20.1, 25.8] 30.6 [10.0, 48.1] RMSE Unsafe% 30 Recall FA/day Median Lead DCLP5 RMSE Unsafe% 60 Recall FA/day Median Lead

31.3 [27.9, 35.1] 1.22 [0.93, 1.57] 0.44 [0.40, 0.48] 1.37 [1.05, 1.73] 5.2 [5.0, 5.8] 44.5 [41.7, 47.5] 3.27 [2.74, 3.85] 0.41 [0.37, 0.46] 1.26 [1.06, 1.50] 5.9 [5.0, 7.1]

29.8 [26.3, 33.9] 1.23 [0.95, 1.60] 0.36 [0.32, 0.40] 1.36 [1.05, 1.72] 5.2 [5.0, 5.8] 42.1 [38.8, 45.7] 3.39 [2.84, 3.98] 0.31 [0.26, 0.35] 1.07 [0.86, 1.29] 6.0 [5.0, 7.2]

26.4 [23.7, 29.5] 26.9 [23.5, 30.9] 26.3 [23.2, 29.8] 28.5 [25.2, 32.2] 32.1 [29.2, 35.2] 1.21 [0.86, 1.62] 1.09 [0.77, 1.48] 1.10 [0.76, 1.49] 1.46 [1.06, 1.90] 1.42 [1.11, 1.75] 0.21 [0.17, 0.26] 0.63 [0.57, 0.68] 0.64 [0.59, 0.68] 0.71 [0.66, 0.75] 0.02 [0.01, 0.03] 1.19 [0.91, 1.48] 1.59 [1.26, 1.96] 1.42 [1.12, 1.73] 1.65 [1.36, 1.97] 1.49 [1.15, 1.83] 6.5 [5.0, 9.0] 7.5 [6.2, 8.5] 6.8 [5.8, 7.8] 9.4 [8.4, 10.4] 21.2 [18.4, 23.9] 39.0 [36.3, 42.0] 39.3 [36.0, 43.0] 38.5 [35.5, 41.8] 39.6 [36.6, 43.1] 46.1 [43.0, 49.2] 3.51 [2.84, 4.27] 3.30 [2.67, 3.97] 3.11 [2.46, 3.80] 3.68 [2.98, 4.41] 4.07 [3.45, 4.73] 0.00 [0.00, 0.00] 0.41 [0.37, 0.45] 0.51 [0.47, 0.55] 0.51 [0.46, 0.56] 0.04 [0.02, 0.06] 0.07 [0.02, 0.14] 1.25 [1.02, 1.51] 1.55 [1.33, 1.79] 1.75 [1.49, 2.06] 1.23 [0.99, 1.47] 52.5 [50.0, 55.0] 6.9 [5.8, 8.1] 9.9 [8.6, 11.1] 11.1 [10.0, 12.5] 43.5 [37.9, 48.5]

RMSE 30.8 [28.1, 33.8] Unsafe% 1.38 [1.17, 1.61] 30 Recall 0.06 [0.04, 0.10] FA/day 1.07 [0.89, 1.25] Median Lead 6.7 [5.0, 9.6] PEDAP RMSE 46.6 [43.2, 50.2] Unsafe% 4.77 [4.19, 5.38] 60 Recall 0.04 [0.03, 0.05] FA/day 0.75 [0.61, 0.89] Median Lead 27.1 [18.1, 37.1]

28.4 [25.7, 31.5] 0.96 [0.81, 1.13] 0.50 [0.46, 0.54] 1.53 [1.29, 1.77] 5.1 [5.0, 5.4] 41.4 [38.2, 44.9] 3.60 [3.14, 4.08] 0.38 [0.34, 0.41] 1.15 [0.97, 1.34] 5.2 [5.0, 5.7]

24.8 [22.6, 27.1] 0.89 [0.75, 1.05] 0.33 [0.28, 0.38] 1.03 [0.85, 1.21] 5.2 [5.0, 5.7] 38.3 [35.6, 41.3] 3.71 [3.23, 4.24] 0.11 [0.08, 0.14] 0.20 [0.15, 0.26] 5.6 [5.0, 6.5]

25.0 [22.7, 27.7] 0.95 [0.80, 1.10] 0.49 [0.44, 0.55] 1.31 [1.07, 1.54] 5.5 [5.0, 6.1] 37.7 [34.7, 41.0] 3.06 [2.66, 3.46] 0.41 [0.37, 0.45] 1.29 [1.04, 1.55] 8.0 [6.8, 9.3]

25.1 [22.8, 27.6] 27.0 [24.6, 29.5] 31.2 [28.7, 33.9] 0.80 [0.67, 0.93] 1.05 [0.88, 1.22] 1.36 [1.15, 1.60] 0.72 [0.67, 0.76] 0.61 [0.57, 0.64] 0.01 [0.01, 0.02] 1.94 [1.61, 2.26] 1.41 [1.15, 1.65] 1.48 [1.22, 1.71] 9.3 [8.3, 10.2] 7.4 [6.1, 9.0] 21.6 [19.1, 24.0] 39.8 [36.8, 43.1] 37.5 [34.5, 40.8] 46.0 [42.7, 49.5] 2.59 [2.24, 2.94] 2.86 [2.48, 3.23] 4.48 [3.93, 5.06] 0.67 [0.63, 0.70] 0.64 [0.60, 0.67] 0.04 [0.03, 0.05] 2.29 [1.91, 2.66] 1.77 [1.47, 2.07] 1.29 [1.08, 1.51] 12.1 [11.0, 13.6] 10.2 [9.5, 11.2] 46.2 [41.7, 50.5]

A Task-Aware Evaluation Framework for Blood Glucose Forecasting

Table 11: Extended real-data results on the nocturnal slice. Unsafe% refers to forecasts falling in Parkes C–E zones. Recall is event-level hypoglycemia recall, FA/day is false alarms per patient-day, and Median Lead is the median warning lead time in minutes. Each cell is reported as mean [95% CI]. Lower is better for RMSE, Unsafe%, and FA/day; higher is better for Recall and Median Lead. Cohort H Metric

ARX

DLin

GRU

iTr

PMLP

Glim

ZOH

RMSE 10.7 [9.7, 11.8] 10.3 [9.5, 11.1] 10.8 [9.9, 11.7] 10.2 [9.3, 11.1] 12.1 [11.3, 13.0] 11.4 [10.5, 12.3] 12.6 [11.4, 14.1] Unsafe% 0.04 [0.02, 0.06] 0.05 [0.02, 0.07] 0.09 [0.04, 0.14] 0.06 [0.03, 0.10] 0.08 [0.04, 0.12] 0.05 [0.02, 0.08] 0.09 [0.04, 0.18] 30 Recall 0.96 [0.93, 0.98] 0.95 [0.91, 0.98] 0.71 [0.61, 0.81] 0.86 [0.71, 0.97] 0.85 [0.77, 0.91] 0.97 [0.93, 1.00] 0.02 [0.01, 0.04] FA/day 0.86 [0.56, 1.26] 0.99 [0.65, 1.40] 0.90 [0.55, 1.32] 1.07 [0.74, 1.44] 0.66 [0.43, 0.93] 1.16 [0.80, 1.56] 1.11 [0.62, 1.86] Median Lead 6.9 [6.0, 7.9] 8.0 [6.7, 9.2] 6.2 [5.4, 7.2] 10.0 [7.3, 13.1] 7.1 [6.0, 8.4] 14.5 [12.0, 16.9] 24.6 [20.9, 28.8] DCLP3 RMSE 23.9 [21.8, 26.5] 17.5 [16.1, 19.0] 17.8 [16.2, 19.7] 16.7 [15.5, 18.0] 19.4 [18.3, 20.6] 19.6 [18.6, 20.7] 20.1 [17.9, 22.9] Unsafe% 0.35 [0.14, 0.74] 0.25 [0.17, 0.34] 0.54 [0.33, 0.79] 0.37 [0.24, 0.53] 0.25 [0.16, 0.35] 0.15 [0.09, 0.21] 0.53 [0.23, 1.04] 60 Recall 0.91 [0.80, 0.98] 0.65 [0.55, 0.76] 0.05 [0.01, 0.10] 0.62 [0.45, 0.76] 0.94 [0.88, 0.98] 0.94 [0.88, 0.98] 0.06 [0.02, 0.10] FA/day 1.72 [1.27, 2.21] 0.50 [0.34, 0.70] 0.11 [0.05, 0.18] 0.60 [0.41, 0.81] 0.95 [0.68, 1.24] 1.20 [0.84, 1.60] 0.68 [0.41, 1.06] Median Lead 24.4 [20.2, 28.7] 8.1 [5.9, 11.1] 15.5 [5.0, 30.0] 16.9 [12.4, 21.4] 24.5 [20.2, 29.2] 26.5 [22.1, 31.5] 41.6 [34.4, 48.9] RMSE 17.6 [14.1, 21.7] 17.7 [14.2, 21.7] 16.1 [13.1, 19.6] 16.4 [13.2, 20.2] 16.1 [12.9, 19.5] 17.4 [14.4, 20.8] 18.0 [14.9, 21.6] Unsafe% 0.27 [0.14, 0.43] 0.35 [0.21, 0.53] 0.51 [0.30, 0.75] 0.37 [0.19, 0.57] 0.36 [0.18, 0.56] 0.45 [0.22, 0.69] 0.30 [0.17, 0.45] 30 Recall 0.56 [0.49, 0.64] 0.19 [0.14, 0.23] 0.27 [0.17, 0.39] 0.67 [0.58, 0.76] 0.75 [0.68, 0.83] 0.86 [0.81, 0.92] 0.06 [0.03, 0.08] FA/day 1.24 [0.95, 1.57] 1.22 [0.95, 1.51] 1.17 [0.92, 1.42] 1.63 [1.27, 2.00] 1.48 [1.16, 1.80] 1.92 [1.54, 2.33] 1.29 [0.99, 1.60] Median Lead 6.0 [5.0, 7.4] 8.3 [5.9, 11.1] 7.4 [5.8, 9.2] 12.1 [10.0, 14.5] 8.8 [7.6, 9.8] 12.4 [10.8, 14.1] 21.9 [18.7, 25.3] DCLP5 RMSE 25.5 [21.4, 30.2] 24.5 [20.6, 29.0] 22.5 [19.0, 26.6] 22.5 [18.8, 26.8] 22.6 [19.1, 26.8] 23.1 [19.7, 27.2] 25.5 [21.8, 29.8] Unsafe% 0.91 [0.63, 1.24] 1.06 [0.77, 1.40] 1.45 [0.95, 2.01] 0.88 [0.54, 1.27] 0.95 [0.57, 1.37] 0.98 [0.57, 1.43] 0.83 [0.56, 1.14] 60 Recall 0.79 [0.72, 0.86] 0.21 [0.15, 0.27] 0.00 [0.00, 0.00] 0.78 [0.70, 0.86] 0.87 [0.82, 0.92] 0.87 [0.81, 0.92] 0.09 [0.06, 0.13] FA/day 1.05 [0.79, 1.35] 0.84 [0.66, 1.02] 0.08 [0.03, 0.15] 1.31 [1.07, 1.56] 1.61 [1.37, 1.86] 1.70 [1.43, 1.99] 0.88 [0.69, 1.09] Median Lead 10.2 [8.5, 12.1] 11.9 [8.0, 16.3] NA 17.9 [14.6, 21.5] 20.2 [16.9, 23.8] 23.0 [20.2, 25.8] 29.1 [24.3, 33.7] RMSE 16.6 [15.4, 17.9] 14.9 [13.7, 16.3] 13.6 [12.7, 14.7] 13.4 [12.4, 14.5] 13.6 [12.6, 14.6] 14.6 [13.6, 15.6] 15.9 [14.7, 17.4] Unsafe% 0.27 [0.20, 0.35] 0.28 [0.21, 0.35] 0.34 [0.24, 0.47] 0.28 [0.20, 0.36] 0.24 [0.16, 0.35] 0.22 [0.15, 0.31] 0.23 [0.17, 0.31] 30 Recall 0.03 [0.01, 0.05] 0.27 [0.22, 0.31] 0.59 [0.54, 0.64] 0.65 [0.60, 0.70] 0.86 [0.81, 0.90] 0.87 [0.84, 0.90] 0.04 [0.03, 0.06] FA/day 1.20 [0.90, 1.55] 1.57 [1.23, 1.98] 1.46 [1.16, 1.87] 1.52 [1.20, 1.93] 2.33 [1.85, 2.94] 1.87 [1.52, 2.29] 1.60 [1.23, 2.04] Median Lead 12.5 [9.4, 15.6] 6.2 [5.1, 7.6] 5.1 [5.0, 5.4] 7.9 [6.5, 9.2] 14.2 [12.4, 16.3] 11.3 [10.0, 12.9] 18.7 [15.8, 21.3] PEDAP RMSE 25.8 [24.1, 27.8] 23.4 [21.7, 25.4] 20.9 [19.5, 22.5] 19.8 [18.3, 21.5] 23.0 [21.6, 24.4] 19.6 [18.2, 21.2] 24.0 [22.1, 26.2] Unsafe% 1.28 [1.01, 1.59] 1.54 [1.23, 1.93] 1.55 [1.18, 2.00] 0.87 [0.66, 1.10] 0.80 [0.59, 1.06] 0.70 [0.51, 0.92] 0.99 [0.74, 1.30] 60 Recall 0.02 [0.01, 0.04] 0.23 [0.18, 0.27] 0.12 [0.09, 0.17] 0.74 [0.69, 0.79] 0.89 [0.87, 0.92] 0.86 [0.81, 0.89] 0.05 [0.03, 0.07] FA/day 0.68 [0.51, 0.88] 1.08 [0.88, 1.32] 0.62 [0.47, 0.80] 1.28 [1.05, 1.53] 2.55 [2.12, 3.07] 1.67 [1.37, 2.05] 1.13 [0.90, 1.39] Median Lead 33.0 [20.5, 45.9] 7.6 [5.7, 10.2] 8.3 [5.8, 11.4] 13.2 [11.1, 16.0] 22.5 [20.4, 24.5] 18.2 [16.1, 20.5] 28.8 [20.7, 37.5]

Table 12: Expanded counterfactual forecasting performance across the perturbation menu at 120 minutes. This table extends Table 4. eRMSE = effect RMSE, SA = sign agreement, and τb = Kendall rank correlation over the action menu. Each cell is reported as mean [95% CI]. Lower is better for eRMSE; higher is better for SA and τb . Best values in each column are bolded. The forecast horizon is 2 hours. eRMSE

τb

Basal pert.

RMSE

ARIMAX 54.18 [43.72, 65.60] 9.83 [7.51, 12.24] 0.43 [0.28, 0.57] -0.17 [-0.46, 0.15] GRU 49.78 [40.01, 59.61] 48.72 [38.75, 57.90] 0.02 [0.00, 0.04] -1.00 [-1.00, -1.00] iTransformer 55.20 [44.79, 65.94] 54.91 [43.99, 65.06] 0.02 [0.00, 0.04] -0.99 [-1.00, -0.97] PatchMLP 51.68 [42.82, 61.20] 49.65 [40.51, 57.93] 0.07 [0.00, 0.17] -0.90 [-1.00, -0.72] Glimmer* 51.90 [43.10, 61.46] 49.87 [40.36, 58.32] 0.02 [0.00, 0.04] -0.98 [-1.00, -0.96]

Bolus pert.

Model

ARIMAX 59.98 [47.02, 73.74] 7.69 [5.54, 10.31] 0.78 [0.72, 0.85] -0.03 [-0.44, 0.33] GRU 21.39 [17.06, 25.97] 18.50 [14.36, 22.68] 0.14 [0.09, 0.19] -0.95 [-1.00, -0.85] iTransformer 17.63 [13.39, 22.57] 15.40 [11.66, 19.45] 0.15 [0.10, 0.20] -1.00 [-1.00, -1.00] PatchMLP 21.17 [16.27, 26.34] 18.01 [13.42, 22.77] 0.26 [0.14, 0.39] -0.95 [-1.00, -0.85] Glimmer* 20.21 [14.80, 25.87] 18.26 [13.43, 23.30] 0.31 [0.19, 0.43] -0.91 [-1.00, -0.79]

Record · ID 151742 · SHA-256 070f8fb4917cfdc1
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.