Beyond Sufficiency: Time Series Explanation with Counterfactual Necessity
Hongnan Ma1 , Yiwei Shi2 , Mengyue Yang2 , Weiru Liu2 2
1 School of Computer Science, University of Bristol School of Engineering Mathematics and Technology, University of Bristol [email protected] {yiwei.shi, mengyue.yang, weiru.liu}@bristol.ac.uk
arXiv:2607.21573v1 [cs.LG] 23 Jul 2026
Abstract Faithful explanations of time-series classifiers should identify subsequences that are not only sufficient to preserve a black-box model’s prediction, but also necessary for maintaining it. However, existing sufficiency-oriented methods can assign high importance to spurious subsequences that support the prediction without being essential to the model’s decision. We introduce TimePNS, a necessityaware framework for time-series explanation. Inspired by Pearl’s counterfactual notion of necessity, TimePNS assesses whether a temporal factor is necessary by intervening on it and measuring whether the original prediction is disrupted. The framework adopts a two-stage design. Stage I learns an identifiable causal generative process together with a sufficiency-oriented explanation mask. Stage II performs counterfactual interventions on temporal factors to derive necessity signals, which supervise a temporal gate that refines the initial explanation by suppressing non-essential components and emphasizing counterfactually necessary ones. Experiments on synthetic and real-world time-series benchmarks show that TimePNS more accurately identifies decision-critical subsequences and consistently improves sufficiency-necessity trade-offs over strong baselines.
1
Introduction
Explaining predictions made by black-box models for multivariate time-series is a fundamental requirement in high-stakes applications such as healthcare, finance, and autonomous systems [Morid et al., 2023, Kim et al., 2025]. In these settings, practitioners must identify which variables and time steps are truly responsible for a model’s output in order to support timely intervention and build trust in automated decision systems [Küsters et al., 2020]. Most existing explanation methods tackle this problem by applying masking and perturbation operations to the input sequence and measuring the resulting change in prediction to infer the importance score of different variables or temporal segments [Crabbé and Van Der Schaar, 2021]. These methods aim to extract a subset of the input sequence that preserves the original prediction while remaining concise and interpretable. However, when viewed through the lens of the probability of necessity and sufficiency (PNS) [Pearl, 2009], existing methods primarily operationalize sufficiency: they seek subsequences that are sufficient to preserve the model’s original prediction when retained [Queen et al., 2023, Liu et al., 2024]. This limitation becomes particularly problematic in real-world time series, where different temporal segments or channels may be highly correlated, giving rise to spurious associations that make multiple subsequences appear informative for the model’s prediction [Zhang et al., 2023]. In such cases, a sufficiency-based explanation method may assign high importance to correlated but non-essential subsequences, leading to explanations that do not faithfully reflect what the model actually relies on. When applied to new time series data, such fragile correlations may no longer persist, making these
explanations unreliable under distribution shift [Yang et al., 2023, Schölkopf et al., 2021]. In contrast, the complementary notion of necessity provides an additional criterion for filtering out correlated but non-essential subsequences. In Pearl’s framework, necessity is captured through a counterfactual ‘what-if’ question: would the outcome fail to occur if the factor of interest were absent [Pearl, 2009]? This counterfactual view sharpens the notion of time series explanation: a factor is not explanatory simply because it can preserve the black-box model’s original prediction, but because removing it changes that prediction. Fig. ?? illustrates this distinction with an ECG example for myocardial infarction detection (MI). The ST-elevation segment in the affected leads is the diagnostically decisive evidence, whereas the reciprocal ST-depression observed in the opposing leads is a frequent co-occurring pattern that is highly correlated with the same underlying condition and therefore also appears predictive to the model. Under a sufficiency-based criterion, both subsequences may be considered adequate explanations, since either alone can preserve the original prediction. A necessity-based analysis, however, reveals a clear asymmetry: in a model that has learned to rely primarily on ST-elevation, masking this segment flips the prediction, whereas masking the correlated reciprocal ST-depression leaves the prediction unchanged. Motivated by this observation, we propose TimePNS, a framework for explanation of black-box timeseries models that moves beyond the prevailing sufficiency-based paradigm. However, distinguishing genuinely necessary temporal subsequences is particularly challenging in time series settings, where raw observations are often highly mixed and naive input-level counterfactual interventions may violate the underlying temporal dependencies [Li et al., 2024]. To address this challenge, TimePNS introduces a two-stage training procedure built upon a learned causal latent representation. In Stage I, a mask-based explainer is trained to select compact subsequences that preserve the black-box model’s original prediction, while a causal representation branch maps the input into a structured latent space and learns both instantaneous and lagged causal mechanisms among latent factors. The second stage refines the initial explanation from the perspective of necessity. Given the learned causal mechanisms, TimePNS estimates a soft probability of necessity score for each temporal factor through latent counterfactual intervention. These scores are then used to supervise a temporal gate, which filters out factors with low estimated necessity and refines the explanation toward decision-critical subsequences. In this way, TimePNS encourages the final explanation to preserve the original prediction while being aligned with counterfactual necessity. The main contributions are summarized as follows: 1) We show that existing time-series explanation methods may select predictive but non-essential subsequences, as they focus mainly on sufficiency rather than necessity. 2) We propose TimePNS, a framework that estimates counterfactual necessity via structured interventions in a learned causal latent space. 3) Experiments show that TimePNS improves critical-subsequence identification and suppresses spurious explanations compared with baselines.
2
Related work
Time Series Explanation. Recent explanation methods for time-series models largely follow a perturbation-based paradigm [Crabbé and Van Der Schaar, 2021]. One line of work directly perturbs the queried time series and feeds the perturbed inputs into the black-box model to estimate the importance of temporal regions or time–feature pairs. Representative methods include Dynamask [Crabbé and Van Der Schaar, 2021] and Extremask [Enguehard, 2023], which learn instance-specific masks and evaluate their relevance through the induced changes in model predictions. Another line of work incorporates perturbations into surrogate-based explanation models. TimeX [Queen et al., 2023] learns an interpretable surrogate via a self-supervised behavioral consistency objective, while TimeX++ [Liu et al., 2024] further introduces an information bottleneck objective and an explanation conditioner to generate in-distribution, label-preserving explanation instances. However, these methods remain largely sufficiency-oriented: they identify regions that can preserve the original prediction, but do not determine whether those regions are truly necessary for the model’s decision. Causal Sufficient and Necessary. PNS was introduced in Pearl’s structural causal model framework as one of the probabilities of causation, together with PN and PS [Pearl, 2009]. Recently, necessity and sufficiency-based ideas have been adopted to identify invariant or robust predictive factors, including representation learning under distribution shift [Yang et al., 2023], graph OOD generalization [Chen et al., 2025]. PNS has also been introduced into explanation and reasoning tasks. In textual causal rationalization, rationale selection is formulated within a structural causal model, where conditional 2
PNS is used to identify non-spurious rationales [Zhang et al., 2023]. More recently, PNS-based intervention analysis has been used to assess whether individual Chain-of-Thought reasoning steps are causally necessary and sufficient for the final answer [Yu et al., 2025]. However, these applications typically operate on observed units, such as tokens or reasoning steps, which can be directly intervened upon. This observation-level formulation is less suitable for multivariate time-series explanation, where raw subsequences are often entangled mixtures of latent causal factors and input-space perturbations may violate instantaneous and lagged temporal dependencies.
3
Notation and Problem Setup
We denote a multivariate time-series instance as X ∈ X = RT ×D , where T is the number of time steps and D is the number of signals. The observation at time step t is denoted by xt ∈ RD , and a contiguous sub-sequence from time step i to time step j is denoted by xi:j ∈ R(j−i+1)×D . Let Fϕ : X → Y be a pretrained black-box predictor. We denote its predicted class by ŷ(X) = arg maxc Fϕ,c (X) where c presents class c. In post-hoc instance-level time-series explanation, the goal is to identify a subset of subsequences of X that explain the prediction of Fϕ on X. Definition 3.1 (Time-Series Mask Matrix). Given an input time-series X ∈ RT ×D , we define a mask matrix M ∈ {0, 1}T ×D , where Mt,d = 1 indicates that the corresponding subsequences are important and can be used for explanation. For example, in Figure ??, if the yellow and blue subsequences are assigned mask value 1, then these subsequences are selected as the explanation. To obtain a meaningful explanation, we learn the explanation mask M under a task-driven objective, such as the sufficiency and necessity objectives function introduced in Sec. 6. Since directly optimizing over the discrete mask space is intractable, we also relax the problem into a continuous optimization via stochastic masking [Queen et al., 2023, Liu et al., 2024]. Specifically, an explanation extractor gθ (·) maps the input to Bernoulli parameters: π = gθ (X) ∈ [0, 1]T ×D ,
4
Mt,d ∼ Bernoulli(πt,d ).
(1)
Causal Sufficiency and Necessity for Time-Series Explanation
We further define such an explanation inspired by Pearl’s counterfactual notions of sufficient and necessary [Pearl, 2009]. To formalize these concepts, we firstly introduce a pair of complementary interventions do(M ) and do(M̄ ). Definition 4.1 (Mask-induced Intervention). Given an input time series X ∈ RT ×D and a mask matrix M ∈ {0, 1}T ×D as in Definition 3.1, we define a mask-induced intervention as replacing the subsequences with mask value 0 by a baseline sample b ∈ RT ×D . The baseline sample is drawn from a distribution BX . In practice, we approximate this distribution by a factorized Gaussian: T Y D Y 2 BX = N µt,d , σt,d , (2) t=1 d=1 2 where µt,d and σt,d are the empirical mean and variance at time step t and feature dimension d, estimated from the training dataset.
We use do(M ) to denote the time series obtained by applying the mask-induced intervention defined in Definition 4.1. The corresponding prediction is denoted by ŷ do(M ) . Conversely, we use do(M̄ ) to denote the complementary intervention, where the subsequences assigned value 1 in M are replaced with a baseline. The corresponding prediction is denoted by ŷ do(M̄ ) . 4.1
Definitions of Sufficiency and Necessity Explanation
Definition 4.2 (Probability of Sufficiency). For an explanation mask M , the Probability of Sufficiency do(M ) do(M̄ ) (PS) is defined as PS(M ) = P ŷ = ŷ ŷ ̸= ŷ , whenever the conditioning event has 3
nonzero probability. A high PS(M ) indicates that the subset of subsequences assigned value 1 by M alone are sufficient to preserve the original prediction. Definition 4.3 (Probability of Necessity). For an explanation mask M , the Probability of Necessity do(M̄ ) do(M ) (PN) is defined as PN(M ) = P ŷ ̸= ŷ ŷ = ŷ , whenever the conditioning event has nonzero probability. A high PN(M ) indicates that the subset of subsequences assigned value 1 by M alone necessary in the sense that removing it changes the original prediction. Definition 4.4 (Probability of Necessity and Sufficiency). For anexplanation mask M , the Probability of Necessity and Sufficiency (PNS) is defined as PNS(M ) = P ŷ do(M ) = ŷ, ŷ do(M̄ ) ̸= ŷ . A high PNS(M ) indicates that the subset of subsequences assigned value 1 by M are both sufficient to preserve the original prediction and necessary as removing it changes the original prediction.
5
Counterfactual Reasoning for Necessary Explanations
For sufficiency-based explanations, we follow the standard masking-based paradigm: an explanation extractor gθ learns a binary mask M over the input time series such that the selected entries preserve the original prediction of the classifier Fϕ [Queen et al., 2023, Liu et al., 2024]. In contrast, necessary explanations are counterfactual: they evaluate whether the original prediction would change after removing the information selected by the explanation [Pearl, 2009]. However, observed variables in time series are often entangled nonlinear mixtures of multiple latent factors [Hyvarinen and Morioka, 2017]. Directly intervening in the observation space simultaneously perturbs multiple latent factors and yield off-manifold counterfactuals. We therefore estimate PN in a latent causal space, where we model the latent generative process, formalize latent dependencies via inverse mechanisms, and propagate counterfactual interventions to compute PN scores. 5.1
Latent Causal Generative Process
We model a multivariate time series as a nonlinear observation of an underlying latent causal process, following [Li et al., 2024]. Let X = {xt }Tt=1 denote an observed time series, where xt ∈ RD . Each observed variable is assumed to be generated from a latent state zt = (zt,1 , . . . , zt,d )⊤ ∈ Rd through a nonlinear observation function h : Rd → RD , such that xt = h(zt ) for t = 1, . . . , T .
Latent variable
The latent generative process is governed by a temporal structural causal model (SCM) with both lagged and instantaneous t-1 Observed variable dependencies, as illustrated in Fig. 1. For each latent component t zt,i , its lagged parents are drawn from previous latent states within the lag window, while its instantaneous parents come Figure 1: Latent variable generative process. The black dashed lines from other components at the same time step. Specifically, represent the latent causal process, while the gray lines indicate the lag inst mixture process. zt,i = fi Pa (zt,i ), Pa (zt,i ), ϵt,i , ϵt,i ∼ pϵi , (3) The exogenous noises ϵt,i are mutually independent over time and latent dimensions. 5.2
Learning an Inverse Mechanism for Latent Dynamics
Rather than explicitly learning the unknown forward structural mechanisms fi in Eq. (3), we adopt an inverse-mechanism formulation, following [Li et al., 2024]. For each latent component zt,i , we learn a neural network ri that maps the current latent state and its lagged context to a recovered residual: ϵ̂t,i = ri (zt−L:t−1 , zt ) ,
(4)
where zt−L:t−1 denotes the latent history within the maximum lag L, and ϵ̂t,i is the residual recovered for the i-th latent component. The inverse-mechanism formulation provides a local characterization of latent dependencies through i the Jacobian of ri . Specifically, derivatives with respect to lagged latent variables, ∂z∂r , indicate t−ℓ,j 4
time-delayed dependencies, whereas off-diagonal derivatives with respect to contemporaneous latent ∂ri for j ̸= i, characterize instantaneous dependencies [Li et al., 2024]. variables, ∂z t,j 5.3
Latent Counterfactual Propagation
Given the factual latent sequence Z = {zt }Tt=1 obtained by encoding X with encη , and the learned inverse mechanisms {ri }di=1 , we construct a counterfactual latent sequence Z cf(τ,k) by intervening on a single latent factor zτ,k , which is then propagated through the learned latent dependency structure. Here, τ denotes the intervention time, k denotes the intervened latent factor, and s ≥ τ denotes a time step reached during counterfactual propagation. More concretely, for each latent factor i, we first compute its factual recovered residual at every time step: ϵ̂factual = ri (zs−L:s−1 , zs ) , s = 1, . . . , T. s,i We then intervene on zτ,k by setting it to a reference value z̃τ,k via do(zτ,k = z̃τ,k ). Under the SCM counterfactual formulation, this intervention replaces the structural equation of zτ,k with the assignment zτ,k = z̃τ,k , while the recovered residuals of all non-intervened latent variables are held fixed at their factual values. Under the topological ordering of the instantaneous graph, an intervention on zτ,k can propagate only to variables ordered after k at the same time step and to variables at future time steps. Thus, states before τ and variables ordered before k at time τ remain unchanged: cf cf zscf = zs (s < τ ), zτ,i = zτ,i (i < k), zτ,k = z̃τ,k . Let Dτ,k = {(τ, i) : i > k} ∪ {(s, i) : s > τ, 1 ≤ i ≤ d} denote the set of potentially affected non-intervened variables. For these variables, we keep the recovered residuals fixed to their factual values: cf(τ,k) ri zs−L:s−1 , zscf(τ,k) = ϵ̂factual , (s, i) ∈ Dτ,k . (5) s,i To obtain a tractable local propagation rule, we linearize Eq. (5) around the factual latent sequence. cf(τ,k) (ℓ) i Let ∆zs,i = zs,i − zs,i , and define Ji,j (s) = ∂z∂r for lagged variables and s−ℓ,j (0) ∂ri Ji,j (s) = ∂z s,j
(zs−L:s−1 ,zs )
(zs−L:s−1 ,zs ) (0) for instantaneous variables. Assuming Ji,i (s) ̸= 0, the perturbation of
each potentially affected variable is computed as L X d X X (0) (ℓ) (0) ∆zs,i = −( Ji,j (s)∆zs−ℓ,j + Ji,j (s)∆zs,j )/Ji,i (s), ℓ=1 j=1
(s, i) ∈ Dτ,k .
(6)
j<i
The first term in the numerator captures propagation through lagged dependencies, while the second term captures propagation throug instantaneous dependencies under the topological ordering. The derivation from the linearized residual-preservation constraint is provided in Appendix ??. 5.4
Counterfactual Necessity Scoring
The counterfactual latent sequence is decoded back to the observation space: X cf(τ,k) = n oT cf(τ,k) decγ zs . The counterfactual sample X cf(τ,k) is then passed to the black-box model F s=1 for necessity scoring. For an input X, We define the necessity score of latent factor zτ,k as h i ∆pτ,k = Fϕ,ŷ(X) (X) − Fϕ,ŷ(X) X cf(τ,k) , [a]+ = max(a, 0). (7) +
A larger ∆pτ,k indicates that intervening on zτ,k leads to a larger reduction in the factual-class confidence, suggesting that this latent factor is more necessary for maintaining the original prediction.
6
TimePNS - PNS Estimation for Faithful Time Series Explanation
Directly optimizing the joint PNS is computationally prohibitive. We therefore decouple the estimation into a two-stage procedure: Stage I learns a sufficient explanation jointly with a structured latent causal model, and Stage II uses counterfactual interventions on that latent causal model to derive a PN signal that refines the explanation. 5
6.1
Stage I: Latent Causal Mechanism Learning and Sufficiency-Oriented Explanation
Latent Causal Mechanism Learning [Li et al., 2024]. This branch trains the latent causal model introduced in Sec. 5.2, which provides the structured intervention space used in Stage II. Given an input time series X, the encoder encη parameterizes an approximate posterior qη (Z | X) over the latent trajectory Z = (z1 , . . . , zT ). A latent trajectory sampled from this posterior is decoded to reconstruct the input, X̃ = decγ (Z). Beyond reconstruction, we regularize the latent dynamics with the inverse transition mechanisms {ri }di=1 introduced in Eq.4. The Jacobian entries of these inverse mechanisms define local instantaneous and lagged dependency scores, which are later reused to propagate latent counterfactual interventions in Stage II. The causal branch is optimized by ! L X normal future inst lag(ℓ) Lcausal = λrec Lrec + λkld Lkld + Lkld + λsp ∥J ∥1 + ∥J ∥1 , (8) ℓ=1
Here, J inst and J lag(ℓ) collect the instantaneous and ℓ-lag Jacobian entries of the inverse mechanisms across time. The reconstruction term preserves information about the input, the KL terms regularize the recovered residuals under the temporal causal prior, and the L1 penalty encourages sparse dependency structures. Detailed definitions of the posterior, residual-based KL terms, temporal causal prior likelihood, and Jacobian penalties are provided in Appendix ??. Sufficiency-Oriented Explanation. Stage I trains the explanation branch using a sufficiency-oriented masking objective built on the mask formulation in Definition 3.1. The explanation extractor gθ outputs Bernoulli parameters π = gθ (X) ∈ [0, 1]T ×D , which define a factorized mask distribution QT QD Pπ (M | X) = t=1 d=1 Bernoulli(Mt,d ; πt,d ). A hard binary mask M ∼ Pπ (· | X) is sampled during the forward pass, and gradients are passed through the mask probabilities using a straightthrough estimator (STE) [Jang et al., 2016]. To avoid theQtrivial all-one mask, we regularize the mask distribution toward a sparse Bernoulli prior Qr (M ) = t,d Bernoulli(Mt,d ; r): X 1 − πt,d πt,d + (1 − πt,d ) log , (9) Lmask = DKL (Pπ (M | X) ∥ Qr (M )) = πt,d log r 1−r t,d
where r ∈ (0, 1) controls the target sparsity. We further encourage temporal smoothness by applying a total-variation penalty to the mask probabilPD PT −1 ities, Lcon = D(T1−1) d=1 t=1 |πt+1,d − πt,d |. Predictive sufficiency is enforced by encouraging the masked input to preserve the frozen black-box predictive distribution, Ljs = EM ∼Pπ (·|X), b∼BX [DJS (Fϕ (X) ∥ Fϕ (XM (b)))] , where XM (b) = M ⊙ X + M̄ ⊙ b and M̄ = 1 − M . Stage I Training Objective. During Stage I, the causal branch minimizes Lcausal , while the explanation branch minimizes the sufficiency-oriented objective Lsuff = λjs Ljs + λmask Lmask + λcon Lcon . Here, Ljs , Lmask , and Lcon correspond to prediction preservation, sparsity, and temporal smoothness, respectively, with λjs , λmask , and λcon controlling their relative weights. 6.2
Stage II: Necessity-Guided Explanation
Necessity Supervision Signal. After Stage I pretraining, we use latent counterfactual interventions to construct a necessity supervision signal. Let ŷ = arg maxc Fϕ,c (X) denote the original blackbox prediction. Necessity refinement is applied only when the Stage I explanation preserves this prediction, indicated by Isuff = 1[arg maxc Fϕ,c (XM ) = ŷ]. For each intervention time τ ∈ Teval and latent factor k, we convert the counterfactual probability drop ∆pτ,k into a soft necessity f τ,k = Isuff ∆pτ,k ∈ [0, 1]. This target supervises the temporal gate by assigning higher target PN preservation weights to time-feature factor whose intervention substantially reduces the factual-class confidence. Necessity-Guided Temporal Gate. We introduce a lightweight gate ρψ over the causal latent trajectory. Given Z c ∈ RT ×d , the gate outputs G = ρψ (Z c ) ∈ [0, 1]T ×d and produces the gated c latent trajectory Z gate = G ⊙ Z c , where Gτ,k controls how much the latent factor zτ,k is preserved. 6
The gate is trained with two objectives. First, for the sufficiency-valid mini-batch Bsuff = {b : P gate )),ŷ(Xb )) b∈Bsuff CE(Fϕ (decγ (Zb gate Isuff,b = 1}, we use a classification-anchor loss Lclf = . max(|Bsuff |,1) This loss encourages the decoded gated trajectory to preserve the original black-box prediction. Second, we align the temporal pattern of the gate with the soft necessity target: Lalign = PB Pd f b=1 k=1 Isuff,b [1−cosT (Gb,Teval ,k ,PNb,Teval ,k )] P . Here, cosT denotes the centered cosine similarity d·max( B I ,1 ) suff,b b=1 computed over the evaluated time steps Teval . Specifically, for two temporal vectors u, v ∈ R|Teval | , it (u−ū)⊤ (v−v̄) is defined as cosT (u, v) = ∥u−ū∥ , where ū and v̄ are the temporal means of u and v, and ϵ 2 ∥v−v̄∥2 +ϵ is a small constant for numerical stability. Finally, a linear projection of Z gate is injected into the representation of gθ , allowing necessity-guided latent information to refine the final mask M . Stage II Training Objective. The PN-guided gate is optimized with Lnecessity = Lgate clf + λcos Lalign , where Lgate preserves the prediction, L aligns the gate with necessity signals, and λcos controls align clf the alignment strength.
7
Experiment
Experiments are structured around two core research questions: RQ1: Does TimePNS produce sufficient and necessary explanations? (Sec. 7.2.1) RQ2: Does TimePNS refine the explanations by removing predictive but unnecessary part?(Sec.7.2.2) 7.1
Experiment setup
We evaluate the quality of our explanations on three synthetic datasets and three real-world datasets. For each metric, bold denotes the best result and underline denotes the second best. All results are reported as mean ± std over 5-fold cross-validation. All experiments in the main body use Vanilla Transformer [Vaswani et al., 2017] with time-based positional encoding as the black-box classifier, with hyperparameter tuned to ensure strong predictive performance. Dataset. We evaluate the performance of our explainer on synthetic datasets with known ground-truth saliency scores from [Queen et al., 2023]: FreqShapes, SeqComb-MV, and Low-VAR. These datasets are carefully designed to cover diverse temporal dynamics in multivariate settings, where correctly identifying salient features at different timesteps is non-trivial. For real-world evaluation, we use three datasets from the UCR Archive [Dau et al., 2019]: Epilepsy, ERing, and ArticularyWordRecognition (WordRec), which span diverse temporal classification domains. Baselines. We compare our method with several baselines, including perturbation-based methods, namely TimeX [Queen et al., 2023], TimeX++ [Liu et al., 2024], Dynamask [Crabbé and Van Der Schaar, 2021], as well as gradient-based methods, including CORTX [Chuang et al., 2023], WinIT [Leung et al., 2021] and SGT+GRAD [Ismail et al., 2021], Evaluation Metrics. We evaluate explanations under two settings. For synthetic datasets, we compare the generated saliency maps with the ground-truth predictive regions specified by the data-generating process. These regions serve as sufficient and necessary explanatory signals. Following Crabbé and Van Der Schaar [2021], we report AUP, AUR, and AUPRC, where higher values indicate better agreement with the ground truth. For real-world datasets without ground-truth saliency, we use occlusion-based evaluation following [Queen et al., 2023]. Sufficiency is evaluated by retaining the top-p salient features and measuring prediction preservation, where higher AUROC and AUPRC is better. Necessity is evaluated by removing the top-p salient features and measuring the degradation of the remaining input, where lower AUROC and AUPRC is better. Detailed dataset descriptions, baseline implementations, and full metric definitions are provided in Appendix ??. 7.2 7.2.1
Experiment Result TimePNS produce sufficient and necessary explanation
Table 1 reports explanation quality on synthetic datasets with ground-truth saliency. TimePNS achieves the best AUPRC on all three datasets and the best AUP on SeqComb-MV and FreqShape, 7
showing that the proposed necessity-aware refinement improves the ranking and localization of salient regions. Compared with traditional methods, Dynamsk, WINIT, CORTX, and SGT show competitive results on individual metrics, such as SGT’s AUR on SeqCombMV and CORTX’s AUPRC on FreqShape, but their performance is less consistent across datasets and metrics. Among stronger recent baselines, TimeX and TimeX++ achieve the best AUR or AUP on LowVER, suggesting that sufficiency-oriented methods can be effective when the synthetic signal is clean and explicit. Nevertheless, TimePNS delivers the most consistent AUPRC gains, which is important for identifying compact and precise explanatory subsequences. For real-world datasets, where ground-truth saliency annotations are unavailable, we evaluate explanations using occlusion-based sufficiency and necessity tests, supplemented by Prediction Shift, measured as the total variation distance between the original and perturbed prediction distributions. As shown in Figure 2, TimePNS achieves the best sufficiency performance across all datasets, indicating that its top-ranked positions preserve the black-box prediction most effectively. Under the necessity test, where lower AUROC and AUPRC indicate a larger performance drop after removing the selected positions, TimePNS induces the strongest degradation across datasets. It also yields the largest Prediction Shift, suggesting that perturbing its selected positions produces the most substantial change in the model output. These results demonstrate that TimePNS identifies positions that are both sufficient for preserving predictions and necessary for maintaining the model’s original behavior. SeqComb-MV
LowV-ER
FreqShape
Method
AUPRC (↑)
AUP (↑)
AUR (↑)
AUPRC (↑)
AUP (↑)
AUR (↑)
AUPRC (↑)
AUP (↑)
AUR (↑)
Dynamsk WINIT CORTX SGT TIMEX TIMEX++ TimePNS
0.3136±0.0019 0.2809±0.0018 0.3629±0.0021 0.4893±0.0005 0.6878±0.0021 0.6663±0.0021 0.7460±0.0015
0.5481±0.0053 0.7594±0.0024 0.5625±0.0006 0.4970±0.0005 0.8326±0.0008 0.8774±0.0006 0.9189±0.0003
0.1953±0.0025 0.2077±0.0021 0.3457±0.0017 0.4289±0.0018 0.3872±0.0015 0.3329±0.0012 0.3156±0.0009
0.1391±0.0012 0.1667±0.0015 0.4983±0.0014 0.3449±0.0010 0.8673±0.0033 0.8957±0.0020 0.9188±0.0029
0.1640±0.0028 0.1140±0.0022 0.3281±0.0027 0.2133±0.0029 0.5451±0.0028 0.8137±0.0020 0.7579±0.0030
0.2106±0.0018 0.3842±0.0017 0.4711±0.0013 0.3528±0.0015 0.9004±0.0024 0.8313±0.0020 0.8034±0.0027
0.2201±0.0013 0.5071±0.0021 0.6978±0.0156 0.5312±0.0019 0.8324±0.0034 0.8286±0.0022 0.8574±0.0026
0.2952±0.0037 0.5546±0.0026 0.4938±0.0004 0.4138±0.0011 0.7219±0.0031 0.7431±0.0021 0.7603±0.0020
0.5037±0.0015 0.4557±0.0016 0.3261±0.0012 0.3931±0.0015 0.6381±0.0022 0.6426±0.0021 0.6396±0.0021
Table 1: Performance comparison on synthetic datasets. Sufficiency AUROC ( ↑ )
Sufficiency AUPRC ( ↑ )
Necessity AUROC ( ↓ )
1.0
Necessity AUPRC ( ↓ )
1.0
Epilepsy
0.9
0.9
0.7
0.8
0.5
0.7
0.7
0.2 0.1
0.4 1.0
0.5
0.2
0.4
0.0
1.0
ERing
0.7
0.2 0.1
0.4 1.0
0.5
0.2
0.4
0.0
1.0
0.2 0.1
WordRec
0.4 0
5
10
15
20
25
Top-saliency positions perturbed (%)
0
5
10
15
20
25
Top-saliency positions perturbed (%)
Random
Dynamask
0.4
0.4
0.6
0.3
0.6
0.6
0.7
0.4
0.5
0.8
0.8
0.5
0.6
1.0
0.8
0.6
0.7
0.0
0.9
0.7
0.8
0.2
1.0
0.8 0.9
0.4
0.4
0.6
0.3
0.6
0.6
0.4
0.5
0.8
0.8
0.5
0.6
1.0
0.8
0.6
0.7
0.0
0.9
0.7
0.8
0.2
1.0
0.8 0.9
0.4
0.4
0.6
0.3
0.5
0.6
0.6
0.4 0.6
0.8
0.8
0.6
0.8
Prediction Shift ( ↑ ) 1.0
1.0
0.8
0.5
0.2
0.4
0.0 0
5
10
15
20
25
0.2 0.0 0
5
10
15
20
25
Top-saliency positions perturbed (%)
Top-saliency positions perturbed (%)
SGT
TimeX++
WinIT
TimeX
0
5
10
15
20
25
Top-saliency positions perturbed (%)
TIMEPNS
Figure 2: Real-world occlusion evaluation on Epilepsy, ERing, and WordRec. 7.2.2
TimePNS refine the explanation
Disagreement-Set Ablation. To further assess whether TimePNS identifies features that are necessary for the black-box predictor, we conduct a disagreement-set ablation analysis. For each sample, we first select the top 25% salient positions identified by the baseline (TimeX or TimeX++) and TimePNS, denoted as Sbaseline and STimePNS , respectively. We then construct three mutually disjoint subsets: positions selected only by the baseline, Sbaseline \ STimePNS ; positions selected only by TimePNS, 8
Epilepsy
ERing
Epilepsy
WordRec
ERing
WordRec.
1.0
Sufficiency AUROC ( ↑ )
0.7
0.5 0.4 0.3
0.7 0.6
0.4 1.0
Sbaseline ∖ TimePNS STimePNS ∖ baseline
Sboth
Sbaseline ∖ TimePNS STimePNS ∖ baseline
Sboth
Sbaseline ∖ TimePNS STimePNS ∖ baseline
Sboth
0.7 0.6 0.5
chance
0.5
0.1 0.0
TimeX++ mean |ΔpGT|
0.8
0.2
Necessity AUROC ( ↓ )
TimeX mean |ΔpGT|
0.6
0.9
0.9 0.8 0.7 0.6 0.5 0.4
0.4
0
5
10
15
20
25
0
Top-saliency positions perturbed (%)
0.3 0.2
5
10
15
20
25
0
Top-saliency positions perturbed (%)
Random
TIMEPNS stage 1
5
10
15
20
25
Top-saliency positions perturbed (%)
TIMEPNS stage 2
0.1 0.0 Sbaseline ∖ TimePNS STimePNS ∖ baseline
Sboth
Sbaseline ∖ TimePNS STimePNS ∖ baseline
Sboth
Sbaseline ∖ TimePNS STimePNS ∖ baseline
(b) Stage I vs. Stage II.
Sboth
(a) Disagreement-set ablation.
Figure 3: Ablation experiment. STimePNS \ Sbaseline ; and positions selected by both methods, Sboth = Sbaseline ∩ STimePNS . We ablate each set by replacing its values with Gaussian samples estimated from the training data, and measure the resulting change in the ground-truth class probability, |∆pGT |. As shown in Figure 3a, TimePNS-exclusive positions induce larger prediction changes than baseline-exclusive positions in most settings, with the confident-sample win rate favoring TimePNS in five out of six comparisons. While the shared set often captures a common core of highly predictive features, the disagreement regions reveal the main difference: positions uniquely selected by TimePNS are more likely to be necessary for preserving the model’s prediction than those uniquely selected by TimeX or TimeX++. Stage II Enhances Necessity-Aware Explanations To isolate the effect of the second training stage, we compare the Stage 1 with the fully trained TimePNS using the same sufficiency and necessity protocol. As shown in Figure 3b, the second stage consistently improves sufficiency across all real world dataset, indicating that the selected positions better preserve the black-box prediction. It also yields lower necessity AUROC, meaning that removing the selected positions causes a larger performance drop. This stronger improvement on the necessity side suggests that the necessary objective encourages saliency to concentrate on positions that the predictor relies on, rather than merely selecting sufficient or correlated regions. These results validate the role of the second stage as a necessity-aware refinement over the Stage I. 7.3
Case Study
Figure ?? in Appendix ?? visualises a representative Epilepsy sample to illustrate the discrepancy between sufficiency and necessity. Using top-50% masks, all TimeX-family methods achieve high sufficiency scores, indicating that their selected timesteps can preserve the original prediction. However, their necessity behavior differs substantially: removing TimeX-selected timesteps causes almost no confidence drop (0.001), whereas removing TimeX++ and TimePNS masks leads to larger drops of 0.514 and 0.740, respectively. This shows that a sufficient mask can still be redundant, while TimePNS better identifies timesteps that are necessary for maintaining the model’s prediction.
8
Conclusion
We introduced TimePNS, a necessity-aware framework for post-hoc explanation of time-series classifiers. Unlike sufficiency-oriented methods that mainly seek subsequences capable of preserving the black-box prediction, it further incorporates counterfactual necessity by refining a sufficiencypreserving mask with necessity signals derived from latent counterfactual interventions. Experiments on synthetic and real-world benchmarks show that TimePNS improves the identification of decisioncritical subsequences and achieves stronger sufficiency–necessity behavior than competitive baselines. Ablation analyses further validate the role of necessity-guided refinement in filtering out predictive but redundant regions. A limitation is the computational cost of repeated counterfactual interventions, which we leave to future work together with extensions to broader temporal architectures.
9
References Xuexin Chen, Ruichu Cai, Kaitao Zheng, Zhifan Jiang, Zhengting Huang, Zhifeng Hao, and Zijian Li. Unifying invariant and variant features for graph out-of-distribution via probability of necessity and sufficiency. Neural Networks, 184:107044, 2025. Yu-Neng Chuang, Guanchu Wang, Fan Yang, Quan Zhou, Pushkar Tripathi, Xuanting Cai, and Xia Hu. Cortx: Contrastive framework for real-time explanation. arXiv preprint arXiv:2303.02794, 2023. Jonathan Crabbé and Mihaela Van Der Schaar. Explaining time series predictions with dynamic masks. In International Conference on Machine Learning, pages 2166–2177. PMLR, 2021. Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293–1305, 2019. Joseph Enguehard. Learning perturbations to explain time series predictions. In International Conference on Machine Learning, pages 9329–9342. PMLR, 2023. Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial intelligence and statistics, pages 460–469. PMLR, 2017. Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning interpretability by saliency guided training. Advances in Neural Information Processing Systems, 34: 26726–26739, 2021. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016. Jongseon Kim, Hyungjoon Kim, HyunGi Kim, Dongjun Lee, and Sungroh Yoon. A comprehensive survey of deep learning for time series forecasting: architectural diversity and open challenges. Artificial Intelligence Review, 58(7):216, 2025. Ferdinand Küsters, Peter Schichtel, Sheraz Ahmed, and Andreas Dengel. Conceptual explanations of neural network prediction for time series. In 2020 International joint conference on neural networks (IJCNN), pages 1–6. IEEE, 2020. Kin Kwan Leung, Clayton Rooke, Jonathan Smith, Saba Zuberi, and Maksims Volkovs. Temporal dependencies in feature importance for time series predictions. arXiv preprint arXiv:2107.14317, 2021. Zijian Li, Yifan Shen, Kaitao Zheng, Ruichu Cai, Xiangchen Song, Mingming Gong, Zhengmao Zhu, Guangyi Chen, and Kun Zhang. On the identification of temporally causal representation with instantaneous dependence. arXiv preprint arXiv:2405.15325, 2024. Zichuan Liu, Tianchun Wang, Jimeng Shi, Xu Zheng, Zhuomin Chen, Lei Song, Wenqian Dong, Jayantha Obeysekera, Farhad Shirani, and Dongsheng Luo. Timex++ learning time-series explanations with information bottleneck. In Proceedings of the 41st International Conference on Machine Learning, pages 32062–32082, 2024. Mohammad Amin Morid, Olivia R Liu Sheng, and Joseph Dunbar. Time series prediction using deep learning methods in healthcare. ACM Transactions on Management Information Systems, 14(1): 1–29, 2023. Judea Pearl. Causality. Cambridge university press, 2009. Owen Queen, Tom Hartvigsen, Teddy Koker, Huan He, Theodoros Tsiligkaridis, and Marinka Zitnik. Encoding time-series explanations through self-supervised model behavior consistency. Advances in Neural Information Processing Systems, 36:32129–32159, 2023. Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612–634, 2021. 10
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. Mengyue Yang, Zhen Fang, Yonggang Zhang, Yali Du, Furui Liu, Jean-Francois Ton, Jianhong Wang, and Jun Wang. Invariant learning via probability of sufficient and necessary causes. Advances in Neural Information Processing Systems, 36:79832–79857, 2023. Xiangning Yu, Zhuohan Wang, Linyi Yang, Haoxuan Li, Anjie Liu, Xiao Xue, Jun Wang, and Mengyue Yang. Causal sufficiency and necessity improves chain-of-thought reasoning. arXiv preprint arXiv:2506.09853, 2025. Wenbo Zhang, Tong Wu, Yunlong Wang, Yong Cai, and Hengrui Cai. Towards trustworthy explanation: On causal rationalization. In International Conference on Machine Learning, pages 41715–41736. PMLR, 2023.
11