Task-Conditional Faithfulness Auditing of Multimodal LLMs for Grid Diagnosis

1

Task-Conditional Faithfulness Auditing of Multimodal LLMs for Grid Diagnosis

arXiv:2607.24539v1 [cs.AI] 27 Jul 2026

Tianqiao Zhao, Senior Member, IEEE, Meng Yue, Member, IEEE, and Jianhui Wang, Fellow, IEEE Abstract—Multimodal large language models (LLMs) can combine topology, measurements, and incident text for grid diagnosis, yet answer accuracy does not establish that taskappropriate evidence was used. This letter proposes a general framework in order to conduct task-conditional faithfulness audit. It compares self-reported reliance, intervention-derived behavioral reliance, and preregistered engineering importance. The framework first registers task-specific evidence requirements and compares them with self-reported reliance and behavioral changes under controlled modality ablations. To resolve detected discrepancies, we design an evidence-gated correction and reaudit mechanism that regenerates failed responses under evidence constraints and independently re-ablates them to verify improved grounding without performance loss. Case studies evaluate three differently scaled LLMs on IEEE 39- and 118-bus scenarios. These results validate the framework ability to detect, diagnose, and correct task-conditional faithfulness failures. Index Terms—Explainable AI, large language models, model faithfulness, multimodal fusion, power systems.

I. I NTRODUCTION

II. M ETHODOLOGY A. Problem Formulation Let xn = (mn,1 , . . . , mn,K , zn ) denote scenario n, with modality mn,i and operating metadata zn ; here K = 3 for topology (T ), measurements (S), and incident text (X). For task j, model Fθ returns a structured response Rj,n containing answer ŷj,n and reported reliance sj,n ∈ ∆K−1 . A task score pj (ŷ, y) ∈ [0, 1] is oriented so that larger is better. The audit object is Aj,n = {pj,n , sj,n , bj,n , gj (xn ), Vj,n , Ij,n }, (1) where b is observable reliance, g is registered engineering importance, V indicates a valid output, and I records intervention validity. Thus, predictive correctness, behavioral grounding, and explanation fidelity are evaluated separately. The framework uses only observable input–output interventions; it does not require access to hidden chain-of-thought or treat self-reports as causal explanations. Aggregate alignment is reported jointly with Pj , validity, and responsiveness so that a model cannot appear faithful by failing to answer difficult cases.

LLM-based grid decision support is emerging across power dispatch, numeric anomaly detection, and multimodal energymanagement-system security [1]–[3]. Related power-system B. Task-Conditioned Reference Registration fault-diagnosis studies also demonstrate the value of combining Before any test query, register heterogeneous measurements rather than relying on a single Tj = (qj , oj , hj , Yj , pj , dj , Ej , Lj , gj ), (2) information source [4]. Multimodal availability, however, does not guarantee task-appropriate evidence use. A correct event where qj is the query; oj , hj parse the answer and reliance; dj label may be copied from an incident narrative even when measures response change; Ej contains admissible evidence; branch-status or dynamic evidence is ignored. This matters and Lj identifies outcome-bearing fields that must not replace because topology identification, security screening, and impact engineering evidence. With modality availability an and estimation require different evidence pathways. More broadly, preregistered task weights wj (zn ), an ⊙ wj (zn ) plausible model explanations may not reflect the factors that . (3) gj (xn ) = ⊤ 1 (an ⊙ wj (zn )) drive a prediction, and multimodal models can over-rely on Structural rules weight the source that defines the answer, one modality [5], [6]. This letter asks whether a multimodal grid LLM uses the generative rules follow the controlled signal-generation process, evidence required by each task and how discrepancies should and conditional rules change weights only through registered be corrected. It contributes: 1) a preregistered task contract metadata such as event stage. Counterfactual siblings preserve that separates engineering evidence requirements from model the physical state and target, ψ(xn ) = ψ(x̃n ) and yn = ỹn , explanations; 2) a validity-aware SR–Abl–Ref audit combining while relocating or contradicting selected evidence. Fractional responsiveness and signed modality effects; 3) a diagnosis and weights permit tasks for which multiple modalities are jointly corrective re-audit gate that requires measurable behavioral necessary; one-hot references are used only when a single improvement without material performance loss; and 4) a source is sufficient. Reference construction is reviewed indepengeneric implementation pipeline that transfers across grid dently of model outputs, and disagreement among engineering systems and LLMs through lightweight system and model rules is retained as a reference interval rather than collapsed adapters. A three-model IEEE 39/118 proof of concept applies post hoc. References, thresholds, and sibling rules are frozen one frozen audit contract and reports output validity explicitly before testing. so that malformed intervention sets cannot inflate faithfulness. C. Intervention-Based Behavioral Audit T. Zhao is with The University of Texas at Arlington, Arlington, TX, USA. M. Yue is with Brookhaven National Laboratory, Upton, NY, USA. J. Wang is with Southern Methodist University, Dallas, TX, USA.

Matched calls evaluate Rn0 = Fθ (xn ), Rn−i = Fθ (Ai (xn )), i = 1, . . . , K, (4)

2

distinguish grounded agreement, unsupported self-report (high SR–Ref but low Abl–Ref), explanation mismatch (high Abl– Ref but low SR–Abl), and shortcut/conflict use (low Abl– ¯ i ). Diagnosis is assigned only when Ref interpreted with ∆ 2. Task Contract and Reference Registration the relevant confidence interval does not cross its calibrated boundary; otherwise the case is marked ambiguous and routed g for additional sampling. A response passes only when   3. Intervention-Based Behavioral Audit ⋆ Vj,n Ij,n = 1, rj,n > τr , pj,n ≥ τP , Gj,n = 1 Abl,Ref . (8) ϕj,n ≥ τB φ Failure triggers diagnosis-specific correction: constrain citations to Ej , mask fields in Lj , request evidence-linked reasoning, 4. Diagnosis and Corrective Re-Audit or invoke power-flow/contingency verification. Unsupported self-report triggers evidence citation and tool verification; explanation mismatch triggers provenance-constrained generation; shortcut/conflict use triggers masking, counterfactual prompting, or human escalation. The regenerated response is 5. Outputs and Portable Adaptation independently re-ablated and accepted only when the lower confidence bound of ∆ϕAbl,Ref is positive and performance is 0 noninferior, ∆P = P + − Pfail ≥ −ϵP . E. Implementation and Adaptation Fig. 1. Task-conditioned audit using full and single-modality interventions. The framework uses a common audit engine with two where Ai neutralizes only modality i using an explicit avail- lightweight interfaces. The system adapter maps grid data ability marker and preserves the query, target, non-intervened into canonical inputs, generates registered counterfactuals, values, schema, and decoding settings. Pair validity Ij,n,i = 1 and checks physical consistency. The model adapter handles requires preserved Q physical invariants and valid full/ablated prompting, parsing, tool syntax, and decoding. Task definitions, ⋆ outputs; Ij,n = i Ij,n,i denotes an entirely valid intervention reference rules, interventions, metrics, gates, and reporting set. Each intervention is accompanied by a sham edit that pre- remain fixed, so a new system or LLM requires only adapter serves the evidence while matching formatting, which screens replacement. The workflow registers tasks and thresholds, for prompt-length and placeholder artifacts. Positive controls validates the adapters, executes matched full-input and ablated confirm that the operator perturbs tasks whose registered calls, computes validity, responsiveness, signed effects, and SR– reference assigns nonzero importance to modality i. Behavioral Abl–Ref scores, then routes failed cases to evidence constraints, change and normalized reliance are physics tools, or human review before independent re-auditing. Ij,n,i δj,n,i Malformed outputs remain invalid, and an execution manifest 0 −i δj,n,i = dj (oj (Rn ), oj (Rn )), bj,n,i = P . records the adapters, prompts, interventions, model settings, I δ j,n,ℓ j,n,ℓ ℓ (5) tool outputs, and scenario identifiers. For categorical tasks, dj is total-variation distance when III. C ASE S TUDY probabilities are available and a hard-label indicator other′ wise; for scalar tasks, d = min(|a − a |/τ , 1). If r = We validate the framework on IEEE 39- and 118-bus j j j,n P systems as a proof of concept using locally served quantized i Ij,n,i δj,n,i ≤ τr , the sample is inconclusive rather than assigned uniform reliance. Responsiveness is ρj = |{n : rj,n > checkpoints: Qwen3 4B Instruct (Q4), Ministral 3 8B (M8), τr }|/|Vj |. Separately, the paired signed utility effect and Gemma 3 12B (G12) [7]. System records are serialized X 0 −i into topology (T ), measurements (S), and text (X), covering ¯ j,i = N −1 ∆ I [p (R ) − p (R )] (6) j,n,i j j n n j,i network structure, operating conditions, and incident narratives. n ¯ > 0), neutral, and harmful (∆ ¯ < 0) The audited tasks are topology-change identification, bottleneck distinguishes useful (∆ influence. Confidence intervals resample matched scenario localization, event classification, N–1 screening, and congestionfamilies. For stochastic decoding, L repeated full/ablated impact estimation. Performance P is accuracy for the first pairs are evaluated with identical sampling settings and δ is four tasks and 1−normalized MAE for the last. Five physical averaged before normalization. The resulting b is interpreted families per task–system cell yield 50 unique families, 150 as marginal dependence under the registered intervention, not model–family trials, 450 model–regime observations, and 2606 registered calls, including re-audits of 89 failed stressed as identification of the model’s latent causal mechanism. cases. Result panels report 95% task-stratified family-bootstrap D. Discrepancy Diagnosis and Corrective Re-Audit intervals from 2,000 resamples, capturing scenario but not The measurement triangle is model-population variation. ϕSR,Ref = sim(s, g), ϕAbl,Ref = sim(b, g), 1. Multimodal Grid Evidence

Topology T | Measurements S | Incident text X | Metadata z

Register task, score, and admissible evidence.

Specify prohibited fields and matched sibling scenarios. Freeze thresholds and engineering reference j.

Run full input and registered single-modality interventions.

−S

−X

Check intervention validity, sham stability, and responsiveness. Compute the three

similarities and signed effects.

Gate on validity, responsiveness, performance, and grounding. Diagnose unsupported, explanation-mismatch, shortcut/conflict, or inconclusive cases.

INDEPENDENT RE-AUDIT

−T

Correct using evidence constraints, masking, physics verification, or human review. Re-ablate; accept only improved grounding with noninferior performance.

Report accepted/rejected cases, coverage, and paired uncertainty.

Swap system/model adapters; task semantics and gates remain fixed.

ϕSR,Abl = sim(s, b), (7) where cosine similarity measures claimed alignment, behavioral grounding, and explanation fidelity. Calibration-set thresholds

A. Subcase A: Cross-Model Validation and Ablation In this subcase, we examined cross-model agreement among performance, claimed alignment ϕSR,Ref , behavioral

ϕ Abl, Ref

0.5 0.0

Q4

0.0

M8

G12

39

118

0.5

ϕ SR, Ref (c) Correction and independent re-audit 0.893

0.452

Performance

1.0

+0.442 [+0.32,+0.54]

0.816

ϕ Abl, Ref

0.979

+0.164 [+0.12,+0.21]

0.0

0.2

0.4

0.6

0.8

Paired mean before / corrected

1.0

(b) Modality-ablation effects 0.50 0.25 0.00

Coverage rate

(a) Claimed alignment vs. behavioral grounding

1.0

Signed effect

3

T aligned

1.0

S aligned

X shortcut

X conflict

(d) Audit and re-audit coverage

0.5 0.0

Strict

Valid set

Responsive Grounding pair

Fig. 2. Measured diagnostics: (a) stressed claimed alignment versus behavioral grounding; (b) signed modality effects across controlled regimes; (c) paired correction gains in performance and grounding; and (d) intervention validity, responsiveness, and audit coverage. TABLE I C ROSS - MODEL TESTS . A/E: ALIGNED / STRESSED . A LL ϕ COLUMNS ARE THE COSINE SIMILARITIES IN (7); CORRECTION CELLS GIVE PAIRED MEAN CHANGE ( VALID - PAIR COUNT ) Mdl. Sys. PA ϕAbl,Ref PE ϕSR,Ref ϕAbl,Ref A E E

∆Pc (n) ∆ϕAbl,Ref (n) c

39 .875 118 .880 39 .787 M8 118 .758 39 .896 G12 118 .875

+.833 (6) +.500 (10) +.555 (15) +.692 (13) +.147 (19) +.250 (12)

Q4

.983 .960 .966 .974 .887 .963

.895 .900 .827 .819 .834 .844

.986 .974 .994 .993 .950 .951

.977 .966 .951 .967 .903 .949

+.190 (6) +.172 (10) +.163 (15) +.128 (13) +.150 (14) +.215 (8)

TABLE II S IGNED MODALITY EFFECTS . A/S H /C F : ALIGNED / SHORTCUT / CONFLICT. Mdl. Sys.

∆A T

∆A S

∆Sh X

Abl,Ref ∆Cf ϕAbl,Ref X ϕSh Cf

39 +.560 +.475 +.040 −.080 118 +.576 +.480 +.093 +.053 39 +.680 +.387 +.160 +.080 M8 118 +.520 +.358 +.082 −.040 39 +.435 +.548 +.000 +.100 G12 118 +.542 +.500 +.000 +.043

Q4

.983 .965 .931 .960 .945 .950

.971 .966 .971 .974 .870 .948

grounding ϕAbl,Ref , and task-conditioned ablation effects, and tested correction on failed stressed cases. On jointly estimable stressed observations, ϕSR,Ref − ϕAbl,Ref = 0.024 [0.011, 0.036]. Correction improves performance from 0.452 to 0.893 (+0.442, [0.324, 0.543]; 75/89) and grounding from 0.816 to 0.979 (+0.164, [0.124, 0.213]; 66/89). Across models, aligned topology and measurement effects are +0.552 and +0.458, respectively. Shortcut text adds utility (+0.062) despite having no registered engineering importance, while conflict-text effects are near zero overall but vary across model– system pairs. These results support cross-model mismatch detection, correction, and signed task-conditional diagnosis, but not uniformly harmful conflict. B. Subcase B: Gate Detection, Correction, and Audit Coverage TABLE III AUDIT COVERAGE AND CONTROLLED - REGIME GATE DETECTION . I ⋆ : VALID INTERVENTION SET; ρ: RESPONSIVENESS GIVEN VALIDITY; BA: BALANCED ACCURACY; S P : SPECIFICITY; ϕB ≡ ϕAbl,Ref ; PAIR COUNTS USE ELIGIBLE FAILED STRESSED CASES . Mdl.

I⋆

ρ

BA

Sp nP /nE nϕB /nE

Q4 1.000 1.000 .869 .938 M8 1.000 1.000 .943 .885 G12 .887 .932 .876 .779

16/16 28/28 31/45

16/16 28/28 22/45

In this subcase, we tested controlled-failure detection, intervention validity and responsiveness, and independent re-audit coverage. Correction targets shortcut or conflict observations failing (8); all 89 eligible failures form the denominator, and missing pairs are not imputed. Balanced accuracy is .869– .943 and specificity .779–.938. Q4/M8 attain complete validity, responsiveness, and pair coverage. G12 remains responsive when valid but retains 31/45 performance and 22/45 ϕAbl,Ref pairs because its strict structured-output rate is .887. The audit therefore transfers cleanly to Q4/M8 while exposing G12 adapter attrition. IV. C ONCLUSION In this letter, we proposed a task-conditional faithfulness audit for multimodal LLM-based grid diagnosis. The framework jointly evaluates performance, reported and behavioral reliance, explanation fidelity, signed modality effects, and engineering reference importance. The results show that reported alignment can exceed behavioral grounding under stress, while targeted correction improves both performance and grounding. Modelspecific responses to shortcut and conflicting text further motivate reporting validity and responsiveness alongside aggregate faithfulness scores. Future work will evaluate the audit in utility-scale grid workflows. R EFERENCES [1] A. Zaboli, J. Hong, A. Stefanov, C.-C. Liu, and C.-S. Hwang, “Large language models for power system security: A novel multi-modal approach for anomaly detection in energy management systems,” IEEE Access, vol. 13, pp. 203558–203585, 2025. [2] Y. Cheng et al., “A large language model for advanced power dispatch,” Sci. Rep., vol. 15, Art. no. 8925, 2025. [3] Y. Liu, H. Wu, and B. Liu, “Evaluation of large language models for numeric anomaly detection in power systems,” arXiv preprint arXiv:2511.21371, 2025. [4] J. Yuan, C. Liu, B. Zheng, and M. Pan, “Multi-modal fusion fault diagnosis for high-voltage transformers based on STFT-ResBIGRUNet,” Sci. Rep., vol. 15, Art. no. 44390, 2025. [5] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain-ofthought prompting,” in Adv. Neural Inf. Process. Syst., vol. 36, 2023. [6] M. Chen, Y. Cao, Y. Zhang, and C. Lu, “Quantifying and mitigating unimodal biases in multimodal large language models: A causal perspective,” in Findings Assoc. Comput. Linguistics: EMNLP, 2024, pp. 16449–16469. [7] Ollama, “Ollama model library,” 2026. [Online]. Available: https://ollama. com/library. Accessed: Jul. 22, 2026.