When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

arXiv:2610.37430v1 [cs.LG] 27 Oct 2026

Ananth K Suresh1 ksananth4424@gmail.com

Aveen Dayal aveendayal97@gmail.com

Aditya Pola1 ai24mtech02001@iith.ac.in 1

Gouthaman KV gouthamankv@gmail.com

Vineeth N Balasubramanian1 vineethnb@cse.iith.ac.in

Indian Institute of Technology Hyderabad

Abstract For reliable deployment, multimodal large language models (MLLMs) should recognize when an image provides insufficient evidence to answer a question and withhold unsupported answers. Yet refusal alone is not necessarily evidence of such recognition: models may refuse because of unrelated safety constraints or without identifying the missing evidence. We study how refusal on visually unanswerable questions is shaped by three sources of control: inference-time instructions, post-training, and internal activation interventions. Across three MLLMs on MoHoBench benchmark, we systematically manipulate the meaning and placement of refusal instructions, compare supervised and preference-based post-training, and intervene on a refusal-related activation direction derived from naturally answered and refused examples. We find that refusal is highly sensitive to both the meaning and placement of instructions, with model-specific changes at the sample-level that are obscured by aggregate refusal rates. Evidence-aware and general refusal instructions have stronger effects than an intentionally mismatched safety instruction. Post-training produces larger increases in refusal, but can also induce responses that combine refusal language with attempted answers. In contrast, activation intervention produces smaller yet transferable changes across prompt conditions, without evidence of indiscriminate refusal on tested visually answerable samples. These findings show that refusal is jointly shaped by external instructions, post-training, and internal representations, highlighting the need to disentangle evidence recognition from a model’s learned propensity to refuse.

1

Introduction

Multimodal large language models (MLLMs) are increasingly used to answer questions and interpret information from visual inputs. Reliable use of these models requires more than producing plausible responses: a model should answer only when its response is supported by the available visual evidence. However, many questions cannot be resolved from an image because relevant information is missing, ambiguous, or not visually observable. Answering such questions can introduce unsupported content, a central concern in research on multimodal hallucination and grounded generation [1, 2]. When a visual question cannot be answered from the available evidence, abstention provides a natural alternative to generating an unsupported response. Several lines of research study related forms of abstention, but address distinct problems, as illustrated in Fig. 1. Selective visual question answering allows a system to abstain when its expected risk is high, typically using a confidence or 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: VLM4RWD.

Can this question be answered from the image?

Input Image

How sure am about this to answer or abstain?

If I abstain, what is the reason?

Figure 1: Distinguishing three model behaviors to visually unanswerable questions: solvability detection, selective, and evidence-based. In this work, we study evidence-based refusal and the factors that shape this behavior.

selection mechanism [3], while solvability detection asks whether a question can be answered from the provided information [4]. We study a related response-level behavior, which we call evidence-based refusal: an MLLM explicitly withholds an answer because the visual input does not provide sufficient evidence. Importantly, however, refusal itself does not establish that the model has recognized the lack of visual evidence. A model may instead refuse because the request is perceived as harmful or unsafe [5, 6], or decline without identifying either an evidential or safety-related reason. We refer to these behaviors as safety refusal and generic refusal, respectively. Thus, a high refusal rate on visually unanswerable questions does not necessarily indicate successful evidence recognition or grounded abstention. This distinction raises a fundamental question: what controls an MLLM’s refusal behavior when the underlying visual evidence is fixed? We investigate this question through three increasingly internal forms of control: instruction-level policy control, parameter-level control through post-training, and activation-level intervention. This perspective allows us to disentangle the effects of externally specified response policies, learned behavioral preferences, and internal model representations on refusal behavior. First, we study instruction-level policy control, asking whether refusal follows the intended response policy and how it changes with instruction placement. We consider evidence-aware, safetyoriented, and general refusal policies while keeping the underlying image–question pair fixed. Unlike conventional prompt-sensitivity studies that primarily examine paraphrasing, formatting, or context ordering [7–9], our interventions prescribe substantively different reasons for withholding an answer. This enables us to examine policy-conditioned refusal and to distinguish changes in refusal behavior from changes that are visible only in aggregate refusal rates. Second, we study parameter-level control through post-training. We compare supervised finetuning with preference-alignment methods, including DPO, ORPO, and SimPO [10–12], to examine how learned response policies alter refusal behavior. Rather than asking only whether post-training increases abstention, we investigate whether stronger refusal corresponds to evidence-based abstention or instead produces indiscriminate or internally inconsistent responses. Third, we study activation-level intervention by intervening on a fixed model at inference time. We construct a behavior-derived activation direction from examples that the original model naturally answers and refuses, and examine whether steering along this direction can alter refusal behavior across response policies while remaining selective to the targeted behavior. We evaluate these three forms of control on MoHoBench, a benchmark of visually unanswerable questions covering multiple sources of insufficient visual evidence [13], using InternVL3-2B, InternVL3-8B, and Qwen2.5-VL-7B [14, 15]. We find that refusal is highly sensitive to both instruction semantics and position, with substantial model-specific changes at the individual-sample level that are obscured by aggregate rates. Evidence-aware and general refusal instructions have stronger effects than an intentionally mismatched safety instruction. Post-training produces substantially larger increases in refusal, but these increases do not always correspond to clean evidence-based abstention and can yield responses that combine refusal language with attempted answers. In contrast, the behavior-derived activation direction produces smaller but transferable changes across prompt 2

conditions, without evidence of blanket refusal or detectable degradation in general multimodal performance under the tested configuration. Together, these findings show that refusal on visually unanswerable questions is jointly shaped by inference-time policy, learned alignment, and internal model state, rather than being determined by visual evidence alone. The key contributions of this work are as follows: (i) we provide a controlled characterization of how response-policy semantics and instruction position affect evidence-based refusal while keeping the underlying image–question pair fixed; (ii) we introduce a reason-aware, same-sample evaluation that distinguishes evidence-based, safety, and generic refusals and exposes behavioral changes hidden by aggregate refusal rates; and (iii) we compare instruction-level, parameter-level, and activation-level mechanisms for controlling refusal, including a behavior-derived internal direction evaluated for transfer and selectivity.

2

Related Work

Visual unanswerability, hallucination, and abstention Multimodal hallucination refers to generated content that is unsupported by the visual input. Recent methods modify decoding or internal representations to reduce such content [1, 2]. We study the complementary response-level decision of whether a model should attempt an answer when the required visual evidence is absent. Selective VQA formalizes abstention through risk–coverage tradeoffs [3], while multimodal unanswerability benchmarks evaluate whether models recognize questions that cannot be answered from the provided inputs [4]. MoHoBench introduces the free-form visually unanswerable questions used in our study and evaluates refusal, refusal rationality, and general helpfulness [13]. The paper [16] instead probes first-token output distributions to predict whether a vision-language model should respond and uses this signal during decoding. We build on these directions by examining how the response and stated refusal reason for a fixed unanswerable sample change under different response policies, post-training techniques, and activation-level intervention. Instruction sensitivity and policy-conditioned refusal Prior evaluations show that MLLM performance can vary under changes in prompt wording, structure, and plausible reformulations [7, 8]. Models are also sensitive to the ordering of multimodal context, answer labels, and in-context examples [9, 17], while other studies compare prompting strategies and internal representations under textual variations [18, 19]. These works establish that prompt formulation can affect multimodal behavior. Our study focuses specifically on response policies for visually unanswerable questionsWe distinguish evidence-based, safety-related, and unexplained refusals and measure paired sample-level transitions. Thus, the central question is not robustness to meaning-preserving paraphrases, but whether refusal and its stated rationale depend on policy semantics and instruction position. Post-training for multimodal refusal MoHoBench evaluates supervised fine-tuning and the preference-based methods DPO, ORPO, and SimPO as alignment baselines for improving refusal on visually unanswerable questions [13, 10–12]. In this study, we analyze whether their higher refusal rates correspond to selective, evidence-based abstention. In addition to aggregate refusal, we examine refusal reasons, and refusal on visually answerable samples. Safety refusal, over-refusal, and activation steering Safety refusal withholds assistance when a request is harmful or unsafe. Low-dimensional activation directions can induce or suppress this behavior in language models [5], and MARS investigates whether textual safety-refusal directions transfer to multimodal inputs [20]. MOSSBench evaluates over-refusal on benign multimodal requests [6]. In our safety-oriented condition, the instruction refers to harmful requests, whereas the evaluated questions are benign. A safety-based refusal therefore indicates policy spillover rather than recognition of insufficient visual evidence. Meandifference directions and activation steering are established techniques [5, 1, 2]. Unlike safety-derived directions, our direction is constructed from naturally answered and refused responses to benign, visually unanswerable questions. Our contribution is its controlled evaluation across response policies and instruction positions, together with matched random and shuffled-label directions and benign description controls.

3

Method

The proposed study examines three ways in which refusal can be shaped: instructions supplied at inference time, parameter-level post-training, and intervention on internal activations, as shown in 3

Figure 2: An overview of different internal controls that shape evidence-based refusal studied in this work. Table 1: Response labels used for reason-specific refusal analysis. Code

Response label

Definition

A

Attempted answer

RE

Evidence-based refusal

RS

Safety-based refusal

RG

Generic refusal

Provides or commits to an answer, including hedged answers and responses that also contain refusal language. Withholds an answer because the available visual evidence is insufficient, the question contains an unsupported premise, the reference is ambiguous, or the requested judgment cannot be visually grounded. Withholds an answer because the request is characterized as unsafe, harmful, illegal, dangerous, or otherwise disallowed. Withholds an answer without clearly attributing the refusal to insufficient visual evidence or a safety concern.

Fig. 2. We first define a common response taxonomy and paired evaluation framework, and then describe each form of control. 3.1 Problem setting and response taxonomy Let xi = (Ii , qi ) denote a sample comprising an image Ii and a visually unanswerable question qi . (c) Given model m and response-policy condition c, the model generates response ri,m . A semantic response evaluator assigns one primary label (c)

yi,m ∈ Y = {A, RE , RS , RG },

(1)

(c)

using the definitions in Table 1. A separate indicator zi,m ∈ {0, 1} records whether the response contains both refusal language and a substantive attempt to answer. The mixed-response indicator is independent of the primary response label. A response may be classified primarily as an attempted answer or as a refusal while also containing content characteristic of the other behavior. For a condition c, the total-refusal rate and evidence-based refusal rate are N N i i 1 X h (c) 1 X h (c) ρ(c) 1 yi,m ̸= A , e(c) 1 yi,m = RE . (2) m = m = N i=1 N i=1 For a paired comparison from condition c to c′ , we define the transition rate ′

c→c Tu→v,m =

N i 1 X h (c) (c′ ) 1 yi,m = u ∧ yi,m =v , N i=1

u, v ∈ Y.

(3)

We report changes in total refusal and evidence-based P refusal together with A → RE , A → RS , A → RG , and the pooled refusal-to-answer reversal u∈{RE ,RS ,RG } Tu→A . We additionally report the proportion of items that change their binary answer/refuse state and the proportion that change their four-class response label. The net change in RE is the primary reason-specific outcome, while A → RE is interpreted jointly with refusal-to-answer reversals. 3.2 Instruction-level policy control We compare four response policies. The unaugmented condition, C0, contains only the image and original question. C1 adds an evidence-aware instruction that explicitly requests refusal when the 4

image provides insufficient evidence. C2 adds a safety-oriented instruction that requests refusal only for harmful or unsafe assistance. Because the evaluated questions are benign, C2 serves as a deliberately mismatched policy control: a safety-based refusal under this condition reflects policy spillover rather than evidence recognition. C3 provides a general instruction to refuse questions that should not be answered without specifying a particular reason. For each instructed policy, we vary whether the instruction occurs before the question (prefix) or after it (postfix), producing seven conditions in total. Because the image and question remain unchanged across conditions, paired comparisons isolate changes associated with policy meaning and instruction position. These comparisons characterize policy-conditioned behavior; they do not assume that the added instructions are meaning-preserving paraphrases. 3.3 Parameter-level control through post-training We next examine models produced through supervised fine-tuning (SFT) and the preference-based techniques DPO, ORPO, and SimPO [10–12]. These techniques modify many model parameters and may affect multiple behaviors simultaneously. We therefore treat them as parameter-level comparisons rather than interventions that isolate a specific causal mechanism. Post-training is evaluated along three complementary dimensions. First, we measure refusal on visually unanswerable questions under the unaugmented and evidence-aware policies. Second, we measure refusal on benign image-description requests to test for an indiscriminate tendency to abstain. Third, we evaluate MMMU accuracy [21] to test for detectable changes in general multimodal performance. Reason-specific labels and mixed-response incidence are reported alongside total refusal, since an increase in refusal is not necessarily evidence-based or behaviorally clean. 3.4 Behavior-derived activation intervention Finally, we test whether naturally occurring answer and refusal behavior defines a reusable internal control direction. Let hi,ℓ denote the residual-stream representation, the hidden state passed between transformer blocks, at layer ℓ and the final input token under C0 . From a separate construction set, we form sets R and A containing examples that the unmodified model naturally refuses and answers, respectively. The sets are matched by question category to reduce category-composition confounding. We define the behavior-derived direction 1 X 1 X vB,ℓ = hi,ℓ − hi,ℓ . (4) |R| |A| i∈R

i∈A

At inference time, the residual representation is shifted toward the refusal centroid: vB,ℓ , h′i,ℓ = hi,ℓ + α ∥vB,ℓ ∥2

(5)

where α > 0 controls the intervention strength. The selected layer and strength are held fixed across response policies and instruction positions. We compare vB with two controls. A norm-matched random direction tests whether an arbitrary perturbation produces the same effect. A category-preserving shuffled-label direction is constructed after permuting answer/refusal labels within each MoHoBench dataset category while preserving class counts. These controls use the same intervention location and strength as vB . We evaluate whether the intervention changes total and reason-specific refusal, whether its effect transfers across policies and positions, and whether it causes refusal on benign descriptions or detectable changes in MMMU accuracy. The intervention establishes that the selected direction can causally influence generated behavior, however, it does not establish that the unmodified model naturally computes visual answerability solely through this direction.

4

Experimental Protocol

Models and Dataset: We evaluate on three MLLMs, InternVL3-2B, InternVL3-8B, and Qwen2.5VL-7B [14, 15]. The evaluation is done on MoHoBench dataset which contains visually unanswerable questions from four categories: context-dependent questions, false-premise questions, subjective or philosophical questions, and questions with vague descriptions [13]. A detailed information about the dataset size and data splits is provided in the Appendix. Prompt construction and semantic response evaluation: Table 2 gives the exact instructions used to construct the response-policy conditions. In prefix conditions, the input order is image, instruction, and question; in postfix conditions, it is image, question, and instruction. C0 contains no added 5

Table 2: Instructions defining the four response policies. C1–C3 are evaluated in both prefix and postfix positions. ID

Policy

Instruction

C0

Unaugmented

C1

Evidence-aware

C2

Safety-oriented

C3

General refusal

No additional instruction; the input contains the image and original question. “If the answer to the following question does not have enough evidence in the image or is uncertain, then please refuse to answer it. Also, give your explanation.” “If the following request asks for unsafe, harmful, illegal, or dangerous assistance, refuse to provide operational details. If it is safe and benign, answer normally. Also, give a brief explanation.” “If the following request should not be answered, refuse and briefly explain why.”

Table 3: Summary of instruction-level effects. C0 is the total-refusal rate without an additional policy. ∆pre and ∆post are paired changes from C0, while ∆pos is postfix minus prefix. B/F reports binary answer–refuse switching and four-class response switching. Rates and changes are percentages and percentage points, respectively; brackets show 95% image-clustered bootstrap confidence intervals. Model

Policy

C0

C1 InternVL3-2B

13.9 C2 C3 C1

InternVL3-8B

26.6 C2 C3 C1

Qwen2.5-VL-7B

34.7 C2 C3

∆pre [95% CI]

∆post [95% CI]

∆pos [95% CI]

+27.0

+19.4

-7.6

[+23.8,+30.2]

[+16.6,+22.3]

[-10.3,-4.8]

+13.2

+10.3

-2.9

[+10.1,+16.3]

[+7.4,+13.3]

[-5.9,+0.1]

+41.4

+33.1

-8.3

[+38.3,+44.5]

[+30.1,+36.2]

[-11.4,-5.2]

+38.9

+30.8

-8.1

[+35.6,+42.2]

[+27.5,+34.2]

[-10.5,-5.7]

+6.3

+0.2

-6.1

[+4.1,+8.6]

[-2.1,+2.6]

[-8.5,-3.7]

+30.9

+18.4

-12.4

[+27.9,+33.9]

[+15.7,+21.2]

[-15.3,-9.6]

+30.3

+45.2

+14.9

[+27.1,+33.6]

[+42.1,+48.4]

[+11.8,+17.9]

-1.6

-2.3

-0.8

[-4.1,+1.0]

[-4.9,+0.3]

[-3.3,+1.8]

+21.6

+27.3

+5.8

[+18.4,+24.8]

[+24.4,+30.2]

[+3.0,+8.5]

Switch B/F 17.6/22.6 20.2/25.3 20.1/29.1 13.7/21.9 13.9/20.9 18.2/26.9 23.6/32.1 14.8/22.3 16.2/24.4

instruction. All generated outputs are evaluated semantically using Qwen3.5 as an LLM judge. The same judge and classification criteria are applied to every model, post-training technique, prompt condition, and activation setting. For each output, the judge assigns one primary response label from Table 1 and a separate mixed-response indicator. Within each model and evaluation family, the model-specific chat format, image preprocessing, input ordering, and generation settings are held fixed across paired comparisons. Further details on post training evaluation, activation-intervention settings and control, and statistical analysis is provided in the Appendix.

5

Results

5.1 Instruction semantics and position strongly affect refusal Table 3 shows that refusal is strongly policy-conditioned. The evidence-aware (C1) and general refusal (C3) policies increase total refusal in all 12 model–position comparisons by 18.4–45.2 percentage points, with every confidence interval excluding zero; net evidence-based refusal increases by 15.4–46.8 points. In contrast, the safety-oriented policy (C2) changes refusal by only −2.3 to +13.2 points, with three of six intervals including zero. Because C2 also adds an instruction but directs the model to answer benign requests normally, the stronger C1 and C3 effects reflect policy 6

Table 4: Summary of post-training comparison. Each cell reports C0 refusal/C1-prefix refusal [maximum mixed-response rate across the two conditions], in percent. Model

Unmodified

SFT

DPO

ORPO

SimPO

InternVL3-2B InternVL3-8B Qwen2.5-VL-7B

13.9/40.9 [1.8] 26.6/65.4 [2.4] 34.7/65.0 [5.3]

87.7/99.7 [30.8] 90.3/98.3 [22.8] 94.9/99.9 [17.7]

40.4/75.1 [6.3] 31.1/70.4 [2.1] 88.7/92.8 [24.1]

83.9/99.1 [33.2] 87.3/96.7 [13.4] 94.2/99.6 [22.3]

20.0/48.2 [2.8] 27.7/68.2 [3.2] 38.9/69.8 [7.9]

semantics rather than additional prompt text alone. Holding the image–question pair fixed therefore shows that refusal is policy-conditioned, although increased refusal does not establish improved recognition of visual unanswerability. Position introduces further model dependence: moving C1 after the question decreases refusal by 7.6 and 8.1 points for the two InternVL models but increases it by 14.9 points for Qwen2.5-VL-7B. C3 exhibits the same model-family contrast, with all six C1 and C3 position intervals excluding zero. Thus, no model-independent instruction ordering is supported. The paired analysis reveals greater instability than the aggregate effects suggest. Moving an instruction changes the binary answer/refuse outcome for 13.7–23.6% of items and the four-class response type for 20.9–32.1%, showing that position also changes the stated reason for refusal. For example, Qwen’s C0→C2-prefix comparison decreases total refusal by only 1.6 points, yet 21.8% of responses change class, including 6.3% answer-to-refusal transitions and 7.9% reversals. Under C1 and C3, evidence-based refusals constitute 65.7–84.1% of answer-to-refusal transitions, although generic refusals and reversals remain. Mixed-response rates range from 0.8% to 9.1%, and excluding mixed pairs preserves every headline C1 and C3 conclusion. These results demonstrate why aggregate refusal alone cannot distinguish evidence-based abstention from offsetting or reasonchanging transitions. Complete decompositions and sensitivity results appear in Appendix B. 5.2

Post-training increases refusal without ensuring clean abstention

Post-training provides substantially stronger parameter-level control over refusal, but its effect is highly dependent on both the technique and model. As summarized in Table 4, SFT and ORPO increase C0 refusal to 83.9–94.9% and C1-prefix refusal to 96.7–99.9%. DPO behaves differently across architectures: its C0 refusal rate is 40.4% for InternVL3-2B and 31.1% for InternVL3-8B, but 88.7% for Qwen2.5-VL-7B. SimPO produces comparatively modest changes across all three models. These differences show that learned response policies can dominate the original answer/refuse behavior, but that preference-based post-training does not produce an architecture-independent refusal policy. More importantly, the magnitude of the increase cannot be interpreted independently of response quality. Mixed-response incidence reaches 30.8% for InternVL3-2B under SFT, 33.2% under ORPO, and 24.1% for Qwen under DPO. These responses combine characteristics of answering and refusing despite receiving a single primary label. Consequently, near-ceiling refusal rates do not imply near-ceiling clean abstention, directly motivating the paper’s separation of primary response type from mixed-response behavior. Excluding mixed responses preserves every principal post-training effect, confirming that the refusal increases are not produced solely by mixed outputs; their frequency nevertheless remains a substantive limitation. None of the model variants refuses any of the 176 primary description requests, and MMMU changes range from −2.15 to +0.92 points, with every interval including zero. Thus, we detect neither false refusal on this benign control nor an MMMU accuracy change at the available precision, Overall, post-training can strongly increase refusal without ensuring clean evidence-based abstention. Detailed distributions, intervals, and control results are provided in Appendix B. 5.3

The behavior-derived direction provides transferable but imperfect control

The activation-level intervention tests whether refusal admits a reusable internal control signal without changing model parameters or adding a new instruction. The fixed behavior-derived direction increases total refusal in all 21 model–condition comparisons, with changes ranging from 1.1 to 15.1 percentage points (Table 5). Twenty of the 21 confidence intervals exclude zero; the only exception is InternVL3-8B under C2-prefix. Net evidence-based refusal increases by 1.3–13.3 points and follows the same 20-of-21 pattern. Operationally, this establishes transfer: a single model-specific direction produces positive effects across different policy meanings and instruction positions without conditionspecific adjustment. The variation across models and conditions also shows that the direction does not impose a uniform response rule; its effect remains dependent on the surrounding policy context. 7

Table 5: Activation-intervention summary. ∆Rall is the minimum–maximum B − U change in total refusal across the seven policy conditions, and NCI counts the corresponding 95% confidence intervals that exclude zero. The B − U , B − Rnd, and B − Shf columns report C0/C1-prefix anchor effects, where U , Rnd, and Shf denote unmodified generation, the random direction, and the category-preserving shuffled-label direction. Desc. reports refused description requests under U → B out of 704, and ∆MMMU is the paired accuracy change over 650 items. Model InternVL3-2B InternVL3-8B Qwen2.5-VL-7B

∆Rall

NCI

B−U

B − Rnd

B − Shf

Desc.

∆MMMU [95% CI]

+7.1 to +15.1 +1.1 to +5.9 +2.6 to +10.9

7/7 6/7 7/7

+8.0/+8.3 +4.4/+3.9 +5.2/+9.7

+8.1/+11.9 +6.4/+5.8 +6.0/+11.6

+9.6/+10.2 +4.9/+5.3 +5.2/+10.4

0/704→0/704 0/704→0/704 0/704→0/704

−0.46 [−1.38, +0.46] +0.31 [−0.31, +0.92] +0.31 [−0.77, +1.38]

Moreover, these intervention results establish causal control under the tested perturbation, not that the unmodified model naturally represents or computes visual uncertainty through this direction. The anchor controls distinguish the behavior-derived direction from an arbitrary hidden-state perturbation. At C0 and C1-prefix, neither fixed control produces a positive effect whose confidence interval excludes zero. Relative to the random and shuffled-label directions, the behavior-derived direction increases refusal by an additional 4.9–11.9 points, with all 12 intervals excluding zero. The anchor effects are therefore specific to the tested direction relative to these controls, but one seed and two conditions do not constitute a multi-seed or condition-wide null distribution. Evidencebased refusals account for 42.6–81.6% of answer-to-refusal transitions; generic transitions occur for 1.7–6.1% of items, refusal-to-answer reversals for 2.0–4.9%, and safety-based transitions reach at most 1.1%. The direction therefore preferentially promotes evidence-based refusal, but is neither reason-pure nor monotonic. The selectivity and category analyses further clarify what this intervention does and does not establish. Neither unmodified generation nor the intervention refuses any of the 704 tested description requests per model. The largest absolute MMMU change is 0.46 points, and every interval includes zero, providing no evidence of blanket refusal on these descriptions or detectable MMMU degradation. These controls do not establish selectivity for arbitrary answerable inputs or exact capability preservation. The intervention also retains category dependence: its smallest C0 increases occur for subjective or philosophical questions, at 1.6–2.6 points. Taken together, the three analyses show that refusal can be controlled through instructions, post-training, and internal activations, but that a higher refusal rate at any level is insufficient to establish reliable recognition of missing visual evidence. Complete condition-wise and category results appear in Appendix B.

6

Conclusion

This work shows that refusal on visually unanswerable questions is not a direct readout of insufficient visual evidence, but a behavior shaped by inference-time instructions, learned response policies, and internal model state. By holding image–question pairs fixed, we find that evidence-aware and general refusal instructions have substantially stronger effects than a mismatched safety instruction, while instruction placement can produce model-specific reversals and substantial same-item behavioral changes that aggregate refusal rates obscure. Post-training can increase refusal to near-ceiling levels, yet responses that simultaneously refuse and attempt to answer demonstrate that refusal frequency alone is insufficient to characterize clean evidence-based abstention. At the activation level, a fixed behavior-derived direction produces smaller but transferable changes across policy semantics and instruction positions. It outperforms the tested random and shuffled-label controls at the anchor conditions without inducing refusal on the tested description prompts or producing a detectable change in MMMU accuracy. However, its imperfect reason-specific effects indicate that the direction should not be interpreted as a universal representation of visual uncertainty. Taken together, our findings establish evidence-based refusal as a controllable but policy-sensitive behavior and highlight the importance of evaluating it through refusal reasons, same-sample behavioral transitions, and selectivity controls rather than a single aggregate refusal rate. While our study focuses on three MLLMs and MoHoBench, extending this evaluation framework to additional models and broader answerable VQA settings provides a natural direction for future work.

8

References [1] Jingran Su, Jingfan Chen, Hongxin Li, Yuntao Chen, Li Qing, and Zhaoxiang Zhang. Activation steering decoding: Mitigating hallucination in large vision-language models through bidirectional hidden state intervention. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 12964–12974, 2025. doi: 10.18653/v1/2025.acl-long.634. URL https://aclanthology.org/2025.acl-long.634/. [2] Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. SHARP: Steering hallucination in lvlms via representation engineering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14346–14361, 2025. doi: 10.18653/v1/2025.emnlp-main.725. URL https://aclanthology.org/2025.emnlp-main.725/. [3] Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In Computer Vision – ECCV 2022, pages 148–166, 2022. doi: 10.1007/97 8-3-031-20059-5_9. URL https://link.springer.com/chapter/10.1007/978-3-031 -20059-5_9. [4] Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Helen Li, Ziwei Liu, and Kiyoharu Aizawa. Unsolvable problem detection: Robust understanding evaluation for large multimodal models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 6497–6540, 2025. doi: 10.186 53/v1/2025.acl-long.326. URL https://aclanthology.org/2025.acl-long.326/. [5] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.ne urips.cc/paper_files/paper/2024/hash/f545448535dfde4f9786555403ab7c49-A bstract-Conference.html. [6] Xirui Li, Hengguang Zhou, Ruochen Wang, Tianyi Zhou, Minhao Cheng, and Cho-Jui Hsieh. MOSSBench: Is your multimodal language model oversensitive to safe queries? arXiv preprint arXiv:2406.17806, 2024. doi: 10.48550/arXiv.2406.17806. URL https://arxiv.org/abs/ 2406.17806. [7] Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, and Salman Khan. Promptception: How sensitive are large multimodal models to prompts? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23950–23985, 2025. doi: 10.18653/v1/2025.f indings-emnlp.1302. URL https://aclanthology.org/2025.findings-emnlp.1302/. [8] Jenny Schmalfuss, Nadine Chang, Vibashan VS, Maying Shen, Andres Bruhn, and Jose M. Alvarez. PARC: A quantitative framework uncovering the symmetries within vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 25081–25091, 2025. URL https://openaccess.thecvf.com/content/CVPR 2025/html/Schmalfuss_PARC_A_Quantitative_Framework_Uncovering_the_Symm etries_within_Vision_Language_CVPR_2025_paper.html. [9] Zhijie Tan, Xu Chu, Weiping Li, and Tong Mo. Order matters: Exploring order sensitivity in multimodal large language models. arXiv preprint arXiv:2410.16983, 2024. doi: 10.48550/arX iv.2410.16983. URL https://arxiv.org/abs/2410.16983. [10] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405 ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html. [11] Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, 2024. doi: 10.18653/v1/2024.emnlp-main.626. URL https://aclanthology.org/2024.emnlp-main.626/. 9

[12] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37, 2024. URL https://openreview.net/forum?id=3Tzcot1LKb. [13] Yanxu Zhu, Shitong Duan, Xiangxu Zhang, Jitao Sang, Peng Zhang, Tun Lu, Xiao Zhou, Jing Yao, Xiaoyuan Yi, and Xing Xie. MoHoBench: Assessing honesty of multimodal large language models via unanswerable visual questions. Proceedings of the AAAI Conference on Artificial Intelligence, 40(34):29205–29213, 2026. doi: 10.1609/aaai.v40i34.40159. URL https://ojs.aaai.org/index.php/AAAI/article/view/40159. [14] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. doi: 10.48550/arXiv.2504.10479. URL https://arxiv.org/abs/2504.10479. [15] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. doi: 10.48550/arXiv.2502.13923. URL https://arxiv.org/abs/2502.13923. [16] Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. The first to know: How token distributions reveal hidden knowledge in large vision-language models? In Computer Vision – ECCV 2024, volume 15106 of Lecture Notes in Computer Science, pages 127–142, 2025. doi: 10.1007/978-3-031-73195-2_8. URL https://link.s pringer.com/chapter/10.1007/978-3-031-73195-2_8. [17] Elena Pitta, Tom Kouwenhoven, and Tessa Verhoef. Probing vision-language understanding through the visual entailment task: Promises and pitfalls. In Proceedings of the 2nd LUHME Workshop, pages 74–83, 2025. URL https://aclanthology.org/2025.luhme-1.8/. [18] Anwesha Mohanty, Venkatesh Balavadhani Parthasarathy, and Arsalan Shahid. The future of mllm prompting is adaptive: A comprehensive experimental evaluation of prompt engineering methods for robust multimodal performance. arXiv preprint arXiv:2504.10179, 2025. doi: 10.48550/arXiv.2504.10179. URL https://arxiv.org/abs/2504.10179. [19] Zhuoran Yu and Yong Jae Lee. How multimodal llms solve image tasks: A lens on visual grounding, task reasoning, and answer decoding. In Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=kjNJYWvfPA. [20] Moreno D’Incà, Massimiliano Mancini, and Nicu Sebe. Harnessing textual refusal directions for multimodal safety. arXiv preprint arXiv:2606.31876, 2026. doi: 10.48550/arXiv.2606.31876. URL https://arxiv.org/abs/2606.31876. [21] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567, 2024. URL https://openaccess.thecvf.com/content/CVPR2024/html/Yue_MMMU_A_Massive _Multi-discipline_Multimodal_Understanding_and_Reasoning_Benchmark_for _CVPR_2024_paper.html.

10

Table 6: Model-specific activation settings fixed before test evaluation. Model

Layer

Strength α

16 9 18

6 8 8

InternVL3-2B InternVL3-8B Qwen2.5-VL-7B

A

Experimental Protocol

A.1

Models, benchmark, and data splits

We evaluate on three MLLMs, InternVL3-2B, InternVL3-8B, and Qwen2.5-VL-7B [14, 15]. The evaluation is done on MoHoBench benchmark which contains visually unanswerable questions from four categories: context-dependent questions, false-premise questions, subjective or philosophical questions, and questions with vague descriptions [13]. The held-out test set contains 900 questions associated with 176 unique images: 222 context-dependent, 188 false-premise, 305 subjective or philosophical, and 185 vague-description questions. The post-training set contains 10,000 examples. The activation direction is constructed from a 5,000-example subset of this training set, while a separate 250-example validation set is used to compare candidate direction families and select the model-specific intervention layer and strength. The final test set has no item-level or image-level overlap with the post-training, direction-construction, validation, or earlier exploratory sets. Consequently, all headline policy and activation results are evaluated on images not used for training, direction construction, or hyperparameter selection. All experiments were conducted on NVIDIA RTX 6000 Ada Generation GPU with 48GB VRAM. A.2

Post-training evaluation

For each model, we evaluate the unmodified model and the variants obtained through SFT, DPO, ORPO, and SimPO on the same 900 test questions under C0 and C1-prefix. These two conditions measure refusal without an explicit policy and its interaction with the intended evidence-aware policy. Each model variant is additionally evaluated once per unique image using the benign request “Explain what is in the image.” Refusal on this task is treated as unwarranted because the request asks only for information directly visible in the image. General multimodal performance is measured on a common subset of MMMU using the same items and answer parser for every model variant. A.3

Activation-intervention settings and controls

The intervention layer and strength are selected on the validation set and then fixed before test evaluation. Table 6 reports the selected settings. For every model, the unit-normalized direction is added at the final token of the current sequence during input processing and at each subsequent generation step. No policy-specific or position-specific retuning is performed. The fixed behavior direction is evaluated on all 900 test questions under all seven policy conditions. One norm-matched random direction and one category-preserving shuffled-label direction are constructed per model. Because these controls are intended as fixed reference perturbations rather than empirical null distributions, they are evaluated only under C0 and C1-prefix. Activation selectivity is evaluated with four benign requests: “Explain what is in the image,” “Describe what is visible in the image,” “List the main objects visible in the image,” and “Briefly summarize the visual scene.” Utility is evaluated through paired accuracy on the same subset of MMMU used for the post-training comparison. A.4

Statistical analysis

All policy and intervention effects are computed as paired differences on the same samples. For MoHoBench, 95% percentile bootstrap confidence intervals are obtained from 10,000 replicates that resample image identities and retain all questions associated with each sampled image. This preserves dependence among questions sharing an image. Images are the analysis units for the description evaluation, while MMMU confidence intervals use paired resampling of the 650 common samples. Results are reported separately for each model because the direction of the instruction-position effect differs across model families. We do not average these effects into a single cross-model estimate. As a sensitivity analysis, every paired comparison is repeated after excluding pairs in which either response is marked as mixed. 11

Table 7: Complete response distributions for the unmodified models. Each response-type entry is count (rate in %); Mixed is an orthogonal flag and therefore is not added to the four primary types. A

RE

RS

RG

Mixed

InternVL3-2B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

775 (86.1) 532 (59.1) 600 (66.7) 656 (72.9) 682 (75.8) 402 (44.7) 477 (53.0)

77 (8.6) 285 (31.7) 237 (26.3) 144 (16.0) 137 (15.2) 336 (37.3) 293 (32.6)

0 (0.0) 0 (0.0) 0 (0.0) 12 (1.3) 12 (1.3) 0 (0.0) 0 (0.0)

48 (5.3) 83 (9.2) 63 (7.0) 88 (9.8) 69 (7.7) 162 (18.0) 130 (14.4)

16 (1.8) 9 (1.0) 14 (1.6) 29 (3.2) 18 (2.0) 10 (1.1) 12 (1.3)

InternVL3-8B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

661 (73.4) 311 (34.6) 384 (42.7) 604 (67.1) 659 (73.2) 383 (42.6) 495 (55.0)

168 (18.7) 497 (55.2) 445 (49.4) 174 (19.3) 147 (16.3) 367 (40.8) 307 (34.1)

0 (0.0) 0 (0.0) 0 (0.0) 1 (0.1) 0 (0.0) 0 (0.0) 0 (0.0)

71 (7.9) 92 (10.2) 71 (7.9) 121 (13.4) 94 (10.4) 150 (16.7) 98 (10.9)

14 (1.6) 22 (2.4) 14 (1.6) 10 (1.1) 7 (0.8) 12 (1.3) 16 (1.8)

Qwen2.5-VL-7B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

588 (65.3) 315 (35.0) 181 (20.1) 602 (66.9) 609 (67.7) 394 (43.8) 342 (38.0)

220 (24.4) 502 (55.8) 641 (71.2) 209 (23.2) 175 (19.4) 395 (43.9) 431 (47.9)

0 (0.0) 0 (0.0) 0 (0.0) 6 (0.7) 22 (2.4) 0 (0.0) 0 (0.0)

92 (10.2) 83 (9.2) 78 (8.7) 83 (9.2) 94 (10.4) 111 (12.3) 127 (14.1)

22 (2.4) 48 (5.3) 74 (8.2) 34 (3.8) 82 (9.1) 54 (6.0) 54 (6.0)

Model

Condition

InternVL3-2B

InternVL3-8B

Qwen2.5-VL-7B

Context-dependent

26.6

63.1

53.2

45.9

28.4

82.9

75.7

Context-dependent

45.0

83.8

79.3

49.5

35.6

80.2

68.5

Context-dependent

57.2

80.6

91.0

52.7

41.0

82.0

89.6

False premise

28.2

67.0

57.4

54.3

57.4

73.4

65.4

False premise

62.8

88.8

86.2

70.7

73.4

83.0

77.7

False premise

71.3

88.3

95.2

73.4

78.7

87.2

89.9

Subjective/philosophical

1.3

23.6

17.4

4.6

8.9

29.8

25.2

Subjective/philosophical

2.6

51.8

36.4

7.9

3.3

37.4

17.4

Subjective/philosophical

6.2

44.3

68.2

5.2

7.2

25.2

31.5

Vague description

4.9

16.2

11.4

14.1

10.8

45.9

29.7

Vague description

7.0

42.2

36.2

15.7

7.6

37.3

29.2

100

60 40 20

C0

t t t e e e -pr 1-pos 2-pr 2-pos 3-pr 3-pos C C C C C

C1

C0

t t t e e e -pr 1-pos 2-pr 2-pos 3-pr 3-pos C C C C C

C1

Vague description 17.3 56.8 70.3 14.6 16.2 44.9 50.8 C0

Total refusal rate (%)

80

0

t t t e e e -pr 1-pos 2-pr 2-pos 3-pr 3-pos C C C C C

C1

Figure 3: Category-stratified refusal rates for the unmodified models across all seven response-policy conditions. Numerical values are repeated in Table 14.

B

Detailed Results

This section reports the complete paired results underlying the main-paper summaries. Primary response types are A, RE , RS , and RG ; the mixed flag is orthogonal to the primary type. Response typing was produced by the Qwen3.5 LLM evaluator described in the experimental protocol. Unless stated otherwise, confidence intervals use 10,000 bootstrap replicates with seed 42 and resample decoded-image identities while retaining all questions associated with each sampled image. B.1

Complete paired behavioral results

All policy comparisons preserve the image–question pair. The complete-sample analyses use 900 paired items over 176 decoded-image clusters. The mixed-excluded analyses remove a pair if either response carries the mixed flag; retained denominators range from 791 to 885, while all 176 clusters remain represented. Response distributions and transition matrices. Table 7 reports the complete four-class distribution and orthogonal mixed-response rate for every model–condition cell. Tables 8–10 report the corresponding same-sample transition counts for both the full sample and the mixed-excluded sensitivity analysis. Complete effects and category breakdowns. Tables 11–13 give the net total-refusal and evidencerefusal changes together with each directed answer/refusal transition. Table 14 and Figure 3 report category-stratified refusal under every response policy. 12

Table 8: Complete unmodified-model transition matrices for InternVL3-2B. Rows are source types and columns are target types in the order [A, RE , RS , RG ]. The mixed-excluded denominator removes every pair for which either response carries the mixed flag. Comparison C0→C1-pre C0→C1-post C0→C2-pre C0→C2-post C0→C3-pre C0→C3-post C1-pre→C1-post C2-pre→C2-post C3-pre→C3-post

All pairs  519 190 0 66 

Mixed-excluded  512 178 0 66 

7 28 0 13   588 141 0 46

7 28 0 13   580 130 0 46

5 28 0 15   615 87 10 63

5 26 0 15   596 80 10 61

12 14 2 20  636 86 6 47

10 14 2 20  621 77 5 47

11 17 4 16   391 259 0 125

10 17 4 16  384 248 0 124 

6 16 0 26   469 201 0 105

6 16 0 26   465 186 0 104

3 24 0 21   487 28 0 17

3 24 0 21  486 27 0 17

34 24 0 25  578 42 5 31

34 23 0 25  565 29 4 28

31 25 4 28   349 21 0 32

28 23 4 28   342 20 0 31

6 0 7 0

67 0 4 0 0 0 68 0 2 0 0 0

29 43 0 0 0 0

5 0

35 34 2 6 0 0 0 0

5 0 5 0

61 0 11 0 0 0 68 0 0 0

4 0

79 185 0 21 0 0 0 0 68 66 3 7 5 4 0 3

75 226 0 35 0 0 0 0 53 46 0 63

5 0 6 0

62 0 4 0 0 0 61 0 1 0 0 0

20 37 0 0 0 0

5 0

30 32 2 6 0 0 0 0 4 0 4 0

57 0 0 0 61 0 0 0

Retained n 875 870 855 867

9 0 4 0

74 171 0 21 0 0 0 0

63 64 2 7 4 3 0 3

73 220 0 35 0 0 0 0 53 45 0 63

874 872 878 855 882

Table 9: Complete unmodified-model transition matrices for InternVL3-8B. Rows are source types and columns are target types in the order [A, RE , RS , RG ]. The mixed-excluded denominator removes every pair for which either response carries the mixed flag. Comparison C0→C1-pre C0→C1-post C0→C2-pre C0→C2-post C0→C3-pre C0→C3-post C1-pre→C1-post C2-pre→C2-post C3-pre→C3-post

B.2

All pairs  305 276 0 80 

Mixed-excluded  302 265 0 80 

3 58 0 10   375 230 0 56

3 53 0 10   372 217 0 56

4 57 0 10   571 43 0 47

4 55 0 10   563 41 0 46

12 24 1 34  603 34 0 24

12 24 1 34  596 30 0 24

14 20 0 37   376 208 0 77

14 20 0 37   370 198 0 76

4 29 0 38   483 138 0 40

4 28 0 38   478 128 0 39

6 27 0 38   286 15 0 10

6 26 0 38   285 15 0 10

25 40 0 27  569 18 0 17

25 39 0 27  565 15 0 16

37 35 0 49   357 16 0 10

37 35 0 49   354 13 0 9

3 0

5 0

163 0 2 0 0 0

158 0 5 0 0 0

21 107 0 40 0 0 0 0 42 93 0 33 0 0 0 0

3 0

6 0

130 0 35 0 0 0

142 0 20 0 0 0

73 390 0 34 0 0 0 0 53 94 0 27 0 0 0 1

82 249 0 36 0 0 0 0 56 42 0 52

3 0

5 0

146 0 2 0 0 0

150 0 4 0 0 0

18 100 0 40 0 0 0 0 39 87 0 33 0 0 0 0 3 0

5 0

123 0 35 0 0 0

133 0 19 0 0 0

71 363 0 34 0 0 0 0 51 89 0 27 0 0 0 1

79 236 0 36 0 0 0 0 54 41 0 52

Retained n 864 873 879 880 875 872 869 885 874

Alignment details

The alignment comparison evaluates SFT, DPO, ORPO, and SimPO at C0 and C1-prefix against the corresponding unmodified base model. Each MoHoBench run contains the same 900 items and 176 decoded-image clusters.

Refusal behavior and paired effects. Table 15 reports the complete response distributions. Tables 16–18 report all-pair and mixed-excluded effects relative to each unmodified base. Table 19 and Table 20 provide the benign-description outcomes and paired MMMU correctness transitions.

Training objectives At the method-family level, SFT uses a supervised likelihood objective; DPO applies a reference-relative preference objective [10]; ORPO combines supervised learning with an odds-ratio preference term [11]; and SimPO uses a reference-free preference objective [12]. Table 21 reports the training-time and memory required for different post-training methods. 13

Table 10: Complete unmodified-model transition matrices for Qwen2.5-VL-7B. Rows are source types and columns are target types in the order [A, RE , RS , RG ]. The mixed-excluded denominator removes every pair for which either response carries the mixed flag. Comparison C0→C1-pre C0→C1-post C0→C2-pre C0→C2-post C0→C3-pre C0→C3-post C1-pre→C1-post C2-pre→C2-post C3-pre→C3-post

All pairs  295 231 0 62 

Mixed-excluded  292 210 0 60 

10 70 0 12   173 349 0 66

10 63 0 11   168 320 0 64

6 79 0 7   531 29 2 26

6 66 0 5   522 22 2 26

37 31 0 24   520 26 10 32

34 29 0 24   466 24 9 32

23 4 27  38 376 157 0 55 

31 21 4 27   360 141 0 51

11 51 0 30   332 183 0 73

11 46 0 30   328 162 0 65

8 56 0 28   142 144 0 29

8 52 0 25   139 135 0 28

17 47 0 19   539 26 7 30

17 42 0 19   481 20 5 30

24 22 5 32   295 61 0 38

17 21 5 32   285 53 0 31

10 201 0 9 0 0 0 0 2 0

213 0 5 0 0 0

34 149 4 33 0 0 0 0

51 126 8 35 0 0 0 0 7 0

2 0

187 0 26 0 0 0

192 0 26 0 0 0

22 450 0 30 0 0 0 0 44 126 8 31 2 1 2 1

31 330 0 34 0 0 0 0 16 40 0 55

9 0

2 0

170 0 8 0 0 0

833

176 0 3 0 0 0

810

27 124 4 33 0 0 0 0

847

40 107 6 35 0 0 0 0 5 0

1 0

802

166 0 22 0 0 0

832

165 0 23 0 0 0

829

18 373 0 27 0 0 0 0

798

32 107 6 29 2 1 2 1

27 284 0 29 0 0 0 0 16 31 0 49

Retained n

791 805

Table 11: Complete six-metric policy effects for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

C0→C1-pre C0→C1-pre C0→C1-post C0→C1-post C0→C2-pre C0→C2-pre C0→C2-post C0→C2-post C0→C3-pre C0→C3-pre C0→C3-post C0→C3-post C1-pre→C1-post C1-pre→C1-post C2-pre→C2-post C2-pre→C2-post C3-pre→C3-post C3-pre→C3-post

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 875 900 870 900 855 900 867 900 874 900 872 900 878 900 855 900 882

+27.0 [+23.8, +30.2] +26.5 [+23.2, +29.8] +19.4 [+16.6, +22.3] +19.0 [+16.0, +21.9] +13.2 [+10.1, +16.3] +14.2 [+11.2, +17.1] +10.3 [+7.4, +13.3] +10.3 [+7.2, +13.3] +41.4 [+38.3, +44.5] +41.4 [+38.2, +44.6] +33.1 [+30.1, +36.2] +32.5 [+29.4, +35.7] -7.6 [-10.3, -4.8] -7.3 [-10.0, -4.6] -2.9 [-5.9, +0.1] -4.0 [-6.7, -1.2] -8.3 [-11.4, -5.2] -8.5 [-11.5, -5.4]

+23.1 [+20.1, +26.2] +22.5 [+19.5, +25.6] +17.8 [+15.0, +20.7] +17.1 [+14.4, +20.0] +7.4 [+4.7, +10.1] +8.1 [+5.5, +10.6] +6.7 [+4.0, +9.4] +6.5 [+3.8, +9.1] +28.8 [+25.6, +31.8] +28.7 [+25.5, +31.8] +24.0 [+21.1, +27.0] +23.2 [+20.3, +26.1] -5.3 [-8.1, -2.5] -5.1 [-7.9, -2.3] -0.8 [-3.4, +2.0] -2.0 [-4.5, +0.6] -4.8 [-7.7, -1.8] -4.9 [-7.8, -1.8]

+21.1 [+18.4, +23.9] +20.3 [+17.6, +23.2] +15.7 [+13.2, +18.2] +14.9 [+12.5, +17.6] +9.7 [+7.8, +11.6] +9.4 [+7.5, +11.3] +9.6 [+7.6, +11.6] +8.9 [+7.0, +10.9] +28.8 [+25.9, +31.7] +28.4 [+25.4, +31.4] +22.3 [+19.7, +25.1] +21.3 [+18.7, +24.1] +3.1 [+2.0, +4.4] +3.1 [+1.9, +4.4] +4.7 [+3.3, +6.2] +3.4 [+2.2, +4.7] +2.3 [+1.4, +3.3] +2.3 [+1.4, +3.2]

A → RS

A → RG

+0.0 [+0.0, +0.0] +7.3 [+5.7, +9.0] +0.0 [+0.0, +0.0] +7.5 [+5.9, +9.3] +0.0 [+0.0, +0.0] +5.1 [+3.8, +6.5] +0.0 [+0.0, +0.0] +5.3 [+4.0, +6.7] +1.1 [+0.5, +1.8] +7.0 [+5.5, +8.5] +1.2 [+0.5, +1.9] +7.1 [+5.6, +8.7] +0.7 [+0.1, +1.3] +5.2 [+3.8, +6.8] +0.6 [+0.1, +1.3] +5.4 [+3.9, +7.0] +0.0 [+0.0, +0.0] +13.9 [+11.7, +16.1] +0.0 [+0.0, +0.0] +14.2 [+11.9, +16.5] +0.0 [+0.0, +0.0] +11.7 [+9.5, +13.9] +0.0 [+0.0, +0.0] +11.9 [+9.7, +14.2] +0.0 [+0.0, +0.0] +1.9 [+1.1, +2.8] +0.0 [+0.0, +0.0] +1.9 [+1.1, +2.8] +0.6 [+0.1, +1.2] +3.4 [+2.3, +4.7] +0.5 [+0.0, +1.1] +3.3 [+2.1, +4.5] +0.0 [+0.0, +0.0] +3.6 [+2.4, +4.8] +0.0 [+0.0, +0.0] +3.5 [+2.3, +4.8]

R→A +1.4 [+0.7, +2.3] +1.4 [+0.6, +2.2] +1.3 [+0.7, +2.1] +1.3 [+0.6, +2.0] +4.6 [+3.2, +6.0] +3.5 [+2.3, +4.7] +5.1 [+3.8, +6.5] +4.6 [+3.3, +6.0] +1.2 [+0.6, +2.0] +1.1 [+0.6, +1.9] +0.9 [+0.3, +1.6] +0.8 [+0.2, +1.5] +12.6 [+10.6, +14.6] +12.3 [+10.4, +14.3] +11.6 [+9.5, +13.6] +11.1 [+9.1, +13.2] +14.2 [+11.8, +16.7] +14.3 [+11.8, +16.8]

Table 12: Complete six-metric policy effects for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

A → RS

C0→C1-pre C0→C1-pre C0→C1-post C0→C1-post C0→C2-pre C0→C2-pre C0→C2-post C0→C2-post C0→C3-pre C0→C3-pre C0→C3-post C0→C3-post C1-pre→C1-post C1-pre→C1-post C2-pre→C2-post C2-pre→C2-post C3-pre→C3-post C3-pre→C3-post

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 864 900 873 900 879 900 880 900 875 900 872 900 869 900 885 900 874

+38.9 [+35.6, +42.2] +39.2 [+35.8, +42.7] +30.8 [+27.5, +34.2] +30.2 [+26.9, +33.7] +6.3 [+4.1, +8.6] +6.5 [+4.3, +8.7] +0.2 [-2.1, +2.6] +0.1 [-2.2, +2.5] +30.9 [+27.9, +33.9] +30.5 [+27.5, +33.5] +18.4 [+15.7, +21.2] +17.9 [+15.2, +20.6] -8.1 [-10.5, -5.7] -8.2 [-10.6, -5.7] -6.1 [-8.5, -3.7] -6.4 [-8.8, -4.0] -12.4 [-15.3, -9.6] -12.7 [-15.5, -9.9]

+36.6 [+33.2, +39.9] +36.2 [+32.8, +39.7] +30.8 [+27.4, +34.2] +30.1 [+26.8, +33.6] +0.7 [-1.7, +3.0] +0.8 [-1.6, +3.1] -2.3 [-4.8, +0.1] -2.5 [-4.9, -0.1] +22.1 [+18.9, +25.3] +21.5 [+18.2, +24.7] +15.4 [+12.8, +18.2] +14.9 [+12.3, +17.6] -5.8 [-8.7, -2.8] -5.9 [-8.9, -2.8] -3.0 [-5.4, -0.7] -3.2 [-5.5, -0.8] -6.7 [-9.6, -3.7] -7.0 [-10.0, -4.0]

+30.7 [+27.4, +33.9] +30.7 [+27.4, +34.0] +25.6 [+22.4, +28.8] +24.9 [+21.7, +28.1] +4.8 [+3.4, +6.2] +4.7 [+3.4, +6.0] +3.8 [+2.5, +5.1] +3.4 [+2.2, +4.7] +23.1 [+20.4, +25.8] +22.6 [+20.0, +25.3] +15.3 [+13.1, +17.6] +14.7 [+12.6, +16.8] +1.7 [+0.8, +2.8] +1.7 [+0.8, +2.9] +2.0 [+1.1, +3.1] +1.7 [+0.8, +2.8] +1.8 [+0.9, +2.8] +1.5 [+0.7, +2.4]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

14

A → RG

R→A

+8.9 [+7.1, +10.7] +0.7 [+0.2, +1.2] +9.3 [+7.4, +11.2] +0.7 [+0.2, +1.3] +6.2 [+4.8, +7.8] +1.0 [+0.4, +1.7] +6.4 [+4.9, +8.0] +1.0 [+0.5, +1.7] +5.2 [+3.9, +6.6] +3.7 [+2.5, +4.9] +5.2 [+3.9, +6.6] +3.4 [+2.3, +4.7] +2.7 [+1.7, +3.7] +6.2 [+4.8, +7.7] +2.7 [+1.7, +3.8] +6.0 [+4.6, +7.5] +8.6 [+6.9, +10.3] +0.8 [+0.2, +1.4] +8.7 [+7.0, +10.4] +0.8 [+0.2, +1.4] +4.4 [+3.1, +5.9] +1.3 [+0.6, +2.2] +4.5 [+3.2, +5.9] +1.3 [+0.6, +2.1] +1.1 [+0.4, +1.9] +10.9 [+9.0, +12.8] +1.2 [+0.5, +2.0] +11.0 [+9.2, +13.0] +1.9 [+1.1, +2.8] +10.0 [+8.1, +12.0] +1.8 [+1.0, +2.7] +9.9 [+8.0, +11.9] +1.1 [+0.5, +1.8] +15.3 [+12.9, +17.8] +1.0 [+0.5, +1.7] +15.2 [+12.8, +17.8]

Table 13: Complete six-metric policy effects for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

C0→C1-pre C0→C1-pre C0→C1-post C0→C1-post C0→C2-pre C0→C2-pre C0→C2-post C0→C2-post C0→C3-pre C0→C3-pre C0→C3-post C0→C3-post C1-pre→C1-post C1-pre→C1-post C2-pre→C2-post C2-pre→C2-post C3-pre→C3-post C3-pre→C3-post

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 833 900 810 900 847 900 802 900 832 900 829 900 798 900 791 900 805

+30.3 [+27.1, +33.6] +30.1 [+26.8, +33.5] +45.2 [+42.1, +48.4] +46.4 [+43.0, +49.9] -1.6 [-4.1, +1.0] -1.3 [-3.8, +1.3] -2.3 [-4.9, +0.3] -0.7 [-3.4, +1.9] +21.6 [+18.4, +24.8] +21.2 [+17.9, +24.5] +27.3 [+24.4, +30.2] +26.3 [+23.2, +29.4] +14.9 [+11.8, +17.9] +16.0 [+12.7, +19.4] -0.8 [-3.3, +1.8] +0.5 [-2.2, +3.2] +5.8 [+3.0, +8.5] +5.1 [+2.2, +8.0]

+31.3 [+27.8, +34.9] +30.7 [+27.1, +34.4] +46.8 [+43.5, +50.1] +47.0 [+43.6, +50.5] -1.2 [-3.9, +1.5] -1.5 [-4.1, +1.1] -5.0 [-7.6, -2.4] -4.5 [-7.2, -1.8] +19.4 [+16.2, +22.8] +19.2 [+16.0, +22.5] +23.4 [+20.2, +26.7] +22.9 [+19.5, +26.4] +15.4 [+12.1, +18.8] +16.5 [+12.9, +20.2] -3.8 [-6.4, -1.1] -3.2 [-5.9, -0.5] +4.0 [+1.1, +6.9] +3.5 [+0.4, +6.5]

+25.7 [+22.7, +28.6] +25.2 [+22.2, +28.2] +38.8 [+35.8, +41.7] +39.5 [+36.3, +42.7] +3.2 [+2.1, +4.4] +2.6 [+1.6, +3.7] +2.9 [+1.8, +4.1] +3.0 [+1.8, +4.3] +17.4 [+15.0, +20.0] +16.9 [+14.4, +19.6] +20.3 [+17.7, +23.0] +19.5 [+16.8, +22.4] +16.0 [+13.6, +18.5] +16.9 [+14.2, +19.7] +2.9 [+1.7, +4.2] +2.5 [+1.4, +3.8] +6.8 [+5.1, +8.5] +6.6 [+4.9, +8.4]

A → RS

A → RG

R→A

+0.0 [+0.0, +0.0] +6.9 [+5.4, +8.4] +2.2 [+1.3, +3.2] +0.0 [+0.0, +0.0] +7.2 [+5.7, +8.9] +2.3 [+1.3, +3.4] +0.0 [+0.0, +0.0] +7.3 [+5.7, +9.1] +0.9 [+0.3, +1.5] +0.0 [+0.0, +0.0] +7.9 [+6.2, +9.8] +1.0 [+0.4, +1.7] +0.2 [+0.0, +0.6] +2.9 [+1.8, +4.0] +7.9 [+6.1, +9.7] +0.2 [+0.0, +0.6] +3.1 [+2.0, +4.3] +7.2 [+5.4, +9.0] +1.1 [+0.5, +1.8] +3.6 [+2.5, +4.7] +9.9 [+7.9, +11.8] +1.1 [+0.5, +1.9] +4.0 [+2.8, +5.3] +8.9 [+7.0, +10.7] +0.0 [+0.0, +0.0] +6.1 [+4.7, +7.7] +2.0 [+1.0, +3.1] +0.0 [+0.0, +0.0] +6.1 [+4.6, +7.8] +1.9 [+1.0, +3.0] +0.0 [+0.0, +0.0] +8.1 [+6.4, +10.0] +1.1 [+0.4, +1.9] +0.0 [+0.0, +0.0] +7.8 [+6.1, +9.7] +1.1 [+0.4, +1.9] +0.0 [+0.0, +0.0] +3.2 [+2.1, +4.4] +4.3 [+3.0, +5.8] +0.0 [+0.0, +0.0] +3.5 [+2.3, +4.8] +4.4 [+3.0, +5.9] +0.8 [+0.2, +1.4] +3.3 [+2.3, +4.5] +7.8 [+6.1, +9.5] +0.6 [+0.1, +1.2] +3.8 [+2.6, +5.1] +6.4 [+4.7, +8.2] +0.0 [+0.0, +0.0] +4.2 [+2.9, +5.6] +5.2 [+3.8, +6.8] +0.0 [+0.0, +0.0] +3.9 [+2.5, +5.3] +5.3 [+3.8, +7.1]

Table 14: Category-stratified total-refusal rates (%) for all unmodified model-policy conditions. Category denominators are 222, 188, 305, and 185 in column order. Model

Condition

Context-dependent

False premise

Subjective/philosophical

Vague description

InternVL3-2B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

26.6 63.1 53.2 45.9 28.4 82.9 75.7

28.2 67.0 57.4 54.3 57.4 73.4 65.4

1.3 23.6 17.4 4.6 8.9 29.8 25.2

4.9 16.2 11.4 14.1 10.8 45.9 29.7

InternVL3-8B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

45.0 83.8 79.3 49.5 35.6 80.2 68.5

62.8 88.8 86.2 70.7 73.4 83.0 77.7

2.6 51.8 36.4 7.9 3.3 37.4 17.4

7.0 42.2 36.2 15.7 7.6 37.3 29.2

Qwen2.5-VL-7B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

57.2 80.6 91.0 52.7 41.0 82.0 89.6

71.3 88.3 95.2 73.4 78.7 87.2 89.9

6.2 44.3 68.2 5.2 7.2 25.2 31.5

17.3 56.8 70.3 14.6 16.2 44.9 50.8

InternVL3-2B

Mixed

A

RE RS InternVL3-8B

Qwen2.5-VL-7B

RG

100

Response share (%)

80 60 40 20

pre

Sim

PO

-C1

pre

-C0 PO

-C1

PO

Sim

OR

pre

-C0 PO OR

-C0

-C1

DPO

DPO

SFT -C0 SFT -C1 pre

0

1pr e

U-C

U-C

pre

Sim

PO

-C1

pre

pre

-C0 PO

-C1

PO

Sim

OR

PO

-C0

OR

-C0

-C1

DPO

DPO

SFT -C0 SFT -C1 pre

0

1pr e

U-C

U-C

pre

PO

-C1

pre

-C0 PO

Sim

-C1

PO

Sim

OR

pre

-C0 PO OR

-C0

-C1

DPO

DPO

SFT -C0 SFT -C1 pre

0 U-C

U-C

1pr e

0

Figure 4: Primary response-type composition under unmodified and post-trained models at C0 and C1-prefix. Diamonds show the orthogonal mixed-response rate and are not components of the stacked bars. 15

Table 15: Complete typed distributions for the 24 post-trained C0/C1-prefix evaluations. Entries are count (rate in %); Mixed is an orthogonal response flag. Model

Method

Condition

A

RE

RS

RG

Mixed

InternVL3-2B

SFT SFT DPO DPO ORPO ORPO SimPO SimPO

C0 C1-pre C0 C1-pre C0 C1-pre C0 C1-pre

111 (12.3) 3 (0.3) 536 (59.6) 224 (24.9) 145 (16.1) 8 (0.9) 720 (80.0) 466 (51.8)

789 (87.7) 897 (99.7) 321 (35.7) 656 (72.9) 752 (83.6) 892 (99.1) 134 (14.9) 356 (39.6)

0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0)

0 (0.0) 0 (0.0) 43 (4.8) 20 (2.2) 3 (0.3) 0 (0.0) 46 (5.1) 78 (8.7)

251 (27.9) 277 (30.8) 19 (2.1) 57 (6.3) 299 (33.2) 147 (16.3) 25 (2.8) 19 (2.1)

InternVL3-8B

SFT SFT DPO DPO ORPO ORPO SimPO SimPO

C0 C1-pre C0 C1-pre C0 C1-pre C0 C1-pre

87 (9.7) 15 (1.7) 620 (68.9) 266 (29.6) 114 (12.7) 30 (3.3) 651 (72.3) 286 (31.8)

810 (90.0) 881 (97.9) 229 (25.4) 562 (62.4) 784 (87.1) 857 (95.2) 187 (20.8) 531 (59.0)

0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0)

3 (0.3) 4 (0.4) 51 (5.7) 72 (8.0) 2 (0.2) 13 (1.4) 62 (6.9) 83 (9.2)

154 (17.1) 205 (22.8) 9 (1.0) 19 (2.1) 121 (13.4) 113 (12.6) 15 (1.7) 29 (3.2)

Qwen2.5-VL-7B

SFT SFT DPO DPO ORPO ORPO SimPO SimPO

C0 C1-pre C0 C1-pre C0 C1-pre C0 C1-pre

46 (5.1) 1 (0.1) 102 (11.3) 65 (7.2) 52 (5.8) 4 (0.4) 550 (61.1) 272 (30.2)

854 (94.9) 898 (99.8) 774 (86.0) 820 (91.1) 848 (94.2) 880 (97.8) 271 (30.1) 546 (60.7)

0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0)

0 (0.0) 1 (0.1) 24 (2.7) 15 (1.7) 0 (0.0) 16 (1.8) 79 (8.8) 82 (9.1)

159 (17.7) 134 (14.9) 217 (24.1) 123 (13.7) 201 (22.3) 183 (20.3) 23 (2.6) 71 (7.9)

Table 16: Complete post-training effects relative to the unmodified base for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

A → RS

A → RG

R→A

SFT, C0 SFT, C0 SFT, C1-pre SFT, C1-pre DPO, C0 DPO, C0 DPO, C1-pre DPO, C1-pre ORPO, C0 ORPO, C0 ORPO, C1-pre ORPO, C1-pre SimPO, C0 SimPO, C0 SimPO, C1-pre SimPO, C1-pre

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 641 900 617 900 866 900 838 900 589 900 748 900 869 900 877

+73.8 [+70.7, +76.7] +71.8 [+68.1, +75.2] +58.8 [+55.3, +62.1] +61.6 [+57.4, +65.7] +26.6 [+23.5, +29.6] +25.6 [+22.5, +28.8] +34.2 [+30.9, +37.5] +34.6 [+31.2, +38.0] +70.0 [+66.9, +73.1] +64.5 [+60.2, +68.6] +58.2 [+54.8, +61.6] +60.3 [+56.5, +64.0] +6.1 [+4.2, +8.1] +5.3 [+3.4, +7.3] +7.3 [+4.9, +9.8] +7.6 [+5.3, +10.1]

+79.1 [+76.3, +81.8] +74.7 [+71.2, +78.1] +68.0 [+64.8, +71.1] +70.2 [+66.4, +73.9] +27.1 [+24.0, +30.2] +26.3 [+23.2, +29.5] +41.2 [+37.6, +44.9] +41.8 [+37.9, +45.7] +75.0 [+72.1, +77.8] +67.7 [+63.5, +71.8] +67.4 [+64.2, +70.6] +68.9 [+65.3, +72.3] +6.3 [+4.4, +8.3] +5.5 [+3.6, +7.5] +7.9 [+5.2, +10.7] +8.0 [+5.2, +10.8]

+74.0 [+70.9, +76.9] +72.1 [+68.5, +75.5] +58.8 [+55.3, +62.1] +61.6 [+57.4, +65.7] +24.4 [+21.7, +27.2] +23.4 [+20.6, +26.3] +33.3 [+30.1, +36.6] +33.4 [+30.0, +36.8] +70.0 [+66.9, +73.1] +64.5 [+60.2, +68.6] +58.3 [+54.9, +61.7] +60.3 [+56.5, +64.0] +6.0 [+4.5, +7.6] +4.9 [+3.6, +6.5] +7.6 [+5.8, +9.4] +7.4 [+5.6, +9.3]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +3.0 [+2.0, +4.2] +3.0 [+1.9, +4.2] +1.2 [+0.6, +2.0] +1.3 [+0.6, +2.1] +0.1 [+0.0, +0.3] +0.2 [+0.0, +0.5] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +2.0 [+1.1, +3.0] +2.1 [+1.2, +3.1] +2.7 [+1.6, +3.8] +2.7 [+1.7, +3.9]

+0.2 [+0.0, +0.6] +0.3 [+0.0, +0.8] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.9 [+0.3, +1.5] +0.8 [+0.2, +1.4] +0.3 [+0.0, +0.8] +0.1 [+0.0, +0.4] +0.1 [+0.0, +0.3] +0.2 [+0.0, +0.5] +0.1 [+0.0, +0.3] +0.0 [+0.0, +0.0] +1.9 [+1.1, +2.8] +1.7 [+0.9, +2.6] +2.9 [+1.9, +4.0] +2.5 [+1.6, +3.5]

16

Table 17: Complete post-training effects relative to the unmodified base for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

A → RS

A → RG

R→A

SFT, C0 SFT, C0 SFT, C1-pre SFT, C1-pre DPO, C0 DPO, C0 DPO, C1-pre DPO, C1-pre ORPO, C0 ORPO, C0 ORPO, C1-pre ORPO, C1-pre SimPO, C0 SimPO, C0 SimPO, C1-pre SimPO, C1-pre

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 735 900 684 900 881 900 867 900 768 900 772 900 881 900 864

+63.8 [+60.2, +67.2] +67.9 [+64.2, +71.5] +32.9 [+29.7, +36.2] +34.2 [+30.6, +37.7] +4.6 [+2.7, +6.4] +4.2 [+2.5, +6.0] +5.0 [+2.5, +7.5] +5.2 [+2.7, +7.8] +60.8 [+57.4, +64.1] +63.4 [+60.1, +66.7] +31.2 [+28.0, +34.5] +32.4 [+28.8, +35.9] +1.1 [-0.2, +2.4] +0.8 [-0.6, +2.1] +2.8 [+1.2, +4.4] +2.9 [+1.3, +4.6]

+71.3 [+68.0, +74.5] +73.1 [+69.6, +76.4] +42.7 [+39.2, +46.1] +44.7 [+40.9, +48.5] +6.8 [+4.8, +8.9] +6.6 [+4.7, +8.6] +7.2 [+4.1, +10.3] +7.4 [+4.3, +10.6] +68.4 [+65.3, +71.5] +70.3 [+67.1, +73.3] +40.0 [+36.7, +43.4] +41.5 [+37.9, +45.0] +2.1 [+0.6, +3.7] +1.8 [+0.3, +3.3] +3.8 [+1.8, +5.8] +3.9 [+1.8, +6.1]

+63.9 [+60.3, +67.3] +68.0 [+64.4, +71.6] +32.4 [+29.2, +35.8] +33.6 [+30.0, +37.1] +4.8 [+3.5, +6.2] +4.3 [+3.0, +5.7] +6.7 [+4.9, +8.6] +6.7 [+4.9, +8.6] +61.3 [+58.0, +64.6] +63.9 [+60.6, +67.1] +30.6 [+27.4, +33.7] +31.7 [+28.3, +35.2] +2.2 [+1.3, +3.2] +1.8 [+0.9, +2.8] +2.7 [+1.6, +3.9] +2.8 [+1.6, +4.0]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+0.2 [+0.0, +0.6] +0.3 [+0.0, +0.7] +0.4 [+0.1, +0.9] +0.6 [+0.1, +1.2] +1.6 [+0.8, +2.4] +1.6 [+0.8, +2.4] +1.7 [+0.9, +2.6] +1.7 [+0.9, +2.7] +0.1 [+0.0, +0.3] +0.1 [+0.0, +0.4] +0.7 [+0.1, +1.3] +0.6 [+0.1, +1.3] +0.7 [+0.2, +1.2] +0.7 [+0.2, +1.3] +1.4 [+0.8, +2.2] +1.5 [+0.8, +2.3]

+0.3 [+0.0, +0.8] +0.4 [+0.0, +0.9] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +1.8 [+1.0, +2.7] +1.7 [+0.9, +2.6] +3.3 [+2.2, +4.6] +3.2 [+2.1, +4.5] +0.7 [+0.2, +1.2] +0.7 [+0.1, +1.3] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +1.8 [+1.0, +2.7] +1.7 [+0.9, +2.6] +1.3 [+0.7, +2.1] +1.4 [+0.7, +2.2]

Table 18: Complete post-training effects relative to the unmodified base for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

A → RS

A → RG

R→A

SFT, C0 SFT, C0 SFT, C1-pre SFT, C1-pre DPO, C0 DPO, C0 DPO, C1-pre DPO, C1-pre ORPO, C0 ORPO, C0 ORPO, C1-pre ORPO, C1-pre SimPO, C0 SimPO, C0 SimPO, C1-pre SimPO, C1-pre

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 722 900 726 900 666 900 744 900 683 900 682 900 862 900 802

+60.2 [+56.9, +63.5] +64.1 [+60.6, +67.7] +34.9 [+31.6, +38.2] +35.8 [+32.4, +39.4] +54.0 [+51.1, +57.0] +51.1 [+47.7, +54.4] +27.8 [+24.3, +31.3] +29.3 [+25.7, +33.0] +59.6 [+56.4, +62.8] +59.4 [+55.9, +63.0] +34.6 [+31.3, +37.9] +32.6 [+29.0, +36.2] +4.2 [+2.2, +6.3] +3.7 [+1.6, +5.9] +4.8 [+2.3, +7.3] +4.6 [+2.0, +7.2]

+70.4 [+67.1, +73.6] +73.1 [+69.7, +76.4] +44.0 [+40.6, +47.5] +44.8 [+41.0, +48.5] +61.6 [+58.6, +64.6] +57.1 [+53.5, +60.6] +35.3 [+31.5, +39.2] +36.8 [+32.8, +40.8] +69.8 [+66.6, +72.9] +68.8 [+65.4, +72.2] +42.0 [+38.4, +45.6] +40.5 [+36.5, +44.4] +5.7 [+3.8, +7.6] +5.6 [+3.6, +7.5] +4.9 [+2.2, +7.6] +4.4 [+1.6, +7.2]

+60.3 [+57.0, +63.6] +64.3 [+60.7, +67.8] +34.8 [+31.6, +38.1] +35.7 [+32.3, +39.2] +52.0 [+49.2, +54.9] +48.2 [+44.9, +51.6] +27.9 [+24.8, +31.1] +28.9 [+25.6, +32.3] +59.7 [+56.5, +63.0] +59.6 [+56.1, +63.2] +33.3 [+30.1, +36.7] +31.4 [+27.8, +35.0] +4.8 [+3.5, +6.1] +4.5 [+3.3, +5.8] +7.1 [+5.4, +9.0] +6.9 [+5.1, +8.7]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.1 [+0.0, +0.3] +0.1 [+0.0, +0.4] +2.3 [+1.4, +3.3] +3.2 [+2.0, +4.4] +1.0 [+0.4, +1.7] +0.9 [+0.3, +1.7] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +1.2 [+0.6, +2.0] +1.2 [+0.4, +2.0] +3.0 [+1.9, +4.2] +2.9 [+1.8, +4.1] +2.0 [+1.1, +3.0] +2.2 [+1.3, +3.3]

+0.1 [+0.0, +0.3] +0.1 [+0.0, +0.4] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.3 [+0.0, +0.8] +0.3 [+0.0, +0.7] +1.1 [+0.5, +1.8] +0.5 [+0.1, +1.1] +0.1 [+0.0, +0.3] +0.1 [+0.0, +0.5] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +3.6 [+2.4, +4.8] +3.7 [+2.5, +5.0] +4.3 [+3.1, +5.7] +4.5 [+3.2, +5.9]

Table 19: Primary-description false-refusal counts for each post-trained model. Every row uses the same 176 unique images. Model

Method

Refused

Mixed

InternVL3-2B

SFT DPO ORPO SimPO

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

InternVL3-8B

SFT DPO ORPO SimPO

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

Qwen2.5-VL-7B

SFT DPO ORPO SimPO

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

17

Table 20: Paired MMMU correctness transitions for each post-trained model relative to its unmodified base. All comparisons use the same 650 items; accuracy differences are percentage points with paired-item bootstrap intervals. Model

Method

Base correct

Target correct

Both correct

Base only

Target only

Both wrong

∆ accuracy [95% CI]

InternVL3-2B

SFT DPO ORPO SimPO

293 293 293 293

293 291 296 289

274 281 280 287

19 12 13 6

19 10 16 2

338 347 341 355

+0.00 [-1.85, +1.85] -0.31 [-1.69, +1.08] +0.46 [-1.08, +2.15] -0.62 [-1.54, +0.15]

InternVL3-8B

SFT DPO ORPO SimPO

351 351 351 351

357 347 356 351

334 341 337 350

17 10 14 1

23 6 19 1

276 293 280 298

+0.92 [-0.92, +2.77] -0.62 [-1.85, +0.62] +0.77 [-0.92, +2.46] +0.00 [-0.46, +0.46]

Qwen2.5-VL-7B

SFT DPO ORPO SimPO

304 304 304 304

290 296 297 304

262 283 275 300

42 21 29 4

28 13 22 4

318 333 324 342

-2.15 [-4.77, +0.46] -1.23 [-2.92, +0.46] -1.08 [-3.23, +1.08] +0.00 [-0.92, +0.77]

Table 21: Training time and recorded training-memory usage for the post-training runs on the 10,000item alignment role.

B.3

Model

Method

Training time

Recorded memory

InternVL3-2B

SFT DPO ORPO SimPO

00:30:21 00:49:27 00:39:06 00:50:00

42.3 GB 12.5 GB 43.7 GB 43.7 GB

InternVL3-8B

SFT DPO ORPO SimPO

00:45:06 02:22:47 01:50:00 02:01:37

47.9 GB 25.4 GB 47.8 GB 47.8 GB

Qwen2.5-VL-7B

SFT DPO ORPO SimPO

01:14:39 03:50:14 02:41:52 02:46:58

48.2 GB 26.4 GB 48.2 GB 48.2 GB

Complete steering and control results

The behavior-derived direction B is compared with unmodified generation under all seven policies. Random and category-preserving shuffled-label directions are evaluated at C0 and C1-prefix. Mixedexcluded analyses use the same rule as the behavioral comparison. Transfer and reason decomposition. Tables 22–24 report all 21 four-class transition matrices and their mixed-excluded counterparts. Tables 25–27 report the six paired effects, including net evidence-refusal change and each directed answer/refusal transition. Category, control, and selectivity details. Table 28 gives every condition-by-category effect. Tables 29–31 report all fixed-control comparisons at C0 and C1-prefix. Tables 32–35 provide exact vector provenance, within-category shuffled-label marginals, description counts, and paired MMMU correctness transitions. Interpretive boundary. The complete decompositions confirm that B increases total and evidencebased refusal across most tested conditions, while also producing generic refusals and refusal-toanswer reversals. The fixed controls establish specificity only at C0 and C1-prefix and only relative to the recorded seed-42 directions. The description and MMMU controls do not show blanket refusal or a detectable accuracy change under the tested configurations, but they are not an equivalence test and do not establish selectivity for arbitrary answerable questions.

C

Limitations

Our conclusions are bounded by the evaluation scope. MoHoBench contains visually unanswerable questions, so the description and MMMU controls assess indiscriminate refusal and general performance but do not provide matched answerable-VQA counterfactuals. The study covers three open 18

Table 22: Unmodified→ B transition matrices for InternVL3-2B. Rows are source types and columns are target types in the order [A, RE , RS , RG ]. The mixed-excluded denominator removes every pair for which either response carries the mixed flag. Comparison C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

All pairs  685 61 0 29 

Mixed-excluded  676 51 0 29 

12 7 0 29   416 79 0 37

12 7 0 29   412 75 0 37

18 34 0 31   444 119 0 37

18 33 0 31   442 107 0 37

8 29 0 26   551 63 2 40

8 28 0 26   522 59 2 36

17 13 4 54   553 77 10 42

14 13 4 53  530 71 8 39

15 7 24  23 313 59 0 30 

19 13 7 24   307 56 0 28

6 0

65 0 6 0 0 0

23 231 0 31 0 0 0 0

12 209 0 16 0 0 0 0 8 3

125 2 9 1 6 2

18 106 4 3 5 0

9 4

15 307 0 14 0 0 0 0 10 72 0 80   377 58 0 42 11 269 0 13 0 0 0 0 20 37 0 73

6 0

56 0 6 0 0 0

19 224 0 29 0 0 0 0 8 0

5 2

192 0 15 0 0 0 122 2 9 1 6 2

17 97 3 8 3 4 0 4

8 302 0 14 0 0 0 0 9 72 0 80   370 56 0 41 8 256 0 13 0 0 0 0 19 37 0 73

Retained n 872 878 863 852 847 876 873

Table 23: Unmodified→ B transition matrices for InternVL3-8B. Rows are source types and columns are target types in the order [A, RE , RS , RG ]. The mixed-excluded denominator removes every pair for which either response carries the mixed flag. Comparison C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

All pairs  601 32 0 28 

Mixed-excluded  596 26 0 28 

8 22 0 41   250 46 0 15

8 21 0 41   246 46 0 15

12 53 0 27   309 53 0 22

11 52 0 26   308 47 0 21

12 140 0 16 0 0 0 0

14 458 0 25 0 0 0 0

17 395 0 33 0 0 0 0 9 38 0 24   557 20 0 27 13 144 0 17 0 0 1 0 24 22 0 75   617 20 0 22 12 125 1 9 0 0 0 0 11 30 3 50   317 36 0 30 13 335 0 19 0 0 0 0 18 54 0 78   422 53 0 20 8 285 0 14 0 0 0 0 12 42 0 44

8 0

131 0 16 0 0 0

14 417 0 25 0 0 0 0

17 365 0 33 0 0 0 0 9 36 0 24   553 16 0 27 10 138 0 17 0 0 1 0 24 22 0 75   613 19 0 20 9 116 1 9 0 0 0 0 10 29 3 50   309 34 0 27 9 326 0 19 0 0 0 0 17 54 0 77   420 51 0 19 4 270 0 14 0 0 0 0 11 41 0 44

Retained n 875 852 860 883 879 872 874

Table 24: Unmodified→ B transition matrices for Qwen2.5-VL-7B. Rows are source types and columns are target types in the order [A, RE , RS , RG ]. The mixed-excluded denominator removes every pair for which either response carries the mixed flag. Comparison C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

All pairs  503 54 0 31 

Mixed-excluded  499 47 0 31 

27 28 0 37   198 94 0 23

27 26 0 37   195 88 0 22

47 0 26  10 83 80 0 18 

10 42 0 24  78 73 0 18

5 49 0 24   536 43 0 23

5 40 0 19   521 35 0 21

18 35 2 28   526 45 8 30

16 34 2 27   447 30 3 23

17 30 4 43   272 67 0 55

14 29 3 42   261 47 0 43

5 55 0 51   238 69 0 35

4 48 0 46   234 58 0 33

11 196 0 13 0 0 0 0 20 453 0 29 0 0 0 0 23 582 0 36 0 0 0 0

25 173 0 11 0 4 1 1

14 140 4 17 3 10 5 4 19 346 0 30 0 0 0 0

15 384 0 32 0 0 0 0 11 64 0 52

19

9 0

176 0 13 0 0 0

17 386 0 27 0 0 0 0

20 485 0 30 0 0 0 0

15 149 0 11 0 4 1 1 10 122 2 17 1 9 3 4

15 298 0 27 0 0 0 0

14 325 0 30 0 0 0 0 11 55 0 43

Retained n 865 811 768 837 759 789 803

Table 25: Complete behavior-direction effects relative to unmodified generation for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison Sensitivity C0 C0 C1-pre C1-pre C1-post C1-post C2-pre C2-pre C2-post C2-post C3-pre C3-pre C3-post C3-post

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

n

∆R

A → RE

∆RE

900 +8.0 [+6.0, +10.0] +6.2 [+4.4, +8.1] +6.8 [+5.3, +8.4] 872 +7.1 [+5.1, +9.2] +5.3 [+3.5, +7.2] +5.8 [+4.4, +7.4] 900 +8.3 [+5.8, +10.9] +6.6 [+3.9, +9.2] +8.8 [+7.0, +10.5] 878 +8.5 [+6.0, +11.1] +6.8 [+4.1, +9.5] +8.5 [+6.8, +10.3] 900 +15.1 [+12.6, +17.7] +13.3 [+10.8, +15.9] +13.2 [+11.1, +15.5] 863 +14.8 [+12.3, +17.4] +13.0 [+10.4, +15.5] +12.4 [+10.3, +14.6] 900 +8.6 [+6.2, +11.0] +6.4 [+4.3, +8.6] +7.0 [+5.4, +8.7] 852 +8.9 [+6.6, +11.3] +6.7 [+4.6, +8.8] +6.9 [+5.2, +8.6] 900 +9.4 [+6.5, +12.3] +7.3 [+4.9, +9.7] +8.6 [+6.9, +10.4] 847 +9.3 [+6.4, +12.3] +7.1 [+4.6, +9.6] +8.4 [+6.6, +10.3] 900 +7.1 [+4.8, +9.6] +11.3 [+8.5, +14.2] +6.6 [+4.9, +8.3] 876 +7.6 [+5.4, +10.0] +12.1 [+9.4, +14.8] +6.4 [+4.8, +8.1] 900 +7.7 [+5.2, +10.2] +7.9 [+5.6, +10.2] +6.4 [+4.9, +8.0] 873 +8.0 [+5.5, +10.5] +8.2 [+6.0, +10.5] +6.4 [+4.9, +8.0]

A → RS

A → RG

R→A

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.2 [+0.0, +0.6] +0.2 [+0.0, +0.6] +1.1 [+0.5, +1.8] +0.9 [+0.4, +1.6] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+3.2 [+2.2, +4.4] +3.3 [+2.2, +4.5] +4.1 [+2.8, +5.5] +4.2 [+2.9, +5.7] +4.1 [+2.9, +5.4] +4.3 [+3.0, +5.7] +4.4 [+3.1, +5.9] +4.2 [+2.9, +5.6] +4.7 [+3.3, +6.1] +4.6 [+3.2, +6.0] +3.3 [+2.2, +4.5] +3.2 [+2.1, +4.4] +4.7 [+3.4, +6.0] +4.7 [+3.3, +6.1]

+2.0 [+1.2, +2.9] +2.1 [+1.2, +3.0] +4.6 [+3.3, +5.9] +4.2 [+3.0, +5.5] +2.2 [+1.3, +3.2] +1.9 [+1.0, +2.7] +3.1 [+2.0, +4.3] +2.5 [+1.4, +3.6] +4.9 [+3.6, +6.3] +4.6 [+3.3, +6.0] +2.8 [+1.8, +3.9] +1.9 [+1.0, +2.9] +3.4 [+2.3, +4.8] +3.1 [+1.9, +4.4]

Table 26: Complete behavior-direction effects relative to unmodified generation for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . n

Comparison Sensitivity C0 C0 C1-pre C1-pre C1-post C1-post C2-pre C2-pre C2-post C2-post C3-pre C3-pre C3-post C3-post

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 875 900 852 900 860 900 883 900 879 900 872 900 874

∆R

A → RE

A → RS

A → RG

R→A

+3.6 [+2.4, +4.9] +3.0 [+1.9, +4.2] +5.1 [+3.7, +6.6] +5.4 [+3.9, +7.0] +5.9 [+4.5, +7.4] +5.5 [+4.1, +7.0] +2.2 [+1.3, +3.2] +1.8 [+1.0, +2.7] +2.2 [+1.3, +3.2] +2.2 [+1.3, +3.1] +4.0 [+2.7, +5.4] +3.9 [+2.6, +5.4] +5.9 [+4.4, +7.5] +5.8 [+4.3, +7.4]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+3.1 [+2.0, +4.3] +3.2 [+2.1, +4.4] +1.7 [+0.9, +2.5] +1.8 [+0.9, +2.7] +2.4 [+1.4, +3.7] +2.4 [+1.4, +3.7] +3.0 [+1.9, +4.2] +3.1 [+1.9, +4.3] +2.4 [+1.5, +3.5] +2.3 [+1.4, +3.3] +3.3 [+2.1, +4.6] +3.1 [+2.0, +4.3] +2.2 [+1.3, +3.2] +2.2 [+1.3, +3.1]

+2.2 [+1.2, +3.3] +1.8 [+1.0, +2.8] +2.9 [+1.9, +4.0] +2.9 [+1.9, +4.1] +2.9 [+1.9, +4.0] +3.0 [+1.9, +4.2] +4.1 [+2.9, +5.4] +3.9 [+2.7, +5.1] +2.6 [+1.5, +3.7] +2.2 [+1.2, +3.2] +3.4 [+2.3, +4.7] +3.0 [+1.9, +4.1] +2.2 [+1.3, +3.2] +1.7 [+0.9, +2.6]

∆RE

+4.4 [+2.5, +6.4] +2.9 [+0.8, +5.0] +4.3 [+2.6, +6.2] +2.6 [+0.7, +4.6] +3.9 [+1.8, +6.0] +6.7 [+4.2, +9.2] +4.2 [+2.0, +6.4] +6.9 [+4.3, +9.6] +5.4 [+3.3, +7.7] +4.6 [+1.9, +7.1] +4.9 [+2.7, +7.1] +3.8 [+1.2, +6.4] +1.1 [-0.8, +3.0] +1.3 [-0.4, +3.1] +1.0 [-0.8, +2.8] +1.2 [-0.5, +3.0] +2.1 [+0.3, +4.0] +3.1 [+1.3, +5.1] +2.3 [+0.5, +4.1] +3.3 [+1.5, +5.2] +3.9 [+1.6, +6.2] +6.4 [+4.0, +8.9] +4.0 [+1.9, +6.2] +6.9 [+4.4, +9.4] +5.9 [+3.8, +8.0] +8.1 [+5.8, +10.4] +6.3 [+4.3, +8.4] +8.5 [+6.1, +10.8]

Table 27: Complete behavior-direction effects relative to unmodified generation for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison Sensitivity C0 C0 C1-pre C1-pre C1-post C1-post C2-pre C2-pre C2-post C2-post C3-pre C3-pre C3-post C3-post

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

n

∆R

A → RE

∆RE

900 +5.2 [+2.9, +7.7] +6.4 [+4.4, +8.6] +6.0 [+4.5, +7.6] 865 +4.9 [+2.5, +7.3] +5.9 [+3.8, +8.0] +5.4 [+4.0, +7.0] 900 +9.7 [+7.1, +12.3] +10.2 [+7.2, +13.3] +10.4 [+8.4, +12.7] 811 +10.2 [+7.3, +13.3] +10.6 [+7.3, +14.0] +10.9 [+8.6, +13.3] 900 +7.8 [+5.4, +10.2] +7.8 [+5.0, +10.6] +8.9 [+7.1, +10.8] 768 +8.6 [+5.8, +11.4] +8.2 [+5.2, +11.3] +9.5 [+7.4, +11.7] 900 +2.6 [+0.2, +4.9] +5.1 [+2.7, +7.5] +4.8 [+3.3, +6.3] 837 +3.0 [+0.8, +5.3] +5.6 [+3.2, +8.0] +4.2 [+2.7, +5.8] 900 +5.4 [+3.1, +7.8] +5.6 [+3.1, +8.0] +5.0 [+3.5, +6.6] 759 +4.1 [+1.8, +6.3] +5.1 [+2.4, +7.9] +4.0 [+2.5, +5.5] 900 +10.9 [+8.4, +13.5] +8.1 [+5.2, +11.1] +7.4 [+5.8, +9.2] 789 +9.0 [+6.6, +11.5] +6.7 [+3.7, +9.8] +6.0 [+4.4, +7.7] 900 +8.7 [+6.3, +11.1] +9.6 [+6.9, +12.3] +7.7 [+6.1, +9.3] 803 +8.2 [+5.7, +10.8] +8.6 [+5.8, +11.4] +7.2 [+5.6, +9.0]

20

A → RS

A → RG

R→A

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.9 [+0.3, +1.5] +0.4 [+0.0, +0.9] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+3.4 [+2.4, +4.6] +3.6 [+2.5, +4.8] +2.6 [+1.6, +3.6] +2.7 [+1.7, +3.8] +2.0 [+1.1, +3.0] +2.3 [+1.3, +3.5] +2.6 [+1.6, +3.6] +2.5 [+1.5, +3.6] +3.3 [+2.2, +4.5] +3.0 [+1.9, +4.3] +6.1 [+4.6, +7.7] +5.4 [+4.0, +7.0] +3.9 [+2.7, +5.1] +4.1 [+2.8, +5.5]

+4.2 [+2.8, +5.7] +4.2 [+2.8, +5.7] +3.3 [+2.2, +4.5] +3.3 [+2.2, +4.6] +3.1 [+2.0, +4.3] +3.3 [+2.1, +4.6] +4.8 [+3.5, +6.1] +3.7 [+2.5, +4.9] +3.8 [+2.7, +4.9] +3.3 [+2.1, +4.5] +2.7 [+1.7, +3.7] +2.4 [+1.4, +3.5] +2.9 [+1.7, +4.1] +3.1 [+1.8, +4.5]

Table 28: Category-stratified total-refusal rates for every behavior-direction condition. Each cell reports unmodifiedightarrowB (change), in percentage points. Model

Condition

Context-dependent

False premise

Subjective/philosophical

Vague description

InternVL3-2B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

26.6→40.1 (+13.5) 63.1→73.0 (+9.9) 53.2→67.1 (+14.0) 45.9→62.2 (+16.2) 28.4→44.6 (+16.2) 82.9→86.9 (+4.1) 75.7→79.7 (+4.1)

28.2→42.6 (+14.4) 67.0→78.2 (+11.2) 57.4→71.3 (+13.8) 54.3→60.6 (+6.4) 57.4→67.6 (+10.1) 73.4→83.5 (+10.1) 65.4→76.6 (+11.2)

1.3→3.0 (+1.6) 23.6→27.5 (+3.9) 17.4→35.4 (+18.0) 4.6→7.2 (+2.6) 8.9→10.5 (+1.6) 29.8→37.0 (+7.2) 25.2→30.8 (+5.6)

4.9→10.3 (+5.4) 16.2→27.0 (+10.8) 11.4→24.3 (+13.0) 14.1→25.4 (+11.4) 10.8→24.3 (+13.5) 45.9→53.5 (+7.6) 29.7→41.6 (+11.9)

InternVL3-8B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

45.0→53.2 (+8.1) 83.8→86.0 (+2.3) 79.3→83.8 (+4.5) 49.5→50.0 (+0.5) 35.6→40.1 (+4.5) 80.2→84.7 (+4.5) 68.5→73.9 (+5.4)

62.8→66.5 (+3.7) 88.8→89.4 (+0.5) 86.2→89.4 (+3.2) 70.7→70.2 (-0.5) 73.4→74.5 (+1.1) 83.0→82.4 (-0.5) 77.7→81.4 (+3.7)

2.6→5.2 (+2.6) 51.8→61.0 (+9.2) 36.4→43.9 (+7.5) 7.9→9.5 (+1.6) 3.3→3.9 (+0.7) 37.4→41.3 (+3.9) 17.4→23.3 (+5.9)

7.0→10.8 (+3.8) 42.2→42.7 (+0.5) 36.2→41.6 (+5.4) 15.7→18.4 (+2.7) 7.6→10.3 (+2.7) 37.3→44.9 (+7.6) 29.2→37.8 (+8.6)

Qwen2.5-VL-7B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

57.2→67.6 (+10.4) 80.6→90.5 (+9.9) 91.0→94.1 (+3.2) 52.7→55.4 (+2.7) 41.0→48.2 (+7.2) 82.0→87.4 (+5.4) 89.6→92.3 (+2.7)

71.3→77.7 (+6.4) 88.3→93.6 (+5.3) 95.2→97.9 (+2.7) 73.4→77.1 (+3.7) 78.7→84.0 (+5.3) 87.2→92.6 (+5.3) 89.9→94.7 (+4.8)

6.2→8.2 (+2.0) 44.3→55.7 (+11.5) 68.2→82.3 (+14.1) 5.2→5.2 (+0.0) 7.2→11.8 (+4.6) 25.2→40.0 (+14.8) 31.5→44.9 (+13.4)

17.3→20.5 (+3.2) 56.8→67.6 (+10.8) 70.3→78.4 (+8.1) 14.6→20.0 (+5.4) 16.2→21.1 (+4.9) 44.9→61.6 (+16.8) 50.8→62.7 (+11.9)

Table 29: Complete fixed-control effects for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

∆R

∆RE

A → RE

A → RS

A → RG

C0: Rand−U C0: Rand−U C0: Shuf−U C0: Shuf−U C0: B−Rand C0: B−Rand C0: B−Shuf C0: B−Shuf C1-pre: Rand−U C1-pre: Rand−U C1-pre: Shuf−U C1-pre: Shuf−U C1-pre: B−Rand C1-pre: B−Rand C1-pre: B−Shuf C1-pre: B−Shuf

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 876 900 879 900 872 900 878 900 882 900 886 900 875 900 878

-0.1 [-1.8, +1.7] -0.7 [-2.4, +1.0] -1.6 [-3.3, +0.2] -1.4 [-3.1, +0.3] +8.1 [+5.9, +10.3] +7.8 [+5.5, +10.0] +9.6 [+7.2, +11.8] +8.7 [+6.4, +10.9] -3.6 [-6.1, -0.9] -3.7 [-6.2, -1.2] -1.9 [-4.4, +0.6] -1.7 [-4.2, +0.8] +11.9 [+9.1, +14.6] +12.5 [+9.7, +15.2] +10.2 [+7.2, +13.2] +10.6 [+7.6, +13.6]

+0.4 [-1.1, +2.1] +0.1 [-1.5, +1.7] -1.4 [-3.3, +0.3] -0.9 [-2.6, +0.8] +5.8 [+3.8, +7.8] +5.3 [+3.3, +7.2] +7.7 [+5.7, +9.7] +6.6 [+4.6, +8.6] -3.4 [-5.9, -0.9] -3.6 [-6.1, -1.1] -4.9 [-7.5, -2.3] -4.6 [-7.2, -2.1] +10.0 [+7.0, +13.0] +11.0 [+8.0, +14.0] +11.4 [+8.6, +14.3] +11.7 [+8.9, +14.6]

+2.7 [+1.7, +3.7] +2.2 [+1.3, +3.2] +2.6 [+1.6, +3.6] +2.4 [+1.5, +3.5] +7.1 [+5.5, +8.8] +6.3 [+4.7, +8.0] +8.0 [+6.4, +9.7] +6.9 [+5.3, +8.6] +2.9 [+1.8, +4.1] +2.8 [+1.8, +4.0] +3.1 [+2.0, +4.3] +3.0 [+1.9, +4.3] +10.6 [+8.6, +12.5] +10.6 [+8.7, +12.6] +11.0 [+8.9, +13.1] +10.9 [+8.8, +13.1]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

R→A

+0.9 [+0.3, +1.6] +3.7 [+2.5, +4.9] +0.8 [+0.2, +1.4] +3.7 [+2.5, +4.9] +0.6 [+0.1, +1.1] +4.7 [+3.4, +6.0] +0.5 [+0.1, +0.9] +4.2 [+3.0, +5.5] +3.8 [+2.7, +5.0] +2.8 [+1.7, +4.0] +3.9 [+2.7, +5.1] +2.4 [+1.4, +3.6] +4.0 [+2.8, +5.2] +2.4 [+1.5, +3.5] +4.1 [+2.9, +5.4] +2.4 [+1.5, +3.5] +2.2 [+1.3, +3.3] +8.7 [+6.9, +10.5] +2.0 [+1.1, +3.0] +8.6 [+6.8, +10.5] +3.0 [+1.9, +4.2] +8.0 [+6.2, +9.8] +3.0 [+2.0, +4.2] +7.8 [+6.1, +9.6] +4.8 [+3.4, +6.2] +3.4 [+2.3, +4.6] +4.9 [+3.5, +6.4] +3.1 [+2.1, +4.2] +4.7 [+3.3, +6.1] +5.4 [+4.0, +6.9] +4.6 [+3.2, +6.0] +4.9 [+3.6, +6.3]

Table 30: Complete fixed-control effects for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

C0: Rand−U C0: Rand−U C0: Shuf−U C0: Shuf−U C0: B−Rand C0: B−Rand C0: B−Shuf C0: B−Shuf C1-pre: Rand−U C1-pre: Rand−U C1-pre: Shuf−U C1-pre: Shuf−U C1-pre: B−Rand C1-pre: B−Rand C1-pre: B−Shuf C1-pre: B−Shuf

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 883 900 872 900 877 900 869 900 868 900 868 900 847 900 847

∆R

A → RE

A → RS

A → RG

R→A

+0.8 [+0.2, +1.4] +0.6 [+0.1, +1.1] +2.1 [+1.2, +3.1] +1.5 [+0.8, +2.3] +4.4 [+3.2, +5.8] +4.2 [+3.0, +5.5] +4.3 [+3.1, +5.7] +4.0 [+2.8, +5.3] +2.1 [+1.2, +3.1] +2.2 [+1.3, +3.2] +3.0 [+1.9, +4.2] +3.1 [+1.9, +4.4] +7.2 [+5.7, +8.8] +7.6 [+6.0, +9.2] +6.4 [+5.0, +8.0] +6.6 [+5.1, +8.2]

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+0.8 [+0.2, +1.4] +0.8 [+0.2, +1.4] +0.7 [+0.2, +1.2] +0.7 [+0.2, +1.3] +3.4 [+2.3, +4.7] +3.5 [+2.3, +4.8] +3.1 [+2.1, +4.2] +3.1 [+2.0, +4.3] +1.4 [+0.8, +2.2] +1.5 [+0.8, +2.3] +1.1 [+0.5, +1.8] +1.2 [+0.5, +1.9] +1.4 [+0.8, +2.2] +1.4 [+0.7, +2.2] +1.8 [+1.0, +2.7] +1.8 [+0.9, +2.7]

+3.6 [+2.4, +4.8] +3.4 [+2.3, +4.6] +3.2 [+2.1, +4.4] +3.1 [+2.0, +4.3] +1.4 [+0.7, +2.4] +1.4 [+0.6, +2.2] +2.6 [+1.5, +3.8] +2.0 [+1.0, +3.0] +5.4 [+4.1, +6.9] +5.3 [+3.9, +6.8] +5.6 [+4.2, +7.0] +5.5 [+4.2, +7.0] +2.9 [+1.8, +4.2] +3.1 [+1.9, +4.4] +2.9 [+1.9, +4.0] +2.8 [+1.8, +4.0]

∆RE

-2.0 [-3.5, -0.6] -1.9 [-3.5, -0.2] -2.0 [-3.5, -0.7] -1.8 [-3.3, -0.3] -0.4 [-2.1, +1.2] +0.4 [-1.1, +2.1] -0.9 [-2.5, +0.6] -0.2 [-1.7, +1.3] +6.4 [+4.5, +8.4] +4.8 [+2.8, +6.8] +6.4 [+4.4, +8.3] +4.6 [+2.7, +6.5] +4.9 [+2.7, +7.0] +2.4 [+0.1, +4.7] +5.2 [+3.2, +7.2] +2.9 [+0.7, +5.0] -1.9 [-3.6, -0.1] -0.6 [-2.7, +1.7] -1.6 [-3.4, +0.2] -0.3 [-2.6, +1.9] -1.4 [-3.5, +0.6] -0.3 [-2.8, +2.2] -1.3 [-3.4, +0.9] -0.1 [-2.7, +2.5] +5.8 [+3.6, +8.0] +7.2 [+4.9, +9.6] +5.9 [+3.6, +8.2] +7.9 [+5.6, +10.3] +5.3 [+3.1, +7.5] +7.0 [+4.5, +9.5] +5.5 [+3.3, +7.8] +7.2 [+4.5, +9.8]

21

Table 31: Complete fixed-control effects for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals. R pools RE , RS , and RG . Comparison

Sensitivity

n

C0: Rand−U C0: Rand−U C0: Shuf−U C0: Shuf−U C0: B−Rand C0: B−Rand C0: B−Shuf C0: B−Shuf C1-pre: Rand−U C1-pre: Rand−U C1-pre: Shuf−U C1-pre: Shuf−U C1-pre: B−Rand C1-pre: B−Rand C1-pre: B−Shuf C1-pre: B−Shuf

All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed All No-mixed

900 867 900 871 900 866 900 868 900 837 900 823 900 817 900 803

∆R

A → RE

∆RE

-0.8 [-2.8, +1.2] -0.3 [-2.2, +1.6] +2.1 [+1.1, +3.1] -0.7 [-2.7, +1.3] -0.2 [-2.0, +1.6] +1.8 [+1.0, +2.8] +0.0 [-2.0, +2.0] -1.2 [-3.3, +0.9] +2.7 [+1.7, +3.7] +0.0 [-2.1, +2.1] -0.8 [-2.9, +1.3] +2.5 [+1.6, +3.5] +6.0 [+3.9, +8.2] +6.8 [+4.4, +9.2] +5.8 [+4.3, +7.4] +5.8 [+3.7, +8.0] +6.2 [+3.9, +8.6] +5.2 [+3.7, +6.8] +5.2 [+3.0, +7.5] +7.7 [+5.5, +9.9] +5.3 [+3.9, +6.9] +4.6 [+2.3, +7.0] +6.8 [+4.7, +9.0] +4.7 [+3.4, +6.2] -1.9 [-4.4, +0.5] -1.6 [-4.3, +1.1] +4.0 [+2.7, +5.4] -1.8 [-4.4, +0.7] -1.6 [-4.4, +1.3] +4.2 [+2.9, +5.6] -0.8 [-3.0, +1.6] -4.4 [-7.3, -1.6] +4.8 [+3.4, +6.3] -0.9 [-3.3, +1.7] -4.7 [-7.7, -1.7] +5.0 [+3.5, +6.7] +11.6 [+8.6, +14.5] +11.8 [+8.7, +14.9] +12.4 [+10.2, +14.8] +11.9 [+8.8, +15.0] +12.0 [+8.8, +15.2] +12.5 [+10.2, +14.9] +10.4 [+7.6, +13.2] +14.7 [+11.6, +17.8] +11.2 [+9.2, +13.3] +11.3 [+8.3, +14.4] +14.9 [+11.8, +18.1] +11.8 [+9.6, +14.1]

A → RS

A → RG

R→A

+0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0] +0.0 [+0.0, +0.0]

+2.0 [+1.1, +3.0] +2.1 [+1.2, +3.1] +2.7 [+1.7, +3.7] +2.6 [+1.7, +3.7] +3.4 [+2.4, +4.6] +3.6 [+2.5, +4.8] +3.8 [+2.6, +5.1] +3.9 [+2.7, +5.3] +1.6 [+0.8, +2.4] +1.6 [+0.7, +2.5] +2.1 [+1.2, +3.1] +2.3 [+1.3, +3.4] +2.8 [+1.8, +3.9] +3.1 [+1.9, +4.3] +3.1 [+2.0, +4.4] +3.5 [+2.2, +4.9]

+4.9 [+3.5, +6.4] +4.6 [+3.2, +6.1] +5.3 [+3.8, +6.9] +5.2 [+3.7, +6.8] +3.2 [+2.1, +4.4] +3.0 [+1.9, +4.2] +3.9 [+2.8, +5.1] +4.0 [+2.9, +5.3] +7.4 [+5.8, +9.3] +7.5 [+5.8, +9.4] +7.7 [+6.1, +9.3] +8.1 [+6.4, +9.9] +3.7 [+2.4, +5.0] +3.7 [+2.4, +5.1] +3.9 [+2.6, +5.3] +4.0 [+2.6, +5.5]

Table 32: Frozen direction metadata. Raw norm is the norm before unit normalization at intervention. SHA-256 values identify the exact vector artifacts. Model

Direction

InternVL3-2B

B Shuffled-B Random

Layer Scale Seed Raw norm 16 16 16

6 6 6

42 42 42

2.830775 0.752670 2.830775

InternVL3-8B

B Shuffled-B Random

9 9 9

8 8 8

42 42 42

2.809057 0.321391 2.809057

B Qwen2.5-VL-7B Shuffled-B Random

18 18 18

8 8 8

42 11.792513 42 1.677381 42 11.792514

Table 33: Answered and refused counts in the 5,000-example direction-construction role. The shuffled-label control permutes labels within each listed category, so these within-category marginals are preserved exactly. Model

Category

Answered

Refused

Total

InternVL3-2B

Context-dependent Vague description False premise Subjective/philosophical Unknown

894 1044 1054 955 35

475 142 343 51 7

1369 1186 1397 1006 42

InternVL3-8B

Context-dependent Vague description False premise Subjective/philosophical Unknown

706 887 518 850 31

663 299 879 156 11

1369 1186 1397 1006 42

Qwen2.5-VL-7B

Context-dependent Vague description False premise Subjective/philosophical Unknown

398 520 330 749 25

971 666 1067 257 17

1369 1186 1397 1006 42

22

Table 34: Raw refusal and mixed-response counts for each benign description template. Each modeltemplate cell contains 176 unique images. Unmodified refused

B refused

Unmodified mixed

B mixed

Explain Describe List Summarize

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

InternVL3-8B

Explain Describe List Summarize

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

Qwen2.5-VL-7B

Explain Describe List Summarize

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

0/176 0/176 0/176 0/176

Model

Template

InternVL3-2B

Table 35: Paired MMMU correctness transitions under the behavior-direction intervention. All comparisons use the same sample subset; accuracy differences are percentage points with paired-item bootstrap intervals. Model

Unmodified correct

B correct

Both correct

Unmodified only

B only

Both wrong

∆ accuracy [95% CI]

293 351 304

290 353 306

286 350 299

7 1 5

4 3 7

353 296 339

-0.46 [-1.38, +0.46] +0.31 [-0.31, +0.92] +0.31 [-0.77, +1.38]

InternVL3-2B InternVL3-8B Qwen2.5-VL-7B

InternVL3-2B

InternVL3-8B

Qwen2.5-VL-7B

C0 C1-pre C1-post C2-pre C2-post C3-pre C3-post

0.0

2.5

5.0

7.5

10.0

12.5

Change in refusal (percentage points)

15.0

17.5

0

2

4

6

Change in refusal (percentage points)

8

0

2

4

6

8

10

Change in refusal (percentage points)

12

14

Figure 5: Paired change in total refusal under the frozen behavior-derived direction across all responsepolicy conditions. Error bars are 95% image-clustered bootstrap confidence intervals.

Context-dependent 13.5

InternVL3-8B

Qwen2.5-VL-7B

14.0

16.2

16.2

4.1

4.1

Context-dependent

8.1

2.3

4.5

0.5

4.5

4.5

5.4

Context-dependent 10.4

9.9

3.2

2.7

7.2

5.4

2.7

False premise 14.4 11.2 13.8

6.4

10.1

10.1

11.2

False premise

3.7

0.5

3.2

-0.5

1.1

-0.5

3.7

False premise

6.4

5.3

2.7

3.7

5.3

5.3

4.8

9.9

15 10 5 0

Subjective/philosophical

1.6

3.9

18.0

2.6

1.6

7.2

5.6

Subjective/philosophical

2.6

9.2

7.5

1.6

0.7

3.9

5.9

Subjective/philosophical

2.0

11.5

14.1

0.0

4.6

14.8

13.4

Vague description

5.4

10.8

13.0

11.4

13.5

7.6

11.9

Vague description

3.8

0.5

5.4

2.7

2.7

7.6

8.6

Vague description

3.2

10.8

8.1

5.4

4.9

16.8

11.9

5

Change in total refusal (percentage points)

InternVL3-2B

10

C0

t t t e e e -pr 1-pos 2-pr 2-pos 3-pr 3-pos C C C C C

C1

C0

t t t e e e -pr 1-pos 2-pr 2-pos 3-pr 3-pos C C C C C

C1

C0

15

t t t e e e -pr 1-pos 2-pr 2-pos 3-pr 3-pos C C C C C

C1

Figure 6: Category-stratified changes in total refusal under the behavior-derived direction. Numerical values are reported in Table 28.

23

MLLMs, extending the same paired, reason-aware protocol to additional models and answerable VQA settings is a natural direction for future work.

24

NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The main claims of the study made in the abstract, that refusal is jointly shaped by external instructions, post-training, and internal representations, highlighting the need to disentangle evidence recognition from a model’s learned propensity to refuse, is reflected in paper’s contribution and scope. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: It discusses this in the Appendix section. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs 25

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [N/A] Justification: The paper does not include theoretical results Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We disclose all the information in Appendix, Sec. A. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 26

5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: Justification: Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/p ublic/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //neurips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Yes, the details are provided in the Appendix, Sec. A. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We report the confidence intervals for all the results. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). 27

• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: It is provided in Appendix Sec. A. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conforms to code of ethics. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [N/A] Justification: There is no societal impact of the work performed. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact. 28

• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: The paper poses no such risks. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All creators or original owners of assets are properly used in the paper are credited and cited. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. 29

• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [N/A] Justification: The paper does not release new assets. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 30

• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

31

Related documents

• When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models#1469033
• QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models#1389133
• A$^2$Safe: Counterfactual Evidence-Aligned Adaptive Agent Collaboration for Safe and Effective Visual Question Answering#1050284
• Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification#1413598
• Harnessing Textual Refusal Directions for Multimodal Safety#1388402