P RIVACY S IM: Evaluating LLM Simulation of User Privacy Behavior
arXiv:2605.12147v1 [cs.CR] 12 May 2026
James Flemings University of Southern California [email protected]
Murali Annvaram University of Southern California [email protected]
Data
§ Code
Abstract Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce P RIVACY S IM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decisionmaking: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model’s response to a data-sharing scenario matches the user’s actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user’s stated privacy attitudes alone may not be the best predictor because they often diverge from the user’s actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. P RIVACY S IM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release P RIVACY S IM to enable further evaluation of LLM privacy simulation.
1
Introduction
Applying large language models (LLMs) to simulate users is a promising direction for evaluating the privacy of data-sharing systems. For example, practitioners designing an LLM healthcare assistant must know whether it is appropriate to share the user’s mental-health information to family members, clinicians, or government agencies. Eliciting the user’s data-sharing preferences through controlled user studies is the gold standard but is slow, expensive, and prone to privacy fatigue, where participants make ineffective privacy decisions when feeling overwhelmed [4, 9]. In contrast, conditioning LLMs with user-specific information to simulate the user’s privacy behavior could substitute for this elicitation, driving early-stage privacy evaluation and development [5, 7, 31]. Human studies can then validate the systems’ privacy implications closer to deployment. However, it remains an open question whether LLMs can sufficiently simulate individual-level privacy behavior to be useful for this purpose. Existing work simulating user behavior typically conditions models on personas, representations of hypothetical individuals [18, 19, 39]. Yet the insights derived from these evaluations may not transfer to simulating privacy behavior. Users routinely deviate from Preprint.
Privacy Persona User Studies
LLM Healthcare Consultation
User Responses
Demographics: -age: 18-24, Privacy Attitudes: concerned with LLM data usage Previous Experiences: frequently uses LLMs
Q1. Neutral Q2. Strongly Agree Q3. …
LLM Responses Q1. Disagree Q2. Strongly Agree Q3. …
LLM conversation agents
Questionnaire
Group chat bots
Q1. Is it appropriate to share your medical information with a chatbot? Q2. …
LLM
Simulation Score
Figure 1: Overview of P RIVACY S IM, an evaluation suite for simulating user privacy behavior. We collect user responses and questionnaires from existing user studies on privacy behavior in LLM and AI contexts. We then condition LLMs on subsets of a user’s privacy persona (demographics, previous experiences with LLMs, and stated privacy attitudes) to simulate their responses to data-sharing questions. Finally, we evaluate the simulation quality by comparing the LLM’s simulated responses to the user’s actual responses. their stated attitudes when making privacy decisions, a pattern known as the privacy paradox [20, 29]. For example, a user concerned about disclosing information to an LLM may still share it to receive personalized responses. Hence, there is a need to go beyond generalized user personas to create privacy personas that are specifically tailored to privacy behavior. Our work seeks to identify a core set of user persona attributes that can be used by an LLM to accurately simulate individual-level privacy behavior. Such privacy-specific persona attributes would enable practitioners to condition LLMs with these attributes to simulate user privacy behavior across various domains. Despite the practical promise, no standardized evaluation suite or methodology exists for assessing LLMs as user-privacy simulators. Recent work either evaluates only populationlevel alignment [23] or uses prior privacy decisions to predict subsequent ones [14]. In contrast, we target individual-level alignment using a core set of persona attributes, without assuming access to in-domain prior user privacy decisions. We address this gap by introducing P RIVACY S IM (shown in Figure 1). At the foundation of this approach is a curated dataset of users’ responses to a set of data-sharing scenarios posed across five existing user studies. These five studies were conducted in diverse areas such as LLM healthcare consultation, conversational agents, and group chat bots. In particular, we sample 200 users from each study and 7–10 data-sharing questions per user, yielding 9,400 user–question pairs. From this curated dataset, we identify three core facets that constitute a user’s privacy persona: demographics (e.g., age), previous experiences (e.g., AI usage), and privacy attitudes (e.g., general data-usage concerns). Crucially, P RIVACY S IM is grounded in real user responses to data-sharing scenarios with a rich set of user attributes, enabling accurate evaluation of whether LLMs can faithfully simulate the nuanced, often inconsistent ways individual users handle privacy [3]. We condition each model on subsets of these facets and measure how closely its simulated responses match the user’s actual responses. We use P RIVACY S IM to organize our analysis around three research questions (RQs): • RQ1. To what extent can a core set of persona facets be used by LLMs to simulate user privacy decisions across data-sharing scenarios? • RQ2. How does LLM simulation quality vary across user privacy persona groups (e.g., users with high stated concern and high AI/chatbot exposure vs. users with low stated concern and high AI/chatbot exposure)? • RQ3. How do LLM characteristics (parameter count and reasoning effort) and privacy-theory prompting affect simulation quality? Our key findings are as follows. First, the top-scoring LLM, Gemini 3.1 Pro, achieves only 40.4% accuracy at simulating individual-level privacy behavior, which motivates further research into richer persona facets. Contrary to expectation, stated privacy attitudes alone are not the strongest signal of a user’s actual privacy behavior. This finding highlights the challenge of handling the privacy 2
paradox. Second, we adopt a (stance, exposure) persona grouping inspired by Dupree et al. [13]. Within this grouping, users with high AI/chatbot experience but low stated privacy attitudes have the lowest simulation quality, indicating that certain user privacy groups are disproportionately difficult to model. Third, larger models and higher test-time reasoning compute yield only modest gains in simulation quality. Prompting models to reason through well-established privacy theories, such as Privacy Calculus [11], also fails to improve simulation quality beyond the signal already provided by the persona facets.
2
Related Work
Privacy-decision theoretical frameworks. A long line of work formalizes how individuals make data-sharing decisions. Contextual Integrity [28] posits that disclosure is appropriate when it conforms to the privacy norms of the originating context. Privacy Calculus [11] models disclosure as a deliberate cost-benefit tradeoff between perceived benefits and risks, while Bounded Rationality accounts [2, 3] argue that users instead rely on heuristics and satisficing. Protection Motivation Theory [34], originally developed for health-behavior research, casts protective behavior as the joint output of a threat appraisal and a coping appraisal. User studies on privacy behavior. A rich body of work investigates user privacy attitudes and behavior in various data-sharing contexts, such as app permissions [15, 41], smart home devices [1, 6], and social media platforms [40]. Recent attention has focused on user privacy behavior in data-sharing contexts involving LLMs and chatbots, which is the focus of our work. Prior work has conducted user studies involving users disclosing information to LLMs [25, 43, 44], disclosure of LLM chat history to other entities [24, 37], and user data access by chatbots and AI agents [10, 42]. These data-sharing contexts are usually vignette-based questionnaires [26] grounded in contextual integrity. LLM simulation of user behavior. A growing body of work uses LLMs to simulate user behavior [5, 7, 8, 12, 35, 17]. Recent studies condition LLMs on synthesized personas from seed human data [18, 19, 31, 32, 39]. Prior work also clusters users into privacy-persona groups with similar attitudes and behaviors [13, 21]. Most recently, two studies investigate using LLMs conditioned on personas to simulate user privacy behavior. Li et al. [23] evaluate population-level alignment of LLMs to simulate privacy attitudes and behavior. They generate synthetic participants by sampling demographics that match the distribution of the original studies. Fawaz et al. [14] generate user personas by synthesizing in-domain prior user privacy decisions to predict new ones [16]. In contrast to both, our work focuses on individual-level alignment using a core set of privacy persona attributes, without assuming access to in-domain prior user privacy decisions.
3
Constructing the P RIVACY S IM Evaluation Suite
P RIVACY S IM grounds LLM privacy simulations in five published user studies, conditions models on a user’s privacy persona, and scores each simulated response against the user’s actual response. We formalize the simulation task in Section 3.1, describe how we curate the evaluation suite in Section 3.2, and define our evaluation metric in Section 3.3. 3.1
Problem Statement
Let Pu denote the privacy persona of a user u, which plausibly captures signal about the user’s privacy behavior (see Section 3.2 for a concrete example). Let Qu be the questionnaire of datasharing scenarios given to user u, and let qi ∈ Qu be the i-th question, paired with the user’s response ru,i . An LLM simulator f takes the persona and the question and produces a simulated response: r̂u,i = f (Pu , qi ). Define ru = [ru,1 , ru,2 , . . . , ru,|Qu | ] and r̂u = [r̂u,1 , r̂u,2 , . . . , r̂u,|Qu | ] to be the user’s and LLM’s complete responses to the questionnaire Qu . We introduce an evaluation function g(r̂u , ru ) to measure the alignment of the LLM’s simulated responses with the user’s responses. We note that neither the data-sharing questions in Qu nor the corresponding user responses appear in Pu . 3.2
Dataset Construction
As mentioned in Section 1, no standardized evaluation suite exists for assessing LLM simulation of user privacy behavior. While one option could be to synthetically generate users using an LLM, such 3
Table 1: Summary of source user studies in P RIVACY S IM. Du , Eu , and Au denote demographics, previous experiences, and privacy attitudes, respectively. “Domain” refers to the data-sharing context posed in the original study. Author
Venue
Country
Du
Eu
Au
Domain
Likert scale
Liu et al. [24] Tran et al. [37] Wu et al. [42] Zufferey et al. [44] Chou et al. [10]
S&P 2025 AIES 2025 S&P 2026 PETS 2025 PETS 2026
China US US Various Various
✗ ✗ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓
LLM Healthcare Consultation LLM Chatbot AI Agent Permissions LLM Conversational Agents Chatbot Group Chat
5-point 100-point 4-point 5-point 5-point
an approach risks missing the complexity inherent in privacy decision-making. Instead, we curate a dataset grounded in existing user studies on privacy behavior. To collect a set of questionnaires Qu and personas Pu to be used in P RIVACY S IM, we searched for prior work that elicited user privacy behavior and released its study results with appropriate licensing for reuse. This search is challenging because many user studies are not publicly available. Synthesizing findings from available user studies, we instantiate a user’s privacy persona Pu as three persona facets that we hypothesize to be plausible predictors of privacy behavior: • Demographics (Du ): user attributes such as age, gender, and education that may correlate with privacy expectations. • Previous Experiences (Eu ): the user’s familiarity with emerging technologies such as AI or chatbots. • Privacy Attitudes (Au ): the user’s stated privacy concerns and preferences regarding data usage. We chose these facets based on the analyses in the source studies and the data available from each study. Taken together, these facets constitute a user’s privacy persona Pu = (Du , Eu , Au ). We required each source study to contain and publicly release (1) user responses for previous experiences (Eu ) and privacy attitudes (Au ) (demographics (Du ) are optional because some publicly released user studies omit them for confidentiality), and (2) user responses to data-sharing scenarios Qu involving LLMs, chatbots, or agents. Given the heightened concern for privacy in AI-mediated interactions, these criteria address the timely question of whether LLMs can understand user datasharing decisions when AI or chatbots are involved. We narrowed our search to five controlled user studies that meet these criteria, which are summarized in Table 1. Notably, each study is published in a top-tier privacy venue, and participants span multiple countries. Each study varies in the type of questions asked and the response format used for demographics, previous experiences, privacy attitudes, and data-sharing scenarios. The example below shows a representative user from Zufferey et al. [44] with one attribute drawn from each of the three persona facets and one paired data-sharing question and response (an example user for each user study can be found in Appendix C). Example: User from LLM Conversational Agents [44] Demographics (Du ). Age: 18–29. Previous experiences (Eu ). Q. Are you using your own local chatbot instance (e.g., Llama, Mistral) based on only locally stored data? A. “Yes, but not exclusively (I also use chatbot services such as ChatGPT).” Privacy attitudes (Au ). Q. It usually bothers me when online companies ask me for personal information. A. “Agree” (7-point: Strongly disagree → Strongly agree). Data-sharing question qi (Q1). How frequently do you share personal identifiers (e.g., name, ID card number, email, phone number) with a chatbot? User response ru,i . “I often share” (5-point: never / rarely / sometimes / often / usually shared; mapped to {0, 1, 2, 3, 4} for evaluation).
We keep each questionnaire presented to the LLM as close as possible to the original questionnaire from the source study. This controls for any variation in how the questions are posed between the original study participants and the LLM. We provide full prompts in Appendix A. To ensure each user study has equal weight in the final evaluation, we sample 200 users per study, since the smallest 4
user study has 203 users; for each user we sample ten data-sharing questions in four of the studies and seven in the fifth, yielding a benchmark of 1,000 users and 9,400 user–question pairs. Lastly, all datasets are publicly released by the original authors under permissive licenses (Creative Commons or MIT) that allow reuse if the original authors are properly attributed. Appendix C gives more information about each study. 3.3
Evaluation Methodology
Because we preserve the original questionnaires, the response scales differ across studies. For example, responses to questions from Zufferey et al. [44] are frequency disclosures on a five-point scale (i.e., ru,i ∈ [0, 4]), whereas Tran et al. [37] elicit agreement with privacy statements on a 0–100 numeric scale. Hence, we cannot simply use exact-match accuracy here because responses on the 100-point scales rarely match exactly. To address this shortcoming, we modify exact-match accuracy so that it measures whether the LLM’s simulated response falls within a certain tolerance bound of the user’s actual response. This bound is defined as a fraction of the responses’ Likert scale. More precisely, let WQu be the upper bound of the response scale used in Qu (e.g., 4 for a 5-point Likert scale, 100 for a 0–100 numeric scale), so each response lies in [0, WQu ]. The simulated tolerance accuracy for a user u with responses ru and simulated responses r̂u is |Qu | 1 X g(r̂u , ru ) = 1[ |ru,i − r̂u,i | ≤ τ · WQu ] . (1) |Qu | i=1 We set τ = 0.15, simplifying (1) to exact-match accuracy for responses on a 4-point or 5-point Likert scale. Effectively, τ has no effect on those Likert-scale responses; on the 100-point scales, the LLM is credited with a match when its prediction lies within ±15 points of the user’s response. We report 100 · g(r̂u , ru ) as a percentage in our experimental results. Evaluation use case. LLM evaluations increasingly rely on synthetic datasets. However, we argue that evaluating LLM privacy decision-making should not rely on synthetic datasets, which are generally generated by LLMs, as such datasets may not fully capture the complexity and nuances of real-world privacy behavior that diverges substantially from general non-privacy contexts. Hence, it is crucial that privacy evaluations be grounded in real user data, and P RIVACY S IM fulfills this need by curating a dataset of real user responses to data-sharing scenarios. By evaluating LLMs on this dataset, we can gain insights into how well LLMs can simulate actual user privacy behavior; such insights are essential for understanding the privacy implications of LLMs in real-world applications.
4
Experiments
We now use P RIVACY S IM to answer the three research questions (RQs) posed in Section 1. Section 4.1 describes our experimental setup, and Sections 4.2, 4.3, and 4.4 present results for RQ1, RQ2, and RQ3, respectively. 4.1
Experimental Setup
Models. We evaluate nine frontier reasoning LLMs, both open-weight and closed-source. The closed models are GPT-5.4 (gpt-5.4-2026-03-05), GPT-5.4 Mini (gpt-5.4-mini-2026-03-17), Claude Sonnet 4.6 (claude-sonnet-4-6), Gemini 3.1 Pro (gemini-3.1-pro-preview), and Gemini 3 Flash (gemini-3-flash-preview), accessed through their respective vendor APIs. The open-weight models are Qwen3.5 27B and Qwen3.5 122B [33], Nemotron 3 120B [30], and Gemma 4 32B, served locally with vLLM [22] on an internal cluster of 8 × A100 (40 GB) GPUs. Decoding parameters. For open-weight models we adopt the temperature T and top-p values recommended by each model provider, namely T = 1.0 and p = 0.9512 . For closed-source models, the temperature parameter is not configurable. Unless stated otherwise, we set the reasoning effort of each model to medium if applicable (Section 4.4 varies this choice), and we set the max output length to 8,192 tokens for closed-source models and 32,768 tokens for open-weight models. 1 https://huggingface.co/google/gemma-4-31B-it#1-sampling-parameters 2 https://huggingface.co/Qwen/Qwen3.5-27B#using-qwen35-via-the-chat-completions-api
5
Tolerance Accuracy (%)
45
40
35
30
none
Gemini 3.1 Pro
Gemini 3 Flash
Eu GPT-5.4
Au Sonnet 4.6
Eu+Au GPT-5.4 Mini
Du Gemma4 32B
Du+Eu Qwen3.5 27B
Du+Au
Nemotron 3 120B
Du+Eu+Au Average
5 studies
3 studies
Figure 2: Average accuracy by prompt type and model across user studies. Du , Eu , and Au denote demographics, previous experiences, and stated privacy attitudes (Section 3.2). Solid bars are the average of all five user studies; hatched bars are the average of the three user studies that contain demographics. The black diamond line is the average across the eight models on the x-axis. For the prompt types containing demographics, the bars appear only as hatched. Our key findings show that (1) Gemini 3.1 Pro achieves the highest accuracy (40.4%) on Eu +Au across the five studies, a 4-percentage-point gain over the no-persona baseline. (2) The no-persona prompt (none) achieves a non-trivial simulation accuracy, suggesting that LLMs can simulate user behavior to some extent even without conditioning on persona facets. (3) The simulation quality can degrade when conditioning only on privacy attitudes compared to previous experiences, consistent with the privacy paradox. Reporting conventions. All reported numbers are tolerance accuracy g(r̂u , ru ) from (1) at τ = 0.15, expressed as percentages. Because a subset of the user studies do not contain demographics, we split the results into two groups: (1) 5 studies containing all the user studies and (2) 3 studies restricted to the domains AI Agent Permissions, Chatbot Group Chat, and LLM Conversational Agents, which all contain demographics. Hence, any claims involving demographics are based on the 3-study subset, while claims about the other facets or overall trends may be based on the full 5-study set. We make this distinction clear in the text and figures. 4.2
RQ1: To what extent can privacy persona attributes drive accurate user simulations?
To answer RQ1, we condition each LLM on every non-empty subset of the three persona facets ({Du , Eu , Au }) plus a no-persona baseline (none), yielding eight prompt types. Figure 2 reports the average accuracy of each model across prompt types, with separate bars for the 5 studies and 3 studies. Figure 3 provides a per-model breakdown for the four strongest models, plotting each prompt type alongside each domain to analyze the cross-domain variance. From these results, we make the following observations. 1. Our hypothesized persona facets provide modest simulation fidelity. At the richest combination of persona facets supported across all five studies (Eu +Au , 5 studies), Figure 2 shows that Gemini 3.1 Pro achieves the highest score at 40.4% accuracy, with Gemini 3 Flash close behind at 40.3%. Across the evaluated models, the average accuracy on the 5 studies is 38.8%. When adding demographics to the persona Du +Eu +Au , the average accuracy across the 3 studies pushes Gemini 3.1 Pro to 44.8% and the model average to 41.2%. We interpret these findings as evidence that LLMs can extract real signal from our privacy persona facets, but remain far from a faithful proxy for individual privacy decision-making, motivating further work on exploring richer persona representations. 2. The performance of the no-persona baseline is non-trivial. We observe that providing Eu +Au to the persona gives a performance boost of ∼ 3 percentage points on average across the five studies, suggesting that LLMs exhibit a non-trivial understanding of general user privacy behavior even without explicit persona conditioning. This finding is consistent with prior works evaluating LLMs to reason about contextual privacy, which showed that LLMs posses some awareness of privacy norms [27, 36]. When analyzing the reasoning traces of the models when no persona facets are 6
Gemini 3.1 Pro
Gemini 3 Flash
GPT-5.4
Sonnet 4.6
Tolerance Accuracy (%)
50
45
40
35
30
ne
no
Eu
A u +A u Eu
D u +E u +A u +A u Du D u +E u Du
AI Agent Permissions
ne
no
Eu
Chatbot Group Chat
A u +A u Eu
D u +E u +A u +A u Du D u +E u Du
LLM Chatbot
ne
no
Eu
A u +A u Eu
LLM Conversational Agents
D u +E u +A u +A u Du D u +E u Du
ne
no
LLM Healthcare Consultation
Eu
A u +A u Eu 5 studies
D u +E u +A u +A u Du D u +E u Du 3 studies
Figure 3: Per-domain accuracy with two averages (solid black: 5-study average; dashed black: 3-study average) across the eight prompt types for the four strongest models: Gemini 3.1 Pro, Gemini 3 Flash, GPT-5.4 (medium), and Claude Sonnet 4.6. Each colored line denotes one user-study domain (LLM Healthcare, AI Agent Permissions, LLM Chatbot, Chatbot Group Chat, and LLM Conversational Agents). Full figure can be found in Figure 4 in Appendix D.
provided, we find that LLMs usually assume an average user’s privacy behavior and then apply population-level privacy norms relevant to the data-sharing question (we provide an example of GPT-5.4’s reasoning trace for no-persona conditioning in Appendix B). Hence, the persona facets are providing a more individualized signal on top of a non-trivial baseline. The fact that every model benefits from persona conditioning suggests that the signal is useful. 3. Privacy attitudes are vulnerable to the privacy paradox. Across nearly every model and most domains (Figures 2 and 3), we observe a consistent dip in accuracy when conditioning on privacy attitudes (Au ) alone compared to conditioning on previous experiences (Eu ) alone. We would expect that privacy attitudes would provide the strongest signal because they are seemingly the most relevant to privacy behavior. We attribute this counterintuitive finding to the privacy paradox, where a user’s stated privacy attitudes may not align with their actual privacy behavior, which was originally observed in some of our evaluated user studies [24, 44]. This paradox makes it difficult for LLMs conditioned solely on self-reported attitudes to simulate user decisions accurately. However, when paired with previous experiences, the model can match and even slightly exceed the performance of previous experiences alone, suggesting that the two facets together can provide a more complete picture of user privacy behavior than either alone. 4. Demographics alone yield the smallest single-facet gain. Our results show that conditioning on demographics alone (Du ) yields marginal gains over conditioning on no persona (none). This finding is inconsistent with prior findings in LLM simulation where demographics generally provide useful predictive signal [39]. We believe this is because of Observation 3, where LLMs already possess some understanding of privacy behavior for an average user. However, when paired with previous experiences and privacy attitudes, Du +Eu +Au achieves the best 3-study average accuracy. 5. Certain privacy behavior tasks are easier to simulate than others. In Figure 3, we observe that all models perform worse on LLM Healthcare Consultation, and conditioning on any user persona facets only worsens the simulation quality. This may be because the healthcare domain involves more complex and high-stakes privacy decisions that are less predictable from the current persona representations provided in the original study. In contrast, the LLM Conversational Agents and Chatbot Group Chat domains show higher accuracy, suggesting that these tasks may involve more consistent and norm-driven privacy behaviors that are easier for LLMs to simulate.
4.3
RQ2: Which user privacy-persona groups are hardest to simulate?
Next, we are interested in understanding LLM simulation quality across different groups of users. To this end, we group users based on their stated previous experiences and privacy attitudes using a privacy persona taxonomy inspired by Dupree et al. [13] and analyze the simulation quality across these groups. We detail below the persona grouping and per-cluster results. 7
Table 2: Accuracy for each privacy persona group averaged over Gemini 3.1 Pro, Gemini 3 Flash, GPT-5.4, and Claude Sonnet 4.6 on the 5 studies. Cluster labels are (stance / exposure) where stance is based on Au and exposure is based on Eu (see Appendix E for more details on mapping). Bold marks the highest and lowest values of acc(Eu +Au ). Cluster (stance / exposure) high / mid mid / mid low / low high / high low / high
n
acc(none)
acc(Eu +Au )
226 187 205 161 221
33.88 39.94 36.00 33.90 35.44
42.41 41.53 39.26 37.25 36.51
Table 3: Simulation accuracy across model families and variants. For GPT-5.4 we report the smaller mini model and the full model under low and medium reasoning effort. We find that reasoning effort has negligible impact and model size can yield modest simulation improvement. Family
Variant
acc(none)
acc(Eu +Au )
GPT-5.4 GPT-5.4 GPT-5.4 Gemini 3 Gemini 3 Qwen3.5 Qwen3.5
Mini Low Medium Flash Pro 27B 122B
36.75 36.14 35.70 37.07 36.32 34.53 35.49
37.99 38.77 39.30 40.27 40.39 36.41 37.34
Persona grouping. We borrow the 2D layout from Dupree et al. [13] to organize users into five clusters along two axes derived from their persona facets: stance (the user’s stated privacy concern, summarized from Au ) and exposure (the user’s self-reported familiarity with AI tools, summarized from Eu ). The five clusters are defined as follows: (1) high-stance / high-exposure, (2) high-stance / mid-exposure, (3) mid-stance / mid-exposure, (4) low-stance / high-exposure, and (5) low-stance / low-exposure. Implementation details for how we map specific values of privacy attitudes and previous experiences to the five clusters are in Appendix E. This grouping yields the following cluster sizes (n=1,000): high-stance / mid-exposure = 226, low-stance / high-exposure = 221, low-stance / low-exposure = 205, mid-stance / mid-exposure = 187, high-stance / high-exposure = 161. Table 2 reports the accuracy at the no-persona baseline (none) and Eu +Au per cluster, averaged over the four strongest models (Gemini 3.1 Pro, Gemini 3 Flash, GPT-5.4 (medium), and Claude Sonnet 4.6) on the 5 studies. From these results, we make the following observations: 1. Low-stance, high-exposure users are the hardest group to simulate. Users with low stated privacy concern but high AI/chatbot exposure achieve the lowest accuracy at Eu +Au (36.51%) and the smallest gain from persona conditioning (+1.07 pp). We hypothesize that because their stated stance is low but their previous experience is shaped by extensive AI/chatbot usage, the LLM infers that they are less concerned about privacy than they actually are. 2. High-stance, mid-exposure users are the easiest group to simulate. In contrast, the high-stance / mid-exposure cluster (high stated privacy concern with moderate AI/chatbot exposure) sees the largest gain from persona conditioning (+8.53 pp) and reaches the highest Eu +Au accuracy in the table (42.41%). We hypothesize that this cluster’s stated attitudes are consistent with their privacy behavior, allowing for more accurate inferences about their privacy decisions. 3. The no-persona baseline systematically under-fits the most privacy-concerned users. At none, the two high-stance clusters (high-stance / high-exposure and high-stance / mid-exposure) achieve the lowest accuracy, both sitting below the remaining three clusters. This shortcoming is because the LLM falls back to an “average user” when not conditioned on any persona attributes, performing poorly on users whose actual privacy behavior is more restrictive. Hence, mid-stance, mid-exposure users (i.e., average users) have the highest simulation quality at none, which aligns with our expectations. However, after persona conditioning, the high-stance / high-exposure cluster achieves the second-highest accuracy gain, while the high-stance / mid-exposure cluster achieves the highest. This improvement suggests that the most privacy-concerned users benefit the most from persona conditioning. 4.4
RQ3: How do LLM characteristics and privacy-informed prompts affect simulation?
Finally, we are interested in understanding how certain model characteristics (such as model size and reasoning effort) and prompts informed by established privacy theories affect simulation quality. All numbers in this section focus on accuracy across all five domains. 8
Reasoning effort has negligible effect. From Table 3, we observe that going from low to medium reasoning on GPT-5.4 changes the accuracy by less than one percentage point at the reported prompt types. This result suggests that simulating user privacy decisions does not benefit from scaling test-time compute. Model size within family yields modest, sometimes negligible gains. At Eu +Au , GPT-5.4 (medium) improves over GPT-5.4 Mini by +1.31 pp, Gemini 3.1 Pro improves over Gemini 3 Flash by only +0.12 pp, and Qwen3.5 122B improves over Qwen3.5 27B by +0.93 pp (Table 3). Hence, scaling model size within a family can improve simulation quality, but the gains are modest. Privacy-informed prompts do not improve simulation quality. We further ask whether conditioning LLMs with prompts containing information about well-established privacy theories would help improve simulation quality. We construct three theory-framed variants of the persona prompt by inserting a privacy decision lens block while leaving the rest of the prompt untouched. Each privacy decision lens is based on the following privacy theory: (1) Privacy Calculus [11]; (2) Bounded Rationality [2, 3]; (3) Protection Motivation Theory (PMT) [34]. Full descriptions of each privacy theory can be found in Section 2 and full prompts can be found in Appendix A.6.
Table 4: Theory-informed prompts (Privacy Calculus, Bounded Rationality, and Protection Motivation Theory (PMT)) compared to the standard persona prompt on Gemini 3 Flash and Gemma 4 32B. Numbers are average accuracy across the 5 studies. We restrict to prompt types that exclude demographics so all 5 studies are comparable. We find that no theory-informed prompt produces a clear gain over standard prompting. Prompt type
baseline
Gemini 3 Flash none 37.07 Eu 39.32 Au 38.60 Eu +Au 40.27
Privacy Bounded Calculus Rationality 36.20 39.20 38.81 40.28
35.61 38.05 39.49 39.95
PMT 37.58 39.23 39.08 40.08
Table 4 reports tolerance accuracy on Gemini 3 Flash and Gemma 4 32B across the prompt types Gemma 4 32B that all 5 studies support. Across both models none 35.85 35.66 34.20 35.87 and every prompt type, no theory variant yields E 37.81 35.90 35.45 37.03 u a meaningful gain. These results suggest that Au 38.04 38.09 37.98 37.34 conditioning LLMs to reason about user privacy Eu +Au 39.04 37.96 37.67 38.31 decisions through the lens of well-established privacy theories does not improve their ability to simulate individual privacy decisions beyond the signal already provided by the persona facets.
5
Discussion and Conclusion
Our experiments reveal that our hypothesized set of persona facets, demographics (Du ), previous experiences (Eu ), and privacy attitudes (Au ), is not enough for reliable LLM simulation of individual privacy behavior. Even the strongest model in our suite achieves only 40.4% accuracy on individual privacy decisions. Therefore, we suggest that LLM simulation is appropriate for low-stakes, earlystage tasks such as comparing design alternatives, surfacing obvious privacy violations, and generating candidate hypotheses, but should be paired with human user studies before any deployment decision. Moreover, finding the right persona facets to condition on is challenging due to idiosyncrasies in user privacy behavior. For example, stated privacy attitudes are the most semantically natural signal to predict user privacy behavior, but conditioning on them alone can lead to suboptimal simulation quality. Pairing them with previous experiences (Eu ) is more reliable. Furthermore, low-stance / high-exposure users see only a +1.07 pp gain from our persona conditioning and remain the hardest cluster to simulate. Together, these results indicate that the gap between what users say about privacy and what they actually do is the central obstacle to faithful simulation. Closing this gap may require richer behavioral signals rather than prompt-based solutions. A limitation of this work is that we tested only three persona facets: demographics, previous experiences, and privacy attitudes. This limitation is due to the small number of openly released user studies, and by the fact that these studies were not originally intended for user privacy simulation. Despite this limitation, P RIVACY S IM is a first step toward identifying what additional facets might be necessary to bridge the gap between LLM simulation and real-world user behavior. Future work eliciting richer privacy persona representations through new user studies designed specifically for this purpose will be crucial for advancing the use of LLMs as user-privacy simulators. 9
References [1] Noura Abdi, Xiao Zhan, Kopo M Ramokapane, and Jose Such. Privacy norms for smart home personal assistants. In Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1–14, 2021. [2] Alessandro Acquisti. Privacy in electronic commerce and the economics of immediate gratification. In Proceedings of the 5th ACM conference on Electronic commerce, pages 21–29, 2004. [3] Alessandro Acquisti and Jens Grossklags. Privacy and rationality in individual decision making. IEEE Security & Privacy, 3(1):26–33, 2005. [4] Alessandro Acquisti, Allan Friedman, and Rahul Telang. Is there a cost to privacy breaches? an event study. ICIS 2006 proceedings, page 94, 2006. [5] Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International conference on machine learning, pages 337–371. PMLR, 2023. [6] Noah Apthorpe, Yan Shvartzshnaider, Arunesh Mathur, Dillon Reisman, and Nick Feamster. Discovering smart home internet of things privacy norms using contextual integrity. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, 2(2):1–23, 2018. [7] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023. [8] James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32(4):401–416, 2024. doi: 10.1017/pan.2024.5. [9] Hanbyul Choi, Jonghwa Park, and Yoonhyuk Jung. The role of privacy fatigue in online privacy behavior. Computers in Human Behavior, 81:42–51, 2018. [10] Kai-Hsiang Chou, Yi-An Wang, Chong Kai Lau, Mahmood Sharif, and Hsu-Chun Hsiao. Bot among us: Exploring user awareness and privacy concerns about chatbots in group chats. Proceedings on Privacy Enhancing Technologies, 2026. [11] Mary J Culnan and Pamela K Armstrong. Information privacy concerns, procedural fairness, and impersonal trust: An empirical investigation. Organization science, 10(1):104–115, 1999. [12] Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [13] Janna Lynn Dupree, Richard Devries, Daniel M Berry, and Edward Lank. Privacy personas: Clustering users via attitudes and behaviors toward security practices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5228–5239, 2016. [14] Kassem Fawaz, Ren Yi, Octavian Suciu, Rishabh Khandelwal, Hamza Harkous, Nina Taft, and Marco Gruteser. Text-based personas for simulating user privacy decisions. arXiv preprint arXiv:2603.19791, 2026. [15] Adrienne Porter Felt, Elizabeth Ha, Serge Egelman, Ariel Haney, Erika Chin, and David Wagner. Android permissions: User attention, comprehension, and behavior. In Proceedings of the eighth symposium on usable privacy and security, pages 1–14, 2012. [16] James Flemings, Ren Yi, Octavian Suciu, Kassem Fawaz, Murali Annavaram, and Marco Gruteser. Personalizing agent privacy decisions via logical entailment. arXiv preprint arXiv:2512.05065, 2025. [17] Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, and Paul Röttger. Simbench: Benchmarking the ability of large language models to simulate human behaviors. arXiv preprint arXiv:2510.17516, 2025. 10
[18] EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. Aligning language models to user opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5906–5919, 2023. [19] Brihi Joshi, Xiang Ren, Swabha Swayamdipta, Rik Koncel-Kedziorski, and Tim Paek. Improving language model personas via rationalization with psychological scaffolds. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China. Association for Computational Linguistics, 2025. [20] Spyros Kokolakis. Privacy attitudes and privacy behaviour: A review of current research on the privacy paradox phenomenon. Computers & Security, 64:122–134, 2017. [21] Ponnurangam Kumaraguru and Lorrie Faith Cranor. Privacy indexes: a survey of westin’s studies. 2005. [22] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [23] Yuxuan Li, Leyang Li, Hao-Ping Lee, and Sauvik Das. How well can llm agents simulate end-user security and privacy attitudes and behaviors? arXiv preprint arXiv:2602.18464, 2026. [24] Zhihuang Liu, Ling Hu, Tongqing Zhou, Yonghao Tang, and Zhiping Cai. Prevalence overshadows concerns? understanding chinese users’ privacy awareness and expectations towards llm-based healthcare consultation. In 2025 IEEE Symposium on Security and Privacy (SP), pages 2716–2734. IEEE, 2025. [25] Lisa Mekioussa Malki, Akhil Polamarasetty, Majid Hatamian, Mark Warner, and Enrico Costanza. Hoovered up as a data point: Exploring privacy behaviours, awareness, and concerns among uk users of llm-based conversational agents. Proceedings on Privacy Enhancing Technologies, 2025. [26] Kirsten Martin and Helen Nissenbaum. Measuring privacy: An empirical test using context to expose confounding variables. Columbia Science & Technology Law Review, 18:176, 2016. [27] Niloofar Mireshghallah, Hyunwoo Kim, Xuhui Zhou, Yulia Tsvetkov, Maarten Sap, Reza Shokri, and Yejin Choi. Can llms keep a secret? testing privacy implications of language models via contextual integrity theory. In The Twelfth International Conference on Learning Representations, 2024. [28] Helen Nissenbaum. Privacy as contextual integrity. Washington Law Review, 79:119, 2004. [29] Patricia A Norberg, Daniel R Horne, and David A Horne. The privacy paradox: Personal information disclosure intentions versus behaviors. Journal of consumer affairs, 41(1):100–126, 2007. [30] NVIDIA. Nvidia nemotron 3: Efficient and open intelligence, 2025. URL https://arxiv. org/abs/2512.20856. White Paper. [31] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. [32] Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109, 2024. [33] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5. [34] Ronald W Rogers. A protection motivation theory of fear appeals and attitude change1. The journal of psychology, 91(1):93–114, 1975. 11
[35] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. [36] Yijia Shao, Tianshi Li, Weiyan Shi, Yanchen Liu, and Diyi Yang. Privacylens: Evaluating privacy norm awareness of language models in action. Advances in Neural Information Processing Systems, 37:89373–89407, 2024. [37] Sarah Tran, Hongfan Lu, Isaac Slaughter, Bernease Herman, Aayushi Dangol, Yue Fu, Lufei Chen, Biniyam Gebreyohannes, Bill Howe, Alexis Hiniker, et al. Understanding privacy norms around llm-based chatbots: A contextual integrity perspective. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 2522–2534, 2025. [38] Sarah Tran, Robert Wolfe, and Nicholas Weber. Replication Data for: Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective, 2025. URL https: //doi.org/10.7910/DVN/M6ABJ3. [39] Pranav Narayanan Venkit, Yu Li, Yada Pruksachatkun, and Chien-Sheng Wu. The need for a socially-grounded persona framework for user simulation. arXiv preprint arXiv:2601.07110, 2026. [40] Yang Wang, Gregory Norcie, Saranga Komanduri, Alessandro Acquisti, Pedro Giovanni Leon, and Lorrie Faith Cranor. “i regretted the minute i pressed share” a qualitative study of regrets on facebook. In Proceedings of the seventh symposium on usable privacy and security, pages 1–16, 2011. [41] Primal Wijesekera, Arjun Baokar, Lynn Tsai, Joel Reardon, Serge Egelman, David Wagner, and Konstantin Beznosov. The feasibility of dynamically granted permissions: Aligning mobile privacy with user preferences. In 2017 IEEE Symposium on Security and Privacy (SP), pages 1077–1093. IEEE, 2017. [42] Yuhao Wu, Ke Yang, Franziska Roesner, Tadayoshi Kohno, Ning Zhang, and Umar Iqbal. Towards automating data access permissions in ai agents. arXiv preprint arXiv:2511.17959, 2025. [43] Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. “it’s a fair game”, or is it? examining how users navigate disclosure risks and benefits when using llm-based conversational agents. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–26, 2024. [44] Noé Zufferey, Sarah Abdelwahab Gaballah, Karola Marky, and Verena Zimmermann. “ai is from the devil.” behaviors and concerns toward personal data sharing with llm-based conversational agents. Proceedings on Privacy Enhancing Technologies, 2025(3):5–28, 2025.
12
Appendix Contents A Persona Prompts
14
A.1 LLM Healthcare Consultation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
A.2 AI Agent Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
A.3 LLM Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
A.4 Chatbot Group Chat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
A.5 LLM Conversational Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
A.6 Theory-framed Prompt Variants for LLM Conversational Agents . . . . . . . . . .
21
A.6.1 Privacy Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
A.6.2 Bounded Rationality . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
A.6.3 Protection Motivation Theory . . . . . . . . . . . . . . . . . . . . . . . .
24
B Example No-Persona Reasoning Trace
26
C More Details About User Studies
27
C.1 LLM Healthcare Consultation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
C.2 AI Agent Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
C.3 LLM Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
C.4 Chatbot Group Chat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
C.5 LLM Conversational Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
D Additional Results
30
E Additional Details on Privacy Persona Cluster
31
13
A
Persona Prompts
Every domain in P RIVACY S IM uses a single prompt template with five labeled sections: a fixed ROLE that frames the LLM as a participant simulator, a USER PRIVACY PERSONA that injects the available facets (Du , Eu , Au ) verbatim from the source study, a TASK & CONSTRAINTS block that enforces JSON-only output, a TARGET OUTPUT FORMAT schema sized to the questionnaire, and the original QUESTIONNAIRE text reproduced from the source paper. Only the persona facets, the response scale, and the questionnaire body change across domains; the surrounding scaffolding is identical so that any cross-domain difference in simulation accuracy is attributable to the persona and questionnaire rather than to prompt engineering. Throughout this appendix, persona facets that the source study did not collect are rendered as Not provided (e.g., demographics in Liu et al. [24] and Tran et al. [37]). The five subsections below show one fully populated prompt for each domain, drawn from a real participant in the corresponding study. A.1
LLM Healthcare Consultation
The fully populated prompt below corresponds to the LLM Healthcare Consultation domain [24]. Domain: LLM healthcare consultation | Persona facets: Eu , Au | Scale: 5-point Likert ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: LLM healthcare consultation - Demographics: Not provided. - Previous AI Experiences: {’Which of the following best describes who you are when using the ,→ healthcare chatbot?’: ’Users consulting on general health knowledge’, ’Did you provide any personal ,→ information, such as age, in the consultation experience? Why?’: ’No [Concerned about privacy; ,→ prefer not to disclose]’, ’Do you have any healthcare consultation experience with any AI-large,→ scale-model except in our survey?’: ’No’, ’Do you have any experience with any AI-large-scale-model ,→ except in our survey?’: ’Yes’, ’Which AI-large-scale-model platforms have you used?’: "ChatGPT | ,→ Bing Chat (Copilot) | Baidu’s Ernie Bot | iFlytek’s Spark | Alibaba’s Qianwen", ’Have you ever ,→ heard of AI-large-scale-model?’: ’(Skipped)’, ’Do you have any healthcare consultation experience ,→ online?’: ’No’, ’Why do you choose to use LLM-based healthcare consultation services?’: "Save time ,→ with quick responses | Reduce healthcare costs | Protecting privacy (don’t have to tell anyone) | ,→ Gain medical knowledge or references", "Why don’t you want to use LLM-based healthcare consultation ,→ services?": ’(Skipped)’, ’Have you experienced a health privacy breach?’: ’No’} - Privacy Attitudes: {’Is online healthcare consultation more likely to compromise privacy than asking ,→ a doctor face-to-face?’: ’Somewhat disagree’, ’You will use reliable LLM-based healthcare ,→ consultation services in the future’: ’Somewhat agree’, ’You understand health privacy’: ’Neutral’, ,→ ’You are concerned about health privacy’: ’Neutral’} TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>", "Q8": "<answer>", "Q9": "<answer>", "Q10": "<answer>" } QUESTIONNAIRE
14
Imagine you use the AI-large-scale-model for healthcare consultation, and the conversation mentions ,→ your mental health information (e.g, stress levels, emotional fluctuations, psychotherapy, ,→ experiences). How acceptable is it for the following recipients to access this information? Your options are ’Completely unacceptable’, ’Somewhat unacceptable’, ’Neutral’, ’Somewhat acceptable’, ,→ and ’Completely acceptable’. Q1. Your family members. Q2. Healthcare providers. Q3. Government agencies. Q4. LLM service providers. How acceptable is it for Government agencies to access your mental health information under the ,→ following conditions? Your options are ’Completely unacceptable’, ’Somewhat unacceptable’, ’Neutral’, ’Somewhat acceptable’, ,→ and ’Completely acceptable’. Q5. If you have given consent, Q6. If you are notified before collection, Q7. If information is kept confidential and not stored, Q8. If information will be stored, Q9. If information is used for health monitoring, Q10. If information is shared with others,
A.2
AI Agent Permissions
The fully populated prompt below corresponds to the AI Agent Permissions domain [42]. Domain: AI agent permissions | Persona facets: Du , Eu , Au | Scale: 4-point preference ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: AI agent permissions - Demographics: {’education’: "Master’s", ’age’: ’18-24’, ’gender’: ’Female’} - Previous AI Experiences: {’On a scale from 1 (Not at all familiar) to 5 (Very familiar), how familiar ,→ are you with existing AI tools, such as ChatGPT and Gemini, in terms of your knowledge and usage? ,→ [1. Not at all familiar, 2. Slightly familiar, 3. Somewhat familiar, 4. Moderately familiar, 5. ,→ Very familiar]’: ’3’, ’How often do you use these tools? [1. Never, 2. Rarely (e.g., once a month ,→ or less), 3. Occasionally (e.g., 1-2 times a week), 4. Frequently (e.g., several times a week), 5. ,→ Daily]’: ’2’, ’For what purposes do you use these tools?’: ’Mostly during school for help with work ,→ /generating ideas’} - Privacy Attitudes: {’On a scale from 1 (Not at all) to 5 (Full trust), how much do you trust these ,→ tools to not violate your privacy?’: ’3’, ’On a scale from 1 (Not important) to 5 (Extremely ,→ Important), how important is privacy to you when using digital tools or services, including AI ,→ tools?’: ’4’, ’If you have any privacy concerns while using these tools, what are they?’: ’The ,→ database being able to access personal data.’, ’Select 3 domains you care the most about your ,→ privacy when using AI tools:’: [’Social’, ’Finance’, ’Smart Home’]} TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>", "Q8": "<answer>", "Q9": "<answer>", "Q10": "<answer>" } QUESTIONNAIRE
15
In year 2049, AI personal assistants are an essential part of people’s daily lives. These AI ,→ assistants handle everything - from helping with work, to managing finances, to organizing travel, ,→ and meal planning. -> "Create a training plan for my upcoming marathon" - The AI assistant might consider user’s health metrics and calendar to create a training plan -> "Plan a stopover in Spain for a two-day vacation with my family on my way back to travel from Japan ,→ " - The AI assistant might consider user’s calendar, their interests, and family info to suggest ,→ a plan -> "Can you compose a bedtime story that my children will like?" - The AI assistant might consider children’s interests to create a story -> "Recommend a TV show or movie I’ll love" - The AI assistant might consider user’s viewing history and their mood for recommendations Now imagine that you’re living in year 2049! Your job is to train a personal assistant named "MIRA" for ,→ yourself. You will be presented several scenarios to personalize Mira. Training MIRA to learn your data sharing preferences Mira learns your data sharing preferences and automatically manages permissions on your behalf, so you ,→ never have to decide each one individually! Please go through the following scenarios and express ,→ your preferences! Let’s begin with the queries:
USER: Plan my weekend in San Francisco. Please include surfing in the plan. MIRA: Absolutely USER! I need the data listed below to address your query. Please express your data sharing and accessing preferences. The permission preferences options are " ,→ Yes, always share", "Yes, but ask me next time", "No, but ask me next time", and "No, never share". ,→ Be careful, Mira or apps may collect incorrect or unnecessary data! Q1. Data: Trip dates; App/MIRA: Weather Q2. Data: Trip location; App/MIRA: Weather Q3. Data: Trip dates; App/MIRA: Local Search Q4. Data: Trip location; App/MIRA: Local Search Q5. Data: Trip activities; App/MIRA: Local Search Q6. Data: Email content; App/MIRA: Local Search USER: Schedule a one-time payment of $1,200 to my landlord next Monday. Please label it "Rent Payment" ,→ and include my address for reference. MIRA: Absolutely USER! I need the data listed below to address your query. Please express your data sharing and accessing preferences. The permission preferences options are " ,→ Yes, always share", "Yes, but ask me next time", "No, but ask me next time", and "No, never share". ,→ Be careful, Mira or apps may collect incorrect or unnecessary data! Q7. Data: Payment note; App/MIRA: Banking Q8. Data: Account number; App/MIRA: Banking Q9. Data: Account email; App/MIRA: Banking Q10. Data: Account password; App/MIRA: Banking
A.3
LLM Chatbot
The fully populated prompt below corresponds to the LLM Chatbot domain [37]. Domain: LLM chatbot | Persona facets: Eu , Au | Scale: 0-100 numeric ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: LLM chatbot - Demographics: Not provided. - Previous AI Experiences: {’How frequently do you use ChatGPT?’: ’Daily’, ’What is the date of your ,→ first use of ChatGPT?’: ’2023 or earlier’, ’What is your ChatGPT subscription type?’: ’ChatGPT Plus ,→ ’, ’Tasks ChatGPT is Used For’: ’Daily Life Skills,Personal Education (e.g., learning new concepts), ,→ Searching for Information,Daily Life Skills (e.g., cooking and managing finances),Asking for Advice, ,→ Other’, ’If you selected "Other" for tasks, please elaborate.’: "Oddly, everything but writing ,→ which is what everyone else seems to use it for. I spend many hours asking about an attenborough ,→ doc Im watching etc. It’s too helpful now so I pay for plus yea", ’Topics Discussed with ChatGPT’: ’ ,→ Health or Wellness Advice,Personal Finances,Mental Health Support,Travel, Hobbies,Entertainment ( ,→ movies, videogames, etc.),Work/Job support,Scientific or academic information,Other ’, ’If you ,→ selected "Other" for topics, please elaborate.’: ’Chat GPT even helped me design and build a ,→ soundproof enclosure for my 3D printer and! a sound proof ventilation system for it which I was ,→ really happy with. I use it for tons now.’}
16
- Privacy Attitudes: {’Online companies collect too much personal information. Please rate your ,→ agreement on a scale of 0 to 100 (0 = Strongly Disagree, 50 = Neutral, 100 = Strongly Agree).’: 82, ,→ ’In general, I trust websites. Please rate your agreement on a scale of 0 to 100 (0 = Strongly ,→ Disagree, 50 = Neutral, 100 = Strongly Agree).’: 0, ’In general, I believe privacy is important. ,→ Please rate your agreement on a scale of 0 to 100 (0 = Strongly Disagree, 50 = Neutral, 100 = ,→ Strongly Agree).’: 100, ’I have privacy concerns about conversations with ChatGPT. Please rate ,→ your agreement on a scale of 0 to 100 (0 = Strongly Disagree, 50 = Neutral, 100 = Strongly Agree).’: ,→ 19, ’In general, I believe that my conversations with chatbots like ChatGPT will remain private. ,→ Please rate your agreement on a scale of 0 to 100 (0 = Strongly Disagree, 50 = Neutral, 100 = ,→ Strongly Agree).’: 40} TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<integer>", "Q2": "<integer>", "Q3": "<integer>", "Q4": "<integer>", "Q5": "<integer>", "Q6": "<integer>", "Q7": "<integer>", "Q8": "<integer>", "Q9": "<integer>", "Q10": "<integer>" } QUESTIONNAIRE You will be shown a series of scenarios describing the use of your ChatGPT history. For each scenario, ,→ please indicate your agreement with the following statement: ’This is an appropriate use of my ,→ ChatGPT data.’ Please rate your agreement on a scale of 0 to 100 (0 = Strongly Disagree, 50 = ,→ Neutral, 100 = Strongly Agree). Q1. Your ChatGPT conversations were shared with a(n) a university computer science researcher based in ,→ China for the purpose of studying AI risks to society. Your conversations with ChatGPT about your ,→ personal health and wellness were shared. You were not informed that your data was collected. Your ,→ conversations were not anonymized before being shared. Personally identifying information from your ,→ conversations was not removed. Q2. Your ChatGPT conversations were shared with a(n) a university social science researcher based in ,→ the US for the purpose of creating a public dataset for AI research. Your conversations with ,→ ChatGPT about social life and personal relationships were shared. You were not informed that your ,→ data was collected. Your conversations were not anonymized before being shared. Personally ,→ identifying information from your conversations was not removed. Q3. Your ChatGPT conversations were shared with a(n) a charitable foundation based in the US for the ,→ purpose of creating a public dataset for AI research. All of your conversations with ChatGPT were ,→ shared. You were asked for consent in advance of the data being collected. Your conversations were ,→ anonymized before being shared. Personally identifying information from your conversations was ,→ automatically removed. Q4. Your ChatGPT conversations were shared with a(n) a hospital based in the US for the purpose of ,→ personalizing advertising. Your conversations with ChatGPT where you used the model to help with ,→ your job were shared. You were not informed that your data was collected. Your conversations were ,→ not anonymized before being shared. Personally identifying information from your conversations was ,→ not removed. Q5. Your ChatGPT conversations were shared with a(n) a university computer science researcher based in ,→ the European Union for the purpose of fighting terrorism. Your conversations with ChatGPT where ,→ you used the model to help with your job were shared. You were asked for consent in advance of the ,→ data being collected. Your conversations were not anonymized before being shared. Personally ,→ identifying information from your conversations was not removed. Q6. Your ChatGPT conversations were shared with a(n) a university social science researcher based in ,→ the European Union for the purpose of training future AI models. Your conversations with ChatGPT ,→ where you sought legal or ethical guidance were shared. You were not informed that your data was ,→ collected. Your conversations were anonymized before being shared. Personally identifying ,→ information from your conversations was automatically removed. Q7. Your ChatGPT conversations were shared with a(n) a university computer science researcher based in ,→ the European Union for the purpose of personalizing advertising. All of your conversations with ,→ ChatGPT were shared. You were not informed that your data was collected. Your conversations were ,→ anonymized before being shared. Personally identifying information from your conversations was ,→ automatically removed. Q8. Your ChatGPT conversations were shared with a(n) a government agency based in the European Union ,→ for the purpose of predicting human behavior. Your conversations with ChatGPT about social life and ,→ personal relationships were shared. You were informed that your data was collected. Your ,→ conversations were not anonymized before being shared. Personally identifying information from your ,→ conversations was not removed.
17
Q9. Your ChatGPT conversations were shared with a(n) a big tech company based in the US for the ,→ purpose of studying AI risks to society. Your conversations with ChatGPT where you sought legal or ,→ ethical guidance were shared. You were not informed that your data was collected. Your ,→ conversations were anonymized before being shared. Personally identifying information from your ,→ conversations was automatically removed. Q10. Your ChatGPT conversations were shared with a(n) a university social science researcher based in ,→ the European Union for the purpose of improving user experience with AI. Your conversations with ,→ ChatGPT where you used the model to help with your job were shared. You were informed that your ,→ data was collected. Your conversations were anonymized before being shared. Personally identifying ,→ information from your conversations was automatically removed.
A.4
Chatbot Group Chat
The fully populated prompt below corresponds to the Chatbot Group Chat domain [10]. Domain: Chatbot group chat | Persona facets: Du , Eu , Au | Scale: 5-point Likert ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: Chatbot group chat - Demographics: {"Age Group": "25 - 34", "Gender - Selected Choice": "Male", "Gender - Prefer to self,→ describe - Text": null, "Highest level of education completed": "Master’s degree", "Technical work ,→ experience": "Yes", "Race and ethnicity - Selected Choice": "White or Caucasian", "Race and ,→ ethnicity - Prefer to self-describe - Text": null} - Previous AI Experiences: {"Please select one instant messaging service from the following list that ,→ you use frequently (more than twice a week) and have seen chatbots on. If more than one applies, ,→ choose the one you use the most. The rest of the survey will be based on your selection here.": " ,→ Discord", "How often do you use the instant messaging service you selected in the previous question ,→ ?": "Daily", "How often do you notice chatbots in group chats, including seeing them send messages ,→ or actively searching for their existence?": "Daily", "Please name 1-2 chatbots you encountered in ,→ group chats and describe them in 1-2 sentences.": "Cleverbot, a chatbot uses the inputs from users ,→ to learn and generate responses.\nXavier., a chatbot that can also interact with audios, images etc. ,→ It can also do internet searches for you.", "How often do you interact with chatbots in group ,→ chats, including actively tagging/mentioning chatbots in your messages, chatbots replying to your ,→ messages, or chatbots reading your messages?": "4-6 times per week", "What types of group chats ,→ have you seen chatbots in? Select all that apply. - Selected Choice": "Educational groups (classes, ,→ study groups),Friend or family groups,Public forums or community groups", "What types of group ,→ chats have you seen chatbots in? Select all that apply. - Other (please specify) - Text": null, " ,→ What purposes do you use chatbots for in group chats? Select all that apply.": "Productivity, ,→ Education,Entertainment & Games,Graphics & Design,Music,Social Networking", "Have you ever noticed ,→ any chatbots in private group chats you have participated in?": "Yes", "How often do you check for ,→ the presence of chatbots whenever you join private group chats?": "Rarely", "Regarding private ,→ group chats - Please elaborate on your answer to the previous question (how often you check for the ,→ presence of chatbots when joining private groups).": "I don’t really go looking for the presence ,→ of chatbots, it’s more something that happens by chance in the discord servers I join. I use them ,→ in the servers I’m already in.", "When do you check for the presence of chatbots in a private group ,→ ? Select all that apply.": null, "Have you ever noticed any chatbots in public group chats you have ,→ participated in?": "Yes", "How often do you check for the presence of chatbots whenever you join ,→ public group chats?": "Rarely", "Regarding public group chats - Please elaborate on your answer to ,→ the previous question (how often you check for the presence of chatbots when joining public groups) ,→ .": "I am not particularly interested in using chatbots in public chats that everyone can read, I ,→ rarely look for them there. I prefer the privacy of private bots, especially if the features are ,→ the same.", "When do you check for the presence of chatbots in a public group? Select all that ,→ apply.": null} - Privacy Attitudes: {"To what extent are you familiar with what information chatbots on Discord can ,→ access in group chats?": "Very knowledgeable", "What type of messages do you believe a chatbot on ,→ Discord has access to?": "All messages sent in the chat, including those that are later deleted.", ,→ "Following up on the previous question, what information do you believe is included with the ,→ message a chatbot has access to on Discord? Select all that apply.": "The content of the message ,→ itself.,The message metadata, such as time sent.,The sender’s username. A user has the same ,→ username in different groups.,The sender’s pseudonym. A user has different pseudonyms in different ,→ groups.,The sender’s metadata (other than the username), such as profile picture, gender, and email ,→ addresses.", "What type of information do you believe a chatbot on Discord can access about the ,→ group? Select all that apply.": "The name of the group.,The list of group members’ usernames.,The ,→ profile pictures of group members.,The roles or statuses of group members (e.g., moderator)."} TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona.
18
2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>" } QUESTIONNAIRE Please rate your level of agreement with the following statements about chatbots in group chats. Your options are ’Strongly disagree’, ’Somewhat disagree’, ’Neither agree nor disagree’, ’Somewhat ,→ agree’, and ’Strongly agree’. Q1. Chatbots from group chats with access to users’ data as listed above would raise my privacy ,→ concerns. Q2. I would be concerned that any personal information I share in the group chat could be collected by ,→ the chatbot and be used inappropriately. Q3. I would stop sharing some sensitive information or talking about controversial topics in group ,→ chat because of privacy concerns about chatbots. Q4. I would still use the chatbot after knowing what information it had access to. Q5. I think chatbots accessing more information than necessary could cause privacy issues in public ,→ group chats. Q6. I think chatbots accessing more information than necessary could cause privacy issues in private ,→ group chats. Q7. I understand how chatbot providers protect the data collected from group chats.
A.5
LLM Conversational Agents
The fully populated prompt below corresponds to the LLM Conversational Agents domain [44]. Domain: LLM conversational agents | Persona facets: Du , Eu , Au | Scale: 5-point frequency ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: LLM conversational agents - Demographics: {’age’: ’18 - 29’, ’country/region’: ’Europe’} - Previous AI Experiences: {’Are you using your own local chatbot instance based on only locally ,→ stored LLM model(s) (e.g., LLaMa, Mistral) and data?\n\n\n\nThis is about setting up your own ,→ chatbot. Apps such as the ChatGPT smartphone app are not included in this category.’: ’Yes, but not ,→ exclusively (I also use chatbot services, as, for example, ChatGPT)’, ’Among the following, for ,→ what purpose do you use your own local chatbot instance? - Selected Choice’: ’General information ,→ seeking and learning,Entertainment (e.g., casual chats, text-based games)’, ’Among the following, ,→ for what purpose do you use your own local chatbot instance? - Other (please specifiy) - Text’: nan, ,→ ’Why do you have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, ,→ Copilot)? - Selected Choice’: ’Performance (e.g., computation time, fine-tuning)’, ’Why do you ,→ have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, Copilot)? ,→ Other (please specifiy) - Text’: nan, ’Please select the chatbots you usually use. - Selected ,→ Choice’: ’ChatGPT’, ’Please select the chatbots you usually use. - Other (please specify, use ,→ commas if multiple answers) - Text’: nan, ’How frequently do you use chatbots?’: ’A few times a ,→ week’, ’On the days you are interacting with chatbots, how long does this interactions last on ,→ average (per day)?’: ’between 1 and 2 hours’, ’Among the following devices, on which one do you ,→ usually interact with chatbots? Several selections are possible. - Selected Choice’: ’Laptop / ,→ Personal computer,Tablet,Smartphone’, ’Among the following devices, on which one do you usually ,→ interact with chatbots? Several selections are possible. - Other (please specify) - Text’: nan, ’ ,→ Among the following, for what purpose do you usually use chatbots? Several selections are possible. ,→ - Selected Choice’: ’General information seeking and learning,Programming support’, ’Among the ,→ following, for what purpose do you usually use chatbots? Several selections are possible. - Other ,→ Text’: nan} - Privacy Attitudes: {’Please indicate to what extent you agree with each of the following statements ,→ [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, ,→ Strongly Agree]. - It usually bothers me when online companies ask me for personal information.’: ’
19
,→ Agree’, ’Please indicate to what extent you agree with each of the following statements [Strongly ,→ agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly ,→ Agree]. - When online companies ask me for personal information, I sometimes think twice before ,→ providing it.’: ’Moderately agree’, ’Please indicate to what extent you agree with each of the ,→ following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - It bothers me to give personal information to so many ,→ online companies.’: ’Agree’, ’Please indicate to what extent you agree with each of the following ,→ statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately ,→ agree, Agree, Strongly Agree]. - I am concerned that online companies are collecting too much ,→ personal information about me.’: ’Agree’, ’Please indicate to what extent you agree with each of ,→ the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - Consumer online privacy is really a matter of ,→ consumers’ right to exercise control and autonomy over decisions about how their information is ,→ collected, used, and shared.’: ’Moderately agree’, ’Please indicate to what extent you agree with ,→ each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor ,→ disagree, Moderately agree, Agree, Strongly Agree]. - Consumer control of personal information lies ,→ at the heart of consumer privacy.’: ’Moderately agree’, ’Please indicate to what extent you agree ,→ with each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree ,→ nor disagree, Moderately agree, Agree, Strongly Agree]. - Companies seeking information online ,→ should disclose the way the data are collected, processed, and used.’: ’Agree’, ’Please indicate to ,→ what extent you agree with each of the following statements [Strongly agree, Disgree, Moderately ,→ disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly Agree]. - A good consumer ,→ online privacy policy should have a clear and conspicuous disclosure.’: ’Agree’} TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>", "Q8": "<answer>", "Q9": "<answer>", "Q10": "<answer>" } QUESTIONNAIRE In the following part of the survey, we will ask you question about how frequently you share different ,→ type of personal data with chatbots. To evaluate this frequency, please think about the number of ,→ times where it would have been relevant to share a given type of data. For example, you should ,→ select ’I usually share’ if you usually share a given type of data when it is relevant for your ,→ chatbot usage at the moment. If you never shared a given type of data in any context, please just ,→ answer ’I never shared’ For all the following types of data related to identifiers and general information, please indicate how ,→ frequently you share such information with a chatbot. Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q1. Personal identifiers (e.g., name, ID card number, email address, phone number) Q2. Account credentials (i.e., usernames, passwords) Q3. Location and mobility (e.g., homeplace, workplace, current location) Q4. Ethnicity and/or citizenship Q5. Criminal records For all the following types of data about standard of living and opinions, please indicate how ,→ frequently you share such information with a chatbot? Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q6. Family life and relationship (e.g., civil status, number of children) Q7. Wealth details (e.g., salary, savings amount, land ownership) Q8. Job / Education Q9. Religion Q10. Political views
20
A.6
Theory-framed Prompt Variants for LLM Conversational Agents
To accompany the theory-framing ablation in Section 4.4 (Table 4), we show the three theory-framed variants of the persona prompt for the same participant from Zufferey et al. [44] as in Section A above, but at prompt type Eu + Au (so demographics are rendered as Not provided.). Each variant differs from the baseline only by an additional PRIVACY DECISION LENS block inserted between the persona section and the task instructions; the role, persona content, output format, and questionnaire are identical across the three variants and the baseline. A.6.1
Privacy Calculus
Theory: Privacy Calculus | Domain: LLM conversational agents | Persona facets: Eu , Au | Scale: 5-point frequency ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: LLM conversational agents - Demographics: Not provided. - Previous AI Experiences: {’Are you using your own local chatbot instance based on only locally ,→ stored LLM model(s) (e.g., LLaMa, Mistral) and data?\n\n\n\nThis is about setting up your own ,→ chatbot. Apps such as the ChatGPT smartphone app are not included in this category.’: ’Yes, but not ,→ exclusively (I also use chatbot services, as, for example, ChatGPT)’, ’Among the following, for ,→ what purpose do you use your own local chatbot instance? - Selected Choice’: ’General information ,→ seeking and learning,Entertainment (e.g., casual chats, text-based games)’, ’Among the following, ,→ for what purpose do you use your own local chatbot instance? - Other (please specifiy) - Text’: nan, ,→ ’Why do you have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, ,→ Copilot)? - Selected Choice’: ’Performance (e.g., computation time, fine-tuning)’, ’Why do you ,→ have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, Copilot)? ,→ Other (please specifiy) - Text’: nan, ’Please select the chatbots you usually use. - Selected ,→ Choice’: ’ChatGPT’, ’Please select the chatbots you usually use. - Other (please specify, use ,→ commas if multiple answers) - Text’: nan, ’How frequently do you use chatbots?’: ’A few times a ,→ week’, ’On the days you are interacting with chatbots, how long does this interactions last on ,→ average (per day)?’: ’between 1 and 2 hours’, ’Among the following devices, on which one do you ,→ usually interact with chatbots? Several selections are possible. - Selected Choice’: ’Laptop / ,→ Personal computer,Tablet,Smartphone’, ’Among the following devices, on which one do you usually ,→ interact with chatbots? Several selections are possible. - Other (please specify) - Text’: nan, ’ ,→ Among the following, for what purpose do you usually use chatbots? Several selections are possible. ,→ - Selected Choice’: ’General information seeking and learning,Programming support’, ’Among the ,→ following, for what purpose do you usually use chatbots? Several selections are possible. - Other ,→ Text’: nan} - Privacy Attitudes: {’Please indicate to what extent you agree with each of the following statements ,→ [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, ,→ Strongly Agree]. - It usually bothers me when online companies ask me for personal information.’: ’ ,→ Agree’, ’Please indicate to what extent you agree with each of the following statements [Strongly ,→ agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly ,→ Agree]. - When online companies ask me for personal information, I sometimes think twice before ,→ providing it.’: ’Moderately agree’, ’Please indicate to what extent you agree with each of the ,→ following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - It bothers me to give personal information to so many ,→ online companies.’: ’Agree’, ’Please indicate to what extent you agree with each of the following ,→ statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately ,→ agree, Agree, Strongly Agree]. - I am concerned that online companies are collecting too much ,→ personal information about me.’: ’Agree’, ’Please indicate to what extent you agree with each of ,→ the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - Consumer online privacy is really a matter of ,→ consumers’ right to exercise control and autonomy over decisions about how their information is ,→ collected, used, and shared.’: ’Moderately agree’, ’Please indicate to what extent you agree with ,→ each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor ,→ disagree, Moderately agree, Agree, Strongly Agree]. - Consumer control of personal information lies ,→ at the heart of consumer privacy.’: ’Moderately agree’, ’Please indicate to what extent you agree ,→ with each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree ,→ nor disagree, Moderately agree, Agree, Strongly Agree]. - Companies seeking information online ,→ should disclose the way the data are collected, processed, and used.’: ’Agree’, ’Please indicate to ,→ what extent you agree with each of the following statements [Strongly agree, Disgree, Moderately ,→ disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly Agree]. - A good consumer ,→ online privacy policy should have a clear and conspicuous disclosure.’: ’Agree’} PRIVACY DECISION LENS -- PRIVACY CALCULUS
21
Evaluate each scenario as this person performing a deliberate cost-benefit calculation about ,→ disclosure. For each request: 1. Identify the perceived BENEFITS of sharing -- service utility, personalization quality, convenience, ,→ time saved, social or relational value. 2. Identify the perceived RISKS or COSTS of sharing -- loss of control over the data, downstream reuse ,→ or profiling, embarrassment, identifiability, future harm, regulatory or employer exposure. 3. Weigh the two through the lens of this person’s stated attitudes and prior experiences; their answer ,→ should reflect whichever side they judge to dominate. 4. When benefits clearly outweigh perceived risks for them, lean toward acceptance; when risks ,→ dominate, lean toward refusal; when the two are comparable, lean toward a conditional or ,→ intermediate option. Pick the answer this person would arrive at after this tradeoff, not the answer a privacy-maximizing or ,→ utility-maximizing agent would give. TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>", "Q8": "<answer>", "Q9": "<answer>", "Q10": "<answer>" } QUESTIONNAIRE In the following part of the survey, we will ask you question about how frequently you share different ,→ type of personal data with chatbots. To evaluate this frequency, please think about the number of ,→ times where it would have been relevant to share a given type of data. For example, you should ,→ select ’I usually share’ if you usually share a given type of data when it is relevant for your ,→ chatbot usage at the moment. If you never shared a given type of data in any context, please just ,→ answer ’I never shared’ For all the following types of data related to identifiers and general information, please indicate how ,→ frequently you share such information with a chatbot. Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q1. Personal identifiers (e.g., name, ID card number, email address, phone number) Q2. Account credentials (i.e., usernames, passwords) Q3. Location and mobility (e.g., homeplace, workplace, current location) Q4. Ethnicity and/or citizenship Q5. Criminal records For all the following types of data about standard of living and opinions, please indicate how ,→ frequently you share such information with a chatbot? Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q6. Family life and relationship (e.g., civil status, number of children) Q7. Wealth details (e.g., salary, savings amount, land ownership) Q8. Job / Education Q9. Religion Q10. Political views
A.6.2
Bounded Rationality
Theory: Bounded Rationality | Domain: LLM conversational agents | Persona facets: Eu , Au | Scale: 5-point frequency ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would.
22
USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: LLM conversational agents - Demographics: Not provided. - Previous AI Experiences: {’Are you using your own local chatbot instance based on only locally ,→ stored LLM model(s) (e.g., LLaMa, Mistral) and data?\n\n\n\nThis is about setting up your own ,→ chatbot. Apps such as the ChatGPT smartphone app are not included in this category.’: ’Yes, but not ,→ exclusively (I also use chatbot services, as, for example, ChatGPT)’, ’Among the following, for ,→ what purpose do you use your own local chatbot instance? - Selected Choice’: ’General information ,→ seeking and learning,Entertainment (e.g., casual chats, text-based games)’, ’Among the following, ,→ for what purpose do you use your own local chatbot instance? - Other (please specifiy) - Text’: nan, ,→ ’Why do you have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, ,→ Copilot)? - Selected Choice’: ’Performance (e.g., computation time, fine-tuning)’, ’Why do you ,→ have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, Copilot)? ,→ Other (please specifiy) - Text’: nan, ’Please select the chatbots you usually use. - Selected ,→ Choice’: ’ChatGPT’, ’Please select the chatbots you usually use. - Other (please specify, use ,→ commas if multiple answers) - Text’: nan, ’How frequently do you use chatbots?’: ’A few times a ,→ week’, ’On the days you are interacting with chatbots, how long does this interactions last on ,→ average (per day)?’: ’between 1 and 2 hours’, ’Among the following devices, on which one do you ,→ usually interact with chatbots? Several selections are possible. - Selected Choice’: ’Laptop / ,→ Personal computer,Tablet,Smartphone’, ’Among the following devices, on which one do you usually ,→ interact with chatbots? Several selections are possible. - Other (please specify) - Text’: nan, ’ ,→ Among the following, for what purpose do you usually use chatbots? Several selections are possible. ,→ - Selected Choice’: ’General information seeking and learning,Programming support’, ’Among the ,→ following, for what purpose do you usually use chatbots? Several selections are possible. - Other ,→ Text’: nan} - Privacy Attitudes: {’Please indicate to what extent you agree with each of the following statements ,→ [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, ,→ Strongly Agree]. - It usually bothers me when online companies ask me for personal information.’: ’ ,→ Agree’, ’Please indicate to what extent you agree with each of the following statements [Strongly ,→ agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly ,→ Agree]. - When online companies ask me for personal information, I sometimes think twice before ,→ providing it.’: ’Moderately agree’, ’Please indicate to what extent you agree with each of the ,→ following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - It bothers me to give personal information to so many ,→ online companies.’: ’Agree’, ’Please indicate to what extent you agree with each of the following ,→ statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately ,→ agree, Agree, Strongly Agree]. - I am concerned that online companies are collecting too much ,→ personal information about me.’: ’Agree’, ’Please indicate to what extent you agree with each of ,→ the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - Consumer online privacy is really a matter of ,→ consumers’ right to exercise control and autonomy over decisions about how their information is ,→ collected, used, and shared.’: ’Moderately agree’, ’Please indicate to what extent you agree with ,→ each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor ,→ disagree, Moderately agree, Agree, Strongly Agree]. - Consumer control of personal information lies ,→ at the heart of consumer privacy.’: ’Moderately agree’, ’Please indicate to what extent you agree ,→ with each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree ,→ nor disagree, Moderately agree, Agree, Strongly Agree]. - Companies seeking information online ,→ should disclose the way the data are collected, processed, and used.’: ’Agree’, ’Please indicate to ,→ what extent you agree with each of the following statements [Strongly agree, Disgree, Moderately ,→ disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly Agree]. - A good consumer ,→ online privacy policy should have a clear and conspicuous disclosure.’: ’Agree’} PRIVACY DECISION LENS -- BOUNDED RATIONALITY Evaluate each scenario as this person under realistic cognitive constraints, not as a perfectly ,→ rational analyst: - People rarely enumerate every risk. They rely on mental shortcuts: "does this look like the kind of ,→ thing I usually allow?", "is this the default?", "the system asked, so it probably needs it", " ,→ everyone seems to use this". - They satisfice -- accept the first option that feels good enough -- rather than optimize. - Stated privacy attitudes and concrete behavior can diverge (the privacy paradox). This person’s ,→ answer in a specific scenario may not perfectly match their abstract stance, particularly when the ,→ request feels routine, low-friction, or socially normal. - If the request feels familiar given their previous AI experiences, the heuristic favors acceptance ,→ even when their attitudes are cautious. If the request feels unfamiliar, intrusive, or salient, the ,→ heuristic flips toward refusal. Pick the option this person -- as a cognitively bounded, heuristic-driven decision-maker -- would most ,→ plausibly land on in the moment, not the option a fully deliberative version of them would compute. ,→ TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT.
23
TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>", "Q8": "<answer>", "Q9": "<answer>", "Q10": "<answer>" } QUESTIONNAIRE In the following part of the survey, we will ask you question about how frequently you share different ,→ type of personal data with chatbots. To evaluate this frequency, please think about the number of ,→ times where it would have been relevant to share a given type of data. For example, you should ,→ select ’I usually share’ if you usually share a given type of data when it is relevant for your ,→ chatbot usage at the moment. If you never shared a given type of data in any context, please just ,→ answer ’I never shared’ For all the following types of data related to identifiers and general information, please indicate how ,→ frequently you share such information with a chatbot. Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q1. Personal identifiers (e.g., name, ID card number, email address, phone number) Q2. Account credentials (i.e., usernames, passwords) Q3. Location and mobility (e.g., homeplace, workplace, current location) Q4. Ethnicity and/or citizenship Q5. Criminal records For all the following types of data about standard of living and opinions, please indicate how ,→ frequently you share such information with a chatbot? Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q6. Family life and relationship (e.g., civil status, number of children) Q7. Wealth details (e.g., salary, savings amount, land ownership) Q8. Job / Education Q9. Religion Q10. Political views
A.6.3
Protection Motivation Theory
Theory: Protection Motivation Theory | Domain: LLM conversational agents | Persona facets: Eu , Au | Scale: 5-point frequency ROLE You are an expert AI simulator acting as a specific human participant in a research study. Your task ,→ is to realistically answer a questionnaire about data-sharing scenarios exactly as this specific ,→ person would. USER PRIVACY PERSONA Internalize the user’s traits and answer each scenario as this specific person would. Draw on whichever ,→ aspects of their profile are most relevant to the particular scenario. - Domain Context: LLM conversational agents - Demographics: Not provided. - Previous AI Experiences: {’Are you using your own local chatbot instance based on only locally ,→ stored LLM model(s) (e.g., LLaMa, Mistral) and data?\n\n\n\nThis is about setting up your own ,→ chatbot. Apps such as the ChatGPT smartphone app are not included in this category.’: ’Yes, but not ,→ exclusively (I also use chatbot services, as, for example, ChatGPT)’, ’Among the following, for ,→ what purpose do you use your own local chatbot instance? - Selected Choice’: ’General information ,→ seeking and learning,Entertainment (e.g., casual chats, text-based games)’, ’Among the following, ,→ for what purpose do you use your own local chatbot instance? - Other (please specifiy) - Text’: nan, ,→ ’Why do you have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, ,→ Copilot)? - Selected Choice’: ’Performance (e.g., computation time, fine-tuning)’, ’Why do you ,→ have your own local chatbot instance and not only use chatbot services (e.g., ChatGPT, Copilot)? ,→ Other (please specifiy) - Text’: nan, ’Please select the chatbots you usually use. - Selected ,→ Choice’: ’ChatGPT’, ’Please select the chatbots you usually use. - Other (please specify, use ,→ commas if multiple answers) - Text’: nan, ’How frequently do you use chatbots?’: ’A few times a ,→ week’, ’On the days you are interacting with chatbots, how long does this interactions last on ,→ average (per day)?’: ’between 1 and 2 hours’, ’Among the following devices, on which one do you ,→ usually interact with chatbots? Several selections are possible. - Selected Choice’: ’Laptop / ,→ Personal computer,Tablet,Smartphone’, ’Among the following devices, on which one do you usually
24
,→ interact with chatbots? Several selections are possible. - Other (please specify) - Text’: nan, ’ ,→ Among the following, for what purpose do you usually use chatbots? Several selections are possible. ,→ - Selected Choice’: ’General information seeking and learning,Programming support’, ’Among the ,→ following, for what purpose do you usually use chatbots? Several selections are possible. - Other ,→ Text’: nan} - Privacy Attitudes: {’Please indicate to what extent you agree with each of the following statements ,→ [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, ,→ Strongly Agree]. - It usually bothers me when online companies ask me for personal information.’: ’ ,→ Agree’, ’Please indicate to what extent you agree with each of the following statements [Strongly ,→ agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly ,→ Agree]. - When online companies ask me for personal information, I sometimes think twice before ,→ providing it.’: ’Moderately agree’, ’Please indicate to what extent you agree with each of the ,→ following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - It bothers me to give personal information to so many ,→ online companies.’: ’Agree’, ’Please indicate to what extent you agree with each of the following ,→ statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, Moderately ,→ agree, Agree, Strongly Agree]. - I am concerned that online companies are collecting too much ,→ personal information about me.’: ’Agree’, ’Please indicate to what extent you agree with each of ,→ the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor disagree, ,→ Moderately agree, Agree, Strongly Agree]. - Consumer online privacy is really a matter of ,→ consumers’ right to exercise control and autonomy over decisions about how their information is ,→ collected, used, and shared.’: ’Moderately agree’, ’Please indicate to what extent you agree with ,→ each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree nor ,→ disagree, Moderately agree, Agree, Strongly Agree]. - Consumer control of personal information lies ,→ at the heart of consumer privacy.’: ’Moderately agree’, ’Please indicate to what extent you agree ,→ with each of the following statements [Strongly agree, Disgree, Moderately disagree, Neither agree ,→ nor disagree, Moderately agree, Agree, Strongly Agree]. - Companies seeking information online ,→ should disclose the way the data are collected, processed, and used.’: ’Agree’, ’Please indicate to ,→ what extent you agree with each of the following statements [Strongly agree, Disgree, Moderately ,→ disagree, Neither agree nor disagree, Moderately agree, Agree, Strongly Agree]. - A good consumer ,→ online privacy policy should have a clear and conspicuous disclosure.’: ’Agree’} PRIVACY DECISION LENS -- PROTECTION MOTIVATION THEORY Evaluate each scenario as this person running two parallel appraisals: THREAT APPRAISAL - Perceived severity: how harmful would exposure of this specific data be for this person, given their ,→ profile (e.g., medical, financial, identity, employment-relevant)? - Perceived vulnerability: how likely does this person believe a harmful outcome -- leak, misuse, ,→ profiling, secondary use -- actually is in this context? COPING APPRAISAL - Response efficacy: does refusing, redacting, or limiting disclosure actually reduce the threat, in ,→ this person’s belief? - Self-efficacy: does this person feel able to refuse or push back without losing the service or the ,→ relationship? - Response cost: what do they give up -- utility, convenience, social standing, continuity of care -,→ by protecting? Combine the appraisals. High threat x high coping efficacy -> protective behavior (refusal or ,→ restriction). Low threat, or high threat but low coping efficacy / high response cost -> likely ,→ acceptance, sometimes reluctantly. Intermediate cases -> conditional acceptance. Pick the answer ,→ this person -- given their stated attitudes, demographics, and previous experiences -- would arrive ,→ at after this combined appraisal. TASK & CONSTRAINTS 1. Answer all data-sharing scenarios in the QUESTIONNAIRE below from the perspective of the User ,→ Persona. 2. Return ONLY valid, parseable JSON. 3. DO NOT wrap your response in markdown code blocks (e.g., do not use ‘‘‘json ... ‘‘‘). 4. DO NOT add any commentary, greetings, or explanations outside the JSON object. 5. Keep field names, nesting, and data types strictly aligned with the TARGET OUTPUT FORMAT. TARGET OUTPUT FORMAT (STRUCTURE ONLY - DO NOT COPY VALUES) { "Q1": "<answer>", "Q2": "<answer>", "Q3": "<answer>", "Q4": "<answer>", "Q5": "<answer>", "Q6": "<answer>", "Q7": "<answer>", "Q8": "<answer>", "Q9": "<answer>", "Q10": "<answer>" } QUESTIONNAIRE
25
In the following part of the survey, we will ask you question about how frequently you share different ,→ type of personal data with chatbots. To evaluate this frequency, please think about the number of ,→ times where it would have been relevant to share a given type of data. For example, you should ,→ select ’I usually share’ if you usually share a given type of data when it is relevant for your ,→ chatbot usage at the moment. If you never shared a given type of data in any context, please just ,→ answer ’I never shared’ For all the following types of data related to identifiers and general information, please indicate how ,→ frequently you share such information with a chatbot. Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q1. Personal identifiers (e.g., name, ID card number, email address, phone number) Q2. Account credentials (i.e., usernames, passwords) Q3. Location and mobility (e.g., homeplace, workplace, current location) Q4. Ethnicity and/or citizenship Q5. Criminal records For all the following types of data about standard of living and opinions, please indicate how ,→ frequently you share such information with a chatbot? Your options are ’I never shared’, ’I rarely shared’, ’I sometimes shared’, ’I often shared’, and ’I ,→ usually shared’. Information about your... Q6. Family life and relationship (e.g., civil status, number of children) Q7. Wealth details (e.g., salary, savings amount, land ownership) Q8. Job / Education Q9. Religion Q10. Political views
B
Example No-Persona Reasoning Trace
We show one reasoning trace (summarized because OpenAI does not provide full traces) from GPT5.4 (medium reasoning effort) on the LLM Healthcare Consultation domain at the no-persona prompt type (none). The trace is taken verbatim from the OpenAI Responses API summary_text blocks. Note how, absent any user-specific signal, the model explicitly chooses to “simulate a generic person with moderate privacy attitudes” and reasons in terms of population-level privacy norms (consent, confidentiality, recipient sensitivity) rather than properties of any individual. Reasoning trace: GPT-5.4 (medium reasoning) | Domain: LLM Healthcare Consultation | Prompt type: none (no persona) **Considering privacy in healthcare** I need to provide an answer as a persona related to healthcare consultation, but I have limited ,→ information. It seems I should simulate a generic person with moderate privacy attitudes, ,→ especially in mental health. I’ll focus on constructing the answer using JSON format and aim for ,→ values that are consistent with typical privacy concerns. It appears that consent and ,→ confidentiality are crucial, but sharing information with family or authorities isn’t acceptable. I ,→ should maintain this perspective throughout. **Creating a cautious privacy stance** I want to craft a balanced and cautious stance regarding privacy in healthcare. For family members, I ,→ find sharing is somewhat unacceptable, while sharing with healthcare providers is completely ,→ acceptable. Government agencies are entirely unacceptable. LLM service providers also fall into the ,→ somewhat unacceptable category. With consent, sharing becomes somewhat acceptable, and being ,→ notified before data collection is neutral. Keeping data confidential yet not stored is somewhat ,→ acceptable, but storing is somewhat unacceptable. Sharing information with others is completely ,→ unacceptable. I need to format this in valid JSON with strings.
Final answer (10-question 5-pt acceptability schema, ru omitted) { "Q1": "Somewhat unacceptable", "Q2": "Completely acceptable", "Q3": "Completely unacceptable", "Q4": "Somewhat unacceptable", "Q5": "Somewhat acceptable", "Q6": "Neutral", "Q7": "Somewhat acceptable",
26
"Q8": "Somewhat unacceptable", "Q9": "Somewhat acceptable", "Q10": "Completely unacceptable" }
C
More Details About User Studies
This appendix gives a per-study description of the five user studies we draw from, summarizing the data-sharing context, the participant pool, what was collected for each persona facet, and the original key findings. C.1
LLM Healthcare Consultation
Context. Liu et al. [24] elicit privacy norms for sharing mental-health information mentioned in an LLM-based healthcare consultation. Each questionnaire asks how acceptable it is for four recipients (family members, healthcare providers, government agencies, LLM service providers) to access this information, and then probes acceptability of access by one specific recipient under six contextual conditions (consent, notification, confidentiality, storage, health-monitoring use, onward sharing) on a five-point Likert scale. Participants. 846 adult users from China recruited online. However, we restrict our analysis to participants who answered questions regarding mental health information or medical information because of greater variance in the responses. Hence, there are 417 participants. Demographics were not released in the public dataset, so Du = ∅ for this domain. Persona facets. Eu captures the participant’s role when consulting (general health vs. personal issue vs. medical professional), prior LLM and online-consultation experience, motivations for/against using LLM healthcare services, and whether they have previously experienced a health-data breach. Au records four agreement items on health-privacy understanding, concern, and willingness to use LLM consultation. Key findings. Despite high stated privacy concern, the prevalence of LLM use overshadows those concerns: many participants disclose personal information to the chatbot anyway, motivated by convenience and a (sometimes mistaken) belief that the LLM will keep information confidential. Acceptability of disclosure to a recipient is strongly conditioned on consent and confidentiality, but drops sharply when the data will be stored or shared onward. Data availability. The raw user-study data is released by the original authors at https://github. com/Cristliu/LLMHealthPrivacy_UserStudy under the MIT License. Example: User from LLM Healthcare Consultation [24] Demographics (Du ). Not provided. Previous experiences (Eu ). Q. Which of the following best describes who you are when using the healthcare chatbot? A. “Users consulting on personal health issues.” Privacy attitudes (Au ). Q. Is online healthcare consultation more likely to compromise privacy than asking a doctor face-to-face? A. “Somewhat agree” (5-point: Strongly disagree → Strongly agree). Data-sharing question qi (Q1). Imagine you use the AI-large-scale-model for healthcare consultation, and the conversation mentions your mental health information (e.g., stress levels, emotional fluctuations, psychotherapy experiences). How acceptable is it for your family members to access this information? User response ru,i . “Somewhat acceptable” (5-point: completely unacceptable / somewhat unacceptable / neutral / somewhat acceptable / completely acceptable; mapped to {0, 1, 2, 3, 4} for evaluation).
C.2
AI Agent Permissions
Context. Wu et al. [42] ask participants to imagine training a future personal AI assistant (“MIRA”) in 2049 and to set permission preferences over (data, app) tuples that the assistant would request 27
to fulfill a natural-language query (e.g., trip planning, banking, fitness, home security, pregnancy support). Each item is answered on a four-point preference scale: Yes, always share; Yes, but ask me next time; No, but ask me next time; No, never share. Participants. 203 U.S. adult participants. The released demographics cover education, age range, and gender. Persona facets. Du contains education, age range, and gender. Eu records self-rated familiarity with current AI tools (1–5), frequency of use (1–5), and free-text purpose of use. Au contains general trust in AI tools (1–5), the importance of privacy when using digital tools (1–5), free-text privacy concerns, and the three application domains the participant cares most about. Key findings. Permission preferences are highly heterogeneous across users and highly contextdependent within a user: the same data type (e.g., location) is approved for one app/intent but not another, and few items receive a uniform allow/deny verdict across the population. This motivates per-user, per-context permission inference rather than global defaults. Data availability. The raw user-study data is released by the original authors at https://github. com/llm-platform-security/ai-agent-permissions under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Example: User from AI Agent Permissions [42] Demographics (Du ). Age: 45–54; Gender: Male; Education: Master’s. Previous experiences (Eu ). Q. On a scale from 1 (Not at all familiar) to 5 (Very familiar), how familiar are you with existing AI tools, such as ChatGPT and Gemini, in terms of your knowledge and usage? A. “3” (Somewhat familiar). Privacy attitudes (Au ). Q. On a scale from 1 (Not at all) to 5 (Full trust), how much do you trust these tools to not violate your privacy? A. “3”. Data-sharing question qi (Q1). You ask your personal AI assistant: “I want to lose weight. Can you create an at-home full-body workout plan for me?” To address this request, your AI assistant wants to access Gender from Fitness Tracking. How willing are you to share this data? User response ru,i . “Yes, always share” (4-point: Yes, always share / Yes, but ask me next time / No, but ask me next time / No, never share; mapped to {3, 2, 1, 0} for evaluation).
C.3
LLM Chatbot
Context. Tran et al. [37] apply Helen Nissenbaum’s contextual-integrity framework to ChatGPThistory sharing. Each scenario is a factorial vignette varying along five contextual-integrity attributes: recipient (university CS/social-science researcher, big tech, insurance company, hospital, charitable foundation, government, among others), location (US, EU, China), purpose (assessing mental health, predicting behavior, training future AI models, advertising, dataset creation, fighting terrorism, AI-risk research), topic of conversation (job, social/personal, legal/ethical, health, all), and three transmission principles (informed/consent/none, anonymized or not, personally identifiable information (PII) automatically removed or not). Participants rate appropriateness on a 0–100 numeric scale. Participants. Du = ∅.
300 U.S. ChatGPT users. Demographics are not released in the public dataset, so
Persona facets. Eu records ChatGPT use frequency, first-use date, subscription type, the tasks the participant uses ChatGPT for, and the topics they discuss with it (with optional free-text elaboration). Au records five 0–100 statements: companies collect too much data, general trust in websites, importance of privacy, ChatGPT-specific privacy concern, and belief that conversations remain private. Key findings. Appropriateness ratings vary sharply with the contextual-integrity attributes, especially recipient and purpose; consent and anonymization raise appropriateness but do not fully compensate for a sensitive recipient/purpose pairing; and stated privacy attitudes only weakly predict appropriateness ratings. 28
Data availability. The raw user-study data is released by the original authors on Harvard Dataverse at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/ DVN/M6ABJ3 [38] under the Creative Commons Zero (CC0 1.0) public-domain dedication. Example: User from LLM Chatbot [37] Demographics (Du ). Not provided. Previous experiences (Eu ). Q. How frequently do you use ChatGPT? A. “Less than monthly.” Privacy attitudes (Au ). Q. Online companies collect too much personal information. A. “70” (0–100: Strongly disagree → Strongly agree). Data-sharing question qi (Q1). Your ChatGPT conversations were shared with a big tech company based in the US for the purpose of fighting terrorism. All of your conversations were shared, you were informed that your data was collected, your conversations were not anonymized, and personally identifying information was not removed. Please rate your agreement with: “This is an appropriate use of my ChatGPT data.” User response ru,i . “0” (0–100 numeric scale: 0 = Strongly disagree, 50 = Neutral, 100 = Strongly agree).
C.4
Chatbot Group Chat
Context. Chou et al. [10] probe user awareness of and concern about chatbots that operate inside group chats on instant-messaging (IM) platforms (Discord, Telegram, WhatsApp, and Slack). After eliciting beliefs about what messages and metadata such chatbots can access, the questionnaire asks the participant to rate agreement with seven statements covering privacy concern, behavior change (would they self-censor?), continued use, and protection-policy understanding, on a five-point Likert scale. Participants. 374 participants worldwide recruited via Prolific. Demographics include age range, gender, race/ethnicity, highest level of education, and a binary technical-work-experience flag. Persona facets. Du contains age, gender, race/ethnicity, education, and technical experience. Eu describes the IM service the participant uses most, frequency of IM and chatbot use, types of group chats and chatbot purposes encountered, free-text descriptions of specific chatbots seen, and habits around checking for chatbot presence in private vs. public groups. Au records the participant’s beliefs about what message content, message metadata, sender metadata, and group-level information chatbots can access on their chosen platform. Key findings. Many participants underestimate the breadth of data accessible to group-chat bots, especially metadata and access to deleted messages; awareness varies substantially across platforms; and even highly aware users typically keep using the chatbots, citing utility. Data availability. The raw user-study data is released by the original authors at https://github. com/csienslab/bot-among-us under the MIT License. Example: User from Chatbot Group Chat [10] Demographics (Du ). Age: 25–34; Gender: Male; Education: Bachelor’s degree; Race/ethnicity: White or Caucasian; Technical work experience: Yes. Previous experiences (Eu ). Q. Please select one instant messaging service you use frequently and have seen chatbots on. A. “Discord.” Privacy attitudes (Au ). Q. What type of messages do you believe a chatbot on Discord has access to? A. “All messages sent in the chat, including those that are later deleted.” Data-sharing question qi (Q1). Please rate your level of agreement with the following statement about chatbots in group chats: “Chatbots from group chats with access to users’ data as listed above would raise my privacy concerns.” User response ru,i . “Neither agree nor disagree” (5-point: Strongly disagree / Somewhat disagree / Neither agree nor disagree / Somewhat agree / Strongly agree; mapped to {0, 1, 2, 3, 4} for evaluation).
29
C.5
LLM Conversational Agents
Context. Zufferey et al. [44] measure how frequently participants share each of ten categories of personal data with general-purpose conversational agents such as ChatGPT and locally hosted LLMs. Categories cover identifiers and general information (personal identifiers, account credentials, location/mobility, ethnicity/citizenship, criminal records) and standard of living and opinions (family/relationship, wealth, job/education, religion, political views), each rated on a five-point frequency scale from I never shared to I usually shared. Participants. 422 participants were recruited internationally with a European skew. We focus on just the chatbot users since they answered questions involving frequency of data disclosure, narrowing the participant pool to 318. Released demographics are intentionally light: age range and country/region. Persona facets. Du contains age and country/region. Eu describes whether the participant runs a local LLM instance (and, if so, why), the chatbot services they use, frequency and average daily duration of chatbot interaction, devices used, and their typical purposes. Au contains the eight items of the Internet Users’ Information Privacy Concerns (IUIPC) privacy-concern scale (collection, control, awareness) on a seven-point agreement scale. Key findings. Self-reported sharing frequency is low overall but varies sharply by category: job/education-related and general-information sharing are common, while credentials, criminal records, religion, and political views are rarely shared. Users running their own local LLMs report more frequent sharing across most categories, consistent with greater perceived control. Data availability. The raw user-study data is released by the original authors on the Open Science Framework at https://osf.io/2vqws/ under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Example user.
D
A representative user from this study is shown in the example box in Section 3.
Additional Results
Figure 3 in the main paper shows per-domain accuracy for the four strongest models. Figure 4 below extends that view to every model evaluated in P RIVACY S IM, adding the three mid-tier frontier models (GPT-5.4 Mini, Gemma 4 32B, Qwen3.5 27B) and the two large open-weight reasoning models (Qwen3.5 122B and Nemotron 3 120B) that were omitted from the main bar chart in Figure 2. All nine subplots share a common y-axis so absolute accuracies are directly comparable. The qualitative ordering of domains is stable across the nine models: LLM Conversational Agents (pink) sits at or near the top in every subplot, LLM Healthcare Consultation (yellow) sits at the bottom, and AI Agent Permissions (blue), Chatbot Group Chat (orange), and LLM Chatbot (green) cluster in the middle. This suggests that the difficulty of simulating privacy behavior is due to features of how each user study was conducted, such as the response scale, the breadth of the question pool, and how predictable the original participants’ decisions were from their persona facets. We observe that the Au -versus-Eu dip identified in Section 4 is visible in eight of the nine models: conditioning on privacy attitudes alone reliably underperforms conditioning on previous experiences alone, and only the Eu +Au combination recovers (and sometimes exceeds) the Eu -only level. The dip is sharpest for the most capable models (Gemini 3.1 Pro, GPT-5.4) and shallow but still present in the open-weight column, which rules out the alternative explanation that the dip is an artifact of frontier-model overfitting to stated attitudes. Across all nine models, the Du -only column drops back toward or below the none baseline, indicating that demographic facets in isolation provide little simulation signal. However, Du +Eu and Du +Eu +Au are typically as good as or better than Eu +Au , so demographics contribute marginal value once paired with behavioral context. This pattern is magnified in the larger open-weight models (Qwen3.5 122B, Nemotron 3 120B). 30
Tolerance Accuracy (%)
Gemini 3.1 Pro
Gemini 3 Flash
45 40 35 30
ne
no
Eu
Au
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
ne
no
Eu
Au
Tolerance Accuracy (%)
Sonnet 4.6
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
ne
no
Eu
Au
GPT-5.4 Mini
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
Gemma4 32B
50 45 40 35 30
ne
no
Eu
Au
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
ne
no
Eu
Au
Qwen3.5 27B Tolerance Accuracy (%)
GPT-5.4
50
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
ne
no
Eu
Qwen3.5 122B
Au
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
Nemotron 3 120B
50 45 40 35 30
ne
no
Eu
AI Agent Permissions
Au
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
Chatbot Group Chat
ne
no
LLM Chatbot
Eu
Au
u +A Eu
Du
u u u +E u+A u+A Du D +E Du
LLM Conversational Agents
ne
no
Eu
Au
u +A Eu
LLM Healthcare Consultation
Du
u u u +E u+A u+A Du D +E Du
5 studies
3 studies
Figure 4: Per-domain tolerance accuracy across all eight prompt types for every model evaluated in P RIVACY S IM, including the two large open-weight models (Qwen3.5 122B and Nemotron 3 120B) omitted from Figure 2. Solid black: 5-study average; dashed black: 3-study average. Each subplot shares the same y-axis so cross-model comparisons are direct.
E
Additional Details on Privacy Persona Cluster
Section 4.3 groups every participant in P RIVACY S IM into one of five privacy-persona clusters using a (stance, exposure) layout inspired by Dupree et al. [13] and asks which clusters are hardest for an LLM to simulate. This appendix walks through the clustering procedure: which user features feed it, how those features are standardized, how the five clusters are then assigned, and what design choices were made to keep the analysis honest. Relation to Dupree et al. and choice of axes. Dupree et al. [13] place each user on two axes: motivation (how strongly they care about protecting their privacy) and knowledge (how much technical understanding they have of privacy risks). Because the source studies in P RIVACY S IM do not include a privacy-knowledge instrument, we cannot operationalize Dupree et al.’s knowledge axis directly. We therefore borrow only the two-axis layout and instantiate the axes from the persona facets we do have: • Stated stance, su : a [0, 1] score summarizing the participant’s privacy attitudes (Au ). • Prior exposure, eu : a [0, 1] score summarizing the participant’s previous experiences (Eu ), capturing AI-tool familiarity rather than privacy knowledge. 31
Throughout the main paper we refer to the resulting clusters by their (stance, exposure) quadrant rather than by Dupree et al.’s original archetype names, since AI-tool exposure is a different construct from privacy knowledge. For continuity with prior work we list the rough correspondence in the centroid table below. From raw items to a [0, 1] score. Privacy-attitude statements across the five studies use either 5-point agreement Likert, familiarity Likert, or numeric scores (1–5, 0–100), all of which are easy to map to [0, 1]. However, a high Likert score can indicate either high or low privacy concern depending on how the statement is framed. We therefore parse each statement and decide its polarity by keyword: if it contains concern-direction words (concerned, worried, compromise, . . .), a high score indicates high concern and the raw [0, 1] value x is kept; if it contains permissive-direction words (trust, willing, will use, . . .), a high score indicates low concern, so we flip it to 1 − x. After this correction, “higher value = more concerned” for every item, and su is the simple mean over all polarity-corrected items. Previous-experience statements use a wider mix of formats (frequency phrases, yes/no, numeric familiarity, free text, multi-select chips), so we parse each statement and decide its score by format. Frequency phrases map to a fractional value (daily → 1, often → 0.75, sometimes → 0.5, . . ., never → 0); a yes / has-used / has-experience answer scores 1; a numeric familiarity rating ≥ 3 on a 1–5 scale scores 1. Anything else, including a “no” answer, a low familiarity rating, free text, or unrecognized multi-select chips, scores 0. We then take eu as the mean of these scores over all non-skipped items. The five source studies use different response vocabularies and different distributions of stance and exposure (e.g., the LLM Conversational Agents pool is heavily European and tech-comfortable; the LLM Healthcare pool skews toward general health users). We therefore z-score su and eu within each domain to obtain ŝu and êu , so that “high stance” and “high exposure” have a consistent meaning regardless of which study a participant came from. Note that this step takes the raw scores out of [0, 1]: ŝu and êu are real-valued, centered at 0 with unit variance within each domain, so a value of 0 means “average for this domain”, positive means above average, and negative means below average. Even with the keyword polarity rule, idiosyncratic instrument wording can flip the sign of su for a particular study. Before clustering, we therefore check the within-domain Pearson correlation between su and the (held-out) revealed-concern signal computed from ru . If that correlation is meaningfully negative (ρ < −0.05), we flip su 7→ 1 − su for that domain only. We chose this small magnitude so that the flip activates only when wording polarity is clearly inverted, leaving near-zero correlations alone. This is a sanity check on the wording of the questionnaire and not a feature of the clustering itself, so it does not introduce circularity (the flip changes the sign of su but is not itself used as a feature). We place the five clusters at fixed canonical centres in (êu , ŝu ) space. Because ŝu and êu are z-scores, the natural “high / mid / low” tiers sit at +0.8, 0, and −0.8, the midpoints of the upper, middle, and lower tertiles of a standard normal, which gives the following centres: Cluster (stance / exposure)
ŝ centre
ê centre
Closest Dupree archetype
high / high high / mid mid / mid low / high low / low
+0.8 +0.8 0.0 −0.8 −0.8
+0.8 0.0 0.0 +0.8 −0.8
Fundamentalist Technician Amateur Lazy Expert Marginally Concerned
Every user is then assigned to the nearest centroid under squared Euclidean distance in (êu , ŝu ). We chose fixed canonical centres (rather than k-means) so that a cluster label has the same meaning across studies. Across the 1,000 participants in P RIVACY S IM, the resulting cluster sizes are high-stance / midexposure = 226, low-stance / high-exposure = 221, low-stance / low-exposure = 205, mid-stance / mid-exposure = 187, high-stance / high-exposure = 161. To check that the qualitative findings in Section 4.3 are not artifacts of the specific centre placement, we also re-ran the assignment with centres pushed to ±0.30 and ±0.55 (corresponding to the conservative and aggressive tertile choices, respectively). The relative ordering of clusters by simulation accuracy at Eu +Au is preserved 32
under both perturbations, with low-stance / high-exposure remaining the lowest-accuracy cluster and high-stance / mid-exposure remaining the cluster that gains the most from persona conditioning.
33