Biased or Personalized? The Impact of Personal Information on AI-driven Development Erfan Entezami
Madeline Endres
University of Massachusetts Amherst Amherst, MA, USA [email protected]
University of Massachusetts Amherst Amherst, MA, USA [email protected]
arXiv:2607.07480v1 [cs.SE] 8 Jul 2026
Abstract Generative AI is increasingly permeating software engineering, enabling developers to generate functions, files, and even entire applications from natural language specifications. AI systems are also becoming more personalized, adapting outputs based on inferred user characteristics and interaction history. While personalization may improve the development experience, it raises concerns that generated software could be shaped by attributes of the developer rather than by task requirements alone. Prior work has shown that generative AI can produce biased software artifacts, but little is known about how developer identity can bias generated code. We characterize three dimensions through which inferred developer attributes can influence generated artifacts: interface design, template content, and code structure. First, through controlled experiments on 800 AI-generated websites, we find that age- and gender-related signals produce significant differences across all three dimensions. Second, we conduct an observational study and follow-up interviews with 20 participants who used AI to create a personal website to both examine how personalization impacts software artifacts in practice, and also to understand how programmers perceive the boundary between personalization and bias. Together, our results show that developer attributes can meaningfully influence generated software beyond stated requirements, highlighting a previously underexplored tension between personalization and fairness in AI-assisted programming.
1
Introduction
What happens when an AI coding assistant knows something about its user? Should the software generated for a 25-year-old woman differ in its implementation from that generated for a 65-year-old man, even when both request the same application? As large language models (LLMs) become more personalized through conversational history and inferred user characteristics [34], these questions are increasingly relevant. Personalization can either improve the software development experience by adapting artifacts to users’ preferences and programming expertise or perpetuate stereotypes and embed bias in generated software. AI-assisted tools have become an integral to many domains, from education [3] to healthcare [1]. Software engineering is no exception. Recent advances in LLMs have led to widespread adoption for tasks such as code generation [29], debugging [65], and documentation [5, 14, 69]. As these models improve, the technical expertise required to use them has decreased, enabling more users to develop software [18]. Users can increasingly transform natural language ideas directly into working software with little or no manual coding, a paradigm commonly known as vibe coding [57].
Simultaneously, LLMs can produce biased outputs that reflect existing stereotypes across a wide array of domains [20, 25, 33, 49, 70]. LLMs can generate different output for different users, inferring user characteristics from information explicitly or implicitly included in the prompt [34, 51, 66], behavior amplified by the increasing use of cross-conversational memory [17]. For example, implicit racial indicators in a prompt can lead state-of-the-art chatbots to recommend different colleges or neighborhoods to otherwise identical users [34]. These concerns naturally extend to software development. Recent work has found that the behavior of generated code at runtime can vary in demographic-sensitive contexts, raising algorithmic fairness concerns [28, 38]. Demographic bias has also been observed in broader software contexts, including how programmers are portrayed [4], what software tasks they are assigned [4, 56], and gendered characteristics in the prompt can even effect how code is evaluated during code review [30]. However, these studies primarily examine the behavior of generated programs, assess single functions, correlate with implicit demographic indicators, or focus on broader sociotechical outcomes. Less is known about if demographic characteristics directly inferred about the developer influences the landscape of software artifacts that the AI system generates. Understanding this phenomenon in software engineering is particularly important as software engineering has historically struggled with demographic inequities [37, 59], and developers with different backgrounds interact with development tools in different ways [7, 48], including when prompting models for code generation [30]. If AI systems infer demographic characteristics and adapt these design decisions using stereotypical associations rather than users’ actual needs, they risk reinforcing existing inequities within software engineering. We help close this gap by investigating the questions: How do demographic characteristics inferred by AI systems influence generated software artifacts, and how do programmers perceive the boundary between personalization and bias? We investigate in the context of AI-assisted web development; web applications permit AI systems to make decisions spanning user interface design, template content, and software implementation, providing a rich setting for studying how demographic characteristics may impact generated software artifacts. We first conduct a controlled experiment on 800 AI-generated websites across two tasks. By varying only age- and gender-related information in prompts, we isolate the effect of demographic attributes on generated artifacts across interface design, template content, and code structure. We then complement this with an
Erfan Entezami and Madeline Endres
observational study of 20 participants using their own ChatGPT accounts to examine how LLM-driven decisions manifest in practice and how developers perceive the boundary between personalization and bias. Our controlled experiments show that changing users’ demographic information (attributes irrelevant to the programming task itself) alters generated software artifacts across all three dimensions: interface design, template content, and code structure. For example, personal websites for older users were more likely to contain photo galleries 𝑝 = 0.003; online shops generated for women contained fewer files and less JavaScript (𝑝 = 0.007); and color differed significantly across groups (e.g., more blue for men). Our user study further found that participants primarily associated personalization with the content rather than higher-level development decisions, such as interface design or code structure, despite evidence that these less visible aspects of the generated software were also influenced by the users’ demographic information. Overall, we show that demographics traits can significantly influence generated software artifacts. This shows that AI coding tools incorporate demographic information irrelevant to the coding task, and motivates future research on the intersection of bias and personalization in software artifacts.
2
How Might Personalization Influence AI-Generated Websites?
Modern AI coding assistants generate substantially more than source code alone, making decisions about user interface design, and software architecture. Personalization in software generation based on user characteristics has the potential to improve developer experience. For example, if a LLM knows the user has less experience with a given API, it could generate more detailed learningfocused comments. To study where and how such personalization or bias may occur in a web development context, we consider three dimensions of AI-generated software artifacts: interface design, template content, and code structure (see Figure 1). These dimensions emerged during preliminary exploratory work conducted prior to our formal experiments, including pilot interactions with AI coding systems. We use these dimensions as an analytical framework for organizing our study, rather than as a comprehensive theory of AI bias and personalization. Interface design captures differences in the visual presentation of generated software, including choices in website layout, and color palette.Template content captures differences in the information populated within generated software artifacts such as placeholder text, example user profiles, product catalogs, or other default content that can shape the resulting software. Finally, code structure captures implementation decisions made by the model, including code volume, file organization, and other characteristics of the generated source code. Together, these dimensions cover both user-facing and developerfacing aspects of AI-generated websites. In this paper, we use this framework to organize and structure our analyses for both our automated experiments and user study.
3
Study Design
We investigated how demographic characteristics inferred by AI coding assistants can influence AI-assisted software development through a two-phase mixed-methods study. First, we conducted controlled automated experiments that systematically varied demographic information while holding all other prompt content constant. Second, we conducted an observational user study in which participants developed personal websites using their own ChatGPT account to examine how personalization manifests in practice and how developers perceived the fine line between personalization and bias. Together, these complementary experiments allow us to study causal instances of demographic bias under controlled conditions and its nuanced effects in a real-world development context.
3.1
Automated Experiment Design
We conducted controlled experiments in which AI models generated websites differing only in the prompter’s demographic characteristics. We focus on web development. We do so because it is a common use case for AI-assisted coding [41, 61, 68], practiced by users with a wide range of ages and backgrounds, making demographic assumptions plausible, and requires decisions about both user-facing interfaces and source code, allowing us to examine demographic bias across all three dimensions of our framework (see Figure 1). To improve generalizability, we consider two tasks: generating a personal website (Task 1) and an online shop (Task 2). Using ChatGPT-4.1 and DeepSeek-V3.2, we generated websites with zeroshot prompting differing only in the user’s persona. We created 20 personas balanced across four groups: young women (YW), older women (OW), young men (YM), and older men (OM). For each task/model/persona combination, we generated ten independent websites, resulting in 800 websites (2 tasks×2 models×20 personas×10 generations). 3.1.1 Automated Experiment Tasks. Figure 2 has our prompt templates. We selected two web development tasks that differ in complexity and opportunities for personalization: a personal website (Task 1) and an online shop (Task 2). Personal websites are identityfocused, and allow for substantial freedom in interface design, aesthetics, and content. To focus analysis on these decisions, we restricted implementations to HTML and CSS. In contrast, the online shop is a more complex application, requiring both frontend and backend components. It also involves data storage and facilitating payment integration. Generated websites typically involved HTML, CSS, JavaScript, Python and data-related files, allowing us to study not only interface design and content but also implementation decisions such as code organization and file structure. 3.1.2 Persona Creation. We constructed personas consisting of a name and age, following prior work on age and gender bias in software engineering [37]. We focus on these characteristics because both have historically been associated with inequities in computing [4, 37, 64]; if AI systems encode such assumptions, they may reinforce existing disparities. We created 20 personas spanning four groups (young women, older women, young men, and older men), with five personas per group. Ages were sampled by selecting birth years from the 1950s
Biased or Personalized? The Impact of Personal Information on AI-driven Development
Figure 1: An AI-assisted coding system may use cues explicitly provided in prompts and from previous chats to personalize software artifacts. We examine this through three dimensions: interface design, template content, and code structure.
Prompt for Task 1: Personal Website: Hi! My name is <persona name>, and I’m <persona age>. I’d like you to design a simple personal website that showcases my hobbies, skills, and activities. Please provide both HTML and CSS files for the website.
Prompt for Task 2: Online Shop: Hi! My name is <persona name>, and I’m <persona age>. I’d like to create a website to introduce and showcase my products as an online shop. Please build me a simple web application for this shop. The front end should use JavaScript to allow customers to browse products, view details, and add items to their shopping cart. The back end should use Python with Flask to handle user sign-ups, log-ins, and payment.
Example Personas Emily, age 24 (young woman) Susan, age 66 (older woman) Joshua, age 21 (young man) Robert, age 71 (older man)
Figure 2: Prompts for the personal website (top) or online shop (middle). Instantiated using one of 20 age and gender personas. Prompts are tightened for space.
(66–75 years old) or 2000s (16–25 years old). Names were the five most common U.S. Social Security Administration names for the corresponding decade and gender [63]. Example personas are shown in Figure 2. 3.1.3 Website Generation Process. To better approximate real-world interactions with AI coding assistants, we followed the vibe coding architecture described by Sapkota et al. [61], generating websites through each model’s chat interface instead of its API. To maintain experimental control, each website was generated in a fresh chat with no access to prior interactions or saved memory, ensuring that
the only demographic information available to the model was the user’s name and age. For each task–model–persona combination, we generated 10 independent websites using zero-shot prompting, making 200 websites per task–model pair and 800 overall. With all non-demographic aspects of the prompt are held constant, differences across groups were attributable to the demographic information embedded in the personas.
3.2
User Study Design
To complement the controlled experiments, an observational user study was conducted in which participants used their own ChatGPT accounts to develop a personal website. This approach captured real-world personalization by allowing ChatGPT to utilize both the current conversation and stored memory from previous interactions [54]. This allows us to investigate how personalization manifests in practice, how users respond to personalized outputs, and the extent to which AI-generated suggestions shape final software artifacts. Personal website development was chosen because it encourages sharing personal information, and is accessible to users with varying levels of technical experience. As in Task 1 of our automated experiments, participants used only HTML and CSS to facilitate comparisons across experiments. 3.2.1 Procedure. Each session lasted up to two hours, and consisted of a website development task (one hour) followed by a semistructured interview (around 30 minutes). During the observation, participants used their own ChatGPT account to create a personal website, interacting with the model as they normally would. After each prompt-response cycle, participants viewed the generated website, rated their satisfaction, and decided if they would continue refining it or finish the task.
Erfan Entezami and Madeline Endres
Table 1: Participant demographic information. ID P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20
Age 23 22 25 36 22 20 19 33 28 34 28 23 20 32 25 21 20 32 22 21
Gender M M W W W W W W W N M M M F F F F F F F
Group Planning Planning Planning Planning Planning Planning Planning Planning Planning Planning No planning No planning No planning No planning No planning No planning No planning No planning No planning No planning
Programming Experience 3 - 5 years 3 - 5 years 6 - 10 years 1 -2 years Less than 1 year N/A 1 -2 years N/A N/A 11 - 20 years 6 - 10 years 1 -2 years N/A 6 - 10 years 6 - 10 years N/A N/A 11 - 20 years Less than 1 year 3 - 5 years
Participants were randomly assigned to either a planning or noplanning condition. Those in planning were given seven minutes before interacting with ChatGPT to sketch their ideas for their website on pen and paper. This manipulation was designed to better understand the extent to which AI-generated suggestions influence users’ final websites. After the development task, we conducted a semi-structured interview focused on the participant’s interaction patterns, personalization perceptions, and opinions regarding ChatGPT’s Memory feature. Participants privately reviewed the information stored in their ChatGPT memory before discussing their impressions of the system’s personalized behavior. The hour for the observation was determined based on the feedback from 4 pilot participants. Our study was approved by our institutional ethics board (IRB), and sessions were in person or over zoom. Participants recieved a $25 gift card.
3.3
User Study: Recruitment and Population
We conducted sessions with 20 participants recruited from a large public university. Participants were at least 18 years old and used a ChatGPT account for at least one year. We chose ChatGPT because, as one of the earliest and most widely adopted conversational LLMs, it was more likely that participants would have experience using the system. Using a single model also ensured consistency across participants. Table 1 has study participant demographics, including age, gender, and programming background. Participants ranged from 19 to 36 years old and had a range of experience, enabling us to consider how programming ability mediates perceptions of personalization and bias in software artifacts.
4
Analysis Methodology
For both the automated experiments and user study, we organize our analyses around three dimensions of where inferred demographic
characteristics of the prompter might influence AI-generated software artifacts: interface design, template content, and code structure (see Section 2).
4.1
Automated Experiments Analysis
To evaluate if generated software artifacts differed across demographic groups, we analyzed all 800 generated websites (Section 3.1). We quantitatively evaluated interface design, template content, and code structure across both tasks (Table 2). We additionally conducted a qualitative analysis of a stratified subsample of 120 personal websites to characterize more nuanced differences in interface design and generated content. Metrics and Statistics: Table 2 summarizes the metrics for each dimension of our framework. For interface design, we analyzed color palette and layout-section presence. For template content, we analyzed generated skills and product categories. For code structure, we analyzed lines of code, and file counts. When testing categorical variables (e.g., color), we used different tests depending on the sample size and number of categories, as is best practice: 𝜒 2 -tests when the expected number of observations in each category without bias was at least 5 [11], Fisher-FreemanHalton [19] exact test for smaller sample sizes, and the proportion’s 𝑧 test when there are only two categories (equivalent to a two-factor 𝜒 2 ). For comparing continuous variables (e.g., code lines) across groups, we used Mann–Whitney 𝑈 [43] tests. We choose this nonparametric test because our data was non-normally distributed. Due to the large number of tests, we use Benjamini-Hochberg (BH) [6] correction within each task and metric family to account for multiple comparisons and limit false positives. Qualitative Subsample: We conducted an analysis of a stratified subsample of 120 personal websites (three websites from each persona–model pair, see Section 3.1). We developed our hierarchical codebook using a multi-pass inductive approach. Two authors independently reviewed a subset to identify recurring interfaces, content, and implementations (e.g., Color: Pink, etc.). Given the conformability of AI generated websites, we quickly reached saturation and a high agreement. After finalizing codes through negotiated agreement, one author coded the remaining websites using NVivo 15. The resulting codes both characterized demographic differences and served as categorical variables for subsequent statistical analyses.
4.2
User Study Analysis
We analyzed participants’ final websites using the same codebook developed for the automated experiments, extending it where necessary to capture additional design features. For the interviews, we followed Detering and Waters 21st Century Approach for coding interviews [12]. Through a series of iterative passes defined by negotiated agreement, we developed a codebook consisting of 58 index codes and subcodes. The codebook captured participants’ experiences with AI-assisted development, perceived personalization, ChatGPT’s Memory feature, and attitudes toward collecting personal information. Once the codebook reached saturation for each index code and all authors agreed on
Biased or Personalized? The Impact of Personal Information on AI-driven Development
Table 2: Overview of metrics for our automated analysis of 800 generated websites across the three bias dimensions. Qual indicates that analyses were conducted on a subsample of 120 personal websites selected for more nuanced review.
Qual.
Task 2 All Websites
Overall layout Section presence Color palette Template Content
✓ ✓ ✓
✓
Personal skills Product categories Code Structure
✓
Metric Interface Design
Lines of code File organization
Task 1 All Websites
✓
✓ ✓
Results—Automated Experiments
We now present the results of our automated experiments testing how inferred developer characteristics can influence generated software artifacts. Overall, we ask How does including explicit indicators of the prompter’s demographic attributes in prompts influence AI software outputs and shape assumptions about users’ preferences? We organize our results around the three dimensions in our conceptual model. Table 2 gives an overview of all metrics analyzed for each dimension.
5.1
GPT 4.1
✓
the sub-code definitions, the first author applied the code-book to the rest of the dataset.
5
DeepSeek V3.2
Bias in Interface Design
We define interface design as the visual organization and aesthetic presentation of the generated websites. As shown in Table 2, we test for bias relating to overall layout (Task 1), layout section presence (Task 1), and color palette (Tasks 1 and 2). These metrics capture both structural design decisions and aesthetic choices that shape users’ first impressions of generated websites. We find significant demographic-related differences for all interface metrics except overall layout.
Figure 3: Personal Website Layout: The typical Task 1 layout generated by DeepSeek (left) and ChatGPT (right).
We find that section presence is associated with persona. For example, while only 10/120 personal websites contained a Photo Gallery (9 DeepSeek, 1 ChatGPT), all were for older personas (DeepSeek: 𝑧 = 3.25, 𝑝 = 0.003). We find similar differences for Contact: 37 of 58 (64%) were for older personas while only 21 (36%) were for younger personas (𝑧 = 2.92, 𝑝 = 0.004). This was driven primarily by the fact that both models rarely generated Contact sections for young women. Of the 21 contact sections for younger personas, 15 were for young men and only 6 were for women (𝑧 = 2.78, 𝑝 = 0.005). These differences suggest that the models vary their selection of website layout and content sections according to demographic attributes included in the prompt (age and gender in our study). As a result, these variations propagate to the final software artifacts, influencing their overall structure and visual presentation.
Personal Websites (Task 1)—Overall Layout: Figure 3 shows the most common personal website layout generated by each model. Layout was highly consistent across personas: every website consisted of a single scrolling page with a colored header followed by sections such as About Me or Skills. Differences were model-specific: ChatGPT favored bulleted lists, while DeepSeek used richer visual elements such as skill bars, timelines, and placeholder profile images.
Personal Websites (Task 1)—Color Palette: From our manual coding of 120 websites, we found significant demographic differences in interface color palettes. Both models used blue more frequently for men and had a wider range of colors for women. In GPT, 24 of 31 (77%) dark blue websites were generated for men; in DeepSeek, (9/35, 83%). Pink and purple were generated exclusively for websites associated with women. We also observe age–gender interactions. Green was associated with older women, purple for young women. These differences were statistically significant (𝑝 < 0.05) with medium-to-large effect sizes (V : 0.36–0.53, Table 3). The observed association between color patterns selected by the models for different persona groups suggests a relationship between demographic information and how models adapt the visual appearance of generated products for users.
Personal Websites (Task 1)—Section Layout: While overall layouts were consistent, we find significant demographic-related differences in section presence and layout. All 120 websites in our qualitative subsample included Hobbies and Skills. However, other sections were less consistent; About Me (108), Activities (107), Contact (58), and Photo Gallery (10).
Online Shops (Task 2)—Color Palette: To test if patterns generalize to Task 2, we automatically extracted CSS colors (e.g., hex, RGB(A), and named colors) from all 400 shops. We then used a HSB-based heuristic to group them by the colors used for Task 1. We manually validated these groupings to ensure they closely matched our qualitative coding.
Erfan Entezami and Madeline Endres
Task 2: Online-Shop Color Presence
GPT
1.0 0.8
Proportion
Table 3: Color distributions from Task 1 qual. subsample. YM, OM, YW, and OW show color frequencies by group (max 15). 𝑃-values are Fisher-Freeman-Halton exact test with BHcorrection. All significant results had medium or large effect (Cramér’s 𝑉 ). We only include colors for which there were at least three examples for a given model.
* p=0.000
* p=0.026
0.6
* p=0.000 * p=0.012
0.4 0.2
0 0 6
3 3 5
Figure 4 shows the proportion of websites with each color category across groups. We again observe significant demographicrelated differences in colors. Overall trends align with those from Task 1—purple is most common for young women, while blue is more common for men. Green, however, is more evenly distributed across groups; for DeepSeek, it is only less common for young women. This generalization indicates that observed biases reflect systematic model behavior, rather than artifacts of a single programming task. Bias In Interface Design: Demographics influenced multiple aspects of AI-generated interfaces. While overall layouts were consistent, age and gender shaped section layout and color choices across both tasks, suggesting systematic bias.
5.2
Bias in Template Content
We define template content as natural-language text generated to populate software artifacts. Although neither task explicitly requested template content, both models routinely generated it. As shown in Table 2, we analyzed generated skills (Task 1, personal website) and product categories (Task 2, online shop). Anecdotally, almost all template text contained significant demographic-related differences. We selected these content types because they were consistently represented in structured formats: skills appeared as its own section on all websites in our qualitative subsample, and products generated as JSON catalogs could be systematically identified and extracted. Personal Websites (Task 1)—Skills: All 120 websites in our qualitative subsample contained a Skills section with AI-generated skills. Manual coding identified 27 unique skills (24 GPT; 21 DeepSeek). Generated skills differed significantly across both age and gender
n
re d
h it e
b ro w
w
ye llo w
or an g e
n /b ei g e
g re y
ta
p in k
* p=0.000
0.2
* * p=0.042p=0.046
n b ro w
re d
0.0 h it e
↓1
0.434 0.384 0.381 0.364 0.239 0.222 0.118
w
4 ↓2
< 0.001 < 0.001 < 0.001 < 0.001 0.054 0.073 0.627
ye llo w
↑ 12
↑ 10
or an g e
↓0 ↑ 15
1 ↑ 10 ↓4 4 4 3 10
n /b ei g e
↓0
* p=0.002
* p=0.007
0.4
g re y
2 ↓0 ↑ 14 ↑ 14 0 0 6
0.6
ta
Green Purple Dark Blue Light Blue Pink Black Has Gradient
Persona group man / old woman / old man / young woman / young
* p=0.000
p u rp le
DeepSeek
0.8
p u rp le
d ar k
0 8
DeepSeek
* p=0.000
1.0
p in k
0 3
↑7
k
0.535 0.481 0.318 0.183
g re en
< 0.001 < 0.001 0.210 0.785
b lu e
↓0
8 1 3 7
b lu e
↑ 13
d ar k
9 ↓0 0 8
Proportion
Dark Blue Green Pink Light Blue
k
↓1
0.0
GPT-4.1
g re en
𝑉
b lu e
𝑝-value
b la c
OW
b la c
YW
b lu e
OM
lig h t
YM
lig h t
Color
Figure 4: Color Bias: Color presence across persona groups 400 generated online-shops (Task 2). Asterisks indicate colors with significant variation across groups after BH correction.
(see Table 4). Older men were associated with physical skills such as woodworking and home repair, while older women were associated with creative skills such as art or crochet. Younger personas were associated with technical skills, including web development, web design, and programming. The only technical skill significantly associated with older personas was “general technology”, which consisted of basic computer skills such as email. Some differences were subtle. For example, young men were associated with web development, while young women were associated with web design—an interesting dichotomy that may relate to the prevalence of gendered computing stereotypes [10, 55]. Patterns were largely consistent between models. Figure 5 visualizes age and gender associations for the most common skills across both models combined. Interestingly, photography (which was one of the most frequently generated skills appearing in 54/120 websites) was one of the few skills showing no significant demographic differences in either model. This contrasts with the significant age-related difference observed for the presence of Photo Gallery sections (see Section 5.1), suggesting that demographic bias may manifest differently across different software artifacts. We consider the implications of this finding in the Discussion in Section 7. Online Shops (Task 2)—Products: We next analyzed personas for which the models generated structured JSON product catalogs. We automatically extracted product names and manually grouped them into 13 categories. Catalog generation rates did not differ by demographic group for either GPT (𝜒 2 = 1.67, 𝑝 = 0.68) or DeepSeek (𝜒 2 = 0.17, 𝑝 = 0.98). GPT generated template JSON catalogs for roughly half of personas in each group, whereas DeepSeek generated catalogs for only 4–5 personas per group. We therefore focus on GPT.
Biased or Personalized? The Impact of Personal Information on AI-driven Development
100
Older Women
Skill associations by age and gender group Younger Women Art
Knitting/Crocheting
Web Design
Gender association (% women − % men)
Language Fluency
Writing
Organizing
50
Baking/Cooking
Skill Photography 0
−50
General Technology Skills
Storytelling
Public Speaking Graphic Design Programming Teamwork Photo/Video Editing Web Development
−100
Woodworking
Older Men −100
−50
0
Age association (% young − % old)
50
Younger Men 100
Figure 5: Template Content Bias: Skills generated for personal websites (Task 1), by age and gender association. x-axis shows the percentage-point difference between young and older personas, while the y-axis shows the difference between women and men. Circle size indicates skill prevalence. Includes skills for which there were at least 10 instances in our qualitative sub-sample. Model-specific statistics are in Table 4. Within GPT-generated catalogs, products varied significantly across persona groups. Clothing was strongly associated with young men, appearing in 19 shops compared with only 1–4 for each other group (𝜒 2 = 37.67, 𝑝 < 0.001). Clothing Accessories and Bath Products were both concentrated among older women (𝑝 < 0.001, 𝑝 = 0.038, respectively). These results demonstrate that observed demographic differences in template content generalize across software tasks. Bias In Template Content: Age and gender significantly influenced generated template content for both tasks. Younger men were associated with web development, younger women with web design, older men with woodworking, and older women with knitting. Although non-executable, it may persist into deployed systems or influence subsequent decisions.
5.3
Table 4: Skills across persona groups for both models. YM, OM, YW, and OW show frequencies by group (max 15). 𝑝-value is Fisher-Freeman-Halton exact test with BHcorrection. 𝑉 is Cramér’s 𝑉 for effect size. Bolded cells drive significant associations. We include significant skills.
Bias in Code Structure
We next investigate if demographic information influences the structure of the generated code. We define code structure as the organization and implementation of the code base. There are many different metrics that could be used to analyze code. As summarized in Table 2, we focus on general structural metrics, such as lines of code and file organization. Personal Websites (Task 1)—Lines of Code: Due to the prompt (Figure 2), all generated personal websites consisted of one HTML file and one CSS file. We thus only compare the number of lines generated for each file type across groups. We find no differences for DeepSeek. However, GPT generated significantly shorter websites
YM
OM
YW
OW 𝑝-value
𝑉
GPT-4.1 Skills Woodworking General Tech Baking/Cooking Photo/Video Editing Web Development Storytelling Art Web Design Programming Teamwork Organizing Knitting, Crocheting Home Repair Problem Solving Public Speaking Writing
↓0
↑ 10
↓0
↓0
↓0
↑ 12
↓0
↑8
↓0
↑9
↓0
↑ 10
↑ 11
↓0
↑9
↓1
↑9
↓0
2
↓0
↓0
↑8
↓0
1
↓0
↓0
↑8
↓0
↓0
↑7
↓0
↑7
↓0
1 ↑8 ↑6 ↑5 ↓2 ↓0 ↓0 3 ↑ 11 ↑ 10
↓1
3
↓0
↓0
↓0
↑4
↑5
↓0
7 2
↓2 ↓1
1 ↓0 ↓0 ↑ 12 ↑5 ↓0 ↓0 ↓2
4
< 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 0.001 0.001 0.002 0.002 0.004 0.004 0.019 0.032 0.047
0.382 0.328 0.298 0.284 0.309 0.312 0.283 0.270 0.240 0.238 0.266 0.244 0.238 0.201 0.201 0.200
< 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 < 0.001 0.003 0.006
0.448 0.443 0.418 0.363 0.353 0.370 0.310 0.300 0.256 0.226
DeepSeek Skills Woodworking Knitting, Crocheting Writing General Tech Graphic Design Web Development Public Speaking Storytelling Baking/Cooking Home Repair
↓0
↑ 15
↓0
↓0
↓0
↓0
↓0
↑ 15
↓0
↓0
↑ 14
↓0
↓0
↑ 14
↓0
↑ 14
↑ 15
↓0
↑ 12
↓0
↑ 14
↓0
↓0
↑ 12
↓0
↓0
↑ 11
↓1
2 ↑4
3 6 ↓0 5 ↓0
↓0
↓0
6 ↑ 12 ↓0
for older personas (Table 5). They had fewer HTML lines (60.1 vs. 65.4), CSS lines (77.3 vs. 99.8), and total lines (137.5 vs. 165.1), a large sized effect for CSS and a medium sized effect for HTML. This suggests that GPT generated simpler or less stylized websites for older personas. Because the effect is larger for CSS than HTML, it is unlikely to be explained solely by shorter generated content, instead suggesting differences in visual presentation. Online Shops (Task 2)—Code Metrics: Compared to Task 1, online shops are substantially more complex, containing HTML, CSS, JavaScript, and Python. The models varied in file organization, sometimes separating languages into multiple files and other times combining them (e.g., embedding CSS in HTML). This lets us analyze code size and project structure. Table 6 summarizes the results. We again observe significant differences in generated code. However, unlike Task 1, the strongest effects occur for DeepSeek; GPT had age-related differences that did not survive comparison correction.
Erfan Entezami and Madeline Endres
Table 5: Task 1 Code Features: Mann-Whitney 𝑈 -tests for code metrics. Cells report p-value and Rank-biserial effect size (𝑒). Highlighted cells are significant after BH correction. The third line reports averages. Age Metric
HTML lines
CSS lines
Gender
GPT
DS
GPT
DS
𝑝 < 0.001 𝑒 = -0.320
𝑝 = 0.762 𝑒 = 0.025
𝑝 = 0.834 𝑒 = 0.017
𝑝 = 0.754 𝑒 = −0.026
Y: 65, O: 60
Y: 149, O: 150
M: 63, W: 63
M: 150, W: 149
𝑝 < 0.001 𝑒 = -0.672
𝑝 = 0.054 𝑒 = −0.158
𝑝 = 0.919 𝑒 = 0.008
𝑝 = 0.882 𝑒 = 0.012
Y:100, O: 77
Y: 300, O: 286
M: 89, W: 88
M: 293, W: 294
For gender, websites generated for men contained significantly more files (5.45 vs. 4.31), including more HTML, CSS, and JavaScript files. Men also received more CSS and JavaScript lines. In contrast, websites generated for women had more Python lines and, interestingly, substantially more lines of HTML despite having fewer files overall (average: woman=281.6 vs man=175.1, 𝑝 < 0.001). This might suggest that DeepSeek more frequently embedded styling within HTML for women rather than separating it into CSS files. As for age, older personas received significantly larger code bases, with more files and more lines of code across nearly every metric. Notably, this is the opposite of GPT in Task 1, where older personas received shorter websites. Together, this suggests that demographic information can influence implementation decisions, but that the direction of those effects depends on both the model and the software task. Bias In Code Structure: Demographic differences extended beyond generated interfaces and template content into the structure of the generated code itself. Although these effects were modeland task-dependent, they suggest that demographic information can influence implementation decisions in ways that are not immediately visible in the final interface.
6
Table 6: Task 2 Code Features: Mann–Whitney 𝑈 tests and Rank-biserial effect size (𝑒). Highlighted cells are significant. Darker cells with bold text remain significant after BH correction. Colors indicate the group with the higher mean: red for men or young, blue for women or old.
User Study
As demonstrated by our automated experiments, demographic attributes can systematically influence the content, design, and code structure of generated software artifacts. Whether these differences constitute beneficial personalization or problematic bias, however, depends on how they are perceived by users. We thus conducted a user study to understand when participants viewed personalized behavior as helpful, when they considered it inappropriate, and how ChatGPT’s Memory feature shaped these perceptions). We ask: User Study RQ1—Personalization in Practice How does personal information affect AI-generated software in practice? User Study RQ2—Perceptions: How do users perceive the implicit collection and use of their personal data for generating personalized software in practice?
Age Metric
Gender
GPT
DS
GPT
DS
HTML files
𝑝 = 0.277 𝑒 = 0.083 Y: 2.32, O: 2.67
𝑝 = 0.043 𝑒 = 0.114 Y: 1.35, O: 1.93
𝑝 = 0.509 𝑒 = 0.050 M: 2.41, W: 2.58
𝑝 < 0.001 𝑒 = -0.129 M: 1.77, W: 1.51
CSS files
𝑝 = 0.045 𝑒 = −0.040 Y: 1.00, O: 0.96
𝑝 < 0.019 𝑒 = 0.150 Y: 0.65, O: 0.80
𝑝 = 0.316 𝑒 = 0.020 M: 0.97, W: 0.99
𝑝 < 0.001 𝑒 = -0.270 M: 0.86, W: 0.59
Python files
𝑝 = 0.025 𝑒 = −0.050 Y: 1.05, O: 1.00
𝑝 = 0.160 𝑒 = 0.020 Y: 0.99, O: 1.01
𝑝 = 0.654 𝑒 = 0.010 M: 1.02, W: 1.03
𝑝 = 1.000 𝑒 = 0.000 M: 1.00, W: 1.00
JS files
𝑝 = 0.021 𝑒 = −0.078 Y: 1.00, O: 0.92
𝑝 < 0.001 𝑒 = 0.312 Y: 1.01, O: 1.83
𝑝 = 1.000 𝑒 = 0.000 M: 0.96, W: 0.96
𝑝 < 0.001 𝑒 = -0.348 M: 1.73, W: 1.11
Total files
𝑝 = 0.307 𝑒 = 0.082 Y: 5.92, O: 6.17
𝑝 < 0.001 𝑒 = 0.290 Y: 4.09, O: 5.67
𝑝 = 0.599 𝑒 = 0.042 M: 5.96, W: 6.13
𝑝 < 0.001 𝑒 = 0.350 M: 5.45, W: 4.31
HTML lines
𝑝 = 0.033 𝑒 = 0.175 Y: 55.46, O: 65.12
𝑝 = 0.932 𝑒 = −0.007 Y: 236.92, O:219.82
𝑝 = 0.637 𝑒 = −0.039 M: 61.25, W: 59.33
𝑝 < 0.001 𝑒 = 0.372 M: 175.13, W: 281.61
CSS lines
𝑝 = 0.978 𝑒 = −0.002 Y: 77.23, O: 72.86
𝑝 < 0.003 𝑒 = 0.247 Y: 97.44, O: 152.31
𝑝 = 0.653 𝑒 = −0.037 M: 74.96, W: 75.13
𝑝 < 0.010 𝑒 = -0.210 M: 139.18, W: 110.57
Python lines
𝑝 = 0.099 𝑒 = −0.135 Y: 73.91, O: 70.64
𝑝 < 0.005 𝑒 = 0.234 Y: 138.39, O: 161.32
𝑝 = 0.550 𝑒 = −0.049 M: 72.79, W: 71.76
𝑝 < 0.017 𝑒 = 0.196 M: 141.50, W: 158.21
JS lines
𝑝 = 0.118 𝑒 = −0.128 Y: 77.66, O: 63.24
𝑝 < 0.016 𝑒 = 0.196 Y: 178.29, O: 228.00
𝑝 = 0.953 𝑒 = 0.005 M: 69.38, W: 71.52
𝑝 < 0.007 𝑒 = -0.221 M: 234.00, W: 172.29
6.1
User Study RQ1— Personalization in Practice
Although all participants created a personal website, their goals varied. Some emphasized academic or professional achievements, others focused on personal interests, and several combined both. Because of this diversity, quantitative analyses like those used for the automated websites (e.g., color or content distributions) would offer limited insight into the model’s personalization decisions. Instead, we conducted a qualitative analysis of the websites’ structure and appearance, triangulated with interview responses, to assess personalization in practice. 6.1.1 What did developers perceive as personalized? During the interviews, users were asked if they had experienced any personalization in their interactions with ChatGPT (e.g., model responses tailored to the participant, without them explicitly requesting it). 13/20 participants (65%) reported experiencing personalization. When asked to elaborate, all 13 emphasized examples of personalized content. For example, “It absolutely did, based on what it knows about me and what information I had given it. . . . I use it for modifying my CV and resume. So it knows a lot about my professional skills. I didn’t have to change that at all.” (P4) 40% of participants (8/20) reported experiencing heavily personalized content, where ChatGPT automatically filled website sections with information it knew about them. An additional 25% of participants (5/20) reported some level of personalization, such as partial customization of website sections or personalized suggestions that were not directly incorporated into the final code.
Biased or Personalized? The Impact of Personal Information on AI-driven Development
6.1.2 What about interface design or code structure? In the interview, participants did not mention any aspect of interface design or code structure as being explicitly personalized. However, qualitatively, we observed patterns in the final generated websites which align with trends in our automated experiments for both framework dimensions. For example, 16/20 participants explicitly specified a color scheme for their websites and one participant provided a general color theme (e.g., warm and light colors). Among the 3 participants who did not specify any colors (all in no planning group), the model defaulted to blue and white, which was consistent with the majority of automatically generated websites in our simulated study. These participants continued developing their websites without changing the color scheme. Furthermore most no-planning participants (7/10) kept the layout generated by ChatGPT during the initial interaction, typically a single-page vertical website with navigation links in the menu bar, similar to the template in Figure 3. These participants primarily modified content and made minor design changes, such as adding profile and background images. In contrast, participants in the planning group produced more diverse website structures, with 6/10 creating multi-page websites with different layouts and visual styles. Our results suggests that bias in interface design or code structure may more difficult for users to recognize than in content. When users do not explicitly plan or specify design preferences, they often adopt the template and visual structure provided by the model. This finding aligns with prior work showing that reliance on LLMgenerated outputs can lead to homogeneous and structurally similar results [2, 47]. Personalization in Practice: Most participants (13/20, 65%) recognized personalized Template Content, whereas personalization in Interface Design and Code Structure was not reported despite qualitative evidence of both. This suggests that these two dimensions of personalization are less recognized to developers and may therefore be more likely to persist unnoticed in generated software artifacts.
6.2
USer Study RQ2—Perceptions
RQ1 showed that personalization influences software artifacts in practice, but does not explain if users perceived those changes as helpful. We thus explored participants’ perceptions of personalization and their reactions to ChatGPT’s memory. 6.2.1 Perceptions of Personalization. Most participants were familiar with ChatGPT personalization (17/20, 85%). Participants expressed diverse opinions, with no consensus between those who found it helpful, those who did not. Participants generally viewed personalization positively when it reduced effort or integrated relevant information for the task at hand. “I think it was very useful. . . it already knew what my background was, so it, actually filled in with the most relevant information for the use case.” (P12) Conversely, they viewed it negatively when it relied on irrelevant prior interactions or retained information across unrelated tasks. “I don’t think it was super useful. . . if there was, general information that it would remember, that would be fine, but if it’s something really specific that I asked it for help with. . . and
then it kept referencing that thing. . . I would just get really annoyed. ” (P20). While personalization could be a helpful starting point, participants edited the generated software content to better match their intent. “I think it’s useful for the structure,. . . but not super useful for writing in a way that feels like me” (P16). Some planning participants indicated that personalization was unhelpful because they already knew what they wanted. 6.2.2 Reactions to Memory. Most participants (15/20) were aware that information could be shared across ChatGPT conversations, although only 12 were familiar with features such as Memory or Temporary Chat. 6 participants had no saved memories, either because they had disabled memory or because no memories had been created. The remaining 14 reviewed their saved ChatGPT memories and reflected on the accuracy and appropriateness of memories for personalization. Participants generally found memories accurate, although several noted that some memories were outdated or captured information from one-off conversations that they did not consider meaningful. Participants were more concerned about privacy and if the memories were appropriate to retain for future personalization. “The only scary thing was that it saved my DOB. Which I didn’t want it to. I’ll delete that.” (P2); Some participants distinguished between project memory and personal memory. They valued persistent context when it helped maintain continuity on for a software task, but were less comfortable with ChatGPT retaining personal information across unrelated conversations. “Remembering stuff from a project based on the code is a useful thing. It doesn’t need to remember personal information too. . . I want it to feel like a machine, I don’t want it to feel like a human” (P15). Some participants said that they might be more selective about what they share with ChatGPT. “Now that I’m aware of it, I would use temporary chats or disable memory for some of the tasks I do” (P2). Others, however, viewed memory as a benefit as long as it led to helpful personalization. “I will not stop them collecting my information. . . As long as they can personalize, I’m happy” (P9). Together, these findings highlight a preference for project-scoped memory over personal memory, and show the importance of balancing the benefits of personalization with control over personal information in a software context. Personalization Perceptions: Participants expressed diverse attitudes towards software personalization. They generally valued personalization when it supported ongoing software tasks or reduced effort, but viewed it negatively when irrelevant personal information persisted across conversations.
7
Discussion
Implications for AI-Assisted Development. Our controlled experiment shows that generated software artifacts depend not only on programming task, but also on demographic information provided by the user. Changing only a user’s age or gender altered interface design, template content, and code structure. As AI coding assistants become increasingly integrated into software development, such
Erfan Entezami and Madeline Endres
differences may affect maintainability, readability, security, developer productivity, and ultimately the quality of deployed systems. We also found evidence of stereotype reinforcement. Websites generated for women had more pink and purple, while blue was more common for men. Older men were skilled at woodworking and home repair, while older women were skilled with knitting. Notably, neither task required models to generate personalized content: they could instead have produced neutral placeholder text (e.g., Lorem ipsum). Many observed differences thus reflect discretionary design decisions rather than task requirements. These concerns are reinforced by our user study which suggests some of these choices often go unnoticed. Participants who had not planned their websites largely retained ChatGPT’s initial designs, indicating that users accept AI-generated design decisions with little modification. Although our controlled experiments manipulated explicit age and gender indicators, real-world coding assistants may infer similar information from conversational history or writing style. Users with different backgrounds, including gender, tend to interact with technology in different ways [7]. Thus, it is conceivable that differences in prompting correlated with demographic attributes may systematically influence the quality of produced code. This has already been observed in practice for code review, where womanassociated prompting styles led to higher acceptance levels in a code review task [30], motivating future work on detecting and mitigating unintended demographic personalization in software artifacts. The Line Between Personalization and Bias. Our findings show that AI coding assistants use personal information when generating software artifacts, raising important questions about which aspects of software development should be personalized and which should remain independent of users’ characteristics. For example, should demographic information influence the structure and organization of generated code, or should personalization be limited to userfacing elements such as content or visual design? Personalization in template content was easier for users to evaluate than implementation decisions. Participants did not notice differences in interface design or code structure. However, they expressed mixed opinions about personalized content: some appreciated the personalized output while others felt it was irrelevant or “uncanny”. While personalization can improve relevance and user experience, its role in code generation is less clear. For example, women personas more often received HTML and CSS in a single file, while men more often received multi-file projects. If these strategies improve the programming experience or instead introduce unnecessary disparities is an open question. We argue that personalization mechanisms should be more transparent and configurable. Allowing users to define the extent and types of personalization they want could help prevent unintended biases while still enabling personalized AI assistance.
8
Limitations & Threats
Our results may have limited generalizability. In the simulated study, we investigated the impact of personal information on AIgenerated software by varying only two demographic factors, age and gender. Although these were selected based on prior software engineering research, users may reveal many other characteristics
that could influence model behavior. Results also may not generalize to different LLMs. To mitigate this, we selected two models that were state-of-the-art at the time of the study. However, given the rapid evolution of generative AI, examining a broader range of models to better understand the consistency and extent of personalization and demographic bias is warranted. To facilitate controlled analyses, we constrained Task 1 to HTML and CSS. While we partially mitigate this with Task 2 which had HTML, CSS, JavaScript, and Python, developers use a much wider variety of languages and frameworks, which may affect model behavior. Our user study was also limited to ChatGPT and HTML/CSS to reduce experimental variability. However, in practice, developers may switch between LLMs and technologies throughout a project. Finally, as with most user studies, participants were aware that they were observed, which may have influenced their interactions. Although we attempted to minimize this effect by clearly explaining the study, allowing participants to withdraw at any time, and offering both in-person and remote sessions, some degree of observation bias is unavoidable.
9
Related Work
LLM-based Tools in Software Engineering. As LLMs have become more capable of understanding and generating code, they have been widely adopted in software engineering [15, 27, 74, 76]. Specialized code models, such as Codex [9], WizardCoder [42] and Coda Llama [60], support tasks including code completion [24, 29, 39], automated code repair [16, 72, 73], code comprehension [5, 8, 50], and generating documentation [5, 14, 69], making it easier for software development teams to collaboratively understand the codebase. As LLMs advance, researchers are exploring the concept of “LLM-asJudge”, where LLMs evaluate various software tasks [26, 32, 46, 67]. LLMs can translate natural language into code, enabling nonprogrammers to build software through vibe coding [21, 22, 44, 57, 61, 62]. Sarkar and Drosos [62] analyze recorded vibe coding sessions from developers to investigate their intent, prompting strategies, and overall workflow. A similar approach has been proposed by Geng et al. [22] where they have conducted an observational study to explore how students in software engineering classes interact with vibe coding tools. We build on this work to examine how bias and personalization can manifest in software artifacts in AI-driven development. Bias in LLM-based Tools. Although revealing and mitigating bias in generative models has been extensively studied for general use cases [20, 25, 33, 49, 70], its impact on software is less explored. Previous work [28, 40, 78] has shown that models can produce socially-biased code behavior for bias-sensitive tasks, including tasks specified through natural language prompts [28]. Qin et al. [58] focus on gender bias in code LLMs, introducing a metric to measure the disparities between model outputs and real-world data. Ling et al. [38] introduce a framework to test social biases in generated code, along with prompting strategies to reduce bias. Du et al. [13] propose a benchmark and metrics to assess bias in code LLMs. Focusing on the developer instead of code output, Bano et al. [4] show that LLMs exhibit racial and gender bias in software recruitment, favoring male and Caucasian candidates. Parziale et
Biased or Personalized? The Impact of Personal Information on AI-driven Development
al. [56] find that candidates’ demographics influence LLM hiring decisions and software task assignment. Closet to our work, Janzen et al. [30] shows that users’ gender shapes prompting style and examine how these differences influence LLM behavior in code generation and code review tasks. Our work extends this literature in two ways. First, we conduct controlled experiments with direct indicators of gender and age across multiple aspects of generated software artifacts; prior work looks only at runtime behavior, single functions, or prompting patterns correlated with gender. Second, we examine users’ perceptions to understand the distinction between helpful personalization and harmful bias. Human-LLM Interaction. Other research investigates how users interact with LLMs, examining prompting and its impact on outcomes [2, 31, 35, 36, 47, 57, 71]. For coding, Pimenova et al. [57] explore why and how users adopt vibe coding, where it falls short, and what practices are emerging to support it. Other papers explore the impact of relying heavily on LLMs on creativity [2, 35, 47], showing that using LLMs across different task stages can lead to more homogeneous outputs. To understand how users’ demographic information influences model outputs, it is first necessary to examine what personal information users share. Recent work [23, 45, 52, 53, 75, 77] has studied the disclosure of personal information to chatbots in everyday use. Mireshghallah et al. [45] analyze real-world chatbot interactions and find that over 70% of queries contain some form of personally identifiable information (PII), even in tasks where it is unnecessary, such as translation or code editing in which users disclose PII in approximately 48% and 16% of queries, respectively. Other works propose strategies for identifying PII in prompts or propose interventions to help users become more aware of the information they share and assist them in removing personal information when necessary[52, 53, 77]. Our work builds on this work by examining how PII can impact generated software artifacts through an observational user study, aiming to better understand programmers’ perceptions of this behavior and to clarify the boundary between personalization and bias.
10
Conclusion
AI coding assistants are increasingly personalized, but little is known about how developer characteristics influence generated software artifacts. Through a series of controlled persona-based experiments with AI-generated websites, we showed that the age and gender of the prompter can significantly and substantially influence AI-generated software across interface design, template content, and code structure. Our user study found that while participants were often able to recognize personalized or potentially biased content, they rarely noticed differences in higher-level design decisions, such as website layout or code structure. We hope this work motivates future research on the appropriate role of personalization in AI-assisted software development, including which aspects of software artifacts should adapt to users’ demographic characteristics and which should remain independent of them.
References [1] Ahmed Al Kuwaiti, Khalid Nazer, Abdullah Al-Reedy, Shaher Al-Shehri, Afnan Al-Muhanna, Arun Vijay Subbarayalu, Dhoha Al Muhanna, and Fahad A AlMuhanna. 2023. A review of the role of artificial intelligence in healthcare. Journal of personalized medicine 13, 6 (2023), 951. [2] Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. 2024. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th conference on creativity & cognition. 413–425. [3] Oyebola Olusola Ayeni, Nancy Mohd Al Hamad, Onyebuchi Nneamaka Chisom, Blessing Osawaru, and Ololade Elizabeth Adewusi. 2024. AI in education: A review of personalized learning and educational technology. GSC Advanced Research and Reviews 18, 2 (2024), 261–271. [4] Muneera Bano, Hashini Gunatilake, and Rashina Hoda. 2025. What does a software engineer look like? Exploring societal stereotypes in LLMs. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 173–184. [5] Suborno Deb Bappon, Saikat Mondal, and Banani Roy. 2024. Autogenics: Automated generation of context-aware inline comments for code snippets on programming q&a sites using llm. In 2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 24–35. [6] Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological) 57, 1 (1995), 289–300. [7] Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beckwith, Irwin Kwan, Anicia Peters, and William Jernigan. 2016. GenderMag: A method for evaluating software’s gender inclusiveness. Interacting with computers 28, 6 (2016), 760–787. [8] Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li. 2023. GPTutor: a ChatGPT-powered programming tool for code explanation. In International conference on artificial intelligence in education. Springer, 321–327. [9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). [10] Sapna Cheryan, Sianna A Ziegler, Amanda K Montoya, and Lily Jiang. 2017. Why are some STEM fields more gender balanced than others? Psychological bulletin 143, 1 (2017), 1. [11] William G Cochran. 1954. Some methods for strengthening the common 𝜒 2 tests. Biometrics 10, 4 (1954), 417–451. [12] Nicole M Deterding and Mary C Waters. 2021. Flexible coding of in-depth interviews: A twenty-first-century approach. Sociological methods & research 50, 2 (2021), 708–739. [13] Yongkang Du, Jen-tse Huang, Jieyu Zhao, and Lu Lin. 2025. Faircoder: Evaluating social bias of llms in code generation. arXiv preprint arXiv:2501.05396 (2025). [14] Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. A comparative analysis of large language models for code documentation generation. In Proceedings of the 1st ACM international conference on AI-powered software. 65–73. [15] Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53. [16] Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1469–1481. [17] Xi Fang, Weijie Xu, Yuchong Zhang, Scott Nickleach, Stephanie Eckman, and Chandan K Reddy. 2026. The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 511–529. [18] Samuel Ferino, Rashina Hoda, John Grundy, and Christoph Treude. 2025. Novice developers’ perspectives on adopting llms for software development: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2025). [19] GH Freeman and John H Halton. 1951. Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika 38, 1/2 (1951), 141–149. [20] Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational linguistics 50, 3 (2024), 1097–1179. [21] Yuyao Ge, Lingrui Mei, Zenghao Duan, Tianhao Li, Yujia Zheng, Yiwei Wang, Lexin Wang, Jiayu Yao, Tianyu Liu, Yujun Cai, et al. 2025. A survey of vibe coding with large language models. arXiv preprint arXiv:2510.12399 (2025). [22] Francis Geng, Anshul Shah, Haolin Li, Nawab Mulla, Steven Swanson, Gerald Soosai Raj, Daniel Zingaro, and Leo Porter. 2025. Exploring Student-AI Interactions in Vibe Coding. arXiv preprint arXiv:2507.22614 (2025).
Erfan Entezami and Madeline Endres
[23] Ece Gumusel. 2025. A literature review of user privacy concerns in conversational chatbots: A social informatics approach: An Annual Review of Information Science and Technology (ARIST) paper. Journal of the Association for Information Science and Technology 76, 1 (2025), 121–154. [24] Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. 2023. Longcoder: A long-range pre-trained language model for code completion. In International Conference on Machine Learning. PMLR, 12098–12107. [25] Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, Vienna, Austria. https://openreview.net/forum?id=kGteeZ18Ir [26] Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246 (2025). [27] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79. [28] Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2025. Bias testing and mitigation in llm-based code generation. ACM Transactions on Software Engineering and Methodology 35, 1 (2025), 1–31. [29] Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. Language models for code completion: A practical evaluation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [30] Lynn Janzen, Üveys Eroglu, Dorothea Kolossa, Pia Knöferle, Sebastian Möller, Vera Schmitt, and Veronika Solopova. 2026. Gendered Prompting and LLM Code Review: How Gender Cues in the Prompt Shape Code Quality and Evaluation. arXiv preprint arXiv:2603.24359 (2026). [31] Andrew Jelson, Daniel Manesh, Alice Jang, Daniel Dunlap, Young-Ho Kim, and Sang Won Lee. 2026. An empirical study to understand how students use ChatGPT for writing essays. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–26. [32] Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T Tan. 2026. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 19416–19448. [33] Mahammed Kamruzzaman, Md Shovon, and Gene Kim. 2024. Investigating subtler biases in LLMs: Ageism, beauty, institutional, and nationality bias in generative models. In Findings of the Association for Computational Linguistics: ACL 2024. 8940–8965. [34] Anjali Kantharuban, Jeremiah Milbauer, Maarten Sap, Emma Strubell, and Graham Neubig. 2025. Stereotype or personalization? user identity biases chatbot recommendations. In Findings of the Association for Computational Linguistics: ACL 2025. 24418–24436. [35] Harsh Kumar, Jonathan Vincentius, Ewan Jordan, and Ashton Anderson. 2025. Human creativity in the age of llms: Randomized experiments on divergent and convergent thinking. In Proceedings of the 2025 CHI conference on human factors in computing systems. 1–18. [36] Zhuoyang Li, Zihao Zhu, Xinning Gui, and Yuhan Luo. 2025. “This is human intelligence debugging artificial intelligence”: Examining how people prompt GPT in seeking mental health support. International Journal of Human-Computer Studies 203 (2025), 103555. [37] Anda Liang, Emerson Murphy-Hill, Westley Weimer, and Yu Huang. 2024. A controlled experiment in age and gender bias when reading technical articles in software engineering. IEEE Transactions on Software Engineering 50, 10 (2024), 2498–2511. [38] Lin Ling, Fazle Rabbi, Song Wang, and Jinqiu Yang. 2025. Bias Unveiled: Investigating Social Bias in LLM-Generated Code. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2025, Philadelphia, PA, USA, February 25 - March 4, 2025, Toby Walsh, Julie Shah, and Zico Kolter (Eds.). AAAI Press, 27491–27499. doi:10.1609/AAAI.V39I26.34961 [39] Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. Repobench: Benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations, Vol. 2024. 47832–47850. [40] Yan Liu, Xiaokang Chen, Yan Gao, Zhe Su, Fengji Zhang, Daoguang Zan, JianGuang Lou, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Uncovering and quantifying social biases in code generation. Advances in Neural Information Processing Systems 36 (2023), 2368–2380. [41] Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2026. Webgenbench: Evaluating llms on generating interactive and functional websites from scratch. Advances in Neural Information Processing Systems 38 (2026). [42] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder:
Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568 (2023). [43] Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60. [44] Christian Meske, Tobias Hermanns, Esther Von der Weiden, Kai-Uwe Loser, and Thorsten Berger. 2025. Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda. IEEE Access 13 (2025), 213242–213259. [45] Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, and Golnoosh Farnadi. 2024. Trust no bot: Discovering personal disclosures in human-llm conversations in the wild. arXiv preprint arXiv:2407.11438 (2024). [46] Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim, and Kyomin Jung. 2026. Don’t judge code by its cover: Exploring biases in llm judges for code evaluation. In Findings of the Association for Computational Linguistics: EACL 2026. 1364–1389. [47] Kibum Moon, Adam E Green, and Kostadin Kushlev. 2025. Homogenizing effect of large language models (LLMs) on creative diversity: An empirical comparison of human and ChatGPT writing. Computers in Human Behavior: Artificial Humans (2025), 100207. [48] Emerson Murphy-Hill, Alberto Elizondo, Ambar Murillo, Marian Harbach, Bogdan Vasilescu, Delphine Carlson, and Florian Dessloch. 2024. Gendermag improves discoverability in the field, especially for women: An multi-year case study of suggest edit, a code review feature. In Proceedings of the IEEE/ACM 46th international conference on software engineering. 1–12. [49] Ranjita Naik and Besmira Nushi. 2023. Social biases through the text-to-image generation lens. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 786–808. [50] Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [51] Vera Neplenbroek, Arianna Bisazza, and Raquel Fernández. 2025. Reading between the prompts: How stereotypes shape llm’s implicit personalization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 20378–20411. [52] Mohammad Hadi Nezhad, Francisco Enrique Vicente Castro, and Ivon Arroyo. 2026. Understanding Users’ Privacy Reasoning and Behaviors During Chatbot Use to Support Meaningful Agency in Privacy. arXiv preprint arXiv:2601.18125 (2026). [53] Ivoline C Ngong, Swanand Ravindra Kadhe, Hao Wang, Keerthiram Murugesan, Justin D Weisz, Amit Dhurandhar, and Karthikeyan Natesan Ramamurthy. 2025. Protecting users from themselves: Safeguarding contextual privacy in interactions with conversational agents. In Findings of the Association for Computational Linguistics: ACL 2025. 26196–26220. [54] OpenAI. [n. d.]. Memory and New Controls for ChatGPT. https://openai.com/ index/memory-and-new-controls-for-chatgpt/ [55] Lyndsey O’Brien, Tanjila Kanij, and John Grundy. 2025. Assessing gender bias in the software used in computer science and software engineering education. Journal of Systems and Software 219 (2025), 112225. [56] Alessandra Parziale, Gianmario Voria, Valeria Pontillo, Amleto Di Salle, Patrizio Pelliccione, Gemma Catolino, and Fabio Palomba. 2026. Once Upon a Team: Investigating Bias in LLM-Driven Software Team Composition and Task Allocation. arXiv preprint arXiv:2601.03857 (2026). [57] Veronica Pimenova, Sarah Fakhoury, Christian Bird, Margaret-Anne Storey, and Madeline Endres. 2025. Good vibrations? A qualitative study of co-creation, communication, flow, and trust in vibe coding. arXiv preprint arXiv:2509.12491 (2025). [58] Zhanyue Qin, Haochuan Wang, Zecheng Wang, Deyuan Liu, Cunhang Fan, Zhao Lv, Zhiying Tu, Dianhui Chu, and Dianbo Sui. 2024. Mitigating gender bias in code large language models via model editing. arXiv preprint arXiv:2410.07820 (2024). [59] Gema Rodríguez-Pérez, Reza Nadri, and Meiyappan Nagappan. 2021. Perceived diversity in software engineering: a systematic literature review. Empirical Software Engineering 26, 5 (2021), 102. [60] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). [61] Ranjan Sapkota, Konstantinos I Roumeliotis, and Manoj Karkee. 2025. Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic ai. arXiv preprint arXiv:2505.19443 (2025). [62] Advait Sarkar and Ian Drosos. 2025. Vibe coding: programming through conversation with artificial intelligence. arXiv preprint arXiv:2506.23253 (2025). [63] Social Security Administration. 2026. Popular Baby Names. https://www.ssa. gov/oact/babynames/index.html. Accessed: 2026-05-29. [64] Ezekiel Soremekun, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2025. Software fairness: An analysis and survey. Comput. Surveys 58, 3 (2025), 1–38.
Biased or Personalized? The Impact of Personal Information on AI-driven Development
[65] Stack Overflow. 2025. 2025 Stack Overflow Developer Survey. Accessed June 2026. https://survey.stackoverflow.co/2025 [66] Manuel Tonneau, Neil KR Seghal, Niyati Malhotra, Sharif Kazemi, Victor OrozcoOlvera, Ana María Muñoz Boudet, Lakshmi Subramanian, Samuel P Fraiberger, Sharath Chandra Guntuku, and Valentin Hofmann. 2026. Different demographic cues yield inconsistent conclusions about LLM personalization and bias. Preprint (2026). [67] Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-ajudge in software engineering. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977. [68] Kai Xu, YiWei Mao, XinYi Guan, and ZiLong Feng. 2025. Web-bench: A llm code benchmark based on web standards and frameworks. arXiv preprint arXiv:2505.07473 (2025). [69] Dayu Yang, Antoine Simoulin, Xin Qian, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, and Grey Yang. 2025. Docagent: A multi-agent system for automated code documentation generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). 460–471. [70] Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2025. Justice or prejudice? quantifying biases in llm-as-a-judge. In International Conference on Learning Representations, Vol. 2025. 102351–102390. [71] Bhada Yun, Renn Su, and April Yi Wang. 2026. AI and My Values: User Perceptions of LLMs’ Ability to Extract, Embody, and Explain Human Values from Casual Conversations. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. 1–38.
[72] Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2024. Pydex: Repairing bugs in introductory python assignments using llms. Proceedings of the ACM on Programming Languages 8, OOPSLA1 (2024), 1100–1124. [73] Quanjun Zhang, Chunrong Fang, Yang Xie, YuXiang Ma, Weisong Sun, Yun Yang, and Zhenyu Chen. 2024. A systematic literature review on large language models for automated program repair. ACM Transactions on Software Engineering and Methodology (2024). [74] Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Shengcheng Yu, Weisong Sun, Yun Yang, and Zhenyu Chen. 2026. A survey on large language models for software engineering. Science China Information Sciences 69, 4 (2026), 141102. [75] Zhiping Zhang, Michelle Jia, Hao-Ping Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li. 2024. “It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26. [76] Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an understanding of large language models in software engineering tasks. Empirical Software Engineering 30, 2 (2025), 50. [77] Jijie Zhou, Eryue Xu, Yaoyao Wu, and Tianshi Li. 2025. Rescriber: Smaller-LLMPowered User-Led Data Minimization for LLM-Based Chatbots. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–28. [78] Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity. arXiv preprint arXiv:2301.12867 (2023).