arXiv CSopen access

How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs

Graphical Abstract How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs Martin Balfroid, Julien Albert, Dzenatan Aliti, Xavier Devroey, Benoît Vanderose

2

1

GitHub

arXiv:2607.26987v1 [cs.SE] 29 Jul 2026

GitHub Actions

3

Collect Stacktraces

Collect bug-fixes

Non-Flaky, Offline Reproducible Bug-Fix Commits

4

Generate Code Tours

Stacktraces

Annotate Code Tours

5 Rationale

Comment Annotations

Rating

Code Tours

Agreement

Annotations Author Qwen2.5-Coder 14B (12) Deepseek-Coder-V2 16B (7) Devstral Small 1.1 24B (7)

Judge (14) Qwen2.5-Coder 14B (19) Deepseek-Coder-V2 16B (19) Devstral Small 1.1 24B

Thoughts

Opinions Developer

Highlights How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs Martin Balfroid, Julien Albert, Dzenatan Aliti, Xavier Devroey, Benoît Vanderose • Developers share preferences in terms of level of detail, text structure, and tone. • Developers seldom have diverging preferences (e.g., imperative mood). • The stack trace steps are not enough for step selection. • Assuming human authorship risks misuse, while assuming AI authorship risks disuse. • LLM judges show sycophancy, confabulation, and incoherence issues when evaluating code tours.

How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs Martin Balfroid∗ , Julien Albert, Dzenatan Aliti, Xavier Devroey and Benoît Vanderose ARTICLE INFO

ABSTRACT

Keywords: Onboarding Code Summarization LLM-as-a-judge Semi-structured Interview Open-weight LLMs

Context: Code tours are interactive, onboarding documentation that guide developers through a codebase. Large Language Models (LLMs) can automatically synthesize code tours. Prior work on code tour generation has not examined developer experience or trust calibration when debugging unfamiliar codebases with code tours generated and evaluated by open-weight LLMs. Objectives: This study surveys how the properties of components in open-weight LLM-authored code tours influence developers’ experiences when debugging unfamiliar codebases. Method: We built a pipeline that generated and evaluated code tours from real reproducible bugs. Twenty-six developers with varying backgrounds participated in a user study. In total, 26 code tours were authored from real Java bugs mined from 2025 GitHub commits, with each tour independently judged by two different LLMs, resulting in 52 evaluated configurations. Participants thought aloud as they explored each tour. Three authors qualitatively coded the interviews to identify recurring themes. Results: Developers generally preferred tours that scaled detail with the code length, avoided merely restating code, were easily scannable, and adopted a guiding tone. However, some preferences were mutually exclusive, such as the use of imperative mood. Stack traces were often insufficient to identify all steps developers found relevant. Developers also trusted descriptions they perceived as humanwritten more than those they believed were AI-generated. Finally, LLM-generated annotations of tour quality were unreliable: sycophancy, confabulation, and incoherence were pervasive. Conclusion: This work lays a basis for future research on fine-tuning open-weight models for code tour generation, personalizing generation to accommodate diverging preferences, selecting relevant steps beyond stack traces, calibrating users’ trust to avoid both disuse and misuse, and improving open-weight LLMs’ ability to be more trustworthy evaluators

1. Introduction Onboarding to an unfamiliar codebase is something every developer has to experience. This can be done through mentoring [18] by developers familiar with the codebase, but they can spend around a third of their time mentoring new developers [1]. Generative AI could offload by generating interactive, context-rich documentation, such as code tours [4, 2, 20], which support onboarding [41]. A code tour [6] consists of sequential steps, each linked to a code segment and accompanied by a summary. Figure 1 illustrates a tour designed to help developers understand a NullPointerException in Apache Commons Lang, a widely used Java library for string manipulation. (1) The first step explains the failing test testLabelFormat, which checks label formatting but throws a NullPointerException. (2) The next step leads us to the render function, which delegates string formatting to a LabelFormatter. (3) The final step identifies the faulty format method, which delegates label formatting to an uninitialized Formatter, resulting in a NullPointerException. Prior works on code tour generation [4, 20] have not investigated how developers interact with code tours to achieve ⋆

This research was supported by the ARIAC project (No. 2010235), funded by the Service Public de Wallonie (SPW Recherche). We gratefully acknowledge the participants for their valuable contributions to this study. ∗ Corresponding author [email protected] (M. Balfroid) ORCID (s): 0000-0002-1318-1184 (M. Balfroid); 0000-0001-6279-5601 (J. Albert)

Balfroid et al.: Preprint submitted to Elsevier

their goals (e.g., understanding, navigating, acting) in an unfamiliar codebase. This paper, therefore, investigates the following research question: How do the properties of a code tour’s components affect the developer experience, with respect to the goals developers pursue when fixing a bug in an unfamiliar codebase? To answer this question, we conducted semi-structured interviews with 26 developers with varying levels of professional and Java experience. Each participant was confronted with six synthetically generated code tours of real Java bugs mined from GitHub commits [35] from 2025. Each tour was authored by one LLM and evaluated by the two other LLMs as judges [48, 43] on three criteria [4]: (Transparency) understanding the codebase; (Scrutability) assessing the correctness and relevance of the explanation; (Efficiency) performing the task efficiently. Participants were asked to rate each rationale on a 1–7 scale and, importantly, to think aloud [22] while doing so. Three independent coders (the first three authors) then performed qualitative coding to extract recurring experiences from the transcripts. The experiences were formulated in first person (following Lejeune [25]) and were further decomposed into four components: the goal (e.g., understanding), the valence (positive, mixed, or negative), the tour’s component it relates to (e.g., description), and the component’s property that affects the goal (e.g., the level of detail in the description).

Page 1 of 22

Human Factors in AI-generated Code Tours StatisticalBarRendererTest.java – testLabelFormat()

public class RendererTest extends TestCase { public void testLabelFormat () { StatisticalBarRenderer renderer = new StatisticalBarRenderer () ; renderer . render () ; } }

Step 1/3 This step initiates the trace within RendererTest.java. The testLabelFormat unit test is designed to verify that the StatisticalBarRenderer correctly processes label formatting. Currently, this test triggers a NullPointerException. We will now examine the render() method to identify the point of failure.

StatisticalBarRenderer.java – render()

public class StatisticalBarRenderer { private LabelFormatter label = new LabelFormatter () ; public void render () { double mean = calculateMean () ; double std = calculateStdDev () ; String fmt = label . format ( " % f + -% f " , mean , std ) ; System . out . println ( fmt ) ; } }

Step 2/3 We are now inside the render() implementation of StatisticalBarRenderer.java. The method calculates the mean and standard deviation before delegating string formatting to the LabelFormatter instance. The execution flow encounters an issue during the call to label.format(). We will step into this method to inspect the internal state of the formatter.

StringFormatter.java – format()

public class LabelFormatter { private java . util . Formatter fmtr ; public String format ( String fmt , Object ... args ) { return this . fmtr . format ( fmt , args ) . toString () ; } }

Step 3/3 The root cause is located within LabelFormatter.java. The format method attempts to delegate the operation to the private member fmtr. However, fmtr is declared but remains uninitialized. This results in a NullPointerException when this.fmtr.format() is invoked. To resolve this, fmtr must be properly instantiated within the constructor or at the point of declaration.

Figure 1: The tutorial code tour. It is designed to help a developer understand a NullPointerException in Apache Commons Lang, a popular Java library for string manipulation, including formatting. It is an interactive document that guides the developer from the failing test case through the rendering logic to the uninitialized variable responsible for the crash.

2. Background Onboarding is the process through which new employees become active members of an organization [5]. New developers usually encounter obstacles such as limited documentation, unfamiliar workflows, and new technologies [29]. Code tours are one of the many software solutions for onboarding [37]: they are structured, in-IDE walkthroughs of key code segments that guide new developers through a

Balfroid et al.: Preprint submitted to Elsevier

codebase interactively [41]. As a form of structured documentation, code tours directly address a lack of documentation, one of the primary onboarding obstacles as identified by Matturro et al. [29]. Beyond documentation, code tours also reduce the burden on senior developers. Indeed, mentorship is a common approach [18], but it can consume up to 30% of the mentor time in some large projects [38] and 10-20 hours per hire [20]. Rather than repeating the same walkthrough for each new hire, code tours crystallize an expert’s walkthrough of the codebase into a reusable artifact [20], allowing it to be authored once and reused across newcomers. Table 1 positions this work relative to Page 2 of 22

Human Factors in AI-generated Code Tours Table 1 Positioning of this work relative to prior work on code tours. No prior study has jointly evaluated fully AI-generated code tours (AI) using open-weight models (OW), with attention to developer experience (DX), trust calibration (Trust), a debugging focus (Debug), and LLM-as-a-judge evaluation (Judge) within the pipeline. ✓ indicates important coverage, ∼ indicates partial coverage (mentioned but not empirically evaluated), and an empty cell indicates no coverage. Study

AI

OW

DX

Taylor and Clarke [41] (2022) Balfroid et al. [4] (2024) Balfroid [2] (2025) Kara et al. [20] (2026)

✓ ∼ ∼

∼ ∼

∼

This work

✓

✓

✓

Trust

Debug

Judge

✓ ∼ ∼

∼

✓

✓

✓

✓

prior work on code tours [41, 4, 2, 20] and highlights the gap that no prior study has jointly evaluated fully AI-generated code tours (AI, Section 2.1) using open-weight models (OW, Section 2.2), with attention to developer experience (DX, Section 2.3), trust calibration (Trust, Section 2.4), a debugging focus (Debug, Section 2.5), and LLM-as-a-judge evaluation (Judge, Section 2.6) within the pipeline.

2.1. AI-generated Code Tours CodeTour [6] is a VS Code extension for creating and navigating guided tours of a project within the IDE, traditionally authored by hand as a sequence of annotated code locations in JSON. Although research on code tour generation is sparse [41, 4, 2, 20], a code tour is essentially a sequence of linked code summaries, i.e., natural-language descriptions of code snippets [47]. Code summarization typically follows a three-step process – modeling the source code, generating summaries, and evaluating the quality of the summaries – and is currently dominated by machinelearning-based techniques [47]. Here we introduce a few notable prior works that will feed the discussion (Section 5). In a large study (∼1000 participants), Leinonen et al. [24] found that students rated GPT-3-generated code summaries as more understandable and accurate than student-authored ones (though the effect was small). There was no difference in length – either objectively (number of characters) or subjectively (participant ratings) – between student-authored and AI-authored descriptions. MacNeil et al. [28] studied three LLM-generated explanation types – line-by-line, key concepts list, high-level summary – within an interactive e-book for a web development course (58 participants). Reading time increased with code complexity, and explanations were most valued when students were unfamiliar with the code. The study was underpowered to conclude that perceived usefulness differed statistically significantly across explanation types. Looking at descriptive statistics: summary and concepts (median of 4) might be perceived as more useful than line-by-line (median of 3.5), though to an extent that is too small to be noticed with 58 participants. Now, back to code tours, Balfroid et al. [4] proposed automatically generating code tours directly from stack traces. Thus, using the trace to select code locations and an LLM to Balfroid et al.: Preprint submitted to Elsevier

author the explanations. Lacy [20] extends this into a hybrid human-AI system. They shift away from a fully AI-generated pipeline by introducing a voice-to-tour feature, in which experts conduct a live walkthrough with an onboardee. Then, AI summarizes it into a reusable tour for newcomers. Although expert-curated code tours lead to measurably better learner comprehension than AI-only tours, developers still perceive fully AI-authored content as sufficient, failing to recognize what is missing. Expert-curated and AI-generated tours are nonetheless considered complementary [20], making the study of AI-generated tours relevant. Although we agree with Kara and İsmail et al. [20] that, ideally, a code tour should be curated by an expert. In practice, this might not be feasible due to the scale. Also, there may not even be any experts involved when it comes to legacy or AIgenerated code. Thus, we continue the line of work of fully AI-generated tours [4, 2].

2.2. Open Weight (OW) for Code Tour Generation Prior work on code tour generation [4, 20] has predominantly relied on LLMs accessed through proprietary external APIs, such as GPT-3.5 and Gemini 2.5 Flash. This reliance raises significant concerns about data privacy, as sensitive source code must be transmitted to third-party servers [1], and about reproducibility, as the opacity of cloud-based models impedes the ability to replicate and verify experimental results [36]. A promising alternative lies in open-weight LLMs, i.e., Large Language Models whose parameters are publicly available. Thus, they can be deployed locally, thereby eliminating exposure of data to third parties and granting full control over the inference pipeline [2]. Although generally smaller in scale, these models have demonstrated competitive performance against their larger, API-based counterparts, particularly in low-resource settings [45], suggesting that local deployment does not sacrifice much in terms of generation quality. In this study, we will consider three open-weight models released in 2025 that specialize in software engineering: Q WEN2.5-CODER 14B, D EEPSEEK-CODER-V2, and D EVSTRAL S MALL 1.1. There are several benchmarks to assess models’ capabilities on tasks. To compare the selected models, we selected two benchmarks that together cover complementary capabilities relevant to generating debuggingfocused code tours: code editing and code comprehension via input/output prediction. AIDER [13] is designed to measure a model’s ability to edit Python code across 133 small exercises that include markdown instructions, a stub Python code that specifies the functions or classes to be implemented, and unit tests. The task is to implement the provided function and class, following the instructions to pass the unit tests. The output is the candidate code to edit the benchmark. There are multiple configurations of edit formats: (i) the whole format, where the entire file is returned; (ii) the diff format, where each edit only provides the change (thus this is more token efficient). CRUXEval [14] (Code Reasoning, Understanding, and eXecution Evaluation) assesses code comprehension through 800 short Python functions, each Page 3 of 22

Human Factors in AI-generated Code Tours

paired with a known input–output example. It comes in two variants: input prediction, where the model must infer an input based on a given output, and output prediction, where the model must predict the output of running the function on a given input. Section 3.4 provides a comparison of the architectures and capabilities of the three models.

2.3. Developer Experience (DX) with Code Tours Taylor and Clarke [41] conducted a controlled experiment with 15 participants: an experimental group (7) received two hand-crafted code tours, while the control group (8) did not. They found that hand-crafted tours help newcomers navigate and understand a codebase. Although they reported participants’ thoughts, they did not perform a systematic qualitative analysis of these excerpts. Kara et al. [20] likewise did not analyze developer experiences in depth, instead observing developers to derive a set of design requirements for Lacy. In contrast, we conduct a systematic qualitative analysis of developer experience (Section 3.6).

2.4. Trust Calibration Trusting an automation system, per Lee & See [23], means believing it will help you achieve your goal in an uncertain and vulnerable situation, which is the case when you are onboarding a new code base. However, not all systems are perfect, so trust should be calibrated to avoid misuse (over-reliance on unreliable automation) and disuse (rejection of capable automation) [23]. Kara et al. [20] noted that developers tended to perceive fully AI-authored content as sufficient without recognizing what is missing. This signals a risk of misuse: over-reliance on unreliable automation [23]. In this study, we investigate more deeply which properties of the code tours components affect the trust of developers (Section 4.5) and discuss how we can better calibrate trust of developers towards fully AI-generated code tours (Section 5.1.4, Section 5.1.5).

2.5. Debugging for Onboarding Developers reported greater efficiency when experimenting with code rather than passively reading documentation [8]. Newcomers can experiment through small tasks such as fixing bugs, adding simple features, or writing unit tests. Building on this, Balfroid et al. [4] focused on generating code tours to explain stack traces. A stack trace records the sequence of nested method calls that lead to an exception, thereby identifying relevant steps for a debugging-focused code tour [2]. While Lacy [20] has shifted from a purely debugging-oriented approach, we continue to focus on debugging because this narrows the study’s scope and makes step selection straightforward using stack traces.

2.6. LLM-as-a-judge AI-generated tours have shortcomings [4] that may go unnoticed by an overtrusting developer [20], raising the question of how to verify the quality of code tours. Expert curation is infeasible at scale. Moreover, for legacy or AIgenerated code, an expert may not even exist. Since code tour Balfroid et al.: Preprint submitted to Elsevier

generation is an open-ended task, it is difficult to evaluate. The LLM-as-a-Judge paradigm[48] addresses this by using LLMs to perform such evaluations. In software engineering, for instance, Weyssow et al. [44] apply this approach to assess alignment with non-functional requirements—such as readability, complexity, and style—in code tasks. While Balfroid [2] proposed using LLM-as-a-judge to evaluate code tours specifically, they did not empirically evaluate it. To the best of our knowledge, this work is the first to investigate how developers experience LLM-generated code tour evaluations.

3. Evaluation Setup 3.1. Overview Now that we have reviewed the literature on code tours, it is time to present our evaluation setup for the code tour generation pipeline, the semi-structured interview, and the qualitative coding to investigate how the different properties of the components of a code tour impact the developer experience. Figure 2 summarizes the pipeline from the bug collection to developer feedback. First (Section 3.2), we collect reproducible bug-fix commits using the Gitbug-Actions pipeline [35]. Second (Section 3.3), for each bug, we run the failing tests to collect stack traces. Third, given the stack trace and the corresponding code context, an LLM generates a structured code tour that explains the execution frames (Section 3.5.1). Fourth, the two remaining LLMs act as judges, assigning ratings and rationales to the generated code tours based on 3 quality criteria (Section 3.5.2). This makes 2 annotations per code tour. Finally (Section 3.6), 26 professional developers freely share their experiences of how the various components of a code tour (including annotations) affect them (Section 3.7).

3.2. Bug Mining Balfroid et al. [4] showed that it was possible to generate synthetic code tours with GPT-3.5 (closed-weight model) to explain the stack trace between a failing test and a faulty method for debugging, a common onboarding task [8]. They collected stack traces from Defects4j [19], a collection of 357 reproducible bugs, by instrumenting the faulty methods to record the frames at method entry, then executing the failing test to record the stack trace. However, this dataset raises concerns about data leakage [36]: given LLMs’ extensive training on vast internet corpora, if a source predates the training cutoff, the LLM is likely to have encountered it, biasing results. Gitbug-Actions [35] is a tool for building up-to-date bug-fix benchmarks, specifically designed to address this problem. This team released GitBug-Java [39], a bug-fix benchmark of fully reproducible bugs from 2023. Nonetheless, as of 2025, this benchmark is already outdated for recent LLMs. Therefore, we analyzed 17,284 recent commits (i.e., committed in 2025) using Gitbug-Actions [35] across 547 Java projects. The mining process resulted in the collection of 110 bugs across 30 projects that can be Page 4 of 22

Human Factors in AI-generated Code Tours

2

1

GitHub

GitHub Actions

3

Collect Stacktraces

Collect bug-fixes

Non-Flaky, Offline Reproducible Bug-Fix Commits

4

Generate Code Tours

Stacktraces

Annotate Code Tours

5 Rationale

Rating

Code Tours

Agreement

Annotations Author Qwen2.5-Coder 14B (12) Deepseek-Coder-V2 16B (7) Devstral Small 1.1 24B (7)

Thoughts

Comment Annotations

Opinions Developer

Judge (14) Qwen2.5-Coder 14B (19) Deepseek-Coder-V2 16B (19) Devstral Small 1.1 24B

Figure 2: Pipeline: (1) collecting reproducible bugs using Gitbug-Actions pipeline [35], (2) executing failing tests to produce candidate stack traces, (3) generating code tour files directly from the stack trace using one of the LLMs as an author, (4) generating annotations — a chain-of-thoughts rationale (text) and a rating (Likert-7) — of code tours for each criterion (Transparency, Efficiency, and Scrutatibility) by each remaining LLMs as judges and (5) collecting developers opinions — excerpts of their thoughts (text) and their agreement (Likert-7) — on code tours and related annotations. (Icons are from flaticon.com)

Figure 3: Information coming from GitBug-Java for ezylang-EvalEx-942ad41ef07c where the first strack trace is the basis of code tour 6. ### Failing Tests - com . ezylang . evalex . functions . datetime . DateTimeFunctionsTest # testDateTimeNewNoParam - java . lang . AssertionError Expecting actual throwable to be an instance of : com . ezylang . evalex . EvaluationException but was : [ FAIL ] DateTimeFunctionsTest # testDateTimeNewNoParam Expected : com . ezylang . evalex . EvaluationException Actual : java . lang . ArrayIndexOutOfBoundsException : Index -1 out of bounds for length 0 at DateTimeNewFunction . validatePreEvaluation ( DateTimeNewFunction . java :92) at Expression . evaluateFunction ( Expression . java :189) at Expression . evaluateSubtree ( Expression . java :147)

-- git a /.../ DateTimeNewFunction . java b /.../ DateTimeNewFunction . java -- a /.../ DateTimeNewFunction . java ++ b /.../ DateTimeNewFunction . java -80 ,6 +80 ,10 @@ int parameterLength = parameterValues . length ; if ( parameterLength == 0) { throw new EvaluationException ( token , " Not enough parameters for function ") ; } if ( parameterLength == 1) { if (! parameterValues [0]. isNumberValue () ) { throw new EvaluationException (

(b) Information about the patch in diff.

(a) Information about failing tests and stack traces, full stack traces are captured dynamically.

reproduced offline and reliably across repetitions (nonflaky).

3.3. Stack trace Collection Balfroid et al. [4] used CodeQL to extract code segments corresponding to stack frames. CodeQL is a query language à la SQL for code, performing both syntactic and semantic analysis: it builds an augmented AST and stores it in a dedicated database, which can then be queried via a proprietary DSL. The database creation step is computationally expensive and is occasionally prone to failure (the authors reported 7 failures in their study), although queries themselves are reasonably efficient. We instead use Tree-sitter to parse concrete syntax trees. This approach is significantly lighter computationally. Nevertheless, syntax trees differ in structure across languages. So, the visitor logic used to extract code segments must be reimplemented for each target language. On the other hand, CodeQL is easily portable across languages thanks to its community, which implements visitors for popular languages. We thus trade portability for efficiency. Balfroid et al.: Preprint submitted to Elsevier

We collect stack traces by running the failing tests of the 110 bugs mined using the GitBug-Java CLI [39]. Each bug is represented as a buggy/fixed commit pair, as given by the gitbug-java info command. For instance, Figure 3 shows the information for the commit 942ad41ef07c that fixes Issue 527 of EvalEx related to a lack of handling of the evaluate function with no parameters. To identify failing tests, we use the info command to generate a list of test names that fail on the buggy commit, then parse the output to extract the individual test identifiers (Figure 3a). For each test, we run it to capture its full resulting stack trace. To determine faulty methods, we use Tree-sitter to parse the source code and extract all method definitions. We then match each method’s body against the code diffs between the buggy and fixed commits, identifying any method whose body spans a diff hunk shared between them (Figure 3b). A single bug can therefore result in multiple stack traces, for example, when several tests fail or when the fix spans across several methods. Meanwhile, some bugs produce no stack traces at all.

Page 5 of 22

Human Factors in AI-generated Code Tours Table 2 Illustrates the architectural and capability differences among the three models. Model

Parameters

Qwen2.5-Coder 14B [17] DeepSeek-Coder-V2 [49] Devstral Small 1.1 [32]

14B 16B 24B

Knowledge Cutoff

Context Window

Size

Quantization

Nov 2024 [11] Nov 2023 [42] Oct 2023 [10]

32K 160K 128K

9 GB 8.9 GB 15 GB

Q4_K_M Q4_0 Q4_K_M

(a) Describe the architectural differences between the models. Knowledge cutoff is an estimate of the date at which the model did not train on new data. Context Window is the number of tokens (in thousands, K) that can be given as input. Size is the amount of RAM, in GB, that the models occupy. Quantization is the process of compressing model weights for making LLMs easier to deploy [21]: Q4_0 is a legacy quantization format that represents each weight in 4 bits; Q4_K_M is a more efficient format that represents each weight in ≈ 4.5 bits with higher fidelity than Q4_0. Model Qwen2.5-Coder 14B [17] DeepSeek-Coder-V2 [49] Devstral Small 1.1 [32]

Code Editing (↑)

Input Prediction (↑)

Output Prediction (↑)

55.7 31.4 40.7

72.9 45.5 64.5

78.8 52.0 71.5

(b) Performance of selected code-generation models on code editing, execution reasoning, and real-world issue-resolution benchmarks. Code Editing capabilities are assessed as the pass@1 resolve rates on the AIDER benchmark (first version [13]) with the whole edit format (10 threads). Input and Output Prediction capabilities are evaluated on pass@1 of CRUXEval [14] with chain of thoughts (temperature set to 0.2 and 10 samples). For all benchmarks, larger numbers indicate better performance (↑).

biggest in terms of size and number of parameters, but has From the 110 reproducible bugs from the previous the oldest knowledge cutoff. step, we collected 243 stack traces across 15 projects. Table 2b provides an overview of the model’s capabilMost of the errors are assertion errors (AssertionError or AssertionFailedError), and there are also NullPointerException, ities in Software Engineering, relating tasks such as code IndexOutOfBoundsException, and ComparisonFailure. The dataset editing and input/output prediction: Q WEN is the best performer, followed by D EVSTRAL, while D EEPSEEK performs is highly imbalanced, with 168 traces (69%) originating from the poorest on those tasks. jhy-jsoup alone. To ensure fair representation across projects The top-p parameter defines a threshold 𝑝 that picks and error categories, we use stratified sampling per projectthe most probable tokens until their cumulative probability error types pairs: for each evaluation round, we randomly reaches at least 𝑝. Setting top-p to 1 ensures that all tokens select one stack trace per project per error type, resulting in 13 samples per round, 26 in total. are considered. A temperature of 1.0 maintains the original token probability distribution and serves as the reference 3.4. Models setting [30]. To explore how models generate and evaluate Three open-weight agentic code models (Q WEN2.5code tour freely, we set the top-p parameter to 1 and the C ODER 14B, D EEPSEEK-C ODER-V2, and D EVSTRAL temperature to 1. SMALL 1.1) from three different providers (Alibaba, 3.5. Tasks Definition Deepseek, and Mistral), with a number of parameters 3.5.1. Code Tour Generation ranging from 14 to 24 billion and knowledge cutoffs Figure 4a presents the prompt used for generating a code before 2025, were selected. The knowledge cutoff is the tour file to explain a stack trace between a faulty method and date after which a model has not been trained on new data. a failing test. The role section defines the persona, implicitly This information is critical to consider to mitigate data setting the expected completion tone [33]. The info section leakage: if data predating the cutoff is used, models may describes the concept of a code tour, while the task section have been trained on it, particularly for widely used datasets provides detailed instructions for the task. such as Defects4J [36]. Even for open-weight models, most Balfroid et al. [4] generated the tour frame-by-frame, providers do not disclose the cutoff date for competitive which sometimes resulted in steps lacking links. This made reasons; community estimates are used instead. Models were sense at the time because the model used in the study was accessed via the Ollama API, hosted on an on-premises GPT3.5-Turbo, which struggled with structured outputs such cluster. The available hardware provides 48 GB of VRAM, as JSON-formatted files [27]. Nowadays, frameworks such which limits the maximum model size that can be studied. as LangChain provide structured output validation. So in our Table 2a provides an overview of the architecture of the approach, we generate the code tour file directly. three models. Q WEN is the smallest model, w.r.t. the number of parameters and the context window length, but has the 3.5.2. Code Tour Evaluation most recent knowledge cutoff. D EEPSEEK has the longest Figure 5 presents the components of the code tour evalcontext window, which is the smallest in terms of size, but uation task prompt. Figure 5a displays the prompt template, it uses a legacy quantization technique that produces poorer which can be filled in with any criterion and rating scale, quality than the one used for the other two. D EVSTRAL is the and is organized into three sections. The role section defines Balfroid et al.: Preprint submitted to Elsevier

Page 6 of 22

Human Factors in AI-generated Code Tours –ROLE– You are a senior developer writing documentation to help onboard a new developer unfamiliar with the

project domain on a debugging task. –INFO– CodeTour is a Visual Studio Code extension that lets you record and play back guided walkthroughs of a codebase. It acts like an interactive table of contents, making it easier to: - onboard to a new project or feature area, - visualize bug reports, or - understand the context of a code review or pull request. A code tour is a sequence of interactive steps. Each step is linked to a specific directory, file, or line of code, and contains a description that explains the relevant context. –TASK– Generate a Code Tour file that walks through the execution path in the provided stack trace. Each step corresponds to a stack frame, described in order. The explanation at each step should give a clear idea of the role of the method, how it broadly works, what it produces, and explain concepts related to it. Mention potential failure causes only when they are directly relevant to the stack trace. After reading the tour, the developer should understand how the error occurs and be able to fix it. In the long term, the developer should get a better understanding of the project domain and be able to contribute more effectively.

(a) Generation prompt. The role section sets the tone for a mentor senior developer; the information section provides information about code tours; and the task section defines the task of generating a code tour. CodeTourFormatter + title: str The display name of the tour, which will be shown in the CodeTour tree view, quick pick, etc. + description: str Overview of the stack trace context and the execution path leading to the error, highlighting how the failure occurred to help developers quickly grasp the bug scenario before exploring detailed steps. + steps: List[CodeTourStepFormatter] An array of tour steps based on the stack frames in the stacktrace, ordered from the entry point to the failure point.

CodeTourStepFormatter + description: str The text which explains the current file/line number, and can include plain text and markdown syntax. + file: str The file path (relative to the workspace root) that this step is associated with. + line: int The 1-based line number that this step is associated with.

(b) Pydantic schema definition of the expected JSON structure. A CodeTourFormatter contains metadata and an ordered list of CodeTourStepFormatter entries, each pointing to a specific file and line in the codebase. Figure 4: Prompt components for the code tour generation task: (a) the prompt and (a) the schema defining the structure in Pydantic.

the synthetic judge as an experienced software engineer. This serves as a cultural anchor, implicitly conveying expectations for tone and technical depth that would otherwise require lengthy, explicit instructions [33], thereby steering the distribution of the next likely tokens. The task section states that the task is to evaluate the criterion <CRITERIA>, which should be filled in with the criterion’s definition. Figure 5b lists definitions of the three criteria evaluated in this study, from [4]: (Transparency) understanding the codebase; (Scrutability) assessing the correctness and relevance of the explanation; and (Efficiency) performing the task. We do not provide subcriteria to reduce potential anchoring bias. This avoids overly constraining either the model or developers with too narrow evaluation dimensions, while offering enough material for developers to respond and stimulate discussion. The definitions are slightly adjusted to be positively framed and formulated with a uniform "To what extent" structure to ensure scoring consistency. The format section defines the expected format and rating scale, using the placeholder <RATING_SCALE>. Figure 5c defines the level definition used in this study: a rating of 7 indicates a very good outcome, whereas a rating of 1 indicates a poor one. The output of the task is to produce an annotation: a rating-rationale pair for a given code tour. The expected JSON schema for an annotation is defined directly in the text, not with Pydantic, since it is much simpler than the Balfroid et al.: Preprint submitted to Elsevier

one for a code tour. It consists of two fields: "rationale" (text) and "rating" (Likert-7). We adopt a 7-point scale, as recommended by Rokeman [34], because it strikes a good balance between simplicity and granularity.

3.6. Developers Evaluation Each stack trace is processed by two LLMs playing complementary roles: one acts as the author (generating a description of the code tour), and the other as the judge (rating the code tour across three criteria, with a rationale). To avoid preference leakage [26], the author and the judge are always two different LLMs. With three LLMs, this makes six (author, judge) configurations. Thus, each sampled stack trace has two configurations, one for each of the other LLMs other than the author’s LLM. Evaluating all annotations by all developers is not feasible. To resolve this limitation, a Balanced Incomplete Block Design (BIBD) [40] is adopted as the assignment scheme. This approach makes sure that each developer evaluates the same number 𝑟 of annotations, and each annotation is evaluated by the same number 𝑘 of developers. Additionally, every pair of developers shares the same number 𝜆 of coevaluated annotations. A BIBD is defined by 𝑣 (the number of developers), 𝑘 (the number of developers per annotation), and 𝜆 (the number of co-evaluated annotations). The BIBD must satisfy two relations: 𝑟(𝑘 − 1) = 𝜆(𝑣 − 1) and 𝑏𝑘 = 𝑣𝑟, Page 7 of 22

Human Factors in AI-generated Code Tours –ROLE– You are a senior software engineer reviewing a code tour. –TASK– Evaluate the quality of this code tour based on the following criterion:

<CRITERION> –FORMAT– Provide your answer on a scale of 1 to 7. <RATING_SCALE> Alongside provide a rationale for your rating. The format of your response should be in JSON: { Rationale: "<rationale>", Rating: "<rating>" }

(a) Evaluation prompt template. The role section sets the tone of a senior software engineer; the task section defines the quality criterion to evaluate, filled with one of the definitions from Figure 5b; and the format section structures the output and anchors the rating scale using the levels defined in Figure 5c. 1. Transparency: To what extent does the code tour clearly explain how the code works to a new developer? 2. Scrutability: To what extent does the code tour enable a developer to critically assess the correctness and relevance of the explanations? 3. Efficiency: To what extent does the code tour help a developer quickly understand and navigate the codebase?

(b) Evaluation criteria, adapted from Balfroid et al. [4]. Each criterion is individually substituted into the <CRITERION> placeholder in the template (Figure 5a). 1. [Terrible] The code tour fails to support the criterion. 2. [Very Poor] The code tour meets the criterion only minimally. 3. [Poor] The code tour provides limited support for the criterion. 4. [Basic] The code tour meets the criterion at a minimal acceptable level. 5. [Good] The code tour supports the criterion reasonably well. 6. [Very Good] The code tour strongly supports the criterion. 7. [Excellent] The code tour fully supports the criterion.

(c) Rating scale, substituted into the <RATING_SCALE> placeholder in the template (Figure 5a). Figure 5: Prompt components for the code tour evaluation task: (a) the template, (b) the quality criteria, and (c) the rating scale.

where 𝑏 represents the total number of annotations. In this context, 𝑏 is the total number of rating-rationale pairs to be evaluated, 𝑣 is the total number of developers acting as evaluators, 𝑟 is the number of annotations assigned to each developer, and 𝑘 is the number of developers assigned to each annotation. Pilot sessions revealed that developers require approximately 5-10 minutes to evaluate a single annotation. Given that people start to lose focus after 60 to 90 minutes, the number of annotations evaluated per developer was set to 𝑟 = 6. Accordingly, values for 𝑘, 𝜆, and 𝑣 were determined to satisfy the BIBD relations 𝑟(𝑘−1) = 𝜆(𝑣−1) and 𝑏𝑘 = 𝑣𝑟, where 𝑏 is the total number of annotations. The number of developers per annotation is set as 𝑘 = 3, and the number of times each pair of developers shares an annotation is set to 𝜆 = 1. Consequently, the assignment scheme is of 𝑣 = 13 developers and 𝑏 = 26 annotations. Developers were recruited based on availability, experience level, and Java proficiency. Focusing mostly on juniorlevel participants, as they represent the main target audience for code tours and are most likely to onboard new codebases. We replicated the assignment scheme in two parallel groups Balfroid et al.: Preprint submitted to Elsevier

of 13 developers each. Each group was assigned a separate set of 26 annotations, structured identically according to the BIBD (See Table 5). Consequently, 26 developers participated, and 52 distinct annotations were evaluated. Demographics are reported in Table 3. We asked them about their highest degree in computer science: 24 have a master’s, and two have a professional bachelor’s. We asked them their self-reported level of development: 13 are juniors, 9 are mediors, and 4 are seniors. And also, they self-reported proficiency in Java: 12 are basic, 12 are intermediate, and 2 are advanced. Inside a design, author-judge configurations are distributed across developers via constraint programming: (i) the two annotations of a given stack trace must never be evaluated by the same developer, and (ii) each developer must be exposed to each configuration equally. The distribution of annotated generations across the three author models is unbalanced because the size of a group (13) is not divisible by the number of author models (three). Assignment is random: Q WEN was assigned to 12 traces (6 in the first group, 6 in the second), D EVSTRAL was assigned to 7 traces (4 in the first, 3 in the second), Page 8 of 22

Human Factors in AI-generated Code Tours

• Component is the component of the code tour the participant refers to (e.g., the description);

Table 3 Demographics split by round (a) Round 1 Participant

Degree

Dev Level

Java Level

P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13

Master Bachelor Master Master Master Master Master Master Master Master Master Master Master

Junior Junior Senior Medior Medior Junior Senior Junior Medior Junior Junior Junior Junior

Basic Basic Advanced Intermediate Intermediate Intermediate Basic Intermediate Intermediate Basic Basic Basic Basic

(b) Round 2 Participant

Degree

Dev Level

Java Level

P14 P15 P16 P17 P18 P19 P20 P21 P22 P23 P24 P25 P26

Master Master Master Master Master Master Master Master Bachelor Master Master Master Master

Senior Medior Junior Medior Medior Junior Senior Medior Junior Medior Medior Junior Junior

Intermediate Intermediate Intermediate Basic Advanced Basic Basic Intermediate Intermediate Intermediate Intermediate Basic Basic

and D EEPSEEK was assigned to 7 traces (3 in the first, 4 in the second). Table 5 reports which traces were assigned to which LLM as an author. Consequently, Q WEN judged 14 code tours (7 from D EEPSEEK and 7 from D EVSTRAL), and D EEPSEEK and D EVSTRAL both judged 19 code tours (12 from Q WEN and 7 from the other). Thus, as each annotation is reviewed by 𝑘 = 3 developers, we collected 42 opinions on Q WEN-generated tours and 57 for the others.

3.7. Qualitative Coding Procedure The interviews were conducted by the first author. All interviews were conducted in French, the native language of both the participants and the interviewer, to ensure the smoothest possible communication, except for one interview conducted in English because the participant did not speak French. No audio recordings were made. Instead, the interviewer took notes on the fly, capturing participants’ comments during the think-aloud sessions. These notes were subsequently translated into English (which we refer to as Excerpts), and entered into a spreadsheet for analysis. Because the excerpts result from this note-taking, translation, and reformulation process, they should be understood as best-effort reconstructions of participants’ reported experiences rather than verbatim transcriptions. The qualitative coding analysis of these excerpts was done by the first three authors following an experience-based approach inspired by Lejeune et al. [25]. Each label (which we referred to as Experiences, e.g., I cannot understand the code when the description lacks detail. (E34).) can be decomposed into four parts: Balfroid et al.: Preprint submitted to Elsevier

• Property is the property of that component being referred to (e.g., the level of detail of the description); • Valence is the positive or negative nature of the experience (e.g., insufficient detail is perceived negatively, whereas adequate detail is perceived positively); • Effect is the specific aspect of the interaction with the component to which the experience relates (e.g., the description helps understand the code because of a sufficient level of detail). After the initial transcription and labeling, the first author of this paper performed a first pass to systematize the labels. Second and third passes by the same author were conducted to systematically verify the consistency of the attributed labels. Then, independently, the second and third authors coded excerpts from their assigned participant groups (Group 1 and Group 2, respectively, each comprising 13 participants), while the first author served as the reference coder, having previously labeled the entire dataset. We computed Krippendorff’s 𝛼 for each Experiences, treating the data as nominal. This coefficient corrects for expected disagreement, which is particularly appropriate in the case where there are rare Experiences, as it is expected to agree more often because it’s mostly negative for all excerpts. The three authors discussed excerpts that led to significant disagreement. After the discussion, they independently revised their codings. We repeated this cycle until all labels achieved strong agreement between authors (𝛼 ≥ 0.8).

4. Results This section reports the results to answer the research question: "How do the properties of a code tour’s components affect the developer experience, with respect to the goals developers pursue when fixing a bug in an unfamiliar codebase?". Following five rounds of discussion, the three coders achieved a high level of pairwise agreement (𝛼 > 0.8) on all labels. During the sixth and final round, the coders reviewed all labels for the overall coding: they discussed possible mergers and splitters and refined the decomposition. In total, 62 labels were formulated to capture user experiences with code tours. The list of labels (denoted hereafter E1 to E62) is available in Table 4 (see Appendix A). These experiences can all be clustered around five goals. Three of which are functional in nature, expressing what developers expect the code tour will help them accomplish, totaling 41 experiences (66.1%). First, developers want to understand the codebase. This goal concerns 25 experiences (40.3%), eight of which focused on understanding efficiently. Second, developers want to act within the codebase, e.g., locating an element or fixing a bug. This goal concerns 8 experiences (12.9%), three focusing on acting efficiently). Third, developers expect to navigate efficiently thanks to a code tour. This goal concerns eight experiences Page 9 of 22

Human Factors in AI-generated Code Tours

Clear

E9

Accessible

E51

E50

Ø E36

Ø E33

Clue

∝

E12

More Details

E48 E49 E52

Understanding

Root Cause

E53

Fix

E4

Ø

E34 E35 E37

Understanding efficiently

E11

E18

∝

Less Details

E6

E6

Long Segment

☆

E62

Short Segment

Acting efficiently

E58 E13

Highlight

☆

Describing Restating

Acting

E14 E15

Assertion Line Fault Line Condition Line JSON Field Color-coding

E16 E17

Ø

E59

☆

Start Line

E61

Figure 6: This diagram illustrates how the properties of the Description component (white rectangles) impact the functional goals (blue ovals). The connecting edges represent participant experiences, marked by label IDs (Ex). Edge color indicates whether the experience was positive (green/solid) or negative (red/dashed), and edge thickness shows how many participants reported it (0.4 points per participant). The ∅ symbol shows the absence of the property. The ∝ symbol shows an interaction between two properties. For instance, the E12 connection reports that for a "Long Segment", the ability to achieve "Understanding efficiently" is proportional to having "More Details." The ⭐ relates to the value of the property, e.g., for E59, it can be understood as "referencing the start line is not needed to act efficiently."

(12.9%). Moreover, two goals are non-functional, relating to the properties that the developers expected of the code tour. First, developers expressed their preferences about code tours. Second, developers expressed what properties of the pipeline affect their trust. Both non-functional goals concern a total of 21 experiences (33.9%): 10 experiences (16.1%) for preference and 11 experiences (17.7%) for trust.

4.1. Understanding the code base 4.1.1. The level of detail should be balanced between being comprehensive and concise. Nearly all participants, with the exception of P21, reported difficulty understanding parts of the codebase at some point due to insufficient detail (E34, E35, E37). 23 participants cited comprehension issues with the code (E34), 9 with the background (e.g., domain concepts, acronyms, etc.) (E35), and 16 with the failure (E37). This is reflected in Figure 6 by the thick red edge linking Less Details to Understanding. For example, many participants, when touring the openFHIR project from medblocks, were puzzled by the acronym FHIR, which is a standard for exchanging medical data. “I don’t know what FHIR is, a small section explaining what it is would be nice.” (P11 – 4.4). Note that Balfroid et al.: Preprint submitted to Elsevier

four participants expressed a lack of detail in descriptions, but they were able to understand in the end, thanks to the judge’s annotations (E54). Three participants indicated they would be unable to resolve the bug at all because the descriptions lacked sufficient detail (E13). Four participants noted the tour was missing details that the judge gave (E28). Conversely, more detailed descriptions often support participants’ understanding of the codebase (E48, E49, E52). 17 participants noted a better understanding of the code (E48), 3 of the background (E49), and 12 of the failure (E52) as areas where the descriptions provided enough details to better understand. This is shown in Figure 6 by the thick green edge from More Details to Understanding. However, there is a balance of details to have: seven participants said that more concise descriptions facilitated faster comprehension (E11), and 13 participants reported they are slower to understand reading a verbose description (E4). Nonetheless, in some cases, four participants explicitly state they doubt they would better understand with more or less details. (E46): “It’s not with more examples that we would understand.” (P25 – 2.2) and “It lacks details, but [...] I’m not sure too many details are desirable” (P8 – 2.5).

Page 10 of 22

Human Factors in AI-generated Code Tours

4.1.2. The expected level of detail scales with the code segment length. As indicated by the ∝ symbol in Figure 6, there is a proportionality between the size of the segment and the level of detail of the description. As segments became longer, three participants reported faster comprehension when reading the summary rather than the code (E12), while five participants experienced difficulty understanding longer segments when details were insufficient (E18). Furthermore, five participants even found descriptions unnecessary for shorter segments (E6), noting that short methods naturally “speak for themselves” (P12 – 1.6). 4.1.3. Descriptions add little value when they merely restate the code. As indicated by the ⭐ symbol in Figure 6, a few experiences relate to properties that are not necessary. Fifteen participants reported being able to understand the code without reading the description (E58). Typically, when the author model simply restates the signature (P8 – 4.7), lists the attributes (P6 – 5.2), and reexplains a clear error message: there is “no need for natural language to explain an error that is clear from the beginning” (P11 – 4.2). P7 was even more critical: “The problem is maintainability, the tour has to evolve with the code base, like every documentation, it is doomed to die after 3 weeks, it does not replace clear and maintainable code” (P7 – G.1). P18 emphasized “I find the description superficial, it simply describes the code, reading it would be faster” (P18 – 4.2). P26 went a bit further, and even questioned the usefulness of the overall tour for understanding: “[...] to what extent should I take into account the fact that one can understand it without going through the steps?” (P26 — 2.2). 4.1.4. Descriptions are expected to be technical, but clear and adapted to the developer’s level. Seven participants praised certain descriptions as accessible and junior-friendly (E50). Though four conceded their domain knowledge aided understanding (E47): “[...] I had an easier time understanding those pieces of code due to more familiar concepts.” (P2 – G.3). Meanwhile, six struggled with descriptions not aligned with their technical level (E33). Difficulties most often pertained to proficiency in Java: “The code tour was not adapted for a Java beginner.” (P1 – G.3) or “It could have explained what a ternary operator is, [...]” (P16 – 4.1). However, some participants also cited challenges with other technologies, for example: “Hard to understand the XML style” (P5 – 5.1). Two participants expected technical tones when discussing code. P9 stated, “I don’t see the issue being too technical, at some point you are in code, but it is more a personal preference.” (P9 – 2.1). P10 added, “I disagree with the transparency part, saying that the description is technical, I do not find it too technical.” (P10 – 3.3). Moreover, P9 added about their fourth code tour “Overall, it’s a code tour. It is less fancy than the previous one, but suitable for developers, though not really for students. . . It’s the kind of thing I can Balfroid et al.: Preprint submitted to Elsevier

see in a company.” (P9 – 4.5). Twenty participants said clear descriptions helped them understand (E51), as shown by the strong green edge between More Clarity and I can understand in Figure 6. Three said clarity also sped up their understanding (E9). Conversely, ten reported that unclear descriptions hindered their understanding (E36).

4.1.5. Root cause analysis is valuable but should be given at the end. Seven participants valued the inclusion of potential failure causes (E53), appreciating how concise root-cause summaries helped them grasp the underlying issue more efficiently: “Just reading it will help me understand root cause analysis... It’s a godsend for developers... having something that sums it up in two words is great[...]” (P6 – 2.2); “It‘s good that it explains the problem, the test, and also a potential cause of the failure . . . it provides a bit of guidance [. . . ] ” (P8 – 3.3); “Here it identify at least the causes, more interesting” (P15 – 3.1); “It is good to have the potential causes [. . . ] because there was no comments in the code” (P26 – 4.1). However, one participant strongly objected to giving a root cause analysis as it prematuraly disrupted their focus and independent reasoning, and arguing root cause analysis would be better placed at the end (E20): “[...] funny it gives potential cause in the first step . . . it bothers me, it gives me clue while I to understand the path [...] it makes me loose focus . . . it is like someone comes behind your shoulder while you are focused [. . . ] why not but at the end then, what’s more is that it is pretty vague [. . . ] it simply lost me with clues” (P18 – 5.1).

4.2. Acting within the code base. 4.2.1. Color-coding and line references (assertions, fault, condition, JSON field,...) support acting (except for referencing the start line) Nine participants reported successfully locating an element when it was highlighted in the description (E14); P9 noted more specifically being able to identify the fault because it was highlighted (E15). Effective cues included referencing the line number (P16 – 6.2, among others) or specifying the field of a JSON object (P22 – 6.3). Using color-coding, for example, highlighting methods in pink within the text (P24 – G.2) is also appreciated (See an example in Figure 10). In contrast, seven participants were unable to locate an element due to insufficient highlighting (E16), nine could not identify the fault at all (E17), and five could not locate an element efficiently (E61). To compensate, some participants resorted to workarounds such as using Ctrl+F to find a method mentioned in the description (e.g., P13 – 3.3). Their suggested improvements converge on a few recurring ideas: referencing line numbers (P5 – 3.1), specifying which assertions (P8 – 2.1) or conditions (P22 – 4.4) failed, and visually linking the relevant line of code to the corresponding text (P11 – 4.3). However, simply mentioning the segment’s start line is seen as redundant (E59) (illustrated by a ⭐ red edge in Figure 6). Also, much like the need for additional detail discussed earlier, the need Page 11 of 22

Human Factors in AI-generated Code Tours H02ProtocolDecoderTest.java — testDecode()

@Test public void testDecode () throws Exception { // [...] Skipping two lines verifyAttribute ( decoder , binary ( " 2 a [...20 bytes later ]4759[...] " ) , Position . KEY_DRIVER_UNIQUE_ID , " % LICENSE$TEST$MR .^^?;6007641111111111119=180919770411=?+1419999958800100?\ r \ n " ) ; // [...] Skipping multiple lines with 83 more verify methods }

^ DRIVING

Step 1/3 Understand the Structure of H02 Protocol Messages

ProtocolTest.java — verifyAttribute()

protected void verifyAttribute ( BaseProtocolDecoder decoder , Object object , String key , Object expected ) throws Exception { Object decodedObject = decoder . decode ( null , null , object ) ; // [...] Skipping 12 lines }

Step 2/3 Analyze the Decoding Process for Different Message Types

H02ProtocolDecoder.java — decode()

@Override protected Object decode ( Channel channel , SocketAddress remoteAddress , Object msg ) throws Exception { ByteBuf buf = ( ByteBuf ) msg ; String marker = buf . toString (0 , 1 , StandardCharsets . US_ASCII ) ; switch ( marker ) { case " * " -> // [...] Skipping 19 lines case " $ " -> // [...] Skipping 1 lines default -> // [...] Skipping 1 lines } }

Step 3/3 Implement Attribute Verification in Tests

Figure 7: Code Tour 18 (traccar-traccar, commit de97c50, trace 1/1) generated by Qwen2.5-Coder 14B. Project. Traccar is an open-source GPS tracking server that supports many GPS device protocols. Commit. This commit adds support for a new device variant (a magnetic card reader called LT32) within the H02 protocol, which devices use to send data like location to the server. Test. The test checks that the H02 decoder correctly handles more than 80 different message types. It fails on LT32 messages (where the binary starts with 2a (*) and bytes 22–23 are 47 59 (GY)): the decoder does not recognize them and returns nothing (null), which then causes a NullPointerException when the test tries to read the result. Tour. The tone of this tour is noteworthy for using the imperative mood (Understand, Analyze, Implement).

for highlighting grows with segment length: “there is no line reference, even though the fragment is larger” (P9 – 2.4).

4.2.2. Suggesting a concrete fix draws mixed reactions in a debugging tour. Six participants appreciated when the tour suggested a concrete fix (E42) and two even trust the suggestion (E45): P14 found that “the solution proposition directly, it’s nice [...] it seems logical as a check, but I’m not sure if you can throw an exception without changing the signature, but it is a good clue. It is clear, and it isn’t a big deal if it doesn’t work” (P14 – 5.2), and P18 simply agreed that “the fix is Balfroid et al.: Preprint submitted to Elsevier

okay, I agree with it” (P18 – 6.2). P23 also said they would fix the bug faster if the description suggested a fix (E62) Three developers distrust a suggested fix that seemed incorrect (E25). Interestingly, in both quotes below (related to code tour in Figure 9), the suggested fix was arguably correct: the root cause was a mismatch between the assertion expecting a UnixPath and a method signature documented as returning a String, raising the legitimate question of whether the signature should be changed just to pass a test. Yet P5 went back and forth on the suggestion, stating “I disagree, your path is automatically a string... [he backtracks] actually, it’s the opposite, he’s right... nitpicking Page 12 of 22

Human Factors in AI-generated Code Tours

Clue

Next Step

Fix

E38

Ø

Optimization

E42

E19

E22 E39

Scannable Structure

Ø

E1

Liking

E41

E40

Incomplete

Inauthentic Rude Disengaged

E21

Pedagogical Guiding

E21 E41

Imperative

Tone

Figure 8: This diagram illustrates how the properties of the Description component (white rectangles) impact the Liking goals (blue ovals). The connecting edges represent participant experiences, marked by label IDs (Ex). Edge color indicates whether the experience was positive (green/solid), mixed (orange/solid-dashed), or negative (red/dashed), and edge thickness shows how many participants reported it (0.4 points per participant). The ∅ symbol shows the absence of the property.

over the wording ‘supposed’... the method signature is a String... it says it’s an implementation problem, but it’s a documentation problem... it should say that the signature is a String” (P5 – 4.1), while P11 (Junior, Beginner Java) similarly resisted, noting that “this method should return a path, but when the signature says it is String, you can not change it [whereas] eval was critical about this” (P11 – 3.3). Four attendees were concerned about being biased by the suggestions made by the tour (E29) However, beyond correctness, some participants did not want the code tour to suggest a fix at all (E38), as P4 stated “I don’t think the code tour should say what needs to be implemented” (P4 – 4.1), and P25 explained that “what I like is when you do not give suggestions, you can do it yourself, while when it gives you ones, you have blinders on, you get biased” (P25 – 6.1).

4.3. Navigating the code base 4.3.1. An overview step and hinting at the next step clarify the link between steps. While seven developers reported navigating more quickly with a well-structured tour (E8), six developers reported being slowed down in their navigation by a poorly structured tour (E2), ten reported being lost because the tour sequence was confusing (E3), and eight could not understand how the tour relates to the reported bug (E32), sometimes because they had to go “back and forth” between steps to understand. Balfroid et al.: Preprint submitted to Elsevier

“[...] It’s a bit difficult to get an idea of the structure [...] I don’t see the link between [step] 2 and 3 [...]” (P13 – 1.1). Moreover, while three users stated navigating more quickly with a tour than without one (E7), 13 participants questioned the added value of the tour structure compared to tools like the IDE or terminal (E57). P5 remarked that “it is just like going through the stack trace, right clicking” (P5 – 6.3), and P11 added “I’m navigating through bits of files. I have less information compared to my terminal” (P11 – 5.3), suggesting that, in its current form, the tour does not always provide insight beyond what an IDE or terminal would already provide. The appropriate structure, though, would depend on the developer’s goal. P15 elaborated on why for them there is “[...] two types of tour: the code tour ([...] “how this factory is initialized”) and the test tours [where] you just want to [...] go to the essential, directly to the faulty method [...] [doing] the trace in opposite direction [...] and skipped some methods that are just “waiters who serve the dishes, it is not them who added too much salt.” . . . the approach is different when you discover a new framework or fix a failing test. [...] code tour would be top down, test tour would be bottom up from the fault.” (P15 – G.1). P14 also added that “[...] When I read a stacktrace, I usually go directly to the root cause by right-click, so I read the stacktrace in the other way around.” (P14 – 1.1). To make the structure more clear, eight participants emphasized they would understand faster with an overview of the tour structure (E55), either with a global explanation (P3 – 3.3), a schematic explanation (P24 – 2.2), a dependency graph (P13 – 1.1), a navigation path (P13 – 4.2), a mini plan (P14 – 1.3), or a “summary of the execution tree showing the path to the error” (P15 — G.2). Another thing that might help is highlighting the next step. Three participants expressed their dissatisfaction that there was no hint of where the tour is going next (E22): “It should indicate clearly what you will investigate next. It would be good to highlight it [because] you lose time finding the place” (P11 – 1.1).

4.3.2. Developers get lost more easily with tours longer than five steps. Three participants reported being faster when the tour was more concise (E10), while two participants reported being slower when the tour was longer (E5). P14 noted “I preferred the direct stack traces, the ones with 7 or 9 explanations, you can get lost” (P14 – G.1), and similarly remarked “There are a lot of steps” (P14 – 1.1) about code tour 21, which is 9 steps long. P23 further corroborated this, stating that “when you are past 4 to 5 steps, you are a bit lost” (P23 – 1.2), while also noting that “it is less scary when you see there are three steps” (P23 – 2.1). 4.3.3. Duplicate steps confuse developers, yet AI judges rarely flag them. A few participants noticed that some tours included duplicate steps. For example, in code tour 33, P23 remarked, Page 13 of 22

Human Factors in AI-generated Code Tours ResultTreeManagerTest.java — getAllTotalMethodsPathTest()

@Test public void getAllTotalMethodsPathTest () { assertEquals ( Path . of ( ResultTreeManager . GLOBAL_RESULT_DIRECTORY_NAME , this . appDirectory , ResultTreeManager . ALL_DIRECTORY_NAME , ResultTreeManager . TOTAL_DIRECTORY_NAME , ResultTreeManager . METHOD_DIRECTORY_NAME ) , this . manager . getAllTotalMethodsPath () ) ;}

Step 1/2 In this step, we are setting up a test case for getAllTotalMethodsPath in the ResultTreeManagerTest class. The test is designed to check if the path returned by getAllTotalMethodsPath matches an expected value. This method should return a sun.nio.fs.UnixPath object but instead returns a java.lang.String. This will be our starting point for debugging.

ResultTreeManager.java — getAllTotalMethodsPath()

public String getAllTotalMethodsPath () { return this . allTotalMethodsPath ; }

Step 2/2 Here, we are looking at the actual implementation of getAllTotalMethodsPath in the ResultTreeManager class. This method is supposed to return a path to the directory where total methods consumption data is stored. However, it currently returns a string. This discrepancy will cause the assertion failure when compared with the expected value.

Figure 9: Code Tour 9 (joular-joularjx, commit 9940cfc, trace 116/150) generated by Devstral Small 1.1. Project. JoularX is a plugin that monitors the power consumption of Java code. Commit. Refactoring of the class generating the path for the results, which are now UnixPath instead of String Test. The test fails because it expected a UnixPath and got a String instead. Tour. This is a noteworthy example of a code tour suggesting a fix.

“the code is twice the same, but the text is different” (P23 – 4.1), an observation echoed by P26 who described it as “the same code [...] two code the same seems like a bug” (P26 – 5.1), and by P18 who questioned “uh the code did not change, is that normal? is it not a bug?” (P18 – 6.1). This issue originates from the StacktraceSkeleton class in the tour generation pipeline (see the replication package), which does not correctly handle synthetic JVM frames. In the scenario of code tour 33, the test calls the patched method using a lambda, assertThrows(..., () -> purl.uriDecode(...)), which the JVM records as a synthetic frame lambda$invalidPercentEncoding$N, where N identifies the lambda within the method. Although developers see this as a redundant step that adds no value (E57), the LLM does not seem to view it as a critical issue. In code tour 22, authored by Q WEN, it was noted that steps 5 and 6 are duplicated, with Step 6 stating that “this step is a duplicate of the previous one. The reason being, in actual stack traces, steps could overlap, and sometimes it might cause confusion on the developer’s end.”. The duplicate step also originates from a lambda call to setField on line 73. D EEPSEEK did not challenge the tour structure when judging the tour; no developers disagreed with the annotations, but P22 reflected: “[...] it is twice the same code [whereas] it should have been one step” (P22 – 1.2). However, D EVSTRAL, acting as the judge, praised the duplicated steps the efficiency section “the inclusion of a duplicate step is notable but does not detract significantly from the tour’s effectiveness, as it serves to emphasize a critical point in the Balfroid et al.: Preprint submitted to Elsevier

debugging process.”. P15 disagreed, giving a 3. The judge also commended the AI-author for flagging it in terms of scrutability “the inclusion of duplicate steps is explained, which is a thoughtful addition to address potential confusion.”. P21 disagreed, giving a 2 and commented: “it’s dumb, it’s the same step [...]” (P21 – 1.3).

4.3.4. The stack trace is not enough as some interesting steps (constructors, branches, concrete implementation, or fields) are usually seen as missing steps. Seven participants reported being impeded in their understanding because the sequence was missing a step (E31). Developers expected a step about the initialization of constructors. For example, in code tour 9, the initialization of the manager is missing: “It lacks code to understand what is going on. For example, I can assume there is a call to a constructor somewhere upstream.” (P3 – 5.1), though they later noted the failure is still comprehensible without it: “However, I have enough context to understand that it failed because a path and a string were compared; the code is enough, in the end.” (P3 – 5.1). Similarly, in code tour 23, “[...] it explains well the failure, [but] the constructor is not accessible [However, it explains that] the constructor has no access [to the] protected [modifier] [...] we do not see the class in the tour [though]” (P23 – 2.2). Some participants complained of missing steps in calls and branches: “[...] it’s missing about where calls and branches are made.” (P5 – 2.2), or a concrete implementation of some interface or Page 14 of 22

Human Factors in AI-generated Code Tours

abstract class: “It lacks a step when you enter the implementation of this interface” (P13 – 6.3). Also, some tours fell short: “[...] it stops there saying the problem is there, but it does not give the numerical value [...]” (P15 – G.3). “In the end, we know nothing about the problem, we need to go deeper, because this field has a problem, it’s just a clue, maybe it is a rabbit hole ... [the judge] said it: we do not get the whole path” (P20 – 2.2).

4.4. Liking 4.4.1. Pedagogical and guiding tones are appreciated, while disengaged, inauthentic, rude or indirect ones give rise to reservations. Six developers reacted positively to tones that felt pedagogical or guiding (E41). P9 valued how such framing broke from conventional documentation: “It feels like someone talking to you, not the sanitized documentation we are used to . . . it’s a teacher, it’s good for onboarding” (P9 – 3.4). More broadly, P9 appreciated the “variety of code tour formalisms” (P9 – G.1), noting that “some [were] more [like] an issue, a document, a Teams conversation, or another natural conversation.” (P9 – G.1). P10 echoed this appreciation for guidance: “I like when it says ’start’ etc [...] guiding you” (P10 – 1.1), adding that “I find it more friendly in the way it guides you step by step . . . Saying things like ’here the start, here the end’ . . . simple language” (P10 – 6.1). Such pedagogical framing, however, comes with a trade-off, as P9 observed: “[...] it is super efficient if you are a beginner, you need to be pedagogical and go into details, but for an expert, it is counterproductive.” In contrast, participants were put off by tones they found disengaged, inauthentic, rude or too indirect (E21). Some perceived disengagement, describing the description tone as “lazy” (P6 – 6.1) or “jaded” (P22 – 5.9; P24 – 4.2). They compared it to writing done out of obligation, such as when the author “was told to” (P13 – 6.3) or the kind of thing you would send to your boss to show it is done (P24 – 6.3). Others considered it inauthentic, calling it “theatrical” (P22 – 5.4). Reading this as unreliable: “There is a tendency to oversell, it lacks sobriety, I feel like it is marketing, I would be cautious” (P13 – 4.4). Sometimes, the description was even felt as rude: “Not very nice to speak like that. I have the impression it is someone who sent me a message . . . I am a bit miffed . . . ” (P25 – 4.1). Some participants dislike indirect phrasing, e.g., using modal verbs such as "might", as it sounds “unconfident” (P12 – 5.2).

4.4.2. Some developers prefer incomplete descriptions and the imperative mood, as they foster more active engagement. Interestingly, four participants actively liked incomplete descriptions because they required more engagement with the material (E40) (Figure 8). As P11 noted “the lack of information allows me to understand. There is a tradeoff between too much info and not enough info because, with too much text, you do not make any effort, whereas when it gives you clues, you do.” (P11 – 6.3). Balfroid et al.: Preprint submitted to Elsevier

While two participants expressed discomfort with the imperative mood (E21), feeling they were being given orders, two others actually received the use of the imperative mood well (E41). P10 said, “It is funny, it feels like it gives instructions . . . it is weird, why is it giving me instructions? Like orders . . . I don’t like it” (P10 – 5.2, see Figure 7). P22 echoed this sentiment, noting “I feel like it gives you orders, it is peculiar, it says ’look at this constant’, well give it to me then . . . it is its job . . . ” (P22 – 5.9). Others, however, read the same directness differently. P23 found it pedagogically useful despite its oddness: “It is funny that it talks to me in imperative, it’s good on a pedagogical level but weird” (P23 – 4.2). P11 went further, identifying it as their preferred style “I don’t know if it’s in the infinitive or imperative mood. . . Of all tours, it is the one I preferred. Here it says ’do this, does that,’ so it forces you to understand the code rather than just accept it’s true. Given that I have no explanation, [. . . ], I do what I normally do, which forces me to understand.” (P11 – 6.1, see Figure 7).

4.4.3. Descriptions should be easily scannable using bullet points and sections. Six participants reported liking a text structure because it is easy to skim (E39). P6 liked how one tour explained “the Arrange Act Assert in bullet points” (P6 – 4.2). P26 found a “structure in bullet points [to be] nice”, making it “clear just by reading” (P26 – 1.3). P21 said that “list helps,” but prefers structure with subtitles, as in their third code tour (P21 – 4.1). They added that, for the list, “it would be nice if the points were [linked to] lines of code, except, of course, if it is a one-line method. [. . . ] It would help with navigation, it helps structure your thoughts [. . . ]”. As for sections, P17 liked the use of subtitles (P17 – 5.1), and P26 found “the structure is better, in 3 paragraphs” (P26 – 4.1). Figure 10 shows Code Tour 32 that illustrates a code tour with such sections. Participants also highlighted how the structure helped them mentally process the code’s logic. P12 valued the mix of “natural language and programming language,” noting that explicitly stating what a line of code does helped them “redo the logical course of the method” (P12 – 4.2). P9 echoed this need for explicit references, stating that the structure mimicked an issue report with clear line recalls and an appreciated “quick summary of the method” (P9 – 4.1). P12 expressed their dissatisfaction when a certain description structure was not easily scannable (E1): “[...] You would prefer a bullet point: the level of information is good but badly structured” (P12 – 2.1). 4.4.4. Suggestions should remain focused on the debugging task. In Code Tour 14 (generated by Q WEN), D EEPSEEK was critical about “minimal mention of optimization or efficiency gains that could be made (e.g., commenting on how certain methods like ‘readNumberingStyles‘ could potentially improve runtime performance through more efficient data structures or algorithms).”. Two participants verbally expressed their disagreement on this point: they do not want Page 15 of 22

Human Factors in AI-generated Code Tours

optimization suggestions (E19). P13 said “[...] we are in debugging [so] this [should] come after [because] we have to understand first. I would be lost getting discourse on optimization before understanding” (P13 – 4.7), and rated 2 for this annotation. P4 was also critical, calling it “strange” to mention, and rated a 5, though. The third developer who had seen this annotation said nothing and even gave it a 7.

4.5. Trusting 4.5.1. Descriptions that seemed human-written were seen as trustworthy, meanwhile those seen as AI-authored were distrusted. The more a description seemed authentic rather than synthetic, the more developers trusted its content. P9 trust the content because it felt human-written (E44), therefore attributed human authorship to the text and granted it unearned authority: “the guy knows better what they are talking about than I do” (P9 – G.1). Three admitted they took claims for granted without independent verification (E60): “This is problematic because you take for granted what it says . . . you do not see what it is talking about [...]” (P11 – 4.3); “[...] I did not check if what is written is true [...]” (P26 – 4.2); and “Given I know nothing in XML, I suppose what it said is true . . . it is nice at least I understand something about style in HTML.” (P21 – 3.2). Conversely, right participants explicitly stated that believing a description was LLM-generated reduced their confidence in it (E24): “It makes me think of AI, so I would not trust it” (P24 – 2.1). Usually, it is stock formulas that betray it is an LLM: ““Please let me know” it’s busted; it is an LLM” (P12 – 2.2). Or forcing a specific structure, such as a worthless summary final line “the last line brings nothing, LLMs sometimes reexplain the high-level point [. . . ]”. Annotations were strikingly homogeneous in phrasing and structure across the three quality criteria. Thus, after repeated exposure, participants can become harsher: “at first did not notice it was LLMs, but gradually I understood it is bullshitting me, I put more extreme values” (P18 – G.1).

4.5.2. Sycophancy (excessive praise), confabulations and incoherence across the pipeline impede trust. 13 participants pointed out factual inaccuracies or confabulations in the content (E23). Others encountered concrete errors, such as an annotation mentioning “screenshots and more detailed explanations of lines in a 2-line code snippet” (P4 – 5.1), or emphasizing the tour for mentioning specific lines whereas “there are no line numbers contrary to what the judge said” (P23 – 3.2). Three participants express distrust when the judge contradicts itself (E26), e.g., “One time it says there are too many details and the other that there are not enough details” (P9 – 5.2). P8 observed “I feel like it is not coherent . . . in transparency, it says it describes more or less the code but lacks details about the algorithm . . . but in the scrutability section, [it says] the level of details helps understand the code” (P8 – 2.4), while P9 similarly noticed that “in the others [criteria], it says it is too technical, and here it says Balfroid et al.: Preprint submitted to Elsevier

we understand clearly the logic behind” (P9 – 2.3), later adding that “it seems like [it] contradicts [itself] from one criterion to the other... One time, [it] says there are too many details and the other that there are not enough details” (P9 – 5.2). P20 further noted that “[the] comment is a bit contradictory, it says it is good and then it is not, I would have been more critical, should have said it is bad from the beginning” (P20 – 6.1). There are also three participants who reported inconsistency in a tour (E30), for instance, “It said it delegates the task OK, then it says he won’t show here, then it shows it in the next frame” (P8 – 2.2). Nine participants cited distrust with the judge being excessively praising and positivity (E27): “It is obvious that it is LLM-generated because of the positive bias [and] lacks [of] negative tone, always saying it is good, even when the generator failed completely” (P5 – G.1). How do the properties of a code tour’s components affect the developer experience? Overall, the analysis of the semi-structured interview with 26 developers shows a nuanced picture of how the properties of a code tour’s components (descriptions, step sequences, judge annotations) affect the developer experience. Across 62 experiences, developers’ reactions cluster around three functional goals (understanding, acting, and navigating) and two non-functional ones (preference and trust). [Understanding] The level of detail must be balanced (Section 4.1.1) and scaled to the segment length (Section 4.1.2). Descriptions should avoid merely restating the code (Section 4.1.3) and are expected to be technical yet clear and accessible (Section 4.1.4). At the final step, they can provide a root cause analysis (Section 4.1.5). [Acting] Color-coding and providing line references to elements of interest (fault, failing condition, assertion, and others but not the start line) is much appreciated and helpful (Section 4.2.1), while suggesting a fix is commendable, but should be done with caution due to potential bias (Section 4.2.2). [Navigating] An overview of the tour sequence and adding hints at where the tour is going improves navigation clarity (Section 4.3.1). The tour should avoid exceeding five steps (Section 4.3.2) and should avoid duplicate steps (Section 4.3.3). It might also be beneficial to include steps that are not typically in the stack trace, such as constructors or the concrete implementations of an abstract class (Section 4.3.4). [Preference] The tone should be guiding, engaged, authentic, direct, and kind (Section 4.4.1). The format should be adapted to different developer personas: for example, some prefer the imperative mood or incomplete information, while others dislike it (Section 4.4.2). The text should be easily scannable, using bullet points and sections (Section 4.4.3), and suggestions should stay within the task scope (Section 4.4.4). [Trust] Developers trusted descriptions that appeared humanwritten more than those seen as AI-authored (Section 4.5.1). Developers noted confabulatory tendencies, incoherence, and sycophantic tendencies that undermined their trust in the system (Section 4.5.2).

Page 16 of 22

Human Factors in AI-generated Code Tours

5. Discussion 5.1. Implications 5.1.1. Aligning a model on general developers’ preferences for code tour generation. Our findings on code tour preferences align with existing work on documentation comprehension and code foraging. This supports the intuition that a code tour is, in essence, a contextualized sequence of code summaries. Thus, code tours may be more closely related to other forms of documentation, which allows us to generalize our findings with greater confidence and draw on a broader body of prior work. First, participants preferred balanced, high-level summaries rather than mere code restatements (Section 4.1.1, 4.1.2, and 4.1.3). This corroborates prior findings that developers favor high-level overviews over line-by-line explanations [28] and prefer concise comments of two to three lines [16]. The ideal ratio between description and code length remains an open question for future work. It is also worth mentioning that there are techniques to reduce the presented code length rather than solely the description length, for example, by folding less-informative blocks [12]. Second, participants wanted scannable, structured formatting (Section 4.4.3), with color-coding and line references pointing to interesting elements (Section 4.2.1), and they wanted the content to focus on the task scope (Section 4.4.4). This supports the view that, in technical documents, structure determines access to information and, consequently, affects usability. Accordingly, readers scan with the intent of answering a question [46], which is "how do I fix this bug ?" in the case of this study. So it is not surprising that descriptions with sections were favored, since a "Role / Execution / Potential Causes" template lets readers jump directly to the relevant part compared to a wall of text. It is also understandable that the optimization suggestions were unwelcome, as they did not answer the user’s question while reading the text. Third, developers appreciated that a guiding, engaged, authentic, direct, and kind tone (Section 4.4.1). Beyond the obvious fact that people prefer to be treated kindly, this echoes evidence that greater politeness within a project is associated with faster issue resolution [9]. In all, these preferences appear to be broadly shared among developers. Thus, fine-tuning a model to align with them is a promising direction for future work. The findings from this work pave the way for aligning a model with developer preferences.

5.1.2. Personalization of code tour generation Conversely, our findings highlight preferences that diverge across developers and might indicate distinct clusters of preferences: some prefer more agency through imperative and incomplete descriptions (Section 4.4.2); developers differ in technical level (Section 4.1.4); and some are cautious about root cause analysis (Section 4.1.5) and fix suggestions (Section 4.2.2). This signals a need for personalization when generating code tour descriptions. Balfroid et al.: Preprint submitted to Elsevier

5.1.3. Steps selection This study provides a bit more insight into the kinds of steps users expect in a code tour. First, the stack trace is not enough: developers are also interested in seeing, for example, the constructor or concrete implementation of an abstract class, which are not usually present in the stack trace (Section 4.3.4. Also, that lambda call related to them results in duplicate steps that developers don’t want (Section 4.3.3). However, we have to be cautious, as developers reported starting to get lost typically after five steps (Section 4.3.2). They also demanded an overview of the path (Section 4.3.1). 5.1.4. Trust calibration for AI-authored code tours The more a description seemed human-written rather than AI-authored, the more developers trusted its content (Section 4.5.1), which might lead to miscalibrated trust. On the one hand, there is a risk of misuse (over-reliance on unreliable automation [23]): when some participants attributed human authorship (whereas all code tours were AI-generated), they granted it unearned authority. On the other hand, there is also a risk of disuse (rejection of capable automation [23]): when some participants attributed AI authorship, they admitted being harsher in their judgment. Nakano et al. [31] revealed a consistent negative shift in perceptions of trust when AI involvement in writing is disclosed, due to the loss of human touch, stylistic awkwardness, diminished author credibility, and a lack of human effort. However, assistive AI authorship can be socially acceptable when the AI contribution is under 50%, and the writing act is descriptive. Code tour writing would fall into the Explore category because the purpose is to develop knowledge. Another mitigating factor was the reader’s AI literacy, and Nakano et al. [31] recommended fostering it through a positive feedback loop by gradually exposing them to content with an increasing ratio of AI-generated parts. Thus, trust in code tours could be calibrated by 1) disclosing what part of the descriptions were AI-generated and human-written (avoiding misuse when AI-authored content passes as human-written); 2) taking into account the AI literacy and code base and technical knowledge when assigning a tour to learners. This mitigates disuse among users with low AI literacy by gradually exposing them, and misuse among users with less knowledge who would over-trust the tour. This could be easily incorporated in onboarding systems such as Lacy [20]. 5.1.5. Mitigating sycophancy, confabulation and incoherence Users’ trust in the code tour generation and evaluation pipeline was undermined by different issues: sycophancy (an overly positive tone), factual confabulations, and logical inconsistencies across the tour description steps and evaluation criteria (Section 4.5.2). You can also add to that the issue of duplicate steps discussed in Section 4.3.3 that the judge did not challenge as harshly as humans would. These issues overlap, e.g., participants repeatedly identified a specific confabulation in which the automated judge Page 17 of 22

Human Factors in AI-generated Code Tours

praised the tour for including line numbers, even though they were none. This pertains to the concept of bullshit described by Hicks et al. [15], who argue that large language model (LLM) outputs can be characterized as bullshit because they are indifferent to truth. According to Cheng et al. [7], sycophancy refers to excessive protection of the user’s ego. These concepts are interconnected: the model’s disregard for truth [15] is influenced by its tendency to protect the user’s image Cheng et al. [7]. Therefore, prioritizing the mitigation of sycophancy is essential to improving the trustworthiness of the pipeline.

junior developers, the population most likely to onboard to an unfamiliar codebase with external assistance and most susceptible to misuse; the number of proficient Java users in our sample is small, and more senior participants, who may be more susceptible to disuse, are underrepresented. Finally, we study agentic code LLMs with a knowledge cutoff before 2025; more recent models are likely to address some of the observed issues. However, the scarcity of reproducible bugs and data leakage as threats that must be taken seriously mean such studies inherently lag behind the state of the art.

5.2. Threats to validity 5.2.1. Internal validity

6. Conclusion

Our excerpts are reconstructions, not verbatim transcripts. Each passes through note-taking, translation, firstperson reformulation, and interpretive decisions. They should therefore be read as best-effort reconstructions of what participants reported, not as their exact words. Also, the interviews were conducted in French and translated into English, which may have introduced a loss of nuance; for instance, French has no neutral pronoun like "it" to refer to a thing, which is relevant given that the perceived humanversus-machine authorship of a description emerged as an influential signal in our study. Because participants thought aloud while rating annotations and exploring tours, this concurrent verbalization and the presence of the interview may have heavily biased their responses. As one participant admitted “I’m not in the same mood as if I were alone [...] sometimes I answer by instinct more than by understanding” (P19 – G.1). Each participant evaluated six tours in a single 30- to 60-minute session without a break. So fatigue and learning effects may have influenced later ratings. Additionally, the study was conducted via a controlled web interface rather than an IDE with full access to the codebase, so behavior may differ in a realistic development setting. So, their perception of the code tour could have been very different after using it regularly for many days. Finally, a parsing bug affecting synthetic JVM (lambda) frames produced duplicate steps that the models rarely flagged. This may have deflated agreement with the judges in those cases, as it represents an atypical scenario. However, it also constitutes a finding in its own right: it signals that the models are not critical enough of the step sequences they are given, whether acting as authors or judges.

5.2.2. Construct validity Our labels may not perfectly capture the intended theoretical concepts, due to anchoring bias during early coding, loss of nuance when forcing experiences into fixed valence categories, grouping heterogeneous excerpts under shared labels, and possible overlap between dimensions. 5.2.3. External validity Our findings may not generalize to other programming languages, to bugs committed after 2025, or to onboarding tasks beyond debugging. Recruitment primarily targeted Balfroid et al.: Preprint submitted to Elsevier

In this paper, we investigated how the properties of LLM-generated code tours affect developer experience when debugging unfamiliar Java codebases. We built a pipeline (Figure 2) that (1) mined reproducible, non-flaky bugfix commits from 2025 GitHub repositories via GitbugActions [35], (2) extracted stack traces from failing tests, (3) generates 26 code tours using three open-weight, locally hosted LLMs (Q WEN2.5-CODER 14B, D EEPSEEK-CODERV2, and D EVSTRAL SMALL 1.1), and (4) has the two nonauthoring LLMs act as judges, producing 52 annotations on Transparency, Scrutability, and Efficiency. (5) Two groups of 13 developers evaluated each a unique set of 6 code tours and annotations while thinking aloud (each combination of tours and annotations was explored by three different developers). Three authors qualitatively coded interviewer notes into 62 first-person experience labels (Krippendorff’s 𝛼 ≥ 0.8 after iterative discussion), clustered around three functional goals (understanding, acting, navigating) and two non-functional goals (liking, trusting). We found that (1) developers want balanced detail scaled to segment length, scannable structure, guiding tone, and no mere code restatement; (2) preferences are sometimes mutually exclusive (e.g., imperative mood, deliberately incomplete descriptions); (3) the stack trace is not enough, developers would like to see snippet such as constructors; (4) perceived human authorship inflated trust (misuse risk) while perceived AI authorship deflated it (disuse risk); and (5) LLM judges exhibited pervasive sycophancy, confabulation, and incoherence issues. Future work should explore (1) aligning a model on general developers’ preferences; (2) using personalization techniques for diverging preferences such as imperative mood; (3) studying how to select steps beyond the stack trace; (4) calibrating user trust between disuse and misuse; and (5) mitigating sycophancy, confabulation, and incoherence issues for open-weight LLMs to be trustworthier judge of code tours.

A. List of Labels Table 4 presents the coding guide with the different labels. Each label represents an experience from the developer’s point of view and is uniquely identified (E). The table describes the developer’s goal (e.g., understanding) and its valence. It also describes the component and the Page 18 of 22

Human Factors in AI-generated Code Tours Table 4 List of labels ID

Name

Goal

Valence

Component

Property

E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E17 E18 E19 E20 E21 E22 E23 E24 E25 E26 E27 E28 E29 E30 E31 E32 E33 E34 E35 E36 E37 E38 E39 E40 E41 E42 E43 E44 E45 E46 E47 E48 E49 E50 E51 E52 E53 E54 E55 E56 E57 E58 E59 E60 E61 E62

I dislike when the text structure is not easily scannable. I navigate more slowly when the sequence is poorly structured. I navigate more slowly when the sequence is confusing. I understand more slowly when descriptions are verbose. I navigate more slowly when the tour has too many steps. I do not need a description to understand a short method. I navigate more quickly when the tour is well structured. I navigate more quickly with a tour than without one. I understand more quickly when the descriptions are clear. I navigate more quickly when the tour has few steps. I understand more quickly when the descriptions are concise. I understand long methods more quickly with a description. I cannot fix the bug when the descriptions lack details. I can locate an element when it is highlighted in the description. I can locate the fault when it is highlighted in the description. I cannot locate an element when it is not highlighted in the description. I can locate the fault when it is not highlighted in the description. I cannot understand a long method when the description lacks sufficient detail. I dislike when the description gives general improvement suggestions. I dislike when the description gives root-cause clues prematurely. I dislike the description because its tone feels inauthentic, rude, or disengaged. I dislike when the description does not hint at the next step. I distrust the content because it contains factual inaccuracies or confabulations. I distrust the content when it feels LLM-generated. I distrust the description because it suggests a fix that appears incorrect. I distrust the judge when it contradicts itself across criteria. I distrust the judge when it is unconditionally positive regardless of tour quality. I distrust the judge because it does not address all relevant aspects of the tour. I distrust the tour because its suggestions may bias my interpretation of the bug. I distrust the tour when it contradicts itself. I cannot understand when the tour is missing a step. I cannot understand how the tour relates to the reported bug. I cannot understand because the description is not adapted to my technical level. I cannot understand the code when the description lacks details. I cannot understand the background when the description lacks details. I cannot understand the description when it is unclear. I cannot understand the failure when the descriptions lack details. I dislike when the description suggests a concrete fix. I like when the text structure is easily scannable. I like when the description is incomplete because it forces me to engage with the code. I like the description because its tone feels pedagogical or guiding. I like when the description gives general improvement suggestions. I understand the code more slowly because some information is given at the wrong time during the tour. I trust the content when it feels human-written. I trust the description because the suggested fix appears correct. I cannot understand the code when the descriptions lack details (while the judge has them) I understand because I have sufficient technical knowledge to compensate for the description. I understand the code when the description has enough details. I understand the background when the description has enough details. I understand the description when it is adapted to my technical level. I understand the description when it is clear. I understand the failure when the description has enough details. I understand the failure because the description provides root-cause clues. I doubt that I would understand better with more or less detail. I would understand faster if the tour included an overview step. I navigate more slowly when descriptions lack details. I find some steps redundant, given what other tools (e.g., IDE or terminal) already provide. I can understand the code without reading the descriptions. I do not need the descriptions to highlight an element. I over-trust the descriptions because I assume they are correct. I cannot locate an element quickly because it is not highlighted. I would fix the bug faster if the description suggests a fix.

Liking Navigating (efficiently) Navigating (efficiently) Understanding (efficiently) Navigating (efficiently) Understanding (efficiently) Navigating (efficiently) Navigating (efficiently) Understanding (efficiently) Navigating (efficiently) Understanding (efficiently) Understanding (efficiently) Acting Acting Acting Acting Acting Understanding Liking Liking Liking Liking Trusting Trusting Trusting Trusting Trusting Trusting Trusting Trusting Understanding Understanding Understanding Understanding Understanding Understanding Understanding Liking Liking Liking Liking Liking Understanding (efficiently) Trusting Trusting Understanding Understanding Understanding Understanding Understanding Understanding Understanding Understanding Understanding Understanding (efficiently) Navigating (efficiently) Navigating (efficiently) Understanding Acting (efficiently) Trusting Acting (efficiently) Understanding (efficiently)

Negative Negative Negative Negative Negative Negative Positive Positive Positive Positive Positive Positive Negative Positive Positive Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Negative Positive Positive Positive Positive Negative Positive Positive Negative Mixed Positive Positive Positive Positive Positive Positive Mixed Negative Negative Negative Negative Negative Negative Negative Mixed

Description Tour Tour Description Tour Ratio Tour Tour Description Tour Description Description Description Description Description Description Description Description Description Description Description Description Content Content Description Judge Judge Judge Tour Tour Tour Tour Description Description Description Description Description Description Description Description Description Description Tour Content Description Tour Description Description Description Description Description Description Description Description Tour Description Tour Description Description Description Description Description

Structure Structure Clarity Details Length Length Structure Value Clarity Length Details Ratio Details Highlight Highlight Highlight Highlight Ratio Suggestion Timeliness Tone Clue Accuracy Authenticity Accuracy Coherence Sycophancy Incomplete Bias Coherence Incomplete Relevance Accessibility Details Details Clarity Details Clue Structure Details Tone Clue Timeliness Authenticity Accuracy Details Accessibility Details Details Accessibility Clarity Details Clue Details Overview Details Value Value Highlight Over-trust Highlight Clue

corresponding property it applies to (e.g., the level of detail in the description).

B. Code tours assignments Table 5 presents the assignment of the generated code tours to the different participants with the experimental Group, Tour ID, Project, and git commit Version. There is a hyperlink for the tours with an example figure in the document. Author denotes the LLM that generated the tour, and Judge denotes the assigned LLM evaluators. Trace represents the stack trace index amongst all the stack traces relating to a project version, and Error denotes the exception type. The Participants (Rank) column lists the assigned human evaluators (P1–P26) and their 0-indexed evaluation order (e.g., P1[0] means Participant 1 evaluated this tour first).

Balfroid et al.: Preprint submitted to Elsevier

Frequency 1 6 10 13 2 5 7 3 3 3 7 3 3 9 1 7 9 5 2 1 7 3 13 8 3 3 9 3 4 3 7 8 6 23 9 10 16 2 6 4 6 6 3 1 2 4 4 17 3 7 20 12 7 4 8 1 13 15 3 3 5 1

CRediT authorship contribution statement Martin Balfroid: Conceptualization, Methodology, Software, Validation, Formal analysis, Data Curation, Investigation, Visualization, Writing – original draft, Writing – review & editing. Julien Albert: Validation, Formal analysis, Data Curation, Writing – review & editing . Dzenatan Aliti: Formal analysis, Data Curation . Xavier Devroey: Supervision, Validation, Writing – review & editing, Project administration, Funding acquisition, Resources. Benoît Vanderose: Supervision, Validation, Writing – review & editing, Project administration, Funding acquisition, Resources.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process. During the preparation of this work, the authors used AI-assisted technologies to (i) polish the language and correct the grammar and LaTeX formatting of the manuscript

Page 19 of 22

Human Factors in AI-generated Code Tours Table 5 Distribution and assignment of the generated code tours. Group

Tour

Author

Project

Version

Trace

Error

Judge

1

1

Qwen

IBM-JSONata4Java

e09ee334c40e

3rd

Assertion

DeepSeek

1

2

Devstral

Nylle-JavaFixture

7184ab8dfacc

1st

Assertion

1

3

Qwen

aws-aws-lambda-snapstart-java-rules

beb0cf891122

1st

Assertion

1

1

1

1

1

4

6

7

9

11

DeepSeek

Qwen

DeepSeek

DeepSeek

Qwen

classgraph-classgraph

ezylang-EvalEx

fast-pack-JavaFastPFOR

joular-joularjx

medblocks-openFHIR

fbe61e01e8fa

942ad41ef07c

964056681432

9940cfc00a68

fdaa43715bb5

1st

1st

1st

116th

2nd

Assertion

Assertion

Assertion

Assertion

Comparison

Participants (Rank) P1 [0], P3 [0], P6 [0]

Devstral

P4 [0], P10 [0], P11 [0]

Qwen

P1 [1], P5 [0], P9 [0]

DeepSeek

P2 [0], P3 [1], P8 [0]

DeepSeek

P1 [2], P10 [1], P12 [0]

Devstral

P3 [2], P5 [1], P13 [0]

Devstral

P2 [1], P4 [1], P5 [2]

Qwen

P3 [3], P9 [1], P10 [2]

DeepSeek

P2 [2], P9 [2], P11 [1]

Devstral

P6 [1], P7 [0], P12 [1]

Qwen

P1 [3], P4 [2], P8 [1]

Devstral

P2 [3], P12 [2], P13 [1]

Devstral

P3 [4], P4 [3], P7 [1]

Qwen

P5 [3], P6 [2], P11 [2]

DeepSeek

P3 [5], P11 [3], P12 [3]

Devstral

P7 [2], P9 [3], P13 [2] P4 [4], P6 [3], P13 [3]

1

14

Qwen

mwilliamson-java-mammoth

b65bc063d98c

5th

Assertion

DeepSeek Devstral

P5 [4], P8 [2], P12 [4]

1

15

Devstral

package-url-packageurl-java

f2e1d7aeaf7f

9th

Assertion

DeepSeek

P1 [4], P11 [4], P13 [4]

Qwen

P5 [5], P7 [3], P10 [3] P1 [5], P2 [4], P7 [4]

Qwen

P6 [4], P8 [3], P9 [4]

1

17

Devstral

traccar-traccar

1a8487184423

1st

Assertion

DeepSeek

1

18

Qwen

traccar-traccar

de97c5099eb3

1st

Assertion

Devstral

P2 [5], P6 [5], P10 [4]

DeepSeek

P7 [5], P8 [4], P11 [5]

DeepSeek

P4 [5], P9 [5], P12 [5]

Qwen

P8 [5], P10 [5], P13 [5]

1

2

2

20

21

22

Devstral

DeepSeek

Qwen

xxDark-jlinker

IBM-JSONata4Java

Nylle-JavaFixture

29d2f1d5a58a

e09ee334c40e

7184ab8dfacc

1st

1st

2nd

Assertion

Assertion

Assertion

2

23

Qwen

classgraph-classgraph

3ed377e7f845

1st

Assertion

2

24

DeepSeek

cloudsimplus-cloudsimplus

0c3ecd7eb81e

1st

Assertion

2

2

2

2

2

25

26

27

29

32

Qwen

fast-pack-JavaFastPFOR

jhy-jsoup

Qwen

DeepSeek

Devstral

Qwen

joular-joularjx

medblocks-openFHIR

mwilliamson-java-mammoth

964056681432

78383995e7cf

9940cfc00a68

f11766986ce4

b65bc063d98c

2nd

91st

130th

6th

4th

Assertion

Assertion

Assertion

Comparison

Assertion

Devstral

P14 [0], P16 [0], P19 [0]

Qwen

P17 [0], P23 [0], P24 [0]

DeepSeek

P14 [1], P18 [0], P22 [0]

Devstral

P15 [0], P16 [1], P21 [0]

DeepSeek

P14 [2], P23 [1], P25 [0]

Devstral

P16 [2], P18 [1], P26 [0]

Devstral

P15 [1], P17 [1], P18 [2]

Qwen

P16 [3], P22 [1], P23 [2]

DeepSeek

P15 [2], P22 [2], P24 [1]

Devstral

P19 [1], P20 [0], P25 [1]

Devstral

P14 [3], P17 [2], P21 [1]

DeepSeek

P15 [3], P25 [2], P26 [1]

Devstral

P16 [4], P17 [3], P20 [1]

Qwen

P18 [3], P19 [2], P24 [2]

Qwen

P16 [5], P24 [3], P25 [3]

DeepSeek

P20 [2], P22 [3], P26 [2]

DeepSeek

P17 [4], P19 [3], P26 [3]

Devstral

P18 [4], P21 [2], P25 [4] P14 [4], P24 [4], P26 [4]

2

33

Devstral

package-url-packageurl-java

f2e1d7aeaf7f

8th

Assertion

DeepSeek Qwen

P18 [5], P20 [3], P23 [3]

2

34

Devstral

thibaultmeyer-cuid-java

d923856da2fb

1st

Assertion

DeepSeek

P14 [5], P15 [4], P20 [4]

Qwen

P19 [4], P21 [3], P22 [4]

Out of Bounds

Devstral

P15 [5], P19 [5], P23 [4]

DeepSeek

P20 [5], P21 [4], P24 [5]

Qwen

P17 [5], P22 [5], P25 [5]

Devstral

P21 [5], P23 [5], P26 [5]

2

2

35

36

Qwen

DeepSeek

traccar-traccar

traccar-traccar

a16ad4de29b3

52cab39d01d3

(Mammouth.ai and Grammarly), and (ii) assist in developing the pipeline and the web interface (Mammouth.ai and GitHub Copilot). All content generated with the assistance of these tools was thoroughly reviewed and edited by the authors to ensure it aligned with their intended meaning and Balfroid et al.: Preprint submitted to Elsevier

1st

1st

Null Pointer

contributions. The authors take full responsibility for the content of the published article.

Page 20 of 22

Human Factors in AI-generated Code Tours

Data availability The code, data, and instructions needed to replicate the experiments are available on Zenodo [3] and GitHub (https: //github.com/balfroim/HumanFactorsCodeTour).

References [1] Azanza, M., Pereira, J., Irastorza, A., Galdos, A., 2024. Can llms facilitate onboarding software developers? an ongoing industrial case study, in: 36th International Conference on Software Engineering Education and Training, CSEE&T 2024, Würzburg, Germany, July 29 - Aug. 1, 2024, IEEE, New York, NY, USA. pp. 1– 6. URL: https://doi.org/10.1109/CSEET62301.2024.10662989, doi:10. 1109/CSEET62301.2024.10662989. [2] Balfroid, M., 2025. Generating code tours using locally-runnable llms, in: Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pp. 1262–1265. [3] Balfroid, M., Devroey, X., Vanderose, B., Albert, J., Dzenatan, A., 2026. How developers experience debugging unfamiliar codebases with code tours generated and evaluated by local llms. URL: https: //doi.org/10.5281/zenodo.21628482, doi:10.5281/zenodo.21628482. [4] Balfroid, M., Vanderose, B., Devroey, X., 2024. Towards llmgenerated code tours for onboarding, in: Izadi, M., Sorbo, A.D., Panichella, S. (Eds.), Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering, NLBSE 2024, Lisbon, Portugal, 20 April 2024, ACM, New York, NY, USA. pp. 65–68. URL: https://doi.org/10.1145/3643787.3648033, doi:10.1145/ 3643787.3648033. [5] Bauer, T., Erdogan, B., 2011. Organizational socialization: The effective onboarding of new employees. APA handbook of industrial and organizational psychology 3, 51–64. doi:10.1037/12171-002. [6] Carter, J., 2020. Codetour - visual studio marketplace. URL: https://marketplace.visualstudio.com/items?itemName= vsls-contrib.codetour. [7] Cheng, M., Yu, S., Lee, C., Khadpe, P., Ibrahim, L., Jurafsky, D., 2025. Elephant: Measuring and understanding social sycophancy in llms. arXiv preprint arXiv:2505.13995 . [8] Dagenais, B., Ossher, H., Bellamy, R.K.E., Robillard, M.P., de Vries, J., 2010. Moving into a new software project landscape, in: Kramer, J., Bishop, J., Devanbu, P.T., Uchitel, S. (Eds.), Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE 2010, Cape Town, South Africa, 1-8 May 2010, ACM, New York, NY, USA. pp. 275–284. URL: https://doi.org/ 10.1145/1806799.1806842, doi:10.1145/1806799.1806842. [9] Destefanis, G., Ortu, M., Counsell, S., Swift, S., Marchesi, M., Tonelli, R., 2016. Software development: do good manners matter? PeerJ Computer Science 2, e73. [10] Docker, 2025a. ai/devstral-small - docker image. URL: https://web.archive.org/web/20260421004604/https://hub.docker. com/r/ai/devstral-small.

[11] Docker, 2025b.

ai/qwen2.5 - docker image.

URL:

https://web.archive.org/web/20260329135354/https://hub.docker. com/r/ai/qwen2.5.

[12] Fowkes, J., Chanthirasegaran, P., Ranca, R., Allamanis, M., Lapata, M., Sutton, C., 2017. Autofolding for source code summarization. IEEE Transactions on Software Engineering 43, 1095–1109. [13] Gaulthier, P., 2025. Aider benchmark. URL: https://aider.chat/ docs/leaderboards/edit.html. [14] Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Synnaeve, G., Wang, S.I., 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065 . [15] Hicks, M.T., Humphries, J., Slater, J., 2024. Chatgpt is bullshit. Ethics and Information Technology 26, 1–10. [16] Hu, X., Xia, X., Lo, D., Wan, Z., Chen, Q., Zimmermann, T., 2022. Practitioners’ expectations on automated code comment generation, in: Proceedings of the 44th international conference on software engineering, pp. 1693–1705.

Balfroid et al.: Preprint submitted to Elsevier

[17] Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al., 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 . [18] Ju, A., Sajnani, H., Kelly, S., Herzig, K., 2021. A case study of onboarding in software teams: Tasks and strategies, in: 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, IEEE, New York, NY, USA. pp. 613–623. URL: https://doi.org/10.1109/ICSE43902.2021.00063, doi:10.1109/ICSE43902.2021.00063. [19] Just, R., Jalali, D., Ernst, M.D., 2014. Defects4j: a database of existing faults to enable controlled testing studies for java programs, in: Pasareanu, C.S., Marinov, D. (Eds.), International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014, ACM, New York, NY, USA. pp. 437–440. URL: https: //doi.org/10.1145/2610384.2628055, doi:10.1145/2610384.2628055. [20] Kara, Z.B., İsmail, A., Ateş, E., Tamcı, İ.N., İyigün, Z., Aslangül, S.Ş., Devran, Ö., Uçar, B.M., Tüzün, E., 2026. Lacy: Simulating expert mentoring for software onboarding with code tours, in: Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering, pp. 450–460. [21] Kurt, U., 2026. Which quantization should i use? a unified evaluation of llama. cpp quantization on llama-3.1-8b-instruct. arXiv e-prints , arXiv–2601. [22] Lallemand, C., Gronier, G., 2015. Méthodes de design UX: 30 méthodes fondamentales pour concevoir et évaluer les systèmes interactifs. Editions Eyrolles. [23] Lee, J.D., See, K.A., 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 50–80. [24] Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., Hellas, A., 2023. Comparing code explanations created by students and large language models, in: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, pp. 124–130. [25] Lejeune, C., 2019. Manuel d’analyse qualitative. De Boeck Supérieur. [26] Li, D., Sun, R., Huang, Y., Zhong, M., Jiang, B., Han, J., Zhang, X., Wang, W., Liu, H., 2025. Preference leakage: A contamination problem in llm-as-a-judge. CoRR abs/2502.01534. URL: https:// doi.org/10.48550/arXiv.2502.01534, doi:10.48550/ARXIV.2502.01534, arXiv:2502.01534. [27] Liu, Y., Li, D., Wang, K., Xiong, Z., Shi, F., Wang, J., Li, B., Hang, B., 2024. Are llms good at structured outputs? a benchmark for evaluating structured output capabilities in llms. Information Processing & Management 61, 103809. [28] MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., Leinonen, J., 2023. Experiences from using code explanations generated by large language models in a web software development e-book, in: Doyle, M., Stephenson, B., Dorn, B., Soh, L., Battestilli, L. (Eds.), Proceedings of the 54th ACM Technical Symposium on Computer Science Education, Volume 1, SIGCSE 2023, Toronto, ON, Canada, March 15-18, 2023, ACM, New York, NY, USA. pp. 931–937. URL: https://doi.org/10.1145/3545945. 3569785, doi:10.1145/3545945.3569785. [29] Matturro, G., Barrella, K., Benitez, P., 2017. Difficulties of newcomers joining software projects already in execution, in: 2017 International Conference on Computational Science and Computational Intelligence (CSCI), IEEE. pp. 993–998. [30] Miller, E., 2024. Adding error bars to evals: A statistical approach to language model evaluations. CoRR abs/2411.00640. URL: https:// doi.org/10.48550/arXiv.2411.00640, doi:10.48550/ARXIV.2411.00640, arXiv:2411.00640. [31] Nakano, H., Takezawa, J., Matulic, F., Yang, C.L., Yatani, K., 2026. Understanding reader perception shifts upon disclosure of ai authorship, in: Proceedings of the 31st International Conference on Intelligent User Interfaces, pp. 2131–2146. [32] Rastogi, A., Yang, A., Jiang, A.Q., Liu, A.H., Sablayrolles, A., Héliou, A., Martin, A., Agarwal, A., Ehrenberg, A., Lo, A., et al., 2025. Devstral: Fine-tuning language models for coding agent applications. arXiv preprint arXiv:2509.25193 .

Page 21 of 22

Human Factors in AI-generated Code Tours [33] Reynolds, L., McDonell, K., 2021. Prompt programming for large language models: Beyond the few-shot paradigm, in: Kitamura, Y., Quigley, A., Isbister, K., Igarashi, T. (Eds.), CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts, ACM, New York, NY, USA. pp. 314:1–314:7. URL: https://doi.org/10.1145/3411763. 3451760, doi:10.1145/3411763.3451760. [34] Rokeman, N.R.M., 2024. Likert measurement scale in education and social sciences: explored and explained. EDUCATUM Journal of Social Sciences 10, 77–88. [35] Saavedra, N., Silva, A., Monperrus, M., 2024. Gitbug-actions: Building reproducible bug-fix benchmarks with github actions, in: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2024, Lisbon, Portugal, April 14-20, 2024, pp. 1–5. URL: https: //doi.org/10.1145/3639478.3640023, doi:10.1145/3639478.3640023. [36] Sallou, J., Durieux, T., Panichella, A., 2024. Breaking the silence: the threats of using llms in software engineering, in: Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, NIER@ICSE 2024, Lisbon, Portugal, April 14-20, 2024, ACM, New York, NY, USA. pp. 102–106. URL: https://doi.org/10.1145/3639476.3639764, doi:10. 1145/3639476.3639764. [37] Santos, I., Felizardo, K.R., Steinmacher, I., Gerosa, M.A., 2025. Software solutions for newcomers’ onboarding in software projects: A systematic literature review. Information and Software Technology 177, 107568. [38] Schuszter, I.C., Cioca, M., 2024. Increasing the reliability of software systems using a large-language-model-based solution for onboarding. Inventions 9, 79. URL: https://doi.org/10.3390/inventions9040079, doi:10.3390/inventions9040079. [39] Silva, A., Saavedra, N., Monperrus, M., 2024. Gitbug-java: A reproducible benchmark of recent java bugs, in: Spinellis, D., Bacchelli, A., Constantinou, E. (Eds.), 21st IEEE/ACM International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, April 15-16, 2024, ACM, New York, NY, USA. pp. 118–122. URL: https: //doi.org/10.1145/3643991.3644884, doi:10.1145/3643991.3644884. [40] Stinson, D.R., 2008. Combinatorial designs: constructions and analysis. ACM SIGACT News 39, 17–21. [41] Taylor, G., Clarke, S., 2022. A tour through code: Helping developers become familiar with unfamiliar code, in: Holland, S., Petre, M., Church, L., Marasoiu, M. (Eds.), Proceedings of the 33rd Annual Workshop of the Psychology of Programming Interest Group, PPIG 2022, The Open University, Milton Keynes, UK & Online, September 5-9, 2022, Psychology of Programming Interest Group, New York, NY, USA. pp. 114–126. URL: https://ppig.org/papers/ 2022-ppig-33rd-taylor/. [42] Wang, H., 2025. Llm knowledge cut-off dates summary. URL: https://web.archive.org/web/20260520072815/https://github. com/HaoooWang/llm-knowledge-cutoff-dates. [43] Wang, R., Guo, J., Gao, C., Fan, G., Chong, C.Y., Xia, X., 2025. Can llms replace human evaluators? an empirical study of llm-as-ajudge in software engineering. Proceedings of the ACM on Software Engineering 2, 1955–1977. [44] Weyssow, M., Kamanda, A., Sahraoui, H.A., 2024. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences. CoRR abs/2403.09032. URL: https:// doi.org/10.48550/arXiv.2403.09032, doi:10.48550/ARXIV.2403.09032, arXiv:2403.09032. [45] Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al., 2024. Laboratoryscale ai: Open-weight models are competitive with chatgpt even in low-resource settings, in: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 1199–1210. [46] Yu, J.S., Yao, Y., 2026. Structured and intelligent technical writing, in: Intelligent Language Services: Theory and Practice with Large Language Models. Springer, pp. 271–296.

Balfroid et al.: Preprint submitted to Elsevier

[47] Zhang, C., Wang, J., Zhou, Q., Xu, T., Tang, K., Gui, H., Liu, F., 2022. A survey of automatic source code summarization. Symmetry 14, 471. [48] Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I., 2023. Judging llm-as-a-judge with mt-bench and chatbot arena, in: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Curran Associates, Inc., Red Hook, NY, United States. p. 29. URL: http://papers.nips.cc/paper_files/paper/2023/ hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_ Benchmarks.html.

[49] Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al., 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931 . Martin Balfroid is a PhD student at the University of Namur. His research investigates AI-in-the-loop approaches to improve software engineering. He earned his master’s degree in Computer Science, with a focus on Data Science, in June 2022. Martin began his PhD in July 2022 with funding from the ARIAC project and is supervised by Assistant Professors Benoît Vanderose and Xavier Devroey. After an initial career as a librarian, Julien Albert transitioned to computer science and obtained a master’s degree from UNamur in 2020. He then worked for one year at UNamur on the EFFaTA-MeM research project, which aims to develop innovative tools for text analysis. In September 2021, he began a Ph.D. in computer science at UNamur under the supervision of Professors Benoît Frenay and Bruno Dumas. His research area is explainability in artificial intelligence. His approach places the user at the center of the concerns by combining machine-learning explainability techniques with methods developed in human-computer interaction. Dzenatan Aliti is a master’s student at the University of Namur. He was a research trainee at the SNAIL Team, working on software testing education through gamification. Xavier Devroey is an associate professor of software engineering at the University of Namur, where, together with Benoît Vanderose, he co-leads the SNAIL Team. His main research interests include automated software testing, test suite generation and augmentation, DevOps, software testing, and variability-intensive systems. Xavier has served as program committee member in various software engineering conferences, including ICST, ASE, EASE, MSR, and as reviewer for various international journals, including JSS, EMSE, TSE, and STVR. Benoît Vanderose is an associate professor of software engineering at the University of Namur, where he co-leads the SNAIL Team with Xavier Devroey. His main research interests include Agile software development, DevEx, and software quality.

Page 22 of 22

Human Factors in AI-generated Code Tours

StylesXmlTests.java — whenMultipleNonNumberingStyleElementsHaveSameStyleIdThenOnlyFirstElementIsUsed()

@Test public void whenMultipleNonNumberingStyleElementsHaveSameStyleIdThenOnlyFirstElementIsUsed () { XmlElement element = element ( " w : styles " , list ( element ( " w : style " , map ( " w : type " , " table " , " w : styleId " , " TableNormal " ) , list ( nameElement ( " Normal Table " ) ) ) , element ( " w : style " , map ( " w : type " , " table " , " w : styleId " , " TableNormal " ) , list ( nameElement ( " Table Normal " ) ) ) )); Styles styles = readStylesXmlElement ( element ) ; assertEquals ( Optional . of ( " Normal Table " ) , styles . findTableStyleById ( " TableNormal " ) . get () . getName () ) ; }

Step 1/3 Start by examining the failing test method in the provided Java codebase. Role: The whenMultipleNonNumberingStyleElementsHaveSameStyleIdThenOnlyFirstElementIsUsed method tests the behavior of the readStyles function when multiple non-numbering style elements have the same style ID. Specifically, it ensures that only the first element with a given ID is used. Execution: The test sets up an XML structure containing two w:style elements with the same w:styleId but different names (Normal Table and Table Normal). It then calls readStylesXmlElement to process this XML and checks that the style returned by findTableStyleById has a name of Normal Table, which is the first encountered. Potential Causes: 1. The readStyles function might not be correctly filtering elements based on their type or ID. 2. The order in which styles are processed could lead to unexpected results, such as picking the second element instead of the first.

StylesXml.java — readStylesXmlElement()

public class StylesXml { public static Styles readStylesXmlElement ( XmlElement element ) { XmlElementList styleElements = element . findChildren ( " w : style " ) ; return new Styles ( readStyles ( styleElements , " paragraph " ) , readStyles ( styleElements , " character " ) , readStyles ( styleElements , " table " ) , readNumberingStyles ( styleElements ) ); }

Step 2/3 Navigate to the readStylesXmlElement method in the provided Java codebase. Role: The readStylesXmlElement method is a factory function that creates a Styles object from an XML element. It processes different types of styles (paragraph, character, table, and numbering) by calling separate helper methods for each type. Execution: The readStylesXmlElement method finds all child elements with the tag w:style within the provided XML element. It then invokes readStyles, readNumberingStyles, and other helpers to extract and store styles of different types in a Styles object. Potential Causes: If any style fails to be processed or an unexpected error occurs during the processing, it could potentially affect the final Styles object returned by this method.

StylesXml.java — readStyles()

private static Map < String , Style > readStyles ( XmlElementList styleElements , String styleType ) { return toMap ( styleElementsOfType ( styleElements , styleType ) , StylesXml :: readStyle ) ; }

Step 3/3 Navigate to the readStyles method in the provided Java codebase. Role: The readStyles method is responsible for filtering and processing w:style elements of a specific type (e.g., paragraph, character, table). It extracts and maps these styles into a Map<String, Style> where each key is the style ID and each value is the corresponding Style object. Execution: The readStyles method takes an XML element list containing multiple w:style elements and filters them based on the given styleType. It then maps these filtered elements into a map of styles, using their IDs as keys by calling the readStyle helper method for each element. Potential Causes: 1. If there are duplicate style IDs within the filtered list, it could lead to unexpected behavior, such as overwriting existing entries in the map with new ones. This could result in a scenario where the last encountered style with a given ID is used instead of the first one.

Figure 10: Code Tour 32 (mwilliamson-java-mammoth, commit b65b...) generated by Qwen2.5-Coder 14B. The tour covers the parsing order of XML styles (supposedly for a Word document, since Mammoth is a Word-to-HTML converter). The test fails because of an AssertionFailedError, expecting <Optional[Normal Table]> but receiving <Optional[Table Normal]>. The structure of the descriptions is noteworthy because it is separated into three sections: role, execution, and potential causes.

Balfroid et al.: Preprint submitted to Elsevier

Page 23 of 22

Related documents

• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks#668134
• CAROL: Context-Aware Online Learning for Fuzzer Scheduling#668135
• An Autonomy Aware Metamodel for Human AI Collaboration in Software Engineering#668136
• A Unified Policy Architecture (UPA): The Governance Kernel for Enterprise AI Operating Systems#668137
• From Reading Code to Reading Spec: A Verified Layer for LLM-Driven Codebase Maintenance#668138