Who Is Doing the Thinking? Characterising Human Agency in First-Year Students' AI Interactions

Who Is Doing the Thinking? Characterising Human Agency in First-Year Students’ AI Interactions Ali Shakiba✉

Thomas Chaffey

ali.shakiba@sydney.edu.au The University of Sydney Sydney, NSW, Australia

thomas.chaffey@sydney.edu.au The University of Sydney Sydney, NSW, Australia

arXiv:2611.04783v1 [cs.SE] 3 Nov 2026

Abstract When students use generative AI to produce their work, the learning process becomes invisible. Established frameworks such as Bloom’s taxonomy then classify the cognitive level of what the AI produced, not what the student contributed. We propose the Human Agency Depth Taxonomy (HADT), a preliminary six-tier taxonomy that classifies the engineering contribution a student makes in a prompt, independent of the AI’s output. HADT combines three dimensions: contextual bandwidth, engineering abstraction and student agency. We applied HADT to 962 prompts from 105 students’ structured AI Journals in an AI-permissive, first-year software engineering course of 145 students, and examined how agency changed over ten journalled weeks. Most prompts (84.2%) carried some student contribution beyond delegation. Agency tracked the development lifecycle. Architecture-discovery prompts dominated the requirements phase (48% of turns) and fell to 5-12% from week 9, while prompts analysing existing systems rose during build and verification (odds ratio 2.88, with turns clustered by student). Two-thirds of the variance in tier lay within students, so agency depended more on the phase of work than on the student. A turn-level rise in delegation in deadline weeks did not hold once turns were clustered by student. Explicitly specifying constraints on the AI’s output was rare throughout, which we interpret as a knowledge bottleneck for novices rather than a gap in instruction. We discuss implications for course and assessment design, and for using HADT as a formative lens on how students engage with AI.

CCS Concepts • Social and professional topics → Software engineering education.

Keywords Generative AI, Human Agency, Software Engineering Education, First-Year Students

1

Introduction

Generative AI (GenAI) has quickly become a mainstream tool in computing education. Large language models (LLM) now solve typical programming exercises with high accuracy [12, 15, 16], and students use them across the software development lifecycle (SDLC) [10, 33], from eliciting requirements to generating designs, code and documentation. In response, many courses have moved from prohibiting AI to permitting, and in some cases teaching its use [25, 29, 40]. This shift exposes an old assessment problem in a new form. Consider two identical arithmetic results, one worked by hand and the other produced with a calculator. Without seeing the working,

an assessor cannot tell them apart. AI-permissive courses face the same difficulty at scale: when only the final artefact is assessed, whether a report, a design or a working application, the process behind it and the student’s contribution to it become invisible[3, 40]. Unrestricted AI use carries an additional risk. Students may offload the cognitive work of a task to the model, producing polished artefacts while conceptual gaps go undetected [2, 21, 35]. Drawing on Vygotsky’s zone of proximal development [46], a zone of no development is created [13]: the AI’s support is so complete that the task no longer requires the student to engage in the learning it was intended to. Reflection instruments can make the process visible again. Structured AI journals [40], in which students log their prompts and outputs and reflect on how they verified them, give educators a record of human-AI interaction. Analyses of such records characterised how novices oversee AI output [39]. Visibility, however, is not a measurement. To determine from these records whether and when students are engaged in the task or delegating it to AI, educators need a way to classify the extent of engineering thinking that students contribute to each interaction. Existing frameworks are poorly suited to this purpose. Taxonomies such as revised Bloom’s [22, 44] and SOLO [4, 28] classify the cognitive level of a task or a response. When AI produces the response, these classifications describe the machine’s output rather than the student’s thinking. Consider a first-year student who prompts: “Help me design the data structure for our Water Rescue Coordination System in Power Apps. The app needs to store incidents, volunteers, assignments, and rescue equipment ...”. Under Bloom’s taxonomy, this is a Create-level task. Yet, the label does not tell us whether the student engaged in any of the designing. They may have specified the entities, constraints and acceptance criteria that shape the answer, or they may have delegated the entire problem to the model. The same label applies in both cases. This paper addresses that gap. We propose the Human Agency Depth Taxonomy (HADT), a six-tier taxonomy that classifies the engineering contribution a student makes through a prompt, independent of what the AI generates. HADT combines three dimensions: Contextual Bandwidth (generic to project-specific), Engineering Abstraction (local to system-wide), and Student Agency (whether the student or the AI defines the problem). We apply HADT to 962 prompts from collected AI journals in a 13-week first-year software engineering course of 145 students, in which AI use was permitted and taught. We address two research questions:

RQ1: What dimensions and levels distinguish the human contribution in students’ prompts, independent of what the AI generates?

Shakiba et al.

RQ2: How is human agency distributed across first-year students’ AI interactions, and how does it change over the semester? This paper makes three contributions: (1) HADT, a taxonomy of human agency in AI prompts, with three dimensions, six tiers, and decision rules. (2) An empirical account of agency across a semester. Most interactions are not passive (84.2% above T0). Agency follows the SDLC, from design-space exploration early to implementation and diagnosis later. (3) Implications for course and assessment design. If agency depends on the situation rather than the student, the timing and structure of assessment shape how students use AI. The rest of the paper is organised as follows: Section 2 reviews the background and literature. Section 3 describes the method: the course and data, and the development of HADT (RQ1). Section 4 reports agency patterns (RQ2). Section 5 discusses implications, Section 6 sets out limitations and threats to validity, and Section 7 concludes with future research directions.

2

Background and Related Work

The widespread availability of GenAI has challenged established approaches to teaching and assessing programming and software engineering. Early studies demonstrated that code-generating models could solve most typical introductory programming exercises and examination questions [15]. Subsequent research has identified opportunities for personalised assistance, feedback, and exercise generation, alongside concerns about academic integrity, over-reliance, and the validity of traditional assessments [12, 33]. In response, educators have adopted approaches ranging from restricting AI use to redesigning learning activities and assessments around its capabilities [25, 40]. A central concern is whether improved performance on AIassisted tasks translates into meaningful learning. Empirical findings remain mixed. In a controlled study, novice programmers using an AI code generator completed more tasks without a corresponding decline in subsequent unaided performance [21]. However, observational research has documented students accepting generated solutions without fully understanding them, highlighting cognitive and metacognitive difficulties in AI-assisted programming [34]. Prather et al. further observed that GenAI could widen differences between learners: students with stronger metacognitive skills benefited from the assistance, whereas others developed an illusion of competence [35]. Similar concerns extend beyond computing education. A large field experiment found that unrestricted chatbot access improved practice performance but reduced subsequent unassisted performance, whereas pedagogically constrained assistance largely mitigated this effect [2]. Together, these findings suggest that successful task completion is not necessarily evidence of learning and that how students engage with AI assistance matters. This distinction can be understood through the literature on cognitive offloading and task delegation. Cognitive offload refers to the use of external resources to reduce cognitive demands, which can free up capacity for other activities while also reducing involvement in important learning processes [36]. Research on task delegation has examined the activities people prefer to assign to AI systems

and the factors that influence these decisions [30]. More recent studies have associated GenAI use with reduced critical-thinking effort among knowledge workers [26] and with patterns of metacognitive disengagement in educational settings, where improvements in task performance are not necessarily accompanied by comparable gains in knowledge or transfer [14, 18, 35]. Although these findings do not establish that delegation is inherently detrimental, they highlight the importance of distinguishing between AI assistance that supports students’ reasoning and assistance that substitutes for it. One way to examine this distinction is through students’ interactions with GenAI systems, particularly their prompts. Existing prompting research has focused primarily on techniques for eliciting effective model responses. Prompt-pattern catalogues and surveys describe strategies such as few-shot examples, role prompting, and chain-of-thought prompting [38, 48]. In computing education, Prompt Problems have been proposed to develop and assess students’ ability to formulate prompts that produce correct code [11]. Other studies have analysed student-AI conversations by classifying prompts according to their conversational function, such as initial requests, follow-up questions, and corrections, or according to the problem-solving issues they address [37]. Prompts have also been mapped to cognitive levels using the revised Bloom’s taxonomy [5]. These approaches provide useful insights into prompting strategies, interaction patterns, and the cognitive demands of requested tasks. However, they do not necessarily distinguish between prompts that delegate an entire problem to AI and those that communicate substantial reasoning or problem-solving work already undertaken by the student. A technically sophisticated prompt, for example, may specify detailed output requirements without providing evidence that the student has independently analysed the underlying engineering problem. Established educational taxonomies offer complementary perspectives on cognitive engagement and learning. The revised Bloom’s taxonomy classifies cognitive processes from remembering to creating [22]. It has been widely applied to programming assessment design, although difficulties in consistently interpreting its categories within computing have motivated domain-specific adaptations [20, 44]. The SOLO (Structure of Observed Learning Outcomes) taxonomy instead examines the structural complexity of learners’ responses [4], including their ability to read, interpret, and explain programs [28, 47]. The ICAP (Interactive, Constructive, Active, Passive) framework approaches learning through observable engagement, distinguishing passive, active, constructive, and interactive behaviours and proposing that deeper forms of engagement generally support stronger learning outcomes [6]. While these frameworks remain valuable, applying them to AIassisted work raises a fundamental question: whose cognitive contribution is being observed? Bloom’s taxonomy describes the cognitive demands of a task, but a prompt asking AI to design a database schema does not establish that the student performed the associated design reasoning. Similarly, the structural complexity of an AI-generated response does not necessarily reflect the student’s understanding. ICAP offers a closer perspective by focusing on learner engagement, but its categories were not developed specifically to distinguish the intellectual contributions of a learner and a generative system. For example, submitting an assignment brief to a chatbot is an observable action, yet provides little evidence that

3 Method 3.1 Context and Data The study took place in ELEC1005 Introduction to Software Engineering at the University of Sydney, a first-year, first-semester course. The cohort comprised 145 students, 140 of whom were in their first semester at university, and no prior programming experience was assumed. The course provides a compressed introduction to the entire SDLC, from planning and requirements through design and implementation to verification and deployment, structured around a team project spanning two assignments (A1 and A2). Rather than teaching programming, the course focuses on engineering processes and uses industry tools: Jira for planning, Figma for prototyping, Git for version control, UML diagrams for design, and a low-code/no-code environment (Power Apps with SharePoint lists) for implementation. Students declared their AI use through a structured AI Journal [40], which records each humanAI conversation in two layers. The first layer, execution tracking, captures the prompt, the AI’s output, the model and version, the environment, and the student’s intent. The second layer, cognitive auditing, asks students to reflect on how they verified the output and their confidence in that verification (1-5), how useful the AI was (1-5), what they changed or rejected, their most important learning moment, and which parts would be hardest to complete

1000 900 800 700 600 500 400 300 200

Interactions retained 100 Students retained

Students retained

the student has independently framed or analysed the underlying engineering problem. Consequently, neither the cognitive level of the requested task, the sophistication of the generated response, nor the observable interaction alone necessarily reveals the extent of the student’s contribution. Recent research has begun adapting established cognitive frameworks to AI-assisted education, including proposals for AI-specific competencies [19] and approaches that use Bloom’s taxonomy to guide the integration of GenAI into software engineering and computing curricula [17]. Such work provides guidance on the competencies students should develop and the types of activities in which AI assistance may be appropriate. However, a complementary need remains for methods that characterise how students distribute problem-solving responsibilities between themselves and AI during actual interactions. Our proposed HADT characterises the student’s observable engineering contribution within a prompt, rather than the sophistication of the prompting technique, the cognitive level of the requested task, or the quality of the generated output. This distinction is important: a carefully structured prompt containing few-shot examples and explicit formatting instructions may still delegate the underlying engineering problem almost entirely to AI, whereas a brief prompt presenting the student’s own analysis, evidence, or hypothesis may demonstrate substantial prior problem framing. By examining what students contribute before requesting assistance, HADT provides a prompt-level perspective on the distribution of engineering work between students and GenAI. It does not directly measure learning, cognitive effort, or the correctness of students’ reasoning; rather, it provides a means of characterising observable patterns of agency and delegation that can inform the interpretation of AI-assisted student work.

Prompts retained

Who Is Doing the Thinking? Characterising Human Agency in First-Year Students’ AI Interactions

80 60 40 20 0

10

20

30

40

50

60

70

80

90 100

0

Minimum % of weeks with AI Journal

Figure 1: Prompts and students retained by minimum share of weeks with an AI Journal; the 50% threshold defines the trajectory subset (27 students, 608 turns) used in the analysis of Section 4.3

again without AI. This paper analyses only the prompts captured in the first layer. AI and human agency were introduced in Week 2 by sharing realworld use cases of inresponsible use of AI as comic books and an in-class padlet activity. In Week 3, students were introduced to the AI Journal, worked through real-world case studies, and completed a warm-up journal, which was excluded from the analysis. From Week 4 onwards, each student submitted a weekly declaration, either reporting that they had not used AI that week or providing an AI Journal documenting their use. The journal was an unmarked hurdle task. Students were explicitly told that their journals would never be used to reduce their marks and could instead serve as evidence of their own contribution, for example, in team disputes. This design was intended to encourage honest disclosure rather than responses optimised for assessment. The dataset comprises all 962 prompts submitted through AI Journals between Weeks 4 and 13 by 105 students (40 students declared they did not use AI at all). These prompts were coded and are analysed in Section 4. The number of submitted prompts declined over the semester, from 187 in Week 4 to between 58 and 75 per week from Week 8 onwards (Table 3). For the longitudinal analysis (Section 4.3), we retain the 27 students who submitted an AI Journal in at least 50% of the eligible weeks, contributing 608 prompts in total (Figure 1).

3.2

Human Agency Depth Taxonomy

HADT classifies each prompt according to the engineering contribution the student makes through it, independent of what the AI generates. We initially coded prompts using the revised Bloom’s taxonomy [22] as part of the course’s routine weekly quality assurance. This approach quickly encountered the measurement problem described in Section 1. Prompts such as the database-schema example could be classified as create, yet they ranged from fully delegated requests to tightly constrained specifications in which the student had already defined much of the problem. We therefore set Bloom’s taxonomy aside and examined what distinguished prompts that contained evidence of student thinking from those that did not.

Shakiba et al.

Table 1: Weekly lecture topics, project activities, assessments, AI instruction, and analysis phases. Wk

Lecture topic

Group Project

Assessment

Explicit AI instruction

Analysis phase

1 2

Intro to SDLC Planning & Agile (Jira) Analysis & UX, user stories, Figma

— Groups form

— —

— GenAI and agency

Not journalled Not journalled

A1 starts

Early-feedback quiz (AI allowed)

Excluded (warm-up)

Requirements engineering, Functional and Non-functional requirements, acceptance criteria, Git Architecture & design, UML Implementation: manual → no-code → AI coding

A1 requirements

—

AI Journal intro + warm-up; GenAI for Requirements engineering (“context starvation”) LLMs and Requirements engineering

A1 design

—

—

Design & prototype

A1 prototype

Mid-sem quiz

AI-assisted coding

Design & prototype · deadline

3

4

5 6

Analysis

Mid-semester break 7

8 9 10

Design patterns, Object-oriented design Verification & validation Quality engineering, CI/CD CI/CD, documentation

A1 due Fri; A2 released

A1 due

—

Design & prototype · deadline

A1 demos; A2 build

A1 demos

—

Build & verify

A2 build

—

—

Build & verify

A2 build; V&V practical

—

—

Build & verify

11

System design & architecture, Architectural decision records

A2 build

—

—

Build & verify

12

Technical debt, spec-driven development Career paths, wrap-up

A2 deliverables

A2 due

Spec-driven AI across the SDLC

Build & verify · deadline

A2 demos; final deliverables Fri

A2 demos

—

Build & verify · deadline

13

Three recurring differences emerged: the amount of projectspecific context supplied by the student, the level of the system addressed by the request, and whether the student or the AI defined the problem. We treated these as the three dimensions of the proposed taxonomy. The first author then grouped recurring combinations of these dimensions into tiers. The tier boundaries and decision rules were refined over two rounds of coding until additional prompts could be classified without requiring further revisions to the scheme. Each dimension describes one aspect of what the student, rather than the AI, contributes: Contextual bandwidth (generic to project-specific). The amount of project-specific context the student contributes, including requirements, data, constraints, earlier decisions,

or evidence. A generic prompt (“What is a primary key?”) could come from any student. A project-specific prompt contains information specific to the system being built. High contextual bandwidth provides evidence that the student is bringing knowledge of their project into the interaction and making that knowledge available to the AI. Engineering abstraction (local to system-wide). The scope at which the request engages with the system, ranging from an isolated component (a formula, a screen, or a table) to that component in the context of the wider system (its interfaces, data flows, requirements, and trade-offs). System-wide prompts provide evidence that the student is considering how multiple parts of the system fit together, a core software engineering competence.

Who Is Doing the Thinking? Characterising Human Agency in First-Year Students’ AI Interactions

Student agency (AI defines the problem to student defines the problem). The extent to which the student, rather than the AI, frames the problem. At the low end, the student forwards a task, often by pasting an assignment brief, lab sheet, or marking criteria, and leaves the AI to interpret, decompose, and solve it. At the high end, the student has already framed and decomposed the problem, specifying what needs to be addressed, and uses the AI to execute, diagnose, or explore within that frame. The dimensions are related but not redundant, and no single dimension increases monotonically across the tiers. A prompt can be highly project-specific yet exhibit low student agency, for example when a student pastes the full project brief and asks the AI to solve it. Similarly, a T4 diagnostic request may concern a single local formula and therefore have lower engineering abstraction than a T3 request framed by system-wide constraints. HADT therefore defines each tier by the combined profile of the three dimensions and the engineering act requested: retrieving knowledge, generating a component, generating under constraints, diagnosing an existing artefact, or exploring the design space (Table 2). The tiers are ordered by the extent of engineering reasoning evident in the student’s prompt before the AI responds. A diagnostic prompt ranks above constrained generation because the student contributes an existing artefact or observation together with a hypothesis about its cause. Design-space exploration ranks highest because the student frames the alternatives or constraints to be explored while retaining responsibility for the design decision. T0 - Delegating the problem. The prompt forwards a task without any student-authored constraints, decomposition, or evidence. Rule: If pasted material (e.g., an assignment brief, lab sheet, or marking criteria) were removed, no student contribution to framing or solving the problem would remain. T1 - Looking up knowledge. The prompt asks a factual or how-to question (e.g., a definition, syntax, or tool feature) that is independent of the student’s project. Rule: The answer would be essentially the same for any student. T2 - Requesting components. The prompt requests a specific, local artefact (e.g., a formula, function, screen, or table) and states the student’s intent for that artefact, but not the wider system constraints it must satisfy. Rule: The student has decomposed the problem into a specific component, and the request is scoped to that component. T3 - Specifying constraints. The prompt requests an artefact and specifies the system constraints it must satisfy (e.g., requirements, interfaces, data model, or acceptance criteria). Rule: The constraints come from the student’s own system and provide criteria against which the output can be evaluated. T4 - Analysing existing systems. The student supplies evidence from a system they have built (e.g., an error, output, or artefact), together with a hypothesis or question, and asks the AI to diagnose or evaluate it.

Rule: The student brings an existing artefact or observation to the interaction and asks the AI for diagnosis or evaluation rather than replacement. T5 - Constraint-driven architecture discovery. The student frames alternative designs, scenarios, or “what-if” constraints and uses the AI to explore the design space and its trade-offs. Rule: The student frames the alternatives or constraints to be explored, and the design decision remains with the student.

3.3

Coding Procedure

The unit of analysis is a single prompt, defined as one student turn within an AI Journal entry. An AI Journal entry records a conversation containing one or more student prompts, each of which was coded separately and assigned exactly one HADT tier. Coders used only the prompt text; other journal fields were not consulted. Content within the prompt was attributed to the student only when it went beyond material provided in the course, such as the student’s own requirements, designs, data, decisions, or evidence from their system. A prompt that reproduced or closely followed teaching material, such as an assignment brief, lab sheet, or marking criteria, without additional student-provided context was coded as T0, regardless of its length. This rule prevents copied task context from being treated as evidence of student engineering contribution. We used the following decision rules for edge cases: • Pasted teaching material with no student-authored addition is assigned T0. • Multi-part prompts are assigned the highest tier supported by their student-authored content. • Follow-up prompts are coded independently using only the text of the current prompt, without reference to earlier prompts in the conversation. A follow-up that itself supplies evidence (e.g., “this formula returns blank when the volunteer list is empty”) can reach T4; a bare “fix it” or “try again” adds no student engineering contribution and is assigned T0. This rule reflects what HADT measures: the engineering contribution added by each prompt. Context already available to the AI from earlier prompts or conversation history is not treated as a new contribution in the current prompt. A short follow-up that states a decision (e.g., “key it on volunteers, not incidents”) is still credited because the decision is explicit in its text. • A reference to an attachment contributes to contextual bandwidth only when the prompt explicitly refers to the attachment and indicates what it contains or why it matters (e.g., “using the attached ER diagram, check whether the Assignments table supports multiple volunteers”). A bare “see attached” does not contribute to contextual bandwidth, and an attachment alone never raises the tier. The first author coded all prompts blind to student identity. To assess inter-rater reliability, the second author independently coded a random sample of approximately 27% of the prompts (𝑛 = 261), using Table 2 and the decision rules above. Because HADT tiers

Shakiba et al.

Table 2: Dimension profile for each HADT tier. The dimensions are not individually monotonic across tiers; tiers are distinguished by their dimension profile together with the engineering act requested. Tier

Contextual bandwidth

Engineering abstraction

Student agency

Engineering act

T5

High (own requirements, constraints)

System-wide

Explore

T4 T3 T2 T1 T0

High (evidence from own artefact) High (own requirements, data model) Medium (names own component) Low (generic) Low (teaching material only)

Local or system-wide System-wide Local Local / none As given by brief

Frames alternatives; keeps decision Frames a hypothesis Fixes the constraints Has decomposed the problem Narrow factual question AI defines the problem

are ordinal, we report linearly weighted Cohen’s 𝜅 [8] as the primary reliability measure, alongside unweighted Cohen’s 𝜅 [7] and percentage agreement. We interpret 𝜅 values using the Landis and Koch bands [24]. The coders agreed on 87.4% of the 261 double-coded prompts. Unweighted Cohen’s 𝜅 was 0.82 (95% CI 0.77 − 0.88), and linearly weighted Cohen’s 𝜅 was 0.88 (95% CI 0.84 − 0.92), indicating almost perfect agreement under the Landis and Koch interpretation. Agreement was complete for T0, T1, and T4. Of the 33 disagreements, 30 occurred between T3 and T5; in 29 of these, the first author assigned T5 and the second author T3. These prompts asked the AI to design an artefact from student-supplied requirements or constraints, creating ambiguity between constrained generation (T3) and design-space exploration (T5). The remaining three disagreements were between T2 and T3. Both coders were also course co-coordinators and lecturers: the first author taught primarily in the second half of the course and the second author in the first half. Their familiarity with the course context may have informed their interpretation of the prompts, although both applied the same documented taxonomy and decision rules.

3.4

Statistical Analysis

All analyses were conducted in Python using SciPy [45] and statsmodels [42]. We use 𝛼 = .05, report exact 𝑝 values, and apply no correction for multiple comparisons. We compared tier distributions across weeks and phases using Pearson’s 𝜒 2 test of independence [32], applied to the week-by-tier (10×6) and phase-by-tier (3×6) √︁ contingency tables of prompt counts. We report Cramer’s 𝑉 = 𝜒 2 /[𝑁 (𝑘 − 1)] as the effect size, where 𝑁 is the number of prompts and 𝑘 is the smaller table dimension [9]. To identify the cells contributing most strongly √︁ to an association, we report Pearson residuals, 𝑟𝑖 𝑗 = (𝑂𝑖 𝑗 − 𝐸𝑖 𝑗 )/ 𝐸𝑖 𝑗 , where 𝑂𝑖 𝑗 and 𝐸𝑖 𝑗 are the observed and expected counts, respectively. Absolute residuals above approximately 2 indicate cells that depart notably from independence [1]. We tested monotonic changes in each tier’s weekly share using Spearman’s rank correlation 𝜌 between week number and the tier’s share in that week [42]. With only ten weekly observations, these tests have low power and describe changes in aggregate weekly shares rather than individual prompts.

Diagnose / evaluate Generate under constraints Generate component Retrieve Delegate

We compared the proportion of T0 prompts between deadline and other weeks using a 2 × 2 𝜒 2 test with Yates’ continuity correction [50], and report the sample odds ratio (OR) as the effect size [1]. The analyses above treat prompts as independent observations, although prompts are nested within students and students contribute different numbers of prompts. We therefore re-estimated the key effects using generalised estimating equations (GEE) [27]. Each model uses a binary outcome indicating whether a prompt is T0, T4, or T5, with week type or phase as the predictor, an exchangeable working correlation within students, and robust (sandwich) standard errors. We report exponentiated coefficients as odds ratios with 95% confidence intervals and draw inferential conclusions only for effects that remain supported after accounting for within-student clustering. Clustered checks were limited to the tiers underlying our main claims (T0, T4, T5); other tier shifts are reported descriptively. For the longitudinal subset (Section 4.3), we estimated the proportion of variation in tier attributable to differences between students rather than variation within students using a linear mixed-effects model with a random intercept for student and phase as a fixed effect [23]. Tier was treated as numeric for this analysis. The intraclass correlation coefficient was calculated as ICC = 𝜎𝑏2 /(𝜎𝑏2 + 𝜎𝑤2 ), where 𝜎𝑏2 is the between-student variance and 𝜎𝑤2 is the withinstudent residual variance [41]. For each student, we also calculated the proportion of T0 prompts in deadline and other weeks and compared these paired proportions using the Wilcoxon signed-rank test [49], excluding students for whom the two proportions were equal.

4

Results

Across 962 coded prompts, 84.2% showed evidence of student engineering contribution (T1-T5). Agency shifted across phases of the SDLC, and these differences remained after accounting for the clustering of prompts within students. T0 was more prevalent during deadline weeks in the unadjusted analysis, but this effect did not remain after accounting for within-student clustering. Most variation in agency occurred within students rather than between students. Sections 4.1 and 4.2 analyse all 962 coded prompts. Section 4.3 examines within-student trajectories for the 27 students who submitted an AI Journal in at least 50% of the eligible weeks, comprising 608 prompts.

Who Is Doing the Thinking? Characterising Human Agency in First-Year Students’ AI Interactions

Table 3: Weekly distribution across categories T0-T5.

4.1

Week

T0

T1

T2 T3

T4

T5 Total

4 5 6 7 8 9 10 11 12 13

28 19 28 19 8 7 6 10 13 14

21 18 40 42 22 35 18 23 9 13

31 27 8 14 17 14 18 14 14 15

12 16 9 10 13 4 11 14 12 18

90 42 27 30 12 3 4 5 7 7

187 130 119 124 75 66 60 70 58 73

Total 152 241 172 51 119 227 % 15.8 25.1 17.9 5.3 12.4 23.6

962 100

5 8 7 9 3 3 3 4 3 6

Overall Distribution

Table 3 reports tier counts by week. Of the 962 prompts, 152 (15.8%) were classified as T0, meaning that 84.2% showed evidence of at least some student engineering contribution (T1–T5). The most frequent tiers were T1 (241; 25.1%) and T5 (227; 23.6%), followed by T2 (172; 17.9%), T4 (119; 12.4%), and T3 (51; 5.3%). T3 was the least frequent tier in every week, except Week 9, when it was tied with T5. Its weekly share ranged from 2.7% in Week 4 to 8.2% in Week 13, compared with 5.3% overall. Thus, students relatively rarely specified the system constraints that an artefact had to satisfy when requesting its generation. The tier distribution differed across weeks (𝜒 2 (45) = 209.8, 𝑝 < .001, Cramer’s 𝑉 = 0.21). Table 1 summarises the course content by week. Based on the development activities in each week, we grouped Weeks 4-13 into three analysis phases: analysis (Week 4: requirements engineering and development of the A1 requirements), design and prototype (Weeks 5-7: architecture, prototyping, implementation, and the A1 submission), and build and verify (Weeks 8-13: verification and validation (V&V), quality engineering, CI/CD, and deployment while students developed A2). Tier distributions differed more strongly across these phases (𝜒 2 (10) = 134.9, 𝑝 < .001, Cramer’s 𝑉 = 0.26). Standardised residuals for the phase-by-tier analysis are shown in brackets: Analysis: design-space exploration. T5 accounted for 48.1% of prompts (+6.9) during Week 4, when the course covered requirements engineering and students developed their A1 requirements. Knowledge lookup was relatively uncommon (T1: 11.2%; −3.8). Design and prototype: a transition. T5 remained substantial (26.5%), while T1 rose to 26.8%. The largest change occurred in Week 6, when T1 increased from 13.8% to 33.6%. At this point, the course moved from prototype design in Figma to implementation using Microsoft Power Apps, SharePoint lists, and AI-assisted coding, while students built their A1 prototypes. Build and verify: diagnosis and components. T4 increased to 17.9% (+3.2) and T2 to 22.9% (+2.4), while T5 fell to 9.5% (−5.8). T1 peaked at 53.0% in Week 9, when the course covered quality engineering and CI/CD.

At the week level, the share of T5 decreased over the semester (Spearman 𝜌 = −0.76, 𝑝 = .011, 𝑛 = 10 weeks), while the share of T4 increased (𝜌 = 0.78, 𝑝 = .008, 𝑛 = 10 weeks). Spec-driven development with AI was introduced in Week 12, followed by the highest observed share of T3 in Week 13 (8.2%). However, Week 13 contained only six prompts, so we do not interpret this pattern further. The number of submitted prompts also declined over the semester, from 187 in Week 4 to 58-75 per week from Week 8 onwards. Consequently, percentage estimates for later weeks are based on fewer observations. Because prompts are nested within students, we re-estimated the two main phase-related shifts using logistic GEE, accounting for within-student clustering. Both remained significant. Compared with the analysis phase, the odds of a T5 prompt were 0.38 times as high in the design and prototype phase (95% CI 0.25 − 0.56) and 0.11 times as high in the build and verify phase (95% CI 0.05 − 0.22; both 𝑝 < .001). Conversely, the odds of a T4 prompt were 2.88 times as high in the build and verify phase (95% CI 1.41 − 5.86, 𝑝 = .004). The increase in the design and prototype phase was smaller and not statistically significant (OR 1.56, 95% CI 0.87 − 2.80, 𝑝 = .14).

4.2

Delegation around Assessment Deadlines

Four journalled weeks contained assessment deadlines (Table 1): Week 6 (mid-semester quiz), Week 7 (A1 due), Week 12 (A2 due), and Week 13 (A2 demonstrations and final deliverables). At the prompt level, T0 accounted for 19.8% of prompts in deadline weeks (74/374), compared with 13.3% in the other six weeks (78/588; 𝜒 2 (1) = 6.82, 𝑝 = .009, OR = 1.61). Week 6 had the highest T0 share of the semester (23.5%). The difference was also pronounced at the end of the semester, with T0 accounting for 20.6% of prompts in Weeks 12-13 compared with 11.4% in Weeks 8-11. However, the deadline effect did not remain statistically significant after accounting for within-student clustering. Because prompts are nested within students, students who submitted more prompts during deadline weeks contributed disproportionately to the prompt-level comparison. In the GEE model, the estimated deadline effect was smaller and not statistically significant (OR = 1.28, 95% CI 0.93 − 1.75, 𝑝 = .13). We therefore treat the apparent increase in delegation during deadline weeks as suggestive rather than conclusive. Section 4.3 examines this pattern within students.

4.3

Within-Student Variation

The longitudinal subset comprised 27 students and 608 prompts. Individual students used a wide range of HADT tiers. The median student used five of the six tiers (range 2-6), and 22 of the 27 students used at least four tiers. Twenty-one students had at least one T0 prompt, but T0 was the most frequent tier for only six. In a random-intercept model adjusted for phase, 32% of the variance in tier was attributable to differences between students (ICC = 0.32), while the remaining 68% occurred within students. Thus, most variation in observed agency occurred within the same students across prompts, although differences between students remained substantial. The SDLC shift was also visible within individual students. From each student’s earliest active phase to the build and verify phase,

Shakiba et al.

Analysis

Design & Prototype ▾

Quiz

▾

Build & Verify A1 due

▾

A2 due

▾

A2 demos

48.1

32.3

22.7

24.2

16.0

4.5

6.7

7.1

12.1

9.6

T4 Analysing existing systems

6.4

12.3

7.6

8.1

17.3

6.1

18.3

20.0

20.7

24.7

T3 Specifying constraints

2.7

6.2

5.9

7.3

4.0

4.5

5.0

5.7

5.2

8.2

T2 Requesting components

16.6

20.8

6.7

11.3

22.7

21.2

30.0

20.0

24.1

20.5

20

T1 Looking up knowledge

11.2

13.8

33.6

33.9

29.3

53.0

30.0

32.9

15.5

17.8

10

T0 Delegating the problem

15.0

14.6

23.5

15.3

10.7

10.6

10.0

14.3

22.4

19.2

0

Week Turns

n=187

W4

W5

n=130

W6

n=119

W7

n=124

W8

n=75

W9

n=66

W10

n=60

W11

n=70

W12

n=58

Share of weekly prompts (%)

T5 Architecture discovery

50 40 30

W13

n=73

Figure 2: Distribution of HADT tiers by week (962 prompts, weeks 4-13). Each cell is the percentage of that week’s prompts at a given tier; columns sum to 100%, and 𝑛 is the number of prompts per week. Brackets mark the analysis phases (Table 1), and ▼ marks assessment deadlines. Mid-semester break was between weeks 6 and 7. the share of T5 decreased for 20 of the 27 students, the share of T4 increased for 11, and both changes occurred for 10. In contrast, the deadline effect was not supported within students. Twelve students had a higher T0 share in deadline weeks than in other weeks, eight had a lower share, and seven showed no change. Although the median T0 share was higher in deadline weeks (16.7% versus 8.3%), the paired difference was not statistically significant (Wilcoxon signed-rank 𝑝 = .94). Only two students used T0 exclusively during deadline weeks. A GEE model fitted to the longitudinal subset produced a similar, non-significant estimate (OR = 1.36, 95% CI 0.98–1.89, 𝑝 = .066).

5

Discussions and Lessons Learnt

The results suggest that first-year students’ agency with AI is neither uniformly low nor fixed. Instead, patterns of agency vary with the work students undertake at different stages of the SDLC. Although delegation was more common during deadline weeks in the unadjusted analysis, this pattern was not supported after accounting for repeated prompts within students. Agency Is Phase-Appropriate, Not a Ladder. The largest shifts in agency followed the SDLC. Design-space exploration (T5) dominated the early requirements and design phase, whereas diagnosis of existing systems (T4) became more common once students had systems to diagnose. A shift towards lower-numbered tiers later in the semester therefore does not necessarily indicate declining engagement. A T4 prompt during build and verification (“here is my error and what I think causes it”) can demonstrate substantial student agency for that phase, just as a T5 prompt can during requirements and design. HADT tiers are therefore better interpreted as a profile in relation to the phase of engineering work than as a score to maximise. In practice, educators could identify the tier profiles expected at different stages of a project and use HADT to

examine where the observed profile of a cohort differs from those expectations. Mid-Semester Knowledge Lookup. Factual and procedural lookup (T1) rose sharply in Week 6 and peaked at 53% of prompts in Week 9. The rise began as the course moved from prototyping in Figma to implementation in a low-code/no-code environment using Power Apps and SharePoint lists, tools with which most students had little prior experience. Using AI as a reference for platform syntax, formulas, and features is a legitimate form of assistance, and T1 is not equivalent to delegation. However, repeated lookup can become problematic if it substitutes for developing the underlying knowledge needed to evaluate and reuse the answers. For a course such as this, a checkpoint on platform fundamentals may therefore be more appropriate than discouraging students from using AI for lookup. Agency Varies Within Students; Deadlines Are a Weak Driver. T0 accounted for between 10% and 24% of prompts each week, so most prompts in every week showed evidence of student engineering contribution. The observed level of agency was also not simply a stable characteristic of individual students. In the phase-adjusted model, 32% of the variance in tier lay between students and 68% within students, while the median student used five of the six tiers over the semester. These results suggest that the same student can adopt substantially different levels and forms of agency across different prompts and stages of work. Deadline pressure, however, provided little evidence of explaining this variation. The prompt-level increase in T0 during deadline weeks became smaller and was no longer statistically significant after accounting for within-student clustering. The within-student comparison was similarly inconclusive: 12 students had a higher T0 share during deadline weeks, eight had a lower share, and seven showed no change. In this dataset, SDLC phase is therefore more clearly associated with patterns of agency than the occurrence of a

Who Is Doing the Thinking? Characterising Human Agency in First-Year Students’ AI Interactions

deadline. Assessment design may nevertheless matter through the kinds of engineering activity that particular stages and deliverables require. Constraints Are Rarely Specified. T3, in which students specify the system constraints that an artefact must satisfy, was the least frequent tier in every week (5.3% overall). Students more often requested a component without its wider system constraints (T2) or framed a design space for exploration (T5) than explicitly translated project requirements into constraints on generated artefacts. This pattern did not arise in the absence of instruction. Students were taught techniques for constraining AI output through examples, output-format specifications, and explicit success criteria at several points in the course. One possible explanation is therefore that T3 depends not only on prompting technique but also on sufficient domain and project knowledge to formulate useful constraints. To specify a meaningful constraint, students may need to know what properties a suitable artefact should have, including the relevant data model, platform limitations, and acceptance criteria. First-year students were developing this knowledge during the semester, so their ability to articulate constraints may have lagged behind their exposure to prompting techniques. This interpretation is consistent with evidence that beginners can struggle to express intent in forms that code LLMs interpret as intended [31]. On this interpretation, teaching prompting techniques alone may be insufficient to increase T3 behaviour. However, the prompt data cannot distinguish students who lacked the knowledge needed to specify constraints from those who knew the relevant constraints but chose not to include them. Using HADT in Teaching. Making HADT visible to students could help them recognise the kinds of engineering contribution evident in their prompts and consider whether those contributions are appropriate for their current phase of work. There is, however, a tension between using HADT pedagogically and using AI Journals as records of authentic practice. If particular tiers are rewarded, students may optimise their prompts to satisfy the taxonomy rather than use AI in the way they otherwise would, an instance of the measurement problem associated with Goodhart’s law [43]. We therefore recommend using HADT primarily for formative feedback, cohort-level analysis, and course design rather than assigning marks for reaching particular tiers. Keeping the AI Journal itself unmarked may help preserve its role as a record of students’ actual AI use. Transferability. The three HADT dimensions are not inherently specific to software engineering. Contextual bandwidth, level of engineering abstraction, and problem ownership can arise wherever students use GenAI to produce or refine disciplinary artefacts. The specific manifestations of these dimensions, however, are likely to be discipline-dependent. HADT may therefore provide a basis for adaptation to other computing and engineering contexts, but its tier definitions and decision rules should be validated before being transferred to substantially different tasks or disciplines.

6

Limitations and Threats

HADT measures the engineering contribution made visible in each prompt, not agency across an entire conversation. Context carried

forward from earlier prompts or retained in model memory is therefore not credited. This is a deliberate construct choice. Coders used only the prompt text: attachments were not recorded in the AI Journal, and other journal fields were not consulted. Constraints or contextual information supplied outside the prompt are therefore invisible to HADT. The tiers are ordinal but do not form a quality scale (Section 5). Analyses that treat tier as numeric, particularly the ICC analysis, should therefore be interpreted as describing the distribution of tiers rather than better or worse prompting. The AI Journal is self-reported, i.e. students chose which AI interactions to record or they could instead declare that they had not used AI. Although the journals were unmarked and students were explicitly told that their disclosures would not reduce their marks, selective reporting remains possible. Delegated interactions may be particularly susceptible to under-reporting, so the observed T0 proportion should not be interpreted as the true prevalence of delegation. Journal participation also declined over the semester, from 187 prompts in Week 4 to 58-75 per week from Week 8 onwards. Later weeks therefore contain fewer observations and may represent a different subset of students from earlier weeks, potentially biasing comparisons across the semester. The clustered models and longitudinal subset reduce some consequences of unequal contributions across students but cannot eliminate selection arising from which students chose to submit journals or which interactions they recorded. The SDLC phases are also confounded with the course activities and instruction delivered during each period (Table 1). Lecture topics, explicit instruction in AI use (e.g., using LLMs for requirements engineering in Week 4), project activities, and assessment demands all changed alongside the lifecycle phase. We therefore cannot isolate whether the observed differences in agency are associated with the engineering phase itself, the accompanying instruction, the assessment context, or some combination of these factors. The prompt-level 𝜒 2 analyses treat prompts as independent even though multiple prompts are nested within students. We therefore report GEE estimates that account for within-student clustering alongside these analyses and base our main inferential conclusions on effects that remain supported after clustering is taken into account. The apparent increase in T0 during deadline weeks did not meet this criterion and is therefore treated as suggestive rather than conclusive. Several statistical tests were also conducted on the same dataset without correction for multiple comparisons, increasing the risk of false-positive findings. Conversely, the week-level trend tests contain only ten observations, and T3 counts are small throughout the semester, limiting power to detect small effects and making week-specific T3 patterns particularly unstable. Finally, the data come from a single first-year course in one semester at one university. Students had no assumed prior programming experience, implemented their projects in a low-code/nocode environment (Power Apps with SharePoint lists) rather than through conventional programming, and were explicitly taught how to use GenAI. Agency profiles may differ in programming-intensive courses, among more experienced students, in other disciplinary contexts, or where AI use is permitted but not explicitly taught. The longitudinal analysis further includes only the 27 students who submitted an AI Journal in at least half of the eligible weeks,

Shakiba et al.

compared with 105 students who contributed at least one analysed journal. This subset may not be representative of all students who used AI or of the cohort as a whole. The longitudinal findings should therefore be interpreted as evidence about variation within this more consistently observed subset rather than as estimates of within-student behaviour for the entire cohort.

7

Conclusions and Future Research Directions

When AI produces the artefact, the cognitive complexity of the task no longer tells educators what the student contributed. We therefore asked a different question: “who is doing the thinking?” We proposed HADT, a six-tier taxonomy that classifies the engineering contribution evident in each prompt along three dimensions: contextual bandwidth, engineering abstraction, and student agency. Applied to 962 prompts from a first-year, AI-permissive software engineering course, HADT showed that most prompts contained evidence of student engineering contribution. Patterns of agency shifted across the development lifecycle, from designspace exploration during requirements to diagnosis during build and verification, and varied more within students than between them. Delegation did not increase reliably during deadline weeks after accounting for within-student clustering, while explicit specification of system constraints remained rare despite being taught. For educators, the central message is that appropriate agency depends on the engineering work students are undertaking. This reinforces the need to redesign AI-permissive assignments carefully around the contributions students are expected to make, rather than simply around the artefacts they produce. HADT provides a lightweight way to make those contributions visible through a tier profile for each week or project phase, interpreted against the engineering work that phase requires rather than as a score to maximise. Future work should refine and validate HADT across courses, institutions, experience levels, and programming contexts. A second direction is to investigate whether coding can be scaled through LLM-assisted classification validated against human labels. Reliable automated coding could allow HADT profiles to support timely formative feedback during a semester, for example through an AI Journal dashboard.

Acknowledgments This project was supported by an Education-Focused Academics SoTL Grant from the University of Sydney.

Ethics and Privacy Statement This study was approved by the Human Research Ethics Committee of the University of Sydney (Approval No. HREC 2025/HE001132substudy 022 v1a). Journal data were de-identified prior to analysis and reported only in aggregated form. Participation formed part of normal course activities; however, all research analyses were conducted on anonymised data in accordance with institutional ethics requirements.

References [1] A. Agresti. 2013. Categorical Data Analysis. Wiley. https://books.google.com. au/books?id=6PHHE1Cr44AC

[2] Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, and Rei Mariman. 2025. Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences 122, 26 (2025), e2422633122. arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.2422633122 doi:10.1073/pnas. 2422633122 [3] Margaret Bearman and Rosemary Luckin. 2020. Preparing University Assessment for a World with AI: Tasks for Human Intelligence. Springer International Publishing, Cham, 49–63. doi:10.1007/978-3-030-41956-1_5 [4] John B Biggs and Kevin F Collis. 2014. Evaluating the quality of learning: The SOLO taxonomy (Structure of the Observed Learning Outcome). Academic press. [5] Zixin Chen, Jiachen Wang, Meng Xia, Kento Shigyo, Dingdong Liu, Rong Zhang, and Huamin Qu. 2025. StuGPTViz: A Visual Analytics Approach to Understand Student-ChatGPT Interactions. IEEE Transactions on Visualization and Computer Graphics 31, 1 (2025), 908–918. doi:10.1109/TVCG.2024.3456363 [6] Michelene T. H. Chi and Ruth Wylie. 2014. The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes. Educational Psychologist 49, 4 (2014), 219–243. arXiv:https://doi.org/10.1080/00461520.2014.965823 doi:10. 1080/00461520.2014.965823 [7] Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46. [8] Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213. [9] Harald Cramér. 1999. Mathematical methods of statistics. Vol. 9. Princeton university press. [10] Marian Daun and Jennifer Brings. 2023. How ChatGPT Will Change Software Engineering Education. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 110–116. doi:10. 1145/3587102.3588815 [11] Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A Becker, and Brent N Reeves. 2024. Prompt Problems: A new programming exercise for the generative AI era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. 296–302. [12] Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2024. Computing Education in the Era of Generative AI. Commun. ACM 67, 2 (Jan. 2024), 56–67. doi:10.1145/3624720 [13] Euzeli C. dos Santos Jr. and Tracey Birdwell. 2025. The Unspoken Crisis of Learning: The Surging Zone of No Development. arXiv:2511.12822 [cs.CY] https://arxiv.org/abs/2511.12822 [14] Yizhou Fan, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yueying Zhao, Yuan Shen, Xinyu Li, and Dragan Gašević. 2025. Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology 56, 2 (2025), 489–530. [15] James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22). Association for Computing Machinery, New York, NY, USA, 10–19. doi:10.1145/3511861.3511863 [16] James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, VIC, Australia) (ACE ’23). Association for Computing Machinery, New York, NY, USA, 97–104. doi:10.1145/3576123.3576134 [17] Vahid Garousi, Zafar Jafarov, Aytan Mövsümova, Leyla Memmedova, and Hüseyn Mirzayev. 2026. Implementing GenAI-Supported Learning in Software Engineering and Computer Science Education using Bloom’s Taxonomy. arXiv:2606.27398 [cs.SE] https://arxiv.org/abs/2606.27398 [18] Michael Gerlich. 2025. AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies 15, 1 (2025), 6. [19] Chahna Gonsalves. 2024. Generative AI’s Impact on Critical Thinking: Revisiting Bloom’s Taxonomy. Journal of Marketing Education (23 Dec. 2024), 1. doi:10. 1177/02734753241305980 [20] Colin G. Johnson and Ursula Fuller. 2006. Is Bloom’s taxonomy appropriate for computer science?. In Proceedings of the 6th Baltic Sea Conference on Computing Education Research: Koli Calling 2006 (Uppsala, Sweden) (Baltic Sea ’06). Association for Computing Machinery, New York, NY, USA, 120–123. doi:10.1145/1315803.1315825 [21] Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J. Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 455, 23 pages. doi:10.1145/3544548.3580919

Who Is Doing the Thinking? Characterising Human Agency in First-Year Students’ AI Interactions

[22] David R. Krathwohl. 2002. A Revision of Bloom’s Taxonomy: An Overview. Theory Into Practice 41, 4 (2002), 212–218. http://www.jstor.org/stable/1477405 [23] Nan M. Laird and James H. Ware. 1982. Random-Effects Models for Longitudinal Data. Biometrics 38, 4 (1982), 963–974. http://www.jstor.org/stable/2529876 [24] J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. [25] Sam Lau and Philip Guo. 2023. From "Ban It Till We Understand It" to "Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 106–121. doi:10.1145/3568813.3600138 [26] Hao-Ping (Hank) Lee, Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson. 2025. The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 1121, 22 pages. doi:10.1145/3706598.3713778 [27] Kung-Yee Liang and Scott L. Zeger. 1986. Longitudinal data analysis using generalized linear models. Biometrika 73, 1 (04 1986), 13– 22. arXiv:https://academic.oup.com/biomet/article-pdf/73/1/13/679793/73-113.pdf doi:10.1093/biomet/73.1.13 [28] Raymond Lister, Beth Simon, Errol Thompson, Jacqueline L. Whalley, and Christine Prasad. 2006. Not seeing the forest for the trees: novice programmers and the SOLO taxonomy. SIGCSE Bull. 38, 3 (June 2006), 118–122. doi:10.1145/1140123.1140157 [29] Jason Lodge, Sarah Howard, {Margaret Lynn} Bearman, Phillip Dawson, Shirley Agostinho, {Simon J.} {Buckingham Shum}, Chris Deneen, Cath Ellis, Tim Fawns, Helen Gniel, Rowena Harper, Michael Henderson, {Danny Yen Ting} Liu, Lina Markauskaite, Jan McLean, Carlo Perrotta, Lambert Schuwirth, Christine Slade, and Associates. 2023. Assessment Reform for The Age of Artificial Intelligence. Tertiary Education Quality and Standards Agency. [30] Brian Lubars and Chenhao Tan. 2019. Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability. arXiv:1902.03245 [cs.AI] https://arxiv.org/abs/1902.03245 [31] Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How beginning programmers and code llms (mis) read each other. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26. [32] Karl Pearson. 1900. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50, 302 (1900), 157–175. [33] James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew LuxtonReilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (Turku, Finland) (ITiCSE-WGR ’23). Association for Computing Machinery, New York, NY, USA, 108–159. doi:10.1145/3623762.3633499 [34] James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. ACM Trans. Comput.-Hum. Interact. 31, 1, Article 4 (Nov. 2023), 31 pages. doi:10.1145/3617367 [35] James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. 2024. The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 (Melbourne, VIC, Australia) (ICER ’24). Association for Computing Machinery, New York, NY, USA, 469–486. doi:10.1145/3632620.3671116 [36] Evan F. Risko and Sam J. Gilbert. 2016. Cognitive Offloading. Trends in Cognitive Sciences 20, 9 (2016), 676–688. doi:10.1016/j.tics.2016.07.002 [37] Andreas Scholl, Daniel Schiffner, and Natalie Kiesler. 2024. Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT. Proceedings of DELFI 2024. doi:10.18420/delfi2024_05 [38] Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco, Giuseppe

Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik. 2025. The Prompt Report: A Systematic Survey of Prompt Engineering Techniques. arXiv:2406.06608 [cs.CL] https://arxiv.org/abs/2406.06608 Beyond Correctness: How [39] Ali Shakiba and Thomas Chaffey. 2026. Novice Software Engineering Students Exercise Oversight of Generative AI. arXiv:2610.32118 [cs.SE] https://arxiv.org/abs/2610.32118 [40] Ali Shakiba and Thomas Chaffey. 2026. Making the Invisible Visible: A Framework for Reflective AI use in Software Engineering Education. The 37th Australasian Association for Engineering Education, Sydney NSW Australia. [41] Tom A. B. Snijders and Roel J. Bosker. 2012. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling (2nd ed.). Sage Publications, London. [42] C. Spearman. 1961. The Proof and Measurement of Association Between Two Things. In Studies in Individual Differences: The Search for Intelligence. AppletonCentury-Crofts, East Norwalk, CT, US, 45–58. Reprint. doi:10.1037/11491-005 [43] Marilyn Strathern. 1997. ’Improving ratings’: audit in the British University system. European Review 5, 3 (1997), 305–321. [44] Errol Thompson, Andrew Luxton-Reilly, Jacqueline L. Whalley, Minjie Hu, and Phil Robbins. 2008. Bloom’s taxonomy for CS assessment. In Proceedings of the Tenth Conference on Australasian Computing Education - Volume 78 (Wollongong, NSW, Australia) (ACE ’08). Australian Computer Society, Inc., AUS, 155–161. [45] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. 2020. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature methods 17, 3 (2020), 261–272. [46] L. S. Vygotsky. 1978. Mind in Society: Development of Higher Psychological Processes. Harvard University Press. http://www.jstor.org/stable/j.ctvjf9vz4 [47] Jacqueline L. Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P. K. Ajith Kumar, and Christine Prasad. 2006. An Australasian study of reading and comprehension skills in novice programmers, using the bloom and SOLO taxonomies. In Proceedings of the 8th Australasian Conference on Computing Education - Volume 52 (Hobart, Australia) (ACE ’06). Australian Computer Society, Inc., AUS, 243–252. [48] Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] https://arxiv.org/abs/2302.11382 [49] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80–83. http://www.jstor.org/stable/3001968 [50] F. Yates. 1934. Contingency Tables Involving Small Numbers and the 𝜒 2 Test. Supplement to the Journal of the Royal Statistical Society 1, 2 (07 1934), 217–235. doi:10.2307/2983604

Related documents

• Who Acts, Who Knows, Who Answers? A Corpus-Assisted Discourse Analysis of Agency, Epistemic Responsibility, and Accountability in Generative AI Higher Education Research#1063395
• The Divided Mind#1192078
• Ethical and Technical Challenges of Generative AI Adoption in Higher Education#221967
• Developmental Ethics and Post-Human Identity: A Thought Experiment on Selective Adoption and "Human-Grounded" AGI Agency#1507443
• Tinkering extended minds: Rethinking direct agency#82999