The Uneven Impact of Generative AI on Student Learning: Examining the Roles of Reliance, Evaluation Literacy, and Course Policy in AI-related Courses Lydia Manikonda1 , Mei Si, Sirajam Munira, Oshani Seneviratne, Kristin Bennett 1
arXiv:2609.18676v1 [cs.AI] 16 Sep 2026
Corresponding author Rensselaer Polytechnic Institute (RPI) Troy, New York, USA {manikl, sim, munirs, senevo, bennek}@rpi.edu Abstract Generative artificial intelligence (GenAI) is changing how students learn, yet the roles of course context, cognitive reliance, evaluation literacy, and early reliance remain underexplored. Using survey responses from 118 students across 12 AI-related courses at our institution, we examined differences in GenAI use and perceived learning experiences. We identified four user clusters: high-use students reporting many benefits, light users reporting less reliance and fewer benefits, and two moderate-use groups reporting different levels of benefit. We also found significant differences between free- and premium-version users, single- and multiple-tool users, and students experiencing different instructor policies. In multivariable regression models, academic benefit was associated with early reliance and academic task support; positive impact was associated with cognitive reliance, academic task support, confidence in GenAI reliability, and instructor policy; and negative impact was associated with early reliance and attitudinal change. The association between early reliance and negative impact became stronger as evaluation literacy increased. Finally, perceptions of GenAI-enhanced learning appear to reflect cognitive, performance, and self-efficacy benefits, while concerns about stress and diminished critical thinking are associated with lower perceived learning benefits. These findings suggest that institutions need better policies to address such inequities so that institutions can enable students to benefit from increasingly capable AI systems.
1
Introduction
GenAI has rapidly become part of everyday learning at higher education institutions. Students use large language models (LLMs) to explain unfamiliar concepts, debug programs, summarize readings, brainstorm ideas, and complete academic work (Cena et al. 2026). In computing and AI-related courses, this development creates an especially consequential educational tension due to the ever-increasing performance quality of GenAI in such course-related topics. The same systems that students study as technical objects are increasingly used as partners in the process of studying. Existing literature has consistently found enthusiasm about GenAI’s usefulness alongside concerns about inaccuracy, academic integrity, overreliance, and reduced engagement with learning (Kim et al. 2025; Marshik et al. 2025; Sah et al. 2025; Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
DeStefano, Hackney, and Moskal 2026). Access is uneven as well: some students use free versions of a single system, while others can draw on premium models or several tools with different capabilities. These differences may shape not only how frequently students use GenAI, but also which learning activities it can support and how much benefit students perceive. Despite this growing literature, three gaps remain. First, much of the evidence comes from broad university populations or from interventions structured around a particular tool or task. Second, reliance is often treated as a single continuum. This can obscure an important distinction between using GenAI during cognitively substantive activities and consulting it before attempting a problem independently or seeking conventional forms of help. Third, evidence about students’ confidence, access conditions, and perceived learning outcomes is rarely considered within a common empirical framework. We address these gaps through a survey of undergraduate and graduate students enrolled in 12 AIand data-intensive courses at Rensselaer Polytechnic Institute (RPI), an R1 university in the United States, spanning artificial intelligence, machine learning, data analytics, decentralized systems, AI ethics, biostatistics, and economic analysis. The study measures academic task support, cognitive and early reliance, evaluation literacy, attitudinal change, confidence without GenAI, GenAI reliability, GenAI policy clarity, positive and negative impacts, academic benefit, limited learning impact, and course-supported effective GenAI use. We define cognitive reliance as using GenAI to recall, organize, combine, evaluate, and apply information or solve problems. We distinguish this from early reliance: consulting GenAI before independent thought, traditional search, or instructor support. We examine how these behaviors relate to students’ perceived learning and whether their experiences differ according to access, number of tools used, academic level, and course context. Accordingly, we ask: 1. RQ1: What distinct patterns of GenAI use, cognitive reliance, and perceived academic benefits and consequences emerge among students in AI-related courses? 2. RQ2: How do factors, including access to free vs. premium GenAI tools, use of single vs. multiple tools, and instructor policies, relate to students’ GenAI learning experiences?
3. RQ3: What factors are associated with students’ perceptions of overall improved learning through GenAI?
2
Related Work
Prior research shows that GenAI can support learning while also raising concerns about cognitive offloading, overconfidence, and reduced independent engagement. Students’ experiences may further vary with confidence, evaluation practices, and instructional context, but these dimensions are often studied separately or in broad university populations. Among the studies reviewed, we did not identify a surveybased analysis that jointly examined GenAI use, learning support, cognitive reliance, evaluation practices, and course policy among students in AI-related courses. Student Use and Perceptions of GenAI University students use GenAI for academic activities such as seeking explanations, finding information, brainstorming, summarizing, revising, programming, and solving problems. Students associate these uses with greater efficiency, access to assistance, confidence, and work quality, while also expressing concerns about inaccurate information, academic integrity, dependence, and reduced engagement (Kim et al. 2025; Marshik et al. 2025; Sah et al. 2025; DeStefano, Hackney, and Moskal 2026). These findings indicate that GenAI use is neither uniformly beneficial nor harmful. However, most surveys examine broad university populations, providing limited evidence about how GenAI use, perceived support, and learning outcomes relate within AI-related courses. Confidence, Cognitive Reliance, and Critical Evaluation Greater confidence in GenAI has been associated with less self-reported critical-thinking effort, whereas greater confidence in one’s own abilities has been associated with more critical engagement (Lee et al. 2025). Critical thinking in AI-assisted work increasingly involves verifying, comparing, and integrating generated outputs, suggesting that selfconfidence, confidence in GenAI, and critical evaluation are related but distinct. GenAI can reduce the cognitive demands of academic work, but excessive reliance may contribute to cognitive offloading and an illusory sense of mastery (Wang and Luo 2026). The timing and structure of assistance also matter: unrestricted assistance may improve performance while GenAI is available without producing corresponding gains in independent performance, whereas learning-oriented safeguards can reduce this effect (Bastani et al. 2025). Together, this literature suggests that reliance should not be treated as a single construct. Using GenAI to organize information, decompose problems, evaluate work, or combine ideas may represent a different form of reliance from consulting GenAI before independent thought, traditional search, or instructor support. Existing surveys rarely distinguish these patterns or examine how they relate to confidence and evaluation practices. Instructor Policy and Student Practices Course policies provide important context for students’ GenAI practices. Students and faculty often share general concerns about GenAI but differ in their perceptions of its use, acceptability, and effects on learning (Kim et al. 2025; Marshik et al. 2025;
Sah et al. 2025; DeStefano, Hackney, and Moskal 2026; Manikonda and Outlaw 2026). Instructors may also misestimate students’ GenAI use, trust, and preferences for AI assistance (Saeli et al. 2026), while students frequently report uncertainty about institutional rules and acceptable uses (Morari, Grimes, and Hawe 2026). Course policy should therefore be considered both as a formal instructional condition and as something students perceive and interpret.
3 3.1
Methodology
Study Design and Data Collection
We surveyed students enrolled in AI- and data-analyticsrelated courses at our institution in the United States during Spring 2026. Students were recruited through course instructors and in-class announcements. Participation was voluntary and anonymous, with no monetary compensation, and the study received approval from the institution’s Institutional Review Board. A total of 143 complete responses were received, and the final sample included 118 undergraduate and graduate students from 12 AI-related courses across multiple disciplines. The survey contained 60 Likert-type items covering GenAI use, perceived impacts, cognitive reliance, evaluation practices, academic benefits, attitudes, and course-policy experiences. These items were organized into eight multi-item constructs and five single-item measures. Likert-type responses were coded on a 1-4 scale, with higher values indicating higher levels of the measured response (e.g., agreement, frequency, confidence, or importance); “Not Applicable” responses were coded as the scale midpoint of 2.5. Multi-item scores were calculated by averaging their included items. Internal consistency was assessed using Cronbach’s alpha, with higher values indicating that the items within a construct produced more consistent responses. Table 1 summarizes the measures, their roles in the regression models, reliability, and descriptive statistics. Additional demographic, access, tooluse, and course-policy variables were used for the group comparisons described in Section 3.2. The complete survey instrument is provided in Supplementary Table S1.
3.2
Analytic Approach
We used four complementary analyses. Clustering identified patterns of GenAI use and experience; group comparisons examined differences by student, access, and course characteristics; multiple regression estimated adjusted associations with the four learning outcomes; and an ensemble model explored potential nonlinear relationships with overall perceived learning. Clustering To identify patterns of GenAI use and experience, we applied hierarchical agglomerative clustering with Ward’s method to students’ responses to the 60 Likert-type items. The four-cluster solution was selected by inspecting the resulting dendrogram and heatmap and considering the interpretability of the response profiles. Cluster-level construct means were then used to describe the four profiles. As described above, Not Applicable responses were coded as 2.5 before clustering.
Table 1: Survey measures, internal consistency, and descriptive statistics. Measure
Items
Included content
Cronbach’s Mean α
SD
Dependent variables Academic benefit
6
Learning confidence; workload management; new perspectives; work quality; grades or performance; retention of course material
.876
2.48
.78
Positive impact
9
Time savings; idea generation; progress; assistance when stuck; work quality; creativity; beyond-expertise tasks; skill confidence; overall learning
.892
2.90
.59
Negative impact
7
Lower learning confidence; stress; unfair advantage; reduced motivation; harmed critical thinking; fear of GenAI use; difficulty adapting output
.768
2.03
.55
Learned GenAI
1
Course helped the student to use GenAI more effectively
–
2.70
.80
Academic task support
12
Learning; information seeking; summarizing; brainstorming; revising; completing and checking assignments; further thinking; course clarification; practice activities; direct content generation
.911
2.97
.55
Cognitive reliance
6
Recalling, organizing, combining, and evaluating information; solving and decomposing problems
.864
2.76
.66
Evaluation literacy
8
Evaluation confidence and reflection; checking sources, concepts, organization, logic, synthesis, and task requirements
.840
3.52
.49
Early reliance
4
Consulting GenAI before traditional search, independent thinking, or instructor support; preferring GenAI explanations
.722
2.22
.72
Attitudinal change
3
Changes in attitudes toward GenAI and interactions with instructors or peers
.683
2.24
.70
Confidence without GenAI
1
Confidence in completing tasks without GenAI
–
3.28
.63
GenAI reliability
1
Confidence in the reliability of GenAI output
–
2.56
.83
GenAI policy clarity
1
Clarity of guidelines for appropriate and inappropriate GenAI use
–
2.92
1.01
Limited GenAI impact
1
Perception that GenAI had little impact on overall learning
–
2.04
.84
Independent variables
Note. N = 118 for all measures. Scores used a 1–4 response scale, with Not Applicable coded as 2.5. Item wording is abbreviated for space. Cronbach’s α is reported only for multi-item constructs.
Item-Level Student Group Comparisons We compared individual Likert-type responses across groups defined by academic level, school, course activity, GenAI access, number of tools used, and perceived instructor policy. The comparisons included graduate versus undergraduate students; students from four schools; lecture-only courses versus courses with an additional GenAI-related activity; premiumaccess versus free-only users; self-paid premium access versus premium access provided by another source; and singletool versus multi-tool users. Instructor policy was analyzed across all five response categories and, separately, across the three substantive policy categories: encouraged, allowed with limits, and discouraged. Two-group comparisons used two-sided Mann– Whitney U tests with rank-biserial correlation as the effect size. Comparisons involving more than two groups used Kruskal–Wallis tests, followed when appropriate by Dunn post-hoc comparisons with Holm adjustment. Benjamini– Hochberg false-discovery-rate correction was applied within each group-comparison family, with adjusted p < .05 considered statistically significant. Multiple Regression Separate multiple linear regressions were estimated for academic benefit, positive impact, negative impact, and course-supported GenAI learning. All predictors were entered simultaneously, with “allowed with limits” serving as the reference category for instructor policy. All variables were standardized, and each model included all 118 respondents. HC3 standard errors were used for the
primary two-sided p-values, with course-clustered standard errors calculated as a robustness check. Raw p-values are reported without multiple-testing correction. Four theoretically motivated interactions were also tested in separate extensions of the corresponding regression models. Ensemble Learning and Feature Rankings As a complementary exploratory analysis, we trained a gradient-boosting classifier to examine potential nonlinear relationships between the survey measures and students’ responses regarding whether GenAI improved their learning. We then used Shapley (SHAP) values (Lundberg and Lee 2017) to rank the features according to their contributions to the model and to show whether higher feature values were associated with higher or lower model predictions. This analysis was used to explore feature importance rather than to replace the adjusted associations estimated by the regression models.
4 4.1
Results
Clustering
Clustering revealed four distinct types of students. Figure 1 summarizes each of the clusters by their average Likert values for the 13 different constructs across the 118 subjects. The rows correspond to the measures defined in Table 1. The four clusters describe four different groups of students with high, medium, and low AI usage characterized by their ranking order on the constructs that describe their AI usage Academic Task Support, Cognitive Reliance, and Early
are clear. Note that usage and benefits may also be partially explained by the percentage of graduate students within each cluster: High Usage/High Benefits (46.4%), Moderate Users/Moderate Benefits (32.6%), Moderate Users/Less Benefits (24.1%), and Low Usage/Low Benefits (16.6%).
4.2
Figure 1: Mean construct scores across the four student clusters.
Reliance. Each group exhibits High, Medium, Less, and Little Benefits as indicated by positive impact and academic benefits. Thus, we refer to the clusters as High Usage/High Benefits, Moderate Usage/Moderate Benefits, Moderate Usage/Less Benefits, and Less Usage/Less Benefits. We see that these constructs correlate with the course specific questions on GenAI clarity and whether they learned GenAI in the course. This suggests that students in courses that teach GenAI with clear usage policies tend to use GenAI for more academic tasks, experience more positive impacts, and are more confident about using GenAI reliably. The greater the usages and benefits of AI reported by students, the more students report that they experienced that AI changed in their interactions with peers and faculty. The clustering shows that confidence in completing tasks without GenAI is inversely correlated with Usage and Benefits, suggesting that higher AI usage and benefits creates students in AI classes that are more reliant on GenAI as part of learning. All students report high Evaluation Literacy, but rather interestingly the students with Moderate Usage/Less Benefits show a more enhanced grasp of AI evaluation that Medium Usage-Medium Benefits cluster. The two medium usage groups illustrate some interesting correlations. They are more likely to say that GenAI impacted their learning as indicated by their disagreement with the statement “GenAI has no impact on learning”. It appears that moderate usage users also report slightly more negative impacts, although all users report relative few negative impacts on average. The clustering results suggest that students’ AI usage, benefits, and impacts vary unevenly across different AI related courses and that these may be related to course specific contexts such as whether GenAI is taught and GenAI policies
Item-Level Differences Across Student Groups
Table 2 summarizes individual survey items that are significant after FDR correction. No FDR-significant item-level differences were observed by academic level, school, course activity, or whether students paid for GenAI access themselves versus receiving access from another source. In contrast, significant differences emerged for premium versus free-only access, single- versus multi-tool use, and perceived instructor policy. The clearest pattern emerged for single- versus multi-tool use. Multi-tool users reported higher responses for nearly all significant items, spanning academic-task support, positive impact, cognitive reliance, early reliance, and academic benefit. The largest differences were for improved academic-work quality (rrb = −0.53), retention of course material (−0.49), using GenAI to evaluate work (−0.48), improving work quality (−0.45), and breaking down complex problems (−0.44). The one exception was confidence without GenAI: single-tool users reported greater confidence in completing tasks without GenAI (rrb = +0.25). This contrast suggests that broader tool use was associated with greater perceived support and reliance on GenAI, but not with greater confidence in working independently without it. Premium access showed fewer but generally positive differences. Premium users reported greater use of GenAI for finding information and brainstorming, and higher perceived benefits for saving time, continuing work when stuck, improving work quality, completing tasks beyond their expertise, and improving academic-work quality. The largest premium-access difference was for completing tasks beyond current expertise (rrb = 0.46), followed by improved academic-work quality (0.40). The three-category substantive policy comparison produced the FDR-significant differences; the five-category analysis yielded no additional substantive pattern. Perceived instructor policy was associated with several items, although effect sizes were generally small and ranged from ϵ2 = 0.06 to 0.19. Differences appeared in task use, confidence in GenAI reliability, cognitive reliance, perceived learning and academic benefits, and whether the course helped students use GenAI more effectively. The largest policy-related effect was for confidence in one’s skillset after using GenAI (ϵ2 = 0.19). Because the omnibus policy test does not indicate direction, these results reflect variation across policy groups rather than a specific ordering among encouraged, limited, and discouraged use.
4.3
Regression Analysis
Main Effects All four regression models included the full sample of 118 respondents. The predictor set explained 68.4% of the variance in academic benefit (adjusted R2 = .641), 79.7% of the variance in positive impact (adjusted R2 = .771), 30.0% of the variance in negative impact (adjusted R2 = .212), and 36.9% of the variance in coursesupported GenAI learning (adjusted R2 = .290). The mod-
Table 2: Significant Individual-Item Differences Across GenAI Access, Tool Use, and Instructor Policy Premium access vs. free-only
Single-tool vs. multi-tool
Perceived instructor policy
Learning about new topics Finding specific facts or information Discovering new insights from information or data Generating summaries of longer content Brainstorming or organizing ideas before starting an assignment Improving or rewriting assignments Completing assignments or significant portions of them Checking whether work meets rules, constraints, or formatting requirements Q14_10 Summarizing lecture slides or notes to clarify materials Q14_11 Generating practice tests or quizzes Q14_12 Generating content for direct use in assignments
– .0354* (+.27) – – .0354* (+.28) – – –
.0016** (-.38) .0326* (-.24) .0096** (-.30) .0034** (-.35) .0006*** (-.41) .0184* (-.28) .0112* (-.30) .0120* (-.29)
– – – .0494* (.07) .0157* (.11) – .0458* (.07) –
– – –
.0016** (-.40) – –
.0458* (.06) .0494* (.07) .0157* (.10)
Positive impact
Q18_1 Q18_2 Q18_3 Q18_4 Q18_5 Q18_6 Q18_7 Q23_1 Q25_4
Save time Generate ideas Make progress in work Continue work when stuck Improve quality of work Be more creative Complete tasks beyond current expertise More confident in skillset after using GenAI GenAI improved learning
.0151* (+.31) – – .0043** (+.38) .0043** (+.39) – .0012** (+.46) – –
.0018** (-.36) .0027** (-.37) – .0023** (-.36) .0003*** (-.45) .0042** (-.35) .0042** (-.35) – .0079** (-.30)
– – .0157* (.09) – .0367* (.08) – – .0017** (.19) –
Confidence without AI
Q15
Confident completing tasks without GenAI
–
.0184* (+.25)
–
Confident in GenAI reliability
*
**
Construct
Item
Academic task support
Q14_1 Q14_2 Q14_3 Q14_4 Q14_5 Q14_6 Q14_7 Q14_8
GenAI reliability
Q16
Survey item
Cognitive reliance
Q19_2 Q19_3 Q19_4 Q19_5 Q19_6
Organize or summarize ideas Combine ideas to create new meanings Evaluate and quality-check work Solve new problems Break down complex problems
Learned GenAI
Q23_6 Course helped use GenAI more effectively
Early reliance
Q24_1 Consult GenAI before traditional search Q24_9 GenAI explains concepts better than instructor Q24_14 Use GenAI before reaching instructor
Academic benefit
Q24_2 Q24_3 Q24_4 Q24_5 Q24_7 Q24_8
Use GenAI to boost confidence GenAI helps manage time/workload GenAI exposes new ideas/perspectives GenAI improves academic work quality GenAI improves grades/performance GenAI helps retain course materials
(-.33)
.0157* (.10)
**
.0016 (-.39) .0061** (-.34) .0001*** (-.48) .0016** (-.38) .0003*** (-.44)
– .0157* (.10) – .0367* (.08) .0458* (.07)
.0349* (+.30)
–
.0113* (.14)
*
.0323 (+.31) – –
**
.0036 (-.34) .0080** (-.31) .0078** (-.31)
– – –
– – –
.0090** (-.30) .0120* (-.29) .0016** (-.39) <.0001*** (-.53) .0060** (-.33) .0001*** (-.49)
– – – – .0358* (.08) .0367* (.08)
.0117 (+.34) – .0354* (+.30) .0120* (+.34) – .0349* (+.29)
.0043** (+.40) – –
.0034
Note. Only individual items with at least one FDR-significant comparison across the three variables shown are included. Each reported cell gives the Benjamini–Hochberg FDRadjusted p-value followed by the effect size in parentheses. * pFDR < .05, ** pFDR < .01, and *** pFDR < .001. A dash indicates no significant difference after FDR correction. For the two-group comparisons, effect size is the rank-biserial correlation (rrb ). For Premium access vs. free-only access, a positive value indicates higher responses among students with premium access, whereas a negative value indicates higher responses among free-only users. For Single-tool vs. multi-tool, a positive value indicates higher responses among single-tool users, whereas a negative value indicates higher responses among multi-tool users. For Perceived instructor policy, eligible responses were Encouraged, Allowed with limits, and Discouraged; effect size is epsilon-squared (ϵ2 ), which indicates magnitude but not direction.
els therefore accounted for more variation in academic benefit and positive impact than in negative impact or coursesupported GenAI learning. Table 3 presents the significant main effects from the mutually adjusted regression models. Each coefficient represents the association between one predictor and the outcome while holding the other predictors constant. Only coefficients with raw HC3 p < .05 are shown; no multiple-testing correction was applied. The significant main effects were as follows: • Academic benefit. Self-reported academic benefit was positively associated with early reliance (β = .301, p < .001) and academic task support (β = .294, p = .008).
• Positive impact. Self-reported positive impact was positively associated with cognitive reliance (β = .495, p < .001), academic task support (β = .242, p = .016), and confidence in GenAI reliability (β = .196, p = .020). Positive impact was lower when instructor policy discouraged GenAI rather than allowed it with limits (β = −.138, p = .017). • Negative impact. Self-reported negative impact was positively associated with early reliance (β = .402, p = .004) and attitudinal change (β = .386, p < .001). • Course-supported GenAI learning. Self-reported course-supported GenAI learning was positively associ-
Table 3: Significant main effects from the multiple regression models. Outcome
β
Predictor
Academic benefit
Early reliance Academic task support
Positive impact
Cognitive reliance Academic task support Instructor policy: discouraged Confidence in GenAI reliability
Negative impact
Attitudinal change Early reliance
Course-supported GenAI learning
Instructor policy: discouraged Attitudinal change
HC3 SE
p
.301 .294
.084 .108
∗∗
< .001 .008∗∗
.495 .242 −.138 .196
.130 .099 .057 .083
< .001∗∗ .016∗ .017∗ .020∗
.386 .402
.108 .138
< .001∗∗ .004∗∗
−.186 .242
.059 .116
.002∗∗ .039∗
Note. Coefficients are standardized and mutually adjusted. ∗ p < .05; ∗∗ p < .01. Reported p-values are raw, two-sided HC3 values.
ated with attitudinal change (β = .242, p = .039). It was lower when instructor policy discouraged GenAI rather than allowed it with limits (β = −.186, p = .002). Course-clustered inference supported most of these findings. The association between academic task support and positive impact was no longer significant after standard errors were clustered by course (p = .060); all other main effects reported in Table 3 remained significant at p < .05. Interaction Effects Four interactions were tested to determine whether the association between a predictor and an outcome changed according to the level of another predictor: academic task support × evaluation literacy, early reliance × evaluation literacy, early reliance × policy clarity, and cognitive reliance × confidence without GenAI. The interaction between early reliance and evaluation literacy was significant for negative impact (β = .307, p < .001). As shown in Figure 2, early reliance was not associated with negative impact at lower evaluation literacy (−1 SD; b = .025, p = .861). The association was positive at average evaluation literacy (b = .357, p = .004) and stronger at higher evaluation literacy (+1 SD; b = .688, p < .001). Thus, early reliance was associated with greater negative impact primarily among students with average or higher evaluation literacy. This interaction remained significant when standard errors were clustered by course (p < .001). The other three interactions were not significant in the primary HC3 analysis. The early reliance × policy clarity interaction was significant only under course-clustered inference (p = .026) and was therefore not treated as a primary finding.
4.4
Impact of Factors on Learning with GenAI
To further understand factors that are highly correlated with the overall influence of GenAI impacting learning, We utilized all the constructs except the target item present in our survey results and train an ensemble learning algorithm via Gradient Boosting Classifier to predict the outcome variable – if “overall, GenAI improved learning.” The SHAP summary plot in Figure 3 shows the feature rankings and their direction of influence on overall how GenAI improved learning. Features are ranked vertically by their cumulative impact on the model. Each dot represents a single observation where,
Figure 2: Association between early reliance and negative impact at lower, average, and higher levels of evaluation literacy. Shaded regions show 95% HC3 confidence intervals.
the color represents the feature value (red for high, blue for low), and the horizontal axis measures the SHAP value (impact on the prediction). For example, high values of strong belief that GenAI improved academic work quality (red dots) strongly correlates with a positive model output. The findings suggest that improved learning through GenAI is associated not simply with its use but with the role it plays in the learning process. SHAP results show that students report greater learning benefits when they perceive GenAI as augmenting their existing capabilities. This includes improving the quality of academic work, being more creative, helping organize and summarize ideas, increasing confidence in their skills, and facilitating the discovery of new insights. In contrast, perceived learning benefits are lower when students experience GenAI as interfering with their cognitive engagement, particularly when they believe it hurts their critical thinking abilities or causes stress. Other factors, such as using GenAI saves time, are also positively associated with positive experiences of using GenAI in learning. Collectively, these findings suggest that the educational value of GenAI seems to be dependent more on whether it supports and extends students’ cognitive processes rather
Figure 3: SHAP summary plot illustrating the top features and their direction of influence on the outcome variable “Overall, GenAI improved learning”
than substitutes for them.
5
Discussion
This study examined how GenAI reliance, evaluation literacy, access conditions, and course policy relate to students’ self-reported learning experience in AI-related coureses. The results show that the reliance on GenAI is not simply a linear dependence such as “high” or “low” but how they engage with it and evaluate its outputs. For instance, early reliance and attitudinal change were associated with greater negative impact. At the same time, premium-access and multiple-tool users reported greater benefits and engagement, but neither group reported significantly greater negative impact. These findings suggest that focusing explicitly on the frequency of usage alone might not be a reliable factor when looking into the impacts of GenAI. Through our insights, it is also uncovered that there are diverse types of tasks that are related to cognitive offloading that may not be harmful to critical thinking in general. For instance, organizing or summarizing ideas and discovering new insights are positively associated with overall improved learning with GenAI, but caution should be exercised when it comes to prolonged reliance on these tools. Also, for this type of offloading to be productive, students should have sufficient domain knowledge and evaluation literacy to determine what has been offloaded. One interesting aspect we found in our analysis was about confidence. Single GenAI-tool users reported greater confidence in completing tasks without GenAI than multi-tool users. This contrast suggests that broader tool use was associated with greater perceived support and reliance on GenAI but not with greater confidence in working independently
without it. This should be a cautionary lesson for instructors and institutes in general on how to leverage GenAI in such a way that it does not harm the ability of independent thinking and confidence in solving tasks independently in environments and careers where there is not always a possibility of accessing GenAI tools. Similarly, self-reported academic benefits were associated with early reliance and academic task support, whereas positive impacts of GenAI use on academic experiences were attributed to cognitive reliance, academic task support, confidence in GenAI reliability, and less restrictive instructor policies. Negative impacts of GenAI use on academic experiences were attributed to early reliance and attitudinal change, with the association between early reliance and negative impacts becoming stronger as students’ evaluation literacy increased. With these types of uneven impact of GenAI on students, We hope that these results will pave the way to eventually address such disparities. Even though the results are insightful our study has certain limitations. This study relies on survey responses from only one R1 university. The insights in this work reflect only of those students who chose to share their experiences and may not represent the broader student population from diverse types of academic institutions. The survey also focused on thirteen different analysis measures – eight multi-item constructs and five single-item measures. Our insights are only limited to these thirteen measures and may not capture all the universally associated factors in the context of learning and relying on GenAI. However, we do believe that given the diverse set of students with diverse experiences of using GenAI, we strongly believe that these insights are robust and even though preliminary can be utilized to expand this survey to a larger set of course at multiple institutes which will be a part of our future work.
6
Conclusion
This study demonstrates that the role of GenAI in student learning at higher education institutions cannot simply be characterized by whether and how students use these tools. Instead, learning experiences are uneven and are shaped by students’ types of reliance, evaluation literacy, positive and negative learning impacts, and the contexts in which GenAI is used. The four user profiles further showed that students with similar levels of use and reliance did not necessarily report similar academic benefits or consequences. How students relied on GenAI also mattered. Students who used it to organize ideas, evaluate their work, or solve problems (cognitive reliance) generally reported more positive academic experiences. In contrast, students who turned to GenAI before thinking independently, searching for information, or asking an instructor (early reliance) reported both academic benefits and negative effects. As GenAI becomes increasingly embedded in higher education, we argue that educators should move beyond policies that simply permit or restrict its use. Policies should clarify appropriate uses, support equitable access, and help students use GenAI to extend rather than replace independent reasoning. The central question is not simply whether students should rely on GenAI, but when and how that reliance can support learning without weakening independent thinking and critical reasoning.
Acknowledgements This research was supported by a grant from Google.org via EMPIRE AI. We thank Dr. Josephine C. Seddon, PhD for her valuable feedback on our survey instrument.
References Bastani, H.; Bastani, O.; Sungu, A.; Ge, H.; Kabakcı, Ö.; and Mariman, R. 2025. Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26): e2422633122. Cena, E.; McParland, A.; Toivo, W.; Dalton, B.; Mundy, M.; O’Connor, P. A.; Robertson, A. E.; Swingler, M.; Wilson, P.; and Duncan, C. 2026. Studying with GenAI: Student views on the opportunities and risks of GenAI in higher education. Education and Information Technologies, 1–27. DeStefano, C. D.; Hackney, J.; and Moskal, P. D. 2026. Understanding the Generative AI Divide: Faculty and Student Perspectives in Higher Education. Online Learning, 30(2): 31. Kim, J.; Klopfer, M.; Grohs, J. R.; Eldardiry, H.; Weichert, J.; Cox, L. A.; and Pike, D. 2025. Examining faculty and student perceptions of generative AI in university courses. Innovative Higher Education, 50(4): 1281–1313. Lee, H.-P.; Sarkar, A.; Tankelevitch, L.; Drosos, I.; Rintel, S.; Banks, R.; and Wilson, N. 2025. The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI conference on human factors in computing systems, 1–22. Lundberg, S. M.; and Lee, S.-I. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30. Manikonda, L.; and Outlaw, D. 2026. Policy Fragmentation or Institutional Alignment? Institutional Governance of AI in Universities and Business Schools. arXiv preprint arXiv:2608.03584. Marshik, T.; McCracken, C.; Kopp, B.; and O’Marrah, M. 2025. Student and instructor perceptions and uses of artificial intelligence in higher education. Teaching of Psychology, 52(3): 339–346. Morari, V.; Grimes, D.; and Hawe, D. 2026. Academic integrity and generative artificial intelligence–views and perceptions of students in an Irish university. Journal of Academic Ethics, 24(1): 17. Saeli, H.; Ringer, J.; Mayer, C.; Cheney, M. B.; and Duncan, G. 2026. Parallel surveys of GenAI and writing: Students’ perspectives and instructors’ perception of student perspectives in writingintensive courses. Written Communication, 43(1): 196–224. Sah, R.; Hagemaster, C.; Adhikari, A.; Lee, A.; and Sun, N. 2025. Generative AI in higher education: student and faculty perspectives on use, ethics, and impact. Issues in Information Systems, 26(2): 373. Wang, Y.; and Luo, X. 2026. Generative AI as a Cognitive CoParticipant: Disciplinary Modulation of EFL Academic Reading Load and Motivation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 40889–40897.
A
Student Survey Instrument
Table A.1: Student Survey Instrument Section / Construct
Item
Question / Statement
Response Format
Consent
Q1
By selecting “I agree” below, I confirm that I am at least 18 years of age and that I consent to participation in the research study described above.
Consent
Demographics
Q2 Q3 Q4
Level of Study School Major
Categorical Categorical Categorical
GenAI Knowledge
Q5 Q6
How would you describe what Generative AI (GenAI) is? How would you describe how Generative AI (GenAI) can be used?
Open-ended Open-ended
Course Info
Q7 Q8 Q9 Q10
Course name and code/number Semester and year Course features Briefly describe the activities and assignments in this course (e.g., coding, writing, discussions, group projects, presentations, labs).
Open-ended Open-ended Multiple selection Open-ended
GenAI Tools Used
Q11 Q11_5_TEXT Q12 Q12_5_TEXT
Which GenAI tools do you use in this course? Other GenAI tool (please specify). Which specific Generative AI tools have you used in this course? Other GenAI tool (please specify).
Multiple selection Open-ended Multiple selection Open-ended
Instructor Policy
Q13
What is your understanding of the instructor’s policy on GenAI use in this course?
Categorical
Academic Task Support
Q14_1 Q14_2 Q14_3 Q14_4 Q14_5 Q14_6 Q14_7 Q14_8
GenAI is helpful in this course for learning about new topics. GenAI is helpful in this course for finding specific facts or information. GenAI is helpful in this course for discovering new insights from information or data. GenAI is helpful in this course for generating summaries of longer content. GenAI is helpful in this course for brainstorming or organizing ideas before starting an assignment. GenAI is helpful in this course for improving or rewriting assignments. GenAI is helpful in this course for completing assignments or significant portions of them. GenAI is helpful in this course for checking whether work meets rules, constraints, or formatting requirements. GenAI is helpful in this course for generating ideas to prompt thinking, even if not directly using any of the generated text. GenAI is helpful in this course for summarizing lecture slides or notes to help explain or clarify course materials. GenAI is helpful in this course for generating practice tests or quizzes to study for assessments in this course. GenAI is helpful in this course for generating content (e.g., text, Python code, or images) for direct use in assignments with modifications.
Likert Likert Likert Likert Likert Likert Likert Likert
I feel confident in my ability to complete tasks without any use of GenAI.
Likert
Q14_9 Q14_10 Q14_11 Q14_12
Confidence Without AI
Q15
Likert Likert Likert Likert
GenAI Reliability
Q16
I feel confident in GenAI’s reliability to complete the course tasks.
Likert
Negative Impact
Q17
I feel it challenging to adapt the content generated by GenAI to be used for this assignment in this course.
Likert
Positive Impact
Q18_1 Q18_2 Q18_3 Q18_4 Q18_5 Q18_6 Q18_7
In this course, GenAI helps me to save time. In this course, GenAI helps me to generate ideas. In this course, GenAI helps me to make progress in my work. In this course, GenAI helps me to continue in my work when I am stuck on next steps. In this course, GenAI helps me to improve the quality of my work. In this course, GenAI helps me to be more creative in my work. In this course, GenAI helps me to complete tasks I do not yet have the expertise to complete on my own.
Likert Likert Likert Likert Likert Likert Likert
Cognitive Reliance
Q19_1 Q19_2 Q19_3 Q19_4 Q19_5 Q19_6
In this course, I rely on GenAI when I need to recall facts and basic concepts. In this course, I rely on GenAI when I need to organize or summarize ideas. In this course, I rely on GenAI when I need to combine ideas to create new meanings. In this course, I rely on GenAI when I need to evaluate and quality-check my work. In this course, I rely on GenAI when I need to solve new problems. In this course, I rely on GenAI when I need to break down complex problems.
Likert Likert Likert Likert Likert Likert
Evaluation Literacy
Q20 Q21
I feel confident in my ability to evaluate the accuracy and quality of GenAI’s outputs. I reflect on and think critically when using GenAI for my coursework.
Likert Likert Continued on next page
Table A.1 – continued from previous page Section / Construct
Item
Question / Statement
Response Format
Q22_1 Q22_2 Q22_3
Likert Likert Likert
Q22_6
How important do you think it is to cross-check GenAI outputs with other trusted sources? How important do you think it is to evaluate whether the GenAI output is well-organized? How important do you think it is to reflect on whether GenAI correctly applies key concepts included in the output? How important do you think it is to evaluate whether the GenAI output ideas or arguments are logically coherent? How important do you think it is to assess how well the GenAI output combines ideas to form new meanings? How important do you think it is to determine whether the GenAI output meets my expectations?
Q23_1 Q23_2 Q23_3 Q23_4 Q23_5 Q23_6
I feel more confident in my skillset after using GenAI in this course. I feel less confident in my learning abilities after using GenAI in this course. Using GenAI in this course causes me stress. I worry that my peers use GenAI to gain an unfair advantage. My attitudes toward GenAI changed over time in this course. This course helped me to use GenAI more effectively.
Likert Likert Likert Likert Likert Likert
Q24_1 Q24_2 Q24_3 Q24_4
Likert Likert Likert Likert
Q24_9 Q24_10 Q24_11 Q24_12 Q24_13 Q24_14
How frequently do you consult GenAI before using traditional search engines or other resources? How frequently do you use GenAI to boost your confidence in learning? How frequently do you find that GenAI helps you manage your time and workload effectively? How frequently do you find that GenAI exposes you to ideas or perspectives you would not have otherwise considered? How frequently do you find that GenAI improves the quality of your academic work? How frequently do you find that GenAI reduces your motivation to learn course materials? How frequently do you find that GenAI has helped improve your grades or performance in this course? How frequently do you find that GenAI helps you retain course materials and achieve long-term learning goals? How frequently do you find that GenAI explains concepts better than your instructor? How frequently have you received clear guidelines about appropriate uses of GenAI in your classes? How frequently do you find that GenAI hurts your critical-thinking abilities? How frequently do you feel scared to use GenAI? How frequently do you use GenAI before thinking about a problem by yourself? How frequently do you use GenAI about a problem before reaching out to your instructor?
GenAI No Impact Positive Impact
Q25_1 Q25_2 Q25_3 Q25_4
GenAI has changed how I interact with my instructors. GenAI has changed how I interact with my peers. GenAI has had little impact on my overall learning experience. Overall, GenAI improved my learning in this class.
Likert Likert Likert Likert
Open Comments
Q26
Do you have any comments for the researchers? If you liked something or disliked something, or something did not work for you, please mention it here.
Open-ended
Q22_4 Q22_5
Positive Impact Negative Impact
Attitudinal Change Learned GenAI Early Reliance Academic Benefit
Negative Impact Academic Benefit
Q24_5 Q24_6 Q24_7 Q24_8
Early Reliance GenAI Policy Clear Negative Impact Early Reliance Attitudinal Change
Likert Likert Likert
Likert Likert Likert Likert Likert Likert Likert Likert Likert Likert