arXiv CSopen access

Investigating Conversational Agents to Support Secondary School Students Learning CSP

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

softwarearchitecture softwareengineering testing

software engineering, software architecture, testing

Investigating Conversational Agents to Support Secondary School Students Learning Computer Science Principles MATTHEW FRAZIER, University of Delaware, USA KOSTADIN DAMEVSKI, Virginia Commonwealth University, USA LORI POLLOCK, University of Delaware, USA Secondary school students enrolled in the AP Computer Science Principles (CSP) course commonly utilize web resources (e.g., tutorials,

arXiv:2604.16213v1 [cs.HC] 17 Apr 2026

Q&A sites) to better understand key concepts in the curriculum. The primary obstacle to using these resources is finding information appropriate for the learning task and student’s background. In addition to web search, conversational agents are increasingly a viable alternative for CSP students. In this paper, we study the potential of conversational agents to aid secondary school students as they acquire knowledge on CSP concepts. We explore general purpose, generative conversational agents (e.g., ChatGPT) and custom, fixed-response conversational agents built specifically to aid CSP students. We present results from classroom use by 45 high school students in grades 9-11 (ages 14-17) across six CSP sections. Our main contributions are in better understanding how conversational agents can help CSP students and an evaluation of the effectiveness and engagement of different approaches for CSP exploratory search. CCS Concepts: • Applied computing → Computer-managed instruction; Interactive learning environments; • Humancentered computing → Empirical studies in HCI. Additional Key Words and Phrases: conversational agent, CS Principles, CSP, exploratory search, ChatGPT ACM Reference Format: Matthew Frazier, Kostadin Damevski, and Lori Pollock. 2018. Investigating Conversational Agents to Support Secondary School Students Learning Computer Science Principles. In . ACM, New York, NY, USA, 34 pages. https://doi.org/XXXXXXX.XXXXXXX

INTRODUCTION

Research indicates that secondary school students who learn Computer Science Principles (CSP), through one of the curricula developed for this Advanced Placement (AP) course for learning foundational concepts of Computer Science (CS) [3], exhibit higher rates of college enrollment and improved skills such as problem-solving [102], response inhibition, planning, and coding [12]. Online educational resources (e.g., curricula, tutorials, documentation, Q&A sites) serve as key sources for secondary school students in learning CSP, enhancing classroom learning spanning concept areas including algorithms, security, and the design of the Internet. An impediment to effectively utilizing these resources is the search for information that aligns with the learning task and the learner’s background. Research shows that secondary school students need support in searching the web and developing information literacy (i.e., find, evaluate, organize, use, and convey information) [64, 65, 114]. Exploratory search is particularly challenging as it goes beyond simple lookup and instead is comprised of learning and investigative search intents [13, 76]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM 1

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

Research also suggests that conversations are a natural interface for conducting exploratory search [33, 60, 127]. For instance, a growing proportion of difficult web search queries, which do not result in successful information retrieval, are reformulated by users into conversation-starting questions [15, 72, 100]. The advantages of using conversations with a search system, compared to conventional lookup information retrieval or recommendation systems, are several: 1) question-asking dialogues are a communication modality that humans are well versed in using; 2) the retrieval system can pose clarifying questions to better understand the user need and context; 3) the user can provide incremental positive or negative (explicit) feedback that can help guide the system in the search process. If designed with customization to build better connections to learning, a conversational agent specializing in exploratory search can also help students relate to the CS concepts by encouraging them to connect CS concepts to their own relevant lived experiences. This paper focuses on the research question: “How do fixed-response and generative conversational agents impact the educational process and learning outcomes of secondary school CSP students performing exploratory search tasks?”. Conversational agents (also known as chatbots) can be differentiated by the nature of their responses: fixed-response, which select from a pre-defined set of responses based on human input, or generative, which use large language models (LLMs) to generate responses based on the input (or prompt). The fixed-response conversational agents are created with custom responses for a specific task. On the other hand, generative conversational agents are typically used without customization, as general-purpose. While generative conversational agents can technically also be customized (fine-tuned), with, e.g., few-shot prompting, doing so for more than a few inputs is difficult and requires a large dataset and powerful computing resources. Fixed-response conversational agents are usually built using a set of popular platforms [85] that provide advanced natural language understanding capabilities, e.g., Rasa [54], Google DialogFlow [32], Microsoft LUIS [79], Amazon Lex [106]. Fixed-response domain-specific conversational agents have previously been shown to help support a specified sphere of knowledge. For instance, these agents can help bridge the gap in the instructor’s background [11, 47, 68, 94]. In addition, researchers have shown that in general, secondary school students have high levels of engagement with chatbots [20]. In particular, they could help alleviate social anxiety for students who do not want to ask instructors questions publicly [39, 77, 78]. Generative conversational agents have become increasingly popular due to the recent demonstrated capabilities of LLM-based approaches of OpenAI’s ChatGPT [81], Meta’s LLAMA [111], Google’s Bard [6], and others. Researchers and educators are still coping with the impact of these agents on learning across the entire educational journey [17]. Generative conversational agents offer the potential advantage of drawing from extensive data sources, enabling them to provide high-quality, pertinent responses to a wide range of student inquiries. However, they also may pose potential drawbacks such as [81]: (1) Factually inaccurate responses: Sometimes, the generated outputs may sound plausible but are incorrect. (2) Unsuitable content for learners: The responses may not always align with the learner’s level or needs. (3) Irrelevant information: The model might include details that are unrelated to the learning task. (4) Prompt sensitivity: The models may require precise prompt engineering, where slight rephrasing can change the output’s correctness. (5) Lack of clarifying questions: Despite having the capability, the models often guess user intent rather than seeking clarification. (6) Model degradation: Over time, the quality of responses can deteriorate due to factors such as overfitting during fine-tuning. 2

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

(7) Model collapse: This refers to a phenomenon where a model’s performance drastically declines, often due to repetitive training on similar data. (8) Fixed knowledge domain: The model’s understanding is constrained by the data it was trained on, limiting its ability to address new or emerging topics. (9) Bias and stereotypes: The models may reflect biases present in the training data, such as political leanings or cultural perspectives, and may perform best in English. In contrast, fixed-response, custom conversational agents consistently deliver curriculum-appropriate responses, given that their responses are purposefully crafted for this context. Nonetheless, they may exhibit inflexibility and limit the exploration of various learning pathways. To answer our research question, we customized a fixed-response conversational agent, called Aida, to a subset of the CSP course. We then evaluated the effectiveness and engagement of Aida, ChatGPT as a representative general purpose, generative conversational agent, and conventional web search using Google. Our study consists of two research hypotheses: (H1) Conversational agents provide better learning experiences than conventional web search for secondary school CSP students performing exploratory search, and (H2) Generative conversational agents lead to better perceived student experiences despite being less engaging than fixed-response conversational agents. Both hypotheses were formulated prior to the study. Hypothesis 2 (H2) is based on preliminary observations indicating that the generative AI’s adaptability could offer more effective scaffolding compared to a fixed system. At the time of the study, research directly comparing retrieval-based agents with generative agents is limited, as the latter technology is still emerging [69, 98, 105, 112, 113]. To evaluate our empirical hypotheses, we recruited 45 high school students in grades 9-11 (ages 14-17) to use the agents to complete two tasks on topics within the CSP curriculum. We also gathered perspectives from these students through a post-survey. Specifically, this paper makes the following contributions to the Computing Education community: 1) a fixed-domain conversational agent for CSP; 2) metrics for the effectiveness and engagement of agents for learning; and 3) a comparative study involving 45 secondary school students using conversational agents and web search in a CSP class. Our results can help researchers to better understand students’ needs for designing future conversational agents for CS learning through exploratory search.

RELATED WORK

Created by the National Science Foundation in 2014, CSP is an AP secondary school curriculum framework for learning foundational concepts of CS [3]. More than a traditional introduction to programming course, CSP challenges secondary school students to explore how computing and technology can impact the world by educating students on ethical, inclusive, and collaborative computing culture. The CSP framework is grounded in five areas: (1) Creative Development, (2) Data, (3) Algorithms and Programming, (4) Computer Systems and Networks, and (5) Impact of Computing. Professional development programs [48, 93] and peer assistants [49] have been shown to help secondary school CSP instructors cope with the challenges of teaching CSP with little prior CS teaching experience, yet additional support is needed for CSP instructors and students [21, 84]. Students and instructors in CSP often search the web to acquire knowledge related to the course [64, 65, 107, 114, 116]. Instead of printed manuals and books, they use a variety of online sources including Q&A forums, CS-related documentation, tutorials, chats, etc [8, 28, 101]. CS students regularly search these sites to lookup information, reuse code, and learn new concepts and skills. While search is routinely used 3

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

by software engineers, support for effective searching for CS-related information is far from a solved problem for a variety of reasons [71]. Literature on ChatGPT in education or learning context has emerged across many subject areas [74, 123] including biochemistry, health professionals, chemistry, business management, nursing, multidisciplinary, social sciences, medicine [19], and computer science [34, 90]. Prather et al. [91] conducted a literature review that explored large language models in computing education. They classify LLM literature into five broad categories: 1) assessing the performance, capabilities, and limitations of LLMs (i.e., programming exercises [30, 31, 61, 75, 83, 88, 89, 96, 108], multiple-choice question answering [103, 117], textbook and exam questions [40, 57], and assisting the learner through hints or Socratic questioning [10, 87, 119]); 2) using LLMs to generate teaching materials [110]; 3) using LLMs to analyze student work [10]; 4) studying the interactions between programmers and LLMs [16, 80]; 5) position papers and surveys/interviews [24, 25, 36, 46, 59, 62, 66, 73, 86, 92, 122, 125]. Researchers have assessed ChatGPT’s capability of generating code in a variety of contexts [23, 35, 44, 70, 99], e.g., Codex GPT-3 LLM, Github Copilot, and Google DeepMind Alphacode. ChatGPT has been shown to easily solve CS1 and CS2 coding problems [91]. While many educators are concerned about the threat that ChatGPT imposes to academic integrity and the learning development of their students, one study showed that code generated by ChatGPT had enough differences from student generated code that it could be detected very accurately [55]. A study by Hou et al. suggests that although LLMs are being rapidly adopted, they have not yet fully eclipsed traditional help-seeking resources [53]. Conversational agents (or chatbots) are increasingly being incorporated into education pedagogy [115, 118] including students’ hybrid web e-learning environments [109, 120], supporting discussion threads [43, 126], providing feedback on programming assessments[18], help requests [52], and summarizing Stack Overflow threads [26, 121]. Educational researchers have observed that chatbots can assist secondary school CS instructors [29], especially those lacking a CS background [11, 47, 68, 94]. In addition, researchers have shown that in general, secondary school students have high levels of engagement with conversational agents [20]. Chatbots could help alleviate social anxiety for students who do not want to ask instructors questions publicly [39, 77, 78]. Research shows that CS reasoning and code explanations created by LLMs are easier to understand than those created by students [67]. In many aspects, HCI researchers have investigated ways to design and implement conversational agents in educational settings [27, 51, 63, 128] including prompt engineering [124]. To our knowledge, no one has studied the effectiveness and student engagement levels of fixed-response versus generative conversational agents for CSP exploratory search tasks. 3

EVALUATING CONVERSATIONAL AGENTS WITH CSP STUDENTS

Our study is designed to answer the research question: “How do fixed-response and generative conversational agents impact the educational process and learning outcomes of secondary school CSP students performing exploratory search tasks?” By recruiting students currently enrolled in CSP courses, we compare three approaches to gather information via exploratory searches: two conversational agents and a web search engine. Specifically, through several metrics, we measured the effectiveness of exploratory search for secondary school CSP and student engagement with: (1) Aida, a customized, fixed-response conversational agent, (2) ChatGPT, a general-purpose, generative conversational agent, and (3) Google’s web search. 3.1

Approaches to Exploratory Search

3.1.1 Aida: Fixed-response Conversational Agent. Since, to our knowledge, there were no existing fixed-response conversational agents for CSP, we designed and built one. Similar to a flow chart, fixed-response conversational agents 4

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

(a) Aida User Interface

(b) ChatGPT User Interface

Fig. 1. Conversational user interfaces for with Aida’s DialogFlow environment and ChatGPT’s HuggingFace environment.

map out the conversation and direct the conversational flow using follow-up questions. Such conversational agents contain a retrieval-based response system, i.e., for each user message, the model detects the intent, identifies key entities (e.g., names, locations, dates), and retrieves a predefined response from its repository. Fixed-response conversational agents are also task-driven, which allows us to direct the conversational flow to align with the CSP curriculum. Through follow-up questions, the conversational scaffolding allows us to organize the information being revealed by the agent to a pedagogically sound framework for delivering computer science lessons. To build the fixed-response agent in our study, Aida (i.e., affectionately named as to “aid a user”), we chose the Google DialogFlow ES platform [32]. Google DialogFlow ES is widely utilized in research [9, 38, 42] and commercial use [32], as it offers a low barrier to entry, a slot-filling intent strategy, session management, an API interface with code examples in various programming languages, and a simple user interface. Specifically for the demographics and purpose of this study, DialogFlow ES meets the requirements for research in pedagogical conversational agents. Following Weber et al.’s taxonomy, a rule-based back-end technology with a text interface and a persona is an appropriate pedagogical conversational agent design for secondary school students when the role of the agent is to act as a tutor for initial or actual learning of conceptual knowledge in CS [118]. Aida’s design aligns with the learner-centered design principles for conversational agents in educational settings, as outlined by Schmitt and Wambsganß et al. [104]. More specifically, Aida is adapted to a specific educational setting, is empathetic, and includes an avatar. Aida identifies and addresses the most frequently asked questions (albeit for predefined topics), has appropriate answers, and offers further useful information for students. At the time of the 5

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

Aida Human Start

Directional Question to Main Topics

Personal Reflection Socratic Question

Main Topic Question

Personal Reflection Answer

Real World Application Socratic Question

Real World Application Answer

Definition

Fig. 2. Pedagogical DialogFlow Follow-Up Intent Design.

study, Aida was not deployed in each school’s identity yet; however, Aida was easily accessible through a web-based application and convenient to use. To respond to student messages, Aida utilizes both rule-based grammar matching and machine learning (ML) intent matching (classification threshold of 0.3). Students accessed Aida’s user interface shown in Figure 1a through one of two custom URLs depending on their assigned task for the study: (1) Binary/Linear Search task or (2) API/Library task. Aida: Pedagogical Intent Design Methodology. Figure 2 shows the pedagogical design of Aida. Aida uses the Socratic method as a teaching strategy to allow students to share their thoughts, expand their understanding, and explore new related ideas in the context of their current work [5]. We designed the rule-based intents to align with Code.org’s CSP pedagogy [4], which follows Kaddoura’s Think-Pair-Share pattern [58], a strategy for enhancing student’s critical thinking skills. In particular, their tools and materials are designed to support exploration and discovery by those without computer science knowledge, so that students can develop an understanding of these concepts through “play” and experimentation [2]. In the Think-Pair-Share pattern, the instructor first asks an open-ended question and students think quietly about it for one to two minutes. Second, students form pairs where they discuss the question for two to five minutes with classmates. Third, the entire class engages in a discussion where each student pair shares their thoughts and ideas they’ve gathered. Last, the instructor facilitates the student’s ideas while summarizing the key points as it relates to the relevant learning objectives. Code.org’s CSP lectures parallel the Think-Pair-Share pattern into their own form of a three-part sequential pattern: 1) Warm-Up; 2) Activity; 3) Wrap-Up. Figure 3 illustrates an example conversation where Aida incorporates the Warm-Up, Activity, and Wrap-Up CSP curricula materials into the DialogFlow intent pedagogical design. Instructors first introduce CS concepts to students through an open-ended question to think about for a brief period as a Warm-Up. Next, students join pairs to work on an Activity where they brainstorm and solve questions while sharing their thoughts with their partners. After the activity, all student pairs share their thoughts with the class, and the instructor summarizes the concepts in the lesson in a Wrap-Up. Aida incorporates this design into its conversational responses by assuming the 6

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

partial role of the instructor and student in our intent design. For primary CS concepts related to the learning task, Aida’s initial response ends in an open-ended question to elicit critical thinking from the student as a Warm-Up. Next, Aida’s responses simulate interactions of pairing with a partner and sharing ideas through conversation by offering real-world examples and fallback phrases related to the open-ended Warm-Up question. Last, Aida summarizes by giving a formal definition of the primary CS concept to Wrap-Up discussion on a topic. To mimic the Think-Pair-Share pattern in Aida, we build follow-up questions in intents to direct the conversational flow to advance the student through each phase of the Think-Pair-Share pattern. The open-ended follow-up questions are designed to probe the learner, through clarifying assumptions or implications, to come up with the answer on their own. In addition to following the pedagogical approach in CSP lectures, we provided intents that show examples of processes on CS concepts. To maintain the scope of the knowledge domain, we provide definitions for related CS concepts. For any non-related CS topic asked to the model, we customized a response to redirect the student’s attention back to the topics relevant to their task. Aida: Intent Content. A key component of Aida’s fixed-response design is conveying content that is appropriate for a secondary school student’s learning level. Given that Aida is a fixed-response agent, the chances of receiving the same response more than once are higher for a given query and thus, it is imperative to curate many responses to a wide variety of inputs such that Aida appears more human-like. To diversify Aida’s responses, we aggregated text data from two CSP curricula consisting of: 1) Code.org’s CSP lecture slides; 2) Code.org’s CSP student web modules; 3) The Beauty and Joy of Computing instructional videos; and 4) The Beauty and Joy of Computing CSP student web modules. We provide each intent with a minimum of 20 training phrases to anticipate a variety of student inputs. Each intent has a minimum of 6 different responses where each response is grounded in different CSP course curricula material. Aida contains 69 intents in total across both tasks in our study (35 intents for Binary/Linear Search Task and 34 intents for API/Library Task). 3.1.2 ChatGPT: Generative Conversational Agent. For the general-purpose, generative conversational agent, we used OpenAI’s ChatGPT (gpt-3.5-turbo). We chose ChatGPT for our generative conversational agent because at the time of writing it was the most popular conversational agent based on large language models. We aim to simulate a real-world experience that students would encounter using ChatGPT on their own. We chose the GPT-3.5-turbo model that powers the free version of ChatGPT which many students are likely to use. The model temperature for ChatGPT’s web interface is 0.7, and we chose 0.7 as the model temperature in our agent. We explicitly considered but decided not to adjust the model temperature in our design. We intentionally chose to reproduce an experience similar to what a student would encounter on the web. Further, to emulate the ChatGPT web interface, we chose not to manipulate the API system prompt or user’s input prompt into the model in any way. We acknowledge that there may be a custom system prompt in the web interface of ChatGPT, however, we were unable to find any resources that support this claim. Further, nothing prohibited students from implementing prompt engineering, as would be required of them when using ChatGPT’s web interface on their own, where a custom system prompt could be initiated by the students. Students communicated with ChatGPT through a HuggingFace [41] environment created specifically for each student in this study. HuggingFace is a popular free platform widely used in research that comes pre-built with common machine learning models, datasets, and applications readily available. We chose HuggingFace to host ChatGPT for its ease of set-up, reliable security, high availability of the platform, and to minimize costs. The user interface contains three components as shown in Figure 1b: (1) a conversation history log, (2) a text box to curate their input (brainstorm questions), and (3) a submit button to send 7

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Source Code.org CSP Curricula Materials

Frazier, et al.

Aida

Human

Thanks Participant 43! How can I help you learn about building APIs or libraries? I can give you some examples, teach you what each is, explain the process, as well as offer advice on when to use it and when you shouldn't. What is an API? Warm-Up Prompt How can you be sure that the users of your library have all they need to use your functions?

Application Program Interfaces (APIs) serves as a guide for how to communicate with a library. How can you be sure that the users of your library have all they need to use your functions?

That's a good question. I'm not sure exactly how you would do it, but I would start by pulling and storing the required information from the library.

Activity Prompt What information does the user need to know in order to use the library functions?

Great! Imagine you want to order a pizza from your favorite pizza place, but you don't want to go there in person. Instead, you call the pizza delivery service and place your order. The pizza delivery service acts as a middleman between you and the pizza place. They take your order, pass it on to the pizza place, and then deliver the pizza to your doorstep. The pizza place provides you with a menu that you can choose from. This is similar to an API. The API (delivery service) helps you interact with the pizza place without you having to directly go there or know how they make the pizza. What information does the user need to know in order to use the library functions? Imagine the effect on a program if a library function adds two numbers, but the user expected the function to format a date. The input would cause an error.

They would need to know the menu, right? Activity A library should have documentation for the included functions: how each function works, a complete list of the parameters, and what (if anything) will be returned.

Wrap-Up Vocabulary API: Application Program Interface specifications for how functions in a library behave and can be used.

A library should have documentation for the included functions: how each function works, a complete list of the parameters, and what (if anything) will be returned. An Application Program Interface (API) is specifications for how functions in a library behave and can be used. Do you want me to explain this again? You can ask me to provide you another explanation of libraries, the role of libraries, show an example, when to use libraries and when you shouldn't, as well as details on creating libraries. Which subject are you interested in studying? If you'd want, we may also discuss procedural abstraction or documentation.

DialogFlow Intent Directional Question to Main Topics

Main Topic Question

Personal Reflection Socratic Question

Personal Reflection Answer

Real World Application Socratic Question

Real World Application Answer

Definition

Directional Question to Main Topics

Fig. 3. Pedagogical Conversation Example of DialogFlow Follow-Up Intents.

messages to OpenAI’s ChatCompletion API [7]. Using a custom interface for ChatGPT instead of general site for this tool allowed us to directly collect the student interactions. 3.1.3 Conventional Web Search. We chose Google as the search engine for our study because it is the world’s largest and most widely used search engine. Research shows that students search the internet more frequently than they ask peers for help, and ask peers for help significantly more frequently than they ask instructors for help [50, 91]. 8

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

3.2

Pilot Sessions

Prior to the main study, we conducted five pilot sessions with students who did not participate in the study. Four sessions used Aida and one used ChatGPT to evaluate their functionality and alignment with the CSP pedagogical framework. Feedback from these sessions led to iterative improvements in Aida’s design, including adjusting the tool to facilitate better conversational flow. These pilot sessions also helped us refine the user interfaces for all tools, ensuring the tasks were appropriate for secondary school students, and also helped us adjust our classroom study procedure. 3.3

Participants

We recruited three high school instructors (each from a different secondary school in a region of the northeastern United States) to conduct our study with the students in their AP CSP classrooms. Each instructor teaches two sections of the course where each section has 15-20 students. Not all students from every section participated in the study; only students who obtained parental consent and child assent prior to the start of the study participated. The AP CSP course contains the same content across all sections, yet depending on the pace of the learning experience, each section may be at a slightly different point within the curriculum. All instructors were willing to have their students participate in our study during one class period. Each instructor reviewed our study protocol, including the purpose of the study, an explanation of exploratory search tasks, and the specific tasks their students would accomplish during the study. Of the 120 students across the six AP CSP course sections, we conducted the study with 45 high school students who obtained parental consent and child assent to participate in our study. Table 1 shows the student demographics by gender, grade, race or ethnicity, and experience level. Students ranged in school year grades 9-11 and had varying levels of prior introductory computer science experience except two students with no programming experience. Each instructor assisted in recruiting the students for the study by facilitating advertisements through an email to the parents of the students. Table 1. Student Participant Demographics by Gender, Grade, Race/Ethnicity, and Experience Level.

Grade

Freshman Grade 9 Sophomore Grade 10 Junior Grade 11 Race/Ethnicity American Indian or Alaska Native Black or African American Other White Experience Program at home on my own (All that Have block-based programming experience apply per Have taken programming courses in middle school student) Have taken programming courses in high school Have programmed in Python or a similar language No programming experience

3.4

Man

Woman

Non-Binary

7 15 9 1 8 6 16 9 20 17 30 17 2

4 6 3 0 1 5 7 2 5 9 13 11 0

0 1 0 0 1 0 0 0 0 1 1 0 0

Task Design

To engage students in exploratory search activities for learning CSP, we designed tasks that necessitate multiple search activity iterations and interpretation using Athukorala et al.’s approach [14] to task design. There are many candidate search activities that have been identified in the literature [76]; however, we focused our study on students completing 9

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

BIG IDEA

Algorithms and Programming

BIG IDEA

TOPIC 3.11

SKILLS

Binary Search

SKILL

1.A

2.B

Investigate the situation, context, or task.

Implement and apply an algorithm.

1.D

Algorithms and Programming

TOPIC 3.14

Libraries

Evaluate solution options.

Required Course Content

AVAILABLE RESOURCES

Required Course Content

§§ External Resource > Searching Algorithms from CS Unplugged

ENDURING UNDERSTANDING

§§ Classroom Resources > Binary Search

ENDURING UNDERSTANDING

AAP-3

AAP-2

Programmers break down problems into smaller and more manageable pieces. By creating procedures and leveraging parameters, programmers generalize processes that can be reused. Procedures allow programmers to draw upon existing code that has already been tested, allowing them to write programs more quickly and with more confidence.

The way statements are sequenced and combined in a program determines the computed result. Programs incorporate iteration and selection constructs to represent repetition and make decisions to handle varied input values.

LEARNING OBJECTIVE

ESSENTIAL KNOWLEDGE

AAP-2.P

AAP-2.P.1

For binary search algorithms:

The binary search algorithm starts at the middle of a sorted data set of numbers and eliminates half of the data; this process repeats until the desired value is found or all elements have been eliminated.

1.D

b. Explain the requirements necessary to complete a binary search. 1.A

LEARNING OBJECTIVE

ESSENTIAL KNOWLEDGE

AAP-3.D

AAP-3.D.1

Select appropriate libraries or existing code segments to use in creating new programs. 2.B

A software library contains procedures that may be used in creating new programs. AAP-3.D.2

Existing code segments can come from internal or external sources, such as libraries or previously written code. AAP-3.D.3

The use of libraries simplifies the task of creating complex programs.

AAP-2.P.2

Data must be in sorted order to use the binary search algorithm.

AAP-3.D.4

Application program interfaces (APIs) are specifications for how the procedures in a library behave and can be used.

AAP-2.P.3

Binary search is often more efficient than sequential/linear search when applied to sorted data.

AP Computer Science Principles Course and Exam Description

AAP-3.D.5

Documentation for an API/library is necessary in understanding the behaviors provided by the API/library and how to use them.

Course Framework V.1 | 83 Return to Table of Contents

AP Computer Science Principles Course and Exam Description

(a) Binary Search

(b) Libraries

Fig. 4. AP College Board Computer Science Principles Curriculum Framework.

two tasks in Marchionini’s “Learn” Exploratory Search category: Comparison and Knowledge Acquisition. We chose these tasks because Comparison and Knowledge Acquisition tasks are suitable for searching as a learning process (i.e., multi-step educational search) since they entail critical and receptive cognitive learning through the student’s learning behavior (i.e., recalling, comprehending, contrasting, aggregating, and synthesizing) and search behavior (i.e., acquiring, comparing, and evaluating usefulness) [97]. We believe that these tasks are representative of common daily tasks that secondary school CSP students take over long periods of time such as reviewing prior lecture materials and completing homework assignments that minimize gaps of knowledge in CS concepts. We chose search algorithms and API/libraries as the two topics for our study because they are more complex topics in the CSP curriculum that motivate exploratory learning. We created two tasks such that information being explored in each task aligns to the Essential Knowledge in the course’s required content for those topics as shown in Figure 4. Table 2 shows the task descriptions. The Binary/Linear Search Task involves analyzing the similarities and differences among the binary and linear search algorithms. This includes comprehension of how each algorithm works, discernment of when to use each algorithm (sorted vs. unsorted lists), and understanding the impacts of each algorithm’s efficiency. The API/Library Task requires students to gain an objective understanding of the requisite components of an effective API/library. This includes comprehension of what APIs/libraries are, how they work, communicating API/library usage through documentation, and understanding that errors may arise due to improper use of APIs/libraries. The expected outcome of each task was for students to write a paragraph summarizing the answer to the question provided, using information gathered through exploratory search during the activity. 10

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 2. Exploratory Search Tasks. Binary/Linear Search Task (Comparison): Imagine that you have been asked to write a program that searches for an item in a list. You have the option to use binary search or linear search. Learn enough to write a short paragraph comparing and contrasting each search algorithm and how each works. What would you have to know about the search problem to know which algorithm to choose? API/Library Task (Knowledge Acquisition): You’re tasked with sharing reusable code from a program that you’ve created with your friends. You will create a library of functions and an API. Learn enough to write a short paragraph on how each works. Write a short paragraph summarizing the key points that you need to keep in mind while you’re creating a library and API. What would you have to know about how your friends intend to use the reusable code from your program to know how to build a library and API?

3.5

Classroom Protocol

The study took place at each secondary school during one CSP class period. We designed the study protocol to fit within 50 minutes (the shortest class duration) to accommodate all of the instructor’s schedules. Each study session consisted of an introduction, sequential completion of the two tasks, and a post-study survey. This study employed a within-subject design, where each participant interacted with both Aida and a comparison tool (either Google or ChatGPT) before completing a survey to assess their experiences. We varied the ordering of the use of the conversational agents and tasks among students to avoid learning bias. We created eight variations of the protocol for each tool/task combination of the study. To minimize risk of introducing unintentional bias, each protocol was randomly distributed to the students. For each protocol, the students were given a different version of the instructions that included the study tasks and directions on how to access the exploratory search tools. Each student was allotted up to 15 minutes to complete each exploratory task. The researcher attended each study field experiment. With the help of each course’s instructor, the researcher led the activity and facilitated the study including monitoring the pace and attention of student’s progress during the study to ensure successful completion and answer student’s questions. Our study’s instructions were online documents stored on the researcher’s university server. The students accessed the instructions using the web browser on computers in the classroom via a unique short URL. Each instruction set contained space for the students to take notes and curate the outcome of their learning as well as a space to store the history of their session which was collected after completing each task. Students were told to not use any handheld devices and close all computer applications while participating in the study with the exception of the web browser. The goal was to avoid any potential distractions as well as allow students the ability to brainstorm and perform notetaking while performing the exploratory search tasks. All students were free to use each tool in any way of their choice. They were not given any instructions on how to interact with each conversational agent, except to avoid jargon. 3.6

Post-Study Survey

We chose not to conduct a pre- and post- study survey of the participant’s knowledge of the task (as in traditional comparative analysis) because, given the timing of this field study in May 2023, the topics were likely already covered in the CSP course. Therefore, the comparative knowledge gained from completing a task with each conversational agent may not reflect the true educational impact of the conversational agents. Instead, we conducted a post-study survey centered around comparing the utility of each of the agents towards completing the tasks. 11

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

The post-study survey gathered information on the effectiveness and the interactivity of the conversational agents. The survey begins with 5-point Likert scale questions that solicit feedback on the students’ perception of how well the conversational agent worked for completing the task including: 1) ratings of how useful the information found by the conversational agent was; 2) if the student perceived that the conversational agent helped them; 3) if the student would use the conversational agent in the future. The survey also contains 5-point Likert scale questions to obtain feedback on the interactivity of the conversational agent including: 1) the student’s ability to understand the conversational agent; 2) the student’s perception on if the conversational could understand them; 3) if the conversational agent exhibited or behaved in a non-human manner; 4) the frequency of the conversational agent’s follow-up questions. We also include open-ended questions to collect any additional feedback on what student’s like or dislike about each conversational agent and any other comments that they want to provide. Next, we collect a comparative conversational agent evaluation where students select their preference of each tool relative to: 1) presenting content that relates concepts to personal experience; 2) the method that made the activity easiest to complete; 3) presenting content that is personalized to secondary school terminology; 4) the method that was the most enjoyable as a learner. Last, we conclude the post-study survey by collecting the student’s demographic data. 3.7

Collected Data

We collected four key kinds of data from each student’s session: (1) the conversational agent’s dialogue, including timestamps, (2) the student’s web browser history including the timestamps of the websites visited and Google search queries, (3) the student’s post-survey answers, and (4) the outcomes of each student’s learning experience (the student’s answers to each task). Aida conversational agent dialogues were collected using the built-in conversation history within the DialogFlow system. Both the ChatGPT dialogues and web browser history were stored in the space provided in each student’s instruction set. 4

DATA ANALYSIS METHODS AND FINDINGS

To answer our research question “How do fixed-response and generative conversational agents impact the educational process and learning outcomes of secondary school CSP students performing exploratory search tasks?”, we analyzed the data we collected to investigate the effectiveness and user engagement, using several different metrics. Prior to our statistical analysis, we tested for a normal distribution by visually checking the distribution of our data through Q-Q plots and histograms of each metric. We conducted an ANOVA analysis comparing the students in the three conditions across both tasks for all metrics of effectiveness and user engagement. A summary of our ANOVA analysis is shown in Table 3. This section describes the metrics and reports the findings; the next section discusses the overall results. 4.1

Effectiveness

To evaluate our first hypothesis (H1) that “Conversational agents provide better learning experiences than conventional web search for secondary school CSP students performing exploratory search”, we analyzed the effectiveness of each approach to exploratory search for secondary school CSP across three dimensions, i.e., the effectiveness of: (1) users achieving the task, (2) helping users learn, and (3) customizing to the CSP course scope. Figure 5 presents our results. 4.1.1 Effectiveness of Achieving the Task. To measure the students’ effectiveness in achieving the task, we analyzed the answers that students provide as the outcome of their learning. We use three measures: Student Answer Completeness, Accuracy, and Extraneousness. 12

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Table 3. Overview of results (mean and standard deviation (SD)) of student answer completeness, accuracy, extraneousness, task duration, exploratory actions, presented examples, and tool scope for both tasks across the three tools. Tool

Aida (A)

ChatGPT (C)

Google (G)

Significance Tukey post-hoc test

Completeness Accuracy Extraneousness Duration Exploratory Actions mean mean mean mean mean (SD) (SD) (SD) (SD) (SD) 0.33 0.86 0.17 11.67 10.56 (0.17) (0.2) (0.26) (3.81) (5.05) 0.48 0.82 0.17 5.77 2.91 (0.19) (0.18) (0.21) (3.78) (1.32) 0.38 0.78 0.27 5.48 6.78 (0.19) (0.3) (0.32) (4.01) (3.37) p <0.01 ns ns p <0.001 p <0.001 A-G, G-C G-C A-G, G-C

Presented Examples mean (SD) 1.58 (1.32) 1.36 (1.97) 1.17 (1.7) ns

Tool Scope mean (SD) 0 (0) 1.59 (1.74) 2.39 (6.35) p <0.025 G-C, C-A

Table 4. Student Answer Completeness Criteria.

Binary / Linear Search Criteria

API / Library Criteria

(1) Binary search is a way to find an item in a list. (2) Binary search at most checks only half of the list. (3) Binary search compares middle value in the list with value being searching for, and eliminates half of the list each comparison, depending on if the value is greater or less than the item being searched for. (4) Linear search is a way to find an item in a list. (5) Linear search at most checks all items in the list. (6) Linear search is sequential and runs from beginning to the end of the list. (7) Binary search is more efficient than linear search. (8) Binary search is better for larger lists. (9) Linear search is better for smaller lists. (10) Binary search is only for sorted lists. (11) Linear search is for lists of any order. (12) When choosing between both algorithms, consider the order or the list. (13) When choosing between both algorithms, consider the size of the list.

(1) APIs are an interface that provides specifications on how to interact with the library. (2) An API transfers data between parties. (3) Libraries are a collection of reusable functions that perform specific tasks. (4) A library works by importing it and calling its functions. (5) Consider testing to prevent library or API misuse or compatibility issues for users. (6) Documentation communicates to others how to interact with a library or API. (7) Consider the formatting of the library or API such as meaningful names, parameters, data types, and return values. (8) Consider the programming language that your users want to use. (9) Consider the specific tasks that users are accomplishing when building a library or API.

Student Answer Completeness. We define the completeness of the student’s answer as the proportion of information in the student’s answer that we expect to be present in their answer. We consider a student’s answer to be “complete” if their answer addresses all information items connected to the task that they were assigned. To ensure objectivity and rigor in developing these criteria, three individuals (one of which is the researcher) with significant CS teaching assistant experience independently generated the expected information items (i.e., the completeness criteria) for each task. These independently developed criteria were then compared and discussed collaboratively to ensure consistency. The researcher merged these sets to create a rubric with the finest granularity of details expected for a complete answer for each task. This collaborative process, involving multiple experienced individuals, was designed to mitigate individual biases and enhance the objectivity of the coding scheme. Notably, the completeness criteria is intentionally exhaustive by design. We require that the student explicitly states each criterion in their answer instead of relying on a pre- and post- study evaluation of student comprehension. Table 4 presents the final criteria for completeness of each of our 13

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00

Frazier, et al.

1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 Aida

ChatGPT

Google

1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00 Aida

(a) Completeness

ChatGPT

Google

(b) Accuracy

Aida

ChatGPT

Google

Fig. 5. Distributions of Student Answer Completeness (left), Accuracy (middle), and Extraneousness (right). The results are aggregated across both tasks for each tool. All participants utilized Aida for one task, and either ChatGPT or Google for one task (Aida N=45, ChatGPT N=22, Google N=23).

study tasks. The researcher applied the detailed criteria as a check list on the corpus of student answers. Each student answer was scored for completeness as: 𝐶𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 =

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜 𝑓 𝐶𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 𝐶𝑟𝑖𝑡𝑒𝑟𝑖𝑎 𝑖𝑛 𝑆𝑡𝑢𝑑𝑒𝑛𝑡 𝐴𝑛𝑠𝑤𝑒𝑟 𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜 𝑓 𝑆𝑡𝑢𝑑𝑒𝑛𝑡 𝐴𝑛𝑠𝑤𝑒𝑟 𝐶𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 𝐶𝑟𝑖𝑡𝑒𝑟𝑖𝑎

Figure 6 shows an example of student answers for the Binary/Linear Search task. The student’s answer meets 8/13 criteria for completeness. This student acknowledges that both algorithms find an item in the list of items (Criteria 1 and 4) by stating “A similarity between linear and binary searches are they both are used to find a certain number within a list”. This student describes that linear search is sequential (Criterion 6) by stating “Linear search goes one by one through the list...” and that the algorithm traverses the entire list for the item being search for (Criterion 5) by continuing that “... and checks each number to see if it is the number that is being searched for.” The student states that linear search is suitable for unsorted lists by answering “The data can be in any order . . . “ (Criterion 11) and that binary search is only suitable on sorted lists answering “Binary search on the other hand must have all the numbers placed in numeric order. . . ” (Criterion 10). The student describes that linear search is better suited for smaller lists (Criterion 9) and binary search is more efficient in time complexity than linear search for larger lists in their answer, stating “One problem with linear [search] is for larger data sets the computing power would be too great to reasonably use it to search.” and “...but with larger datasets it requires much less computing power.” (Criterion 8). The student’s answer neither states that binary search at most checks half of the list (Criterion 2) nor does the student describe how the binary search algorithm works (Criterion 3). Although the student explains usage on larger lists, the student does not acknowledge that binary search is more efficient overall (Criterion 7). The student does not state that choosing between the two algorithms depends on the size of the list (Criterion 12) and the ordering of the list (Criterion 13). In Figure 5a, we see that the student answers for those students that used ChatGPT scored higher on completeness than both Aida and Google, which were similar (median of 0.5 for ChatGPT compared to median of 0.33 for both Aida and Google). The small proportions of complete answers, in absolute terms, is attributed only to the detailed and exhaustive nature of our completeness criteria. Our ANOVA analysis comparing the students in the three conditions regarding their answer’s completeness across both tasks is shown in Table 3. The test revealed that students who interacted with ChatGPT produced significantly more complete student answers compared to both Aida (𝑀𝐴𝑖𝑑𝑎 = 0.33, 𝑀𝐶ℎ𝑎𝑡𝐺𝑃𝑇 = 0.478, 𝑡 = −3.146, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = −0.84, 𝑝 < 0.01) and Google (𝑀𝐺𝑜𝑜𝑔𝑙𝑒 = 0.384, 𝑀𝐶ℎ𝑎𝑡𝐺𝑃𝑇 = 0.478, 𝑡 = −1.705, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = −0.508, 𝑝 < 0.01) having a large and medium effect size, respectively. 14

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY A similarity between linear and binary searches are they both are used to find a certain number within a list. Linear search goes one by one through the list and checks each number to see if it is the number being searched for. The data can be in any order and that makes it easier to add more values to the list. One problem with linear search is for larger data sets the computing power would be too great to reasonably use it to search. Binary search on the other hand must have all the numbers placed in numeric order which does make it harder to add new values but with larger datasets it requires much less computing power.

Fig. 6. Student Answer Example for Binary/Linear Search Task.

Student Answer Accuracy. We define the accuracy of the student’s answer as the proportion of expected information items in a student’s answer that are factually accurate. We consider a student’s answer to be “accurate” if the information items in their answer semantically matches the expected information in the student answer completeness criteria for the task being explored. There must be no indication of AI hallucination, misinformation, or inaccurate information from web sources. The researcher assigned a score to each student’s answer representing the ratio of the total count of accurate information items present in their answer out of the total number of expected criteria in the answer. Each student answer received a score, computed as: 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜 𝑓 𝐴𝑐𝑐𝑢𝑟𝑎𝑡𝑒 𝐶𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 𝐶𝑟𝑖𝑡𝑒𝑟𝑖𝑎 𝑖𝑛 𝑆𝑡𝑢𝑑𝑒𝑛𝑡 𝐴𝑛𝑠𝑤𝑒𝑟 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜 𝑓 𝐶𝑜𝑚𝑝𝑙𝑒𝑡𝑒𝑛𝑒𝑠𝑠 𝐶𝑟𝑖𝑡𝑒𝑟𝑖𝑎 𝑖𝑛 𝑆𝑡𝑢𝑑𝑒𝑛𝑡 𝐴𝑛𝑠𝑤𝑒𝑟

As noted earlier, the student’s answer in Figure 6 meets 8/13 criteria for completeness: Criteria 1, 4-6, and 8-11. Of the eight criteria completed in the student’s answer, the student’s answer meets 8/8 in accuracy. This student accurately states that both algorithms are for searching for items (Criteria 1 and 4) by stating “A similarity between linear and binary searches are they both are used to find a certain number in a list.”. The student accurately answers that the linear search is sequential and traverses the whole list by stating “Linear search goes one by one through the list and checks each number to see if it is the number that is being searched for.” (Criteria 5 and 6). The student accurately answers that linear search is for unsorted lists and that binary search is for sorted lists by stating “The data can be in any order . . . ” (Criterion 11) and “Binary search on the other hand must have all the numbers placed in numeric order. . . ” (Criterion 10). The student accurately states that linear search is better for smaller lists (Criterion 9) and binary search is more efficient for larger lists by answering “One problem with linear [search] is for larger data sets the computing power would be too great to reasonably use it to search.” and “...but with larger datasets it requires much less computing power” (Criterion 8). Figure 7 contains an example of inaccurate information in a student’s answer related to the binary search algorithm. This student inaccurately states that the binary search algorithm involves a sorting algorithm as part of binary search process as opposed to a sorted list being a requirement for utilizing binary search by writing “However for a binary search the program will sort the array or list into different parts and then search for the item being searched for...”. The student inaccurately meets Criterion 3 of the Student Answer Completeness Criteria by including a search algorithm in the divide and conquer methodology of the binary search algorithm. In Figure 5b, we see that student answer accuracy is overall high among the three tools. Students using Aida produced slightly more accurate answers. Aida’s median accuracy was 1.0 compared to 0.86 for ChatGPT and 0.83 for Google. Our ANOVA analysis (𝑝 = 0.368, 𝑝 > 0.1) showed no significant difference in student answer accuracy across both tasks in the three conditions as shown in Table 3. Student Answer Extraneousness. We consider a student’s answer to contain “extraneous” information if the student provides extra information in their answer that is unrelated to the task being explored, as compared to the student 15

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

Binary and linear searches are both algorithms used to search for a specific item in an array or list. A linear search is when the program will go through each item in the array or list until it finds what the program is looking for. However for a binary search the program will sort the array or list into different parts and then search for the item being searched for, this is much faster than a linear search because it doesn't go through every single item in the list or array.To choose between the 2 searches you would need to know how big the list or array is, how much data is going to be searched through,how much time you have, and what the program is searching for.

Fig. 7. Example of Student Answer containing inaccurate information for the Binary/Linear Search Task.

answer completeness criteria for the task being explored. Supporting information that justify and demonstrate the student’s comprehension of the task being explored, such as examples, are considered to be relevant to the student answer completeness criteria. The researcher assigned a score to each student’s answer representing the ratio of the total count of extraneous sentences present in their answer out of the total number of sentences in the student’s answer. Each student answer received a score, computed as: 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜 𝑓 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒𝑠 𝐵𝑒𝑦𝑜𝑛𝑑 𝐴𝑛𝑠𝑤𝑒𝑟𝑖𝑛𝑔 𝑡ℎ𝑒 𝑇 𝑎𝑠𝑘 𝑃𝑟𝑜𝑚𝑝𝑡 𝑖𝑛 𝑡ℎ𝑒 𝑆𝑡𝑢𝑑𝑒𝑛𝑡 𝐴𝑛𝑠𝑤𝑒𝑟 𝑇𝑜𝑡𝑎𝑙 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜 𝑓 𝑆𝑒𝑛𝑡𝑒𝑛𝑐𝑒𝑠 𝑖𝑛 𝑡ℎ𝑒 𝑆𝑡𝑢𝑑𝑒𝑛𝑡 𝐴𝑛𝑠𝑤𝑒𝑟 Of the extraneous information in the student’s answer, the extraneous text could be reflection, copy-paste facts from 𝐸𝑥𝑡𝑟𝑎𝑛𝑒𝑜𝑢𝑠𝑛𝑒𝑠𝑠 =

the tool’s response, or paraphrased facts. To gain more insight, we also characterize the text in the student’s answer beyond the text that semantically achieves the completeness criteria. We count the number of student answers that contain each kind of text and then compute the proportion of extraneous sentences in each session of each kind. In Figure 6, the student’s answer meets 8/13 criteria for completeness and 8/8 in accuracy, however, the student provides 1/5 extraneous sentences. Along with the student answering that linear search can be performed on unsorted lists, the student provides the first extraneous information item on the difficulty of inserting items into an unsorted list by stating “...and that makes it easier to add more values to the list” (0.5 sentence). Additionally, along with the student answering that binary search must be performed on sorted lists, the student provides the second extraneous information item on the difficulty of inserting items into a sorted list by stating “...which does make it harder to add new values” (0.5 sentence). We consider both items of information unrelated to the task being explored because the task is to compare and contrast binary search and linear search algorithms, not to compare and contrast inserting items into unsorted and sorted lists. Of extraneous sentences in the student’s answer in Figure 6, the sentences are categorized as 1 “Paraphrased Facts”. The student describes the difficulty of inserting items into an unsorted list in their own paraphrased words by answering “...and that makes it easier to add more values to the list”. The student next describes the difficulty of inserting items into an sorted list in their own paraphrased words by answering “...which does make it harder to add new values”. Both extraneous sentences in this student answer are in the “Paraphrased Facts” category. Therefore, the student receives 1 “Paraphrased Facts” for the presence of this kind of text in their answer. In Figure 5c, we see that participants using Aida included the fewest extraneous sentences in their answers. The median extraneousness using Aida was 0.0 and mean was 0.17. The answers of students using ChatGPT had a median of 0.1 and mean of 0.19, while students using Google had an even higher extraneousness median of 0.2 and mean of 0.27. Table 3’s results of our ANOVA analysis (𝑝 = 0.324, 𝑝 > 0.1) showed no significant difference between the means of the three conditions across both tasks in student answer extraneousness. When we examined the type of extraneous sentences, we observed that using Google resulted in a very high relative proportion of copy/paste extraneous sentences. 16

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY 10 9 8 7 6 5 4 3 2 1 0

25 20 15 10 5 0 Aida

ChatGPT

Google

Aida

(a) Number of Presented Examples

ChatGPT

Google

(b) Amount of Out of Scope Information

Fig. 8. Distribution of the Number of Presented Examples (left) and Computer Science Principles Exclusion Statement Occurrences per Session (right). The results are aggregated across both tasks for each tool. All participants utilized Aida for one task, and either ChatGPT or Google for one task (Aida N=45, ChatGPT N=22, Google N=23).

In addition, Google student answers exhibited slightly fewer than the others’ sentences that contained reflection and paraphrased facts. 4.1.2 Effectiveness of Providing Examples to Students. In addition to the answers that students provided as the outcome of their learning, we examined the effectiveness of each tool during the actual learning journey. Learning occurs through the presentation of real-life examples to the user [36, 45, 95], so we examine the number of examples provided by the agent. Providing examples alongside the explanation of concepts bolsters and reinforces learning through demonstration of key characteristics associated with the concept. We show the number of examples provided by each tool per student session in Figure 8a. The results show that Aida is consistently showing at least one example to students in each session (median of 1, mean of 1.6). ChatGPT and Google, similar to each other, show examples slightly less often and less consistently. In addition, Aida presented examples in 73.33% of total sessions while ChatGPT and Google presented in 43.48% and 45.45% of total sessions, respectively. These percentages represent specifically the proportion of sessions in which each tool provided relevant, illustrative examples in response to students’ inquiries. However, our ANOVA test (𝑝 = 0.604, 𝑝 > 0.1) in Table 3 showed no significant difference between the means of the three conditions across both tasks in presented examples to students. 4.1.3 Effectiveness of Customizing to the CSP Course Scope. As a proxy for the effectiveness of customizing the user’s exploration to the CSP course scope, we further analyzed the student’s learning journey by measuring the appropriateness of the agent’s responses in relation to information intentionally excluded from the CSP course curriculum. The AP College Board CSP Framework defines exclusion content that does not need to be included in the course via specific “exclusion statements”. The content described in the exclusion statements should not be covered by CSP and will not be assessed on the AP Computer Science Principles Exam [1], e.g., see Table 5. When exploring a task with a conversational agent, we considered the agent’s responses to be “out of scope” if the agent reveals information that is within CSP exclusion statements. When exploring a task with web search, we considered the information retrieved from web sources to be “out of scope” if the web page visited or search query contains information that is within the AP College Board’s exclusion statements for the task being explored. We counted the number of “out 17

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

of scope” information in each session of each tool in Figure 8b. It shows that ChatGPT usually generates the most sentences on topics in the exclusion statements per student session. While Google and Aida median number of “out of scope” instances is low, Google has a few student sessions with very high occurrence of exclusion statement content. We conducted an ANOVA comparing the students in the three conditions regarding the “out of scope” information presented during their sessions. Our ANOVA analysis (i.e., Table 3) indicates that students who interacted with Aida were exposed to significantly less “out of scope” information compared to ChatGPT (𝑀𝐴𝑖𝑑𝑎 = 0, 𝑀𝐶ℎ𝑎𝑡𝐺𝑃𝑇 = 1.59, 𝑡 = 4.297, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = 1.612, 𝑝 < 0.025) having a large effect size. A subsequent comparison in our ANOVA analysis in Table 3 shows that students were exposed to significantly more “out of scope” information with Google compared to ChatGPT (𝑀𝐺𝑜𝑜𝑔𝑙𝑒 = 2.39, 𝑀𝐶ℎ𝑎𝑡𝐺𝑃𝑇 = 1.59, 𝑡 = 0.582, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = 0.17, 𝑝 < 0.025) having a small effect size. Table 5. Advanced Placement College Board Computer Science Principles Exclusion Statements. Binary Search Linear Search Algorithmic Efficiency

(EK AAP-2.P.1) Specific implementations of the binary search are outside the scope of the course and the AP Exam. (EK AAP-2.0.1) Traversing multiple lists at the same time using the same index for both (parallel traversals) is outside the scope of this course and the AP Exam. (EK AAP-4.A.3) Formal analysis of algorithms (Big-O) and formal reasoning using mathematical formulas are outside the scope of this course and the AP Exam. (EK AAP-4.A.9) Specific heuristic solutions are outside the scope of this course and the AP Exam.

Figure 9 shows excerpts of student interactions with the tools. We designed Aida’s fixed-responses to not include any “out of scope” information. There are 3 “out of scope” information in ChatGPT’s responses during this student’s exploration of the task. Not all interactions are shown, however, Figure 9b shows 2 of the 3 “out of scope” information included in ChatGPT’s response. ChatGPT describes the binary search and linear search time complexity in Big O notation which violates the College Board Essential Knowledge Exclusion Statement EK AAP-4.A.3 as shown in Table 5 when replying “...Because binary search reduces the search space by half at each step, it has a time complexity of O(log n), which is much faster than linear search, which has a time complexity of O(n).”. Figure 9c shows a student’s Google search history including the search queries and websites visited when exploring the task. There are 20 “out of scope” information on the websites visited during the student’s exploration. While the websites that the student visited do include visual diagrams of both search algorithms, the websites also contain several coding implementations of binary search and linear search in iterative and recursive techniques in several programming languages, as well as Big O Notation for time complexity and space complexity for the search algorithms. These details are “out of scope” and violate the College Board Essential Knowledge Exclusion Statements EK AAP-2.P.1 and EK AAP-4.A.3 as shown in Table 5. 4.2

User Engagement

To evaluate our second hypothesis (H2) that “Generative conversational agents lead to better perceived student experiences despite being less engaging than fixed-response conversational agents”, we analyzed the user engagement metrics alongside the participant post-study survey perspectives of the conversational agents. We also report a comparison to Google search for a comprehensive analysis to enhance our discussion in the subsequent section. We analyzed each student’s engagement with the tools, where engagement is defined as the extent to which a student is involved in the exploratory learning experience with the tool. We measured engagement through two metrics computed from their conversation 18

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Human

Aida Thanks Participant 43! How can I help you learn about building APIs or libraries? I can give you some examples, teach you what each is, explain the process, as well as offer advice on when to use it and when you shouldn't.

10:47 AM

12:31 PM

Hello! How can I assist you today?

What is an API?

12:33 PM

12:37 PM

10:49 AM

12:40 PM

A software library is like a pencil box. Just as a pencil box contains different pencils, pens, and erasers for different tasks, a software library contains pre-written code for different programming tasks. Each tool in a pencil box is like a function in a software library. Just as each tool has a specific purpose, each function in a software library performs a specific task. Whether you should build a library of functions depends on a variety of factors. Let's explore them together. Can you think of one characteristic that makes a good library function? Here is a hint of one quality. Imagine if a library showed the incorrect calculation or result.

(a) Aida Conversation Excerpt

12:38 PM

What is a search algorithm?

A search algorithm is a technique used to find a specific item or piece of information in a collection of data. These collections of data can be anything from arrays and linked lists to databases and search engines. The purpose of a search algorithm is to locate the target item as quickly and efficiently as possible. There are many different types of search algorithms, each with their own strengths and weaknesses depending on the size and structure of the data, and the type of item being searched for. Common search algorithms include linear search, binary search, hash tables, and tree-based searches such as depth-first search and breadth-first search. The choice of search algorithm depends on the specific requirements of the task at hand, as well as the size and structure of the data being searched.

10:59 AM

12:34 PM

How can you answer my questions?

I am an AI language model developed by OpenAI, called GPT-3. I can answer questions using natural language processing and machine learning algorithms to understand the context and intent of your questions. My responses are generated based on patterns and information present in large amounts of data that I have been trained on.

You could provide them with all the necessary information, and you would need to share how to use the library and what the code means, right?

12:32 PM

What is binary search?

Binary search is a searching algorithm used to find a specific element in a sorted collection of elements. It works by dividing the collection into two halves and checking if the target element is in the first or second half. If the target element is in the first half, the algorithm continues searching in the first half; otherwise, it continues in the second half. This process is repeated until the target element is found or the algorithm determines that the element is not in the collection. Because binary search reduces the search space by half at each step, it has a time complexity of O(log n), which is much faster than linear search, which has a time complexity of O(n).

10:47 AM

That's a good question. I'm not sure exactly how you would do it, but I would start by pulling and storing the required information from the library.

10:59 AM

ChatGPT Hello

10:44 AM

Application Program Interfaces (APIs) serves as a guide for how to communicate with a library. How can you be sure that the users of your library have all they need to use your functions?

10:49 AM

Human

12:41 PM

(b) ChatGPT Conversation Excerpt

Human

Google

8:38 AM

how to binary search

8:39 AM

https://www.geeksforgeeks.org/binary-search/

8:41 AM

what is linear search

8:41 AM

https://www.tutorialspoint.com/data_structures_algorithms/linear_search_algorithm.htm

8:42 AM

compare and contrast linear and binary searches

8:42 AM

https://www.javatpoint.com/ds-linear-search-vs-binary-search

(c) Google Search History Excerpt Fig. 9. Example student sessions: Aida conversation, ChatGPT conversation, and Google search history.

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

logs. First, we analyze the number of their exploratory actions with each tool. Second, we assess their time spent on each task. We define the number of student exploratory actions as the sum of all attempts that a student takes to continue achieving the task being explored. In sessions where the task is explored with a conversational agent, we consider the student’s “exploratory action” as the total number of messages that the student sends the conversational agent related to the task being explored. In sessions where the task is being explored with a conventional web search engine, we consider the student’s “exploratory action” as the total number of websites visited during their session (counting both search result and web content pages). Each student’s session received a score representing the count of the total number of exploratory actions for each task being explored. We define the task duration as the amount of time that a student interacts with one of the tools to perform one of the exploratory search tasks. We measure the task duration as the difference between their last recorded interaction time and their first interaction time with a tool. In sessions where a task is being performed with a conversational agent, each speaker’s message receives a timestamp at the time that the message is sent. In sessions where a task is being performed with Google, the web browser’s history records the time when a web page is visited. Each student’s task duration was represented as the total number of minutes the student explored for a specific task. For instance, while not all the interactions are shown for space, there are 9 student interactions related to exploring the task with Aida as shown in Figure 9a. The student either asks a question related to the task being explored (e.g., “What is an API?”) or responds to Aida’s question (e.g., “That’s a good question. I’m not sure exactly how you would do it, but I would start by pulling and storing the required information from the library.” ). The session duration for this conversation is 15 minutes, the difference between the last recorded interaction time in the conversation, 10:59 AM, and the first recorded interaction time, 10:44 AM. Figure 9b shows excerpts of student interactions with ChatGPT. There are 4 student interactions with ChatGPT related to the task, i.e., student interactions such as “What is binary search?” and “What is a search algorithm?” are related to completing the task and counted as exploratory actions. Conversely, student interactions such as “Hello” and “How can you answer my questions” are unrelated to the task being explored and are not counted towards this student’s exploratory actions. The session duration for this conversation is 10 minutes (12:31 PM - 12:41 PM). There are 6 student interactions with Google shown in Figure 9c. The student either searches Google for the task being explored (e.g., “how to binary search”) or visits a web page for information on the task being explored (e.g., “https://www.geeksforgeeks.org/binary-search/”). The session duration is 4 minutes (8:38 AM - 8:42 AM). Figure 10 presents the results for the number of student exploratory actions and task duration per session over both tasks for all students. We see that Aida’s engagement was higher than ChatGPT and Google. Students interacted with Aida more (median of 10 exploratory actions) compared to ChatGPT and Google (medians of 4 and 6 exploratory actions, respectively) and spent more time using the tool (median of 12 minutes). On the other hand, ChatGPT generated the fewest number of actions, by a wide margin, and the shortest task duration. As indicated in Table 3, results of our ANOVA analysis comparing the students in the three conditions regarding the task duration and number of student exploratory actions shows that students who interacted with Google completed their tasks significantly in less time than compared to ChatGPT (𝑀𝐺𝑜𝑜𝑔𝑙𝑒 = 5.478, 𝑀𝐶ℎ𝑎𝑡𝐺𝑃𝑇 = 5.773, 𝑡 = −0.254, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = −0.076, 𝑝 < 0.001) having a small effect size. Regarding the number of student exploratory actions, we found that students interactions with Aida produced significantly more exploratory actions compared to Google (𝑀𝐴𝑖𝑑𝑎 = 10.556, 𝑀𝐺𝑜𝑜𝑔𝑙𝑒 = 6.783, 𝑡 = 3.663, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = 0.827, 𝑝 < 0.001) and student interactions with Google produced significantly more exploratory actions compared to ChatGPT (𝑀𝐺𝑜𝑜𝑔𝑙𝑒 = 6.783, 𝑀𝐶ℎ𝑎𝑡𝐺𝑃𝑇 = 2.909, 𝑡 = 5.012, 𝐶𝑜ℎ𝑒𝑛 ′𝑠 𝑑𝑠 = 1.472, 𝑝 < 0.001), both having a large effect sizes (i.e., Table 3). 20

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY 25

20 18 16 14 12 10 8 6 4 2 0

20 15 10 5 0 Aida

ChatGPT

Google

Aida

(a) Number of Student Exploratory Actions

ChatGPT

Google

(b) Task Duration

Fig. 10. Results regarding measures of user engagement including the Number of Student Exploratory Actions (left) and Task Duration (right). The results are aggregated across both tasks for each tool. All participants utilized Aida for one task, and either ChatGPT or Google for one task (Aida N=45, ChatGPT N=22, Google N=23).

4.3

Student Perspectives

After all students completed both tasks, each student answered a survey to obtain their perceptions of the tools in terms of effectiveness and the interactiveness of the two kinds of conversational agents. Each participant used only two of the exploratory search methods. We conducted an Wilcoxon Rank-Sum Test comparing the students’ ordinal Likert scale responses of the two conversational agents across both tasks for all post-survey metrics of effectiveness and interactiveness as shown in Table 6. Table 6. Overview of results (mean and standard deviation (SD)) of post-study survey on conversational agent preference regarding effectiveness (i.e., providing useful information, helpfulness, using it in the future) and engagement (i.e., understandability, understanding the user, behaving awkwardly, number of questions asked) across both tasks.

Tool

Aida

ChatGPT

Significance (Cohen’s ds, t-value)

Useful Information mean (SD) 3.53 (.87) 4.41 (0.8) p <0.005 (-1.35, -4.10)

Effectiveness Helped Me

Future Use

mean (SD) 3.31 (1.08) 4.59 (0.59) p <0.001 (-1.34, -6.25)

mean (SD) 2.51 (1.16) 4.50 (0.6) p <0.001 (-1.96, -9.26)

Understand Tool mean (SD) 3.36 (1.03) 4.32 (0.78) p <0.01 (-1.01, -4.26)

Interactiveness Understand Behaved Me Awkwardly mean mean (SD) (SD) 2.71 3.09 (1.06) (1.24) 4.36 2.45 (0.66) (0.91) p <0.001 ns (-1.74, -7.83)

Too Many Questions mean (SD) 3.58 (1.20) 1.32 (0.57) p <0.001 (2.18, 10.48)

4.3.1 Quantitative Insights. Figure 11 presents the results of survey questions about the two conversational agents’ (Aida and ChatGPT) interactions and effectiveness. Student perspectives on ChatGPT are considerably more favorable than Aida for all of these questions: understanding the user, understandability, behaving awkwardly, number of questions asked, providing useful information, helpfulness, and using it in the future. There is a clear participant preference towards ChatGPT based on these results. There was a significant difference among students for each survey question with regard to preference of ChatGPT over Aida (𝑝 < 0.01) as shown in Table 6 with the exception of student’s 21

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

Effectiveness

Interactiveness

The conversational agent was able to find useful information. (Useful Information)

I could understand the conversational agent well. (Understand Tool)

Aida

53%

Aida

20%

ChatGPT

91%

ChatGPT

-100%I feel like-50% 0% the conversational agent50% helped me. 100%

49% 90%

-100%

(Helped Me) Aida

20%

44%

Aida

42%

ChatGPT

95%

ChatGPT

-100%

Aida

38%

ChatGPT

-100%

-50%

50%

I would use the conversational agent in the future. (Future Use)

-50%

50%

100%

Aida

38%

96%

ChatGPT

55%

100%

Strongly Agree

Agree

Neutral

50%

100%

90%

-50% 0% 50% The conversational agent behaved awakwardly. (Behaved Awkwardly)

100%

40% 14%

-100% The conversational -50% 0% agent asked50% too many

100%

questions. (Too Many Questions)

Percentage

Response

22%

-100%

20%

-50%

The conversational agent could understand me well. (Understand Me)

Disagree

Strongly Disagree

Aida

24%

ChatGPT

96%

56% 0%

-100%

-50%

50%

100%

Percentage

Fig. 11. Students’ Perception of Conversational Agent Interactions. All participants utilized Aida for one task, and either ChatGPT or Google for one task (Aida N=45, ChatGPT N=22, Google N=23).

perceptions of the conversational agent that behaved awkwardly, each having large effect sizes. There was no significant difference between the conversational agent in terms of displaying awkward or non-humanlike behavior (𝑝 > 0.2). The survey results from questions comparing the tools are shown in Table 7. It is notable that for each question, each tool was preferred by at least some subset of the participants. ChatGPT was chosen by the highest percentage of participants as enabling them to understand more based on their personal experience, making it easier to complete the task, learning more from the tool, and providing more enjoyment than Aida or Google Search. Aida, the fixed-response conversational agent was chosen the lowest percentage of the time for all these characteristics. Table 7. Overview of students’ comparisons on their preferences of tools regarding the method that enabled understanding more based on personal experience, the method that made it easier to complete the task, the method that they learned more from, and the method that was more enjoyable to use. All participants utilized Aida for one task, and either ChatGPT or Google for one task (Aida N=45, ChatGPT N=22, Google N=23). Tool Aida ChatGPT Google No Preference

Personal Experience 20% 29% 24% 27%

Easiest To Use 16% 44% 36% 4%

Learn More 16% 40% 31% 13%

Enjoy More 13% 42% 29% 16%

4.3.2 Thematic Analysis. The researcher of this paper used a reflexive process of thematic analysis, as described by Braun and Clarke [22], to analyze the open-ended question responses in our post-study survey. The reflexive analysis process is a theoretically flexible method to developing, analyzing, and interpreting patterns in qualitative data. Reflexivity involves drawing upon your experiences, pre-existing knowledge, and social position and critically interrogating how these aspects influence and contribute to the research process and potential insights into the data [37]. Given that the qualitative data set in our study was three open-ended questions per student for each tool (i.e., Aida, 22

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

ChatGPT, Google), the researcher labeled parts of the student responses that captured important features of the data. The process involved organizing the open-ended responses by question and systemically coding each response by question. Lastly, the labels were grouped into broader themes for reporting and discussion. Qualitative Insights. We asked students to describe what they liked and disliked about their experience (i.e., effectiveness and interactivity) completing the task, as well as to provide any other additional comments on each tool. The students’ open-ended responses indicate their positive perception of the effectiveness of conversational agents, and do not support positive perception of the effectiveness of conventional web search. Here, Px refers to Participant x. Participants responded positively to Aida’s responses and particularly enjoyed the examples that Aida provides. P60 writes “It[Aida] felt like a conversation and it was smooth, it gave me answers and connected the answers to real world things that I could relate to which was helpful.”). Students also expressed positive perceptions of ChatGPT responses’ effectiveness. P79 writes “I liked how ChatGPT would give me what I needed while give even more info which let me ask more and better questions”. The most common response from students related to Google’s effectiveness was that the information retrieved from Google was too difficult to understand. For instance, P90 writes “There was no easy way to condense the information into small, readable chunks. Also, there was no way to get the websites to explain the concept in a way I might understand better, just the one explanation they have for everyone.” Further evidence in the open-ended responses shows that students were skeptical of the credibility of websites from their Google search (i.e., P64 writes “One thing I think I disliked was that, since there were so many sources all right next to each other, it was sometimes troublesome to decide if one was more correct/legitimate than the other.”). However, compared to the credibility of the information retrieved from web search, students’ perceptions were that the information retrieved from both conversational agents was credible. P78 writes “I liked that Aida was specifically for programming and seemed to only have information about what I was searching for. It made it so everything was concise and I knew that whatever I was searching for would have an answer.” No students commented on not trusting the credibility of ChatGPT with the exception of P43 who observed that ChatGPT hallucinated. The students’ open-ended responses support the perception of the interactions with conversational agents and conventional web search provided in the closed question results presented earlier. The responses about Aida’s interactions were mixed and polarized, where the primary theme revolved around the pedagogical design of Aida compared to ChatGPT, as well as Aida’s intent matching. Specifically, some students really enjoyed that Aida guided them to arriving at their own answers without divulging the answer while others did not enjoy the frequency of Aida’s questions. P75 dislikes Aida’s interactions writing “I disliked how linear it[Aida] felt, like it was searching for an exact answer and if it didn’t receive it, it would just throw the question back to me with a limited amount of change. It made getting information hard, as I would answer to the best of my ability and be roadblocked. When I tried to ask it other questions related to the subject, it again railroaded me onto the track of questions I had first received. In conversation, a real human wouldn’t just keep saying the exact same thing back to someone when they are trying to answer questions.”. Responses regarding the infrequency of Aida matching the appropriate DialogFlow intents include P66 who writes “I feel like Aida didn’t always understand what I was saying and there was some confusion”. Nearly all students interactions were positive with ChatGPT. While students did largely enjoy the responses from ChatGPT, two students commented that ChatGPT required additional prompt engineering to appropriately gather the information that they were seeking. For instance, P89 writes “i had to be very specific to find what i was looking for”. Those students who enjoyed the interaction with Aida commented vastly different. P58 writes “I think that this[Aida] is a great educational tool, and that it is almost like an online teacher. The answers it gave were also easy to understand 23

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

and comprehend.” P78 enjoyed Aida’s Socractic questioning writing “I also like that it[Aida] was friendly and would sometimes ask questions that test your understanding and make you think. It made it so I could contribute and made me comfortable in knowing that it was providing helpful, relevant information.”. P64 writes “I liked that fact that, unlike when I was using the Google search browser, AIDA was more interactive than of course the search browser did. Being able to drive a discussion about what I was researching made things easier in some cases.”. Nearly all students enjoyed Aida’s speed of responses (i.e., fixed-response intent matching) compared to the speed of ChatGPT’s generative responses. Regarding ChatGPT, P69 writes “It[ChatGPT] took up to 30 seconds to respond”. On the other hand, regarding Aida, P59 writes “The responses were very quick and almost instantaneous”. Not surprisingly, students’ feedback supports their positive perception of the interactions with conventional web search, yet interestingly students voiced concerns with the struggles of information literacy in web search. Many students acknowledge that they are already familiar with using Google and, therefore, they found it easy to use (i.e., P54 writes “I liked using google because I am more familiar with how to ask an efficient question.”). While nearly all students enjoyed the breadth of sources that conventional web search provides, the variety of sources was a double-edged sword that made it difficult to find useful information and the search process overwhelming for students (i.e., P86 writes “The amount of sources for the task were few and far between.”). P70 expressed a considerable amount of query reformulation in their Google queries by writing “I disliked that it was sometimes difficult to get Google to show me the answer I was looking for. Sometimes I had to try to reword my question multiple times to get the information I was looking for.”. 5

DISCUSSION

This study aimed to investigate two research hypotheses: (H1) Conversational agents provide a better learning experiences than conventional web search for secondary school CSP students performing exploratory search, and (H2) Generative conversational agents lead to better perceived student experiences despite being less engaging than fixedresponse conversational agents. The effectiveness and engagement results as well as qualitative analysis of our study support both hypotheses. Our study reveals a few salient points related to the use of conversational agents in CSP, as the computing education field copes with the rapidly evolving technology that generative AI provides. We present a discussion of opportunities and challenges in leveraging the benefits of conversational agents while mitigating risks. 5.1

Opportunities In Using Conversational Agents For CSP Exploratory Search

First, conversational agents have the opportunity to act as a single knowledge source (i.e., personalized tutor) for students who struggle with their conventional information gathering experience. We observed that ChatGPT performed well on the effectiveness measures, i.e., it produced the most complete answers, and was by far the most liked tool by the students. Clearly, this supports our first hypothesis (H1) and indicates that LLMs have a lot of potential for use in student exploratory search for information related to CSP. However, ChatGPT was outperformed by Aida in terms of the higher accuracy and lower extraneousness of the answers. This indicates that Aida’s custom responses lead to high quality of learned information by the students. Nonetheless, Aida’s rigid conversational structure was generally disliked by the students and supports our second hypothesis (H2). Relative to both of the conversational agents, Google performed poorly across the board in effectiveness as well as in student perceptions. The learning tasks in our study were intentionally designed to span across multiple searches and require exercising information literacy skills such as locating, aggregating, organizing, and communicating acquired knowledge. In their experiences with Google search, students looked for a single web source that consolidated the information to complete the task. 24

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Second, conversational agents present an opportunity to leverage the existing trust that students have in them and provide an enhanced learning experience for students. Students exhibited high levels of trust in conversational agents compared to information retrieved from conventional web search. Unlike Aida, there is a potential for ChatGPT to hallucinate. Although hallucination is far from a solved problem, given how powerful AI models are becoming and the rapid pace at which new models are being released, we anticipate that the amount of hallucination will become less of a problem in the near future. Only one student questioned the credibility of ChatGPT. Comparatively, students questioned the information retrieved from websites in their Google search exploration. Beyond our study, CS educators are increasingly worried that their students will become over-reliant on LLMs for automated responses [91]. Notably, although this was not the case that we observed in our study, CS educators fear “blind trust” where students do not question LLMs responses given how advanced their natural language capabilities are. Additionally, the discrepancy in students’ training to evaluate information from web search results versus conversational agents like ChatGPT is a significant factor in understanding their trust in these tools. While students are often taught to critically assess the credibility of web search results, similar instruction for conversational agents is generally lacking. This educational gap may contribute to the higher level of trust observed in responses generated by ChatGPT, despite the potential for inaccuracies. As conversational agents become more integrated into educational settings, it is crucial to provide students with explicit instruction on how to critically evaluate the information produced by these tools, just as they would with traditional web search results. Addressing this gap will better equip students to navigate the complexities of information retrieval across different platforms and foster more informed use of AI-driven technologies. Third, conversational agents have the opportunity to broaden participation in CS. Students reported that all exploratory learning methods enabled them to learn based on their personal experience, relatively equally. This indicates that students of all backgrounds can utilize learning with conversational agents. Further, there is an opportunity for conversational agents to reduce the workload of instructors by providing personalized help to students who are struggling and who would otherwise consume a considerable amount of instructor time [91]. Given a more personalized experience, the inherent biases of LLMs should be considered when broadening participation (e.g., the impact on the model’s behavior to differing ethnic student names when added to a prompt) [91]. We also observed that ChatGPT did not ask any clarifying questions to students during any of the sessions. However, ChatGPT has the ability to ask clarifying questions, which could be an opportunity for the agent to gather information about its audience in an effort to better tailor its responses. The lack of clarifying questions is more akin to a conventional information retrieval or recommendation system as opposed to a conversational agent, which contradicts its name, “ChatGPT”. Incorporating timely and effective Socratic questioning into ChatGPT through clarifying questions could better align LLMs to the CSP pedagogical framework. In contrast, Aida asked clarifying questions based on the defined intents. 5.2

Challenges In Using Conversational Agents For CSP Exploratory Search

Despite ChatGPT’s high effectiveness for task completion, using LLMs for exploratory search poses several challenges as a learning tool. The first challenge is that LLMs do not scope their responses to their audience because they are a general-purpose tool for broad audiences. Unbeknownst to the students in our study, ChatGPT provided the highest count of sessions with “out of scope” responses (most consistent), which were deemed inappropriate to include in CSP according to the College Board’s Curriculum Framework, relative to the other tools. Google provided the largest average amount of “out of scope” responses, however, the mean was driven by outlier sessions with high quantities of 25

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

“out of scope” responses. Further, students reported that when using Google search, they largely did not understand the information retrieved which is indicative of “out of scope” knowledge. Without guardrails or additional customization, the information retrieved from generative conversational agents have the potential to become too difficult for secondary school students to understand as well. To this end, in the same manner that students should learn how to effectively use Google search, students using LLMs for exploratory search should be instructed in prompt engineering, the process of effectively structuring text so that it can be interpreted by an AI model. Comparatively, at the expense of being less human-like, Aida structures the dialogue to explore topics sequentially. Another challenge relates to convenience in completing the task. Students favor convenience in their learning journey and largely agree that ChatGPT is the easiest and shortest duration method to complete the task. It can be perceived that speed and ease of use are benefits of LLMs. However, CS educators believe LLMs pose a risk of creating “lazy learners” in introductory CS educational settings (i.e., Do students actually learn more using generative conversational agents or do students agree that their learning is “better” because it is easier?) [91]. There are advantages to a high degree of natural language processing and natural language understanding such as decreased task duration. However, there are also tradeoffs, such as a decreased number of exploratory actions. Expert interviews indicate that CS educators believe students should learn the core concepts in introductory CS courses prior to expanding learning to leverage generative AI in higher level CS courses [91]. Especially considering that students may be required to use LLMs in professional settings, yet LLMs will not be available during job interviews, students should have some level of exposure in higher level course settings to prepare them for a successful transition into the workforce. Students in our study agree that utilizing Google to complete the task was easy; however, their opinions were influenced by having prior familiarity with already using the tool. Students noted that Aida was the most challenging to use (i.e., the least convenient). Interestingly, the Think-Pair-Share pedagogical approach is already implemented within in the CSP course lectures and activities, however, the students did not prefer it when it came to their own individual information gathering. With more exposure to gaining knowledge using the Think-Pair-Share pattern through conversational agents, there is a possibility students will adapt and change perceptions in future iterations of the tool whilst mitigating the challenge of creating “lazy learners”. The next challenge is the ethical implications of utilizing LLMs in an educational context. In our study, we observed that in some cases students entered the full task prompt or large portions of the task prompt directly into both conversational agents. This level of effectiveness has risks hindering the efficacy of using LLMs as a learning tool and imposes ethical concerns on the outcomes of student’s tasks. This is evidenced by students presenting a model’s answers directly as their own. Nothing prohibits LLMs from completing the task directly for the student as opposed to supporting the student to explore completing the task. Research shows that presenting generative AI’s knowledge as your own ideas is not plagiarism, yet it constitutes a form of falsification of the student’s ideas [91]. Further, many schools have not yet produced well-formed academic policies on LLMs. As a result, schools simply prohibit use of LLMs for learning, which poses risk of students violating unauthorized resource usage policies since CS educators believe that students will use LLMs anyway [91]. While in both Aida and Google, we also observed that students similarly directly reported parts of the information retrieved as their own. However, there were no intents in Aida to match entering the full task prompt in order to generate a full succinct answer. Similarly, only a few students responded that they were able to eventually navigate to a single website that was able to provide enough information they needed to complete task; the majority of students were unsuccessful at finding a single website with a full succinct answer.

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

THREATS TO VALIDITY

Results may vary with different tools representing fixed-response and generative conversational agents and search engines. In particular, the high number of “out of scope” responses from the generative agent could correspond to the choice of model. At the time of the study (May 2023), GPT-4 [82], whose training data is significantly greater than GPT-3.5, was not yet publicly available. We mitigated this risk by using the largest GPT conversational model publicly available at the time of the study. Further, the lack of clarifying questions can be fine-tuned through prompt engineering. We mitigate this risk by allowing students to provide any input into the model. A fixed-response dialog structure inherently imposes a constraint to multi-turn exploratory conversations, which will, by design, appear less engaging and less human-like compared to generative conversational agents. Due to the potentially wide variety of intents that the intended users may have, there is always a possibility a user may ask a question that is not supported by the fixed-response conversational agent. To mitigate this, each Aida model was trained on a variety of diverse phrases for each intention, and each intent contained a variety of fixed responses to reduce the likelihood that a student would receive the same response. We also piloted Aida’s interactions, the study procedure, and survey on several non-authors and made improvements. Recognizing the spectrum of AI design possibilities, we focused on the specific configurations of Aida and ChatGPT that reflect deliberate design choices by OpenAI for ChatGPT and the pedagogy used by Code.org for Aida. Although alternative designs could have led to different outcomes, we also found the default configurations compelling based on our own experimentation. Future work could explore varying system prompts, customization options, and student interaction patterns to further understand the impact of AI design choices on learning experiences. We acknowledge that the number of examples presented to a learner is only a proxy for the effectiveness of helping users learn. We cannot guarantee that the user read or understood the examples presented. Yet, Denny et al. [36] proposes that exemplar solutions from generative AI in computing education has the potential opportunity to benefit learners by saving time from instructors on generating examples [35] and helping learners appreciate the trade-offs of different solutions approaches [56]. To mitigate against threats to construct validity, we constructed our exploratory tasks based on Athukorala et al. which lend themselves to exploratory search [14]. The tasks were chosen because they are more complex topics in the CSP curriculum that motivate exploratory learning. Internal threats due to possible task/tool use ordering bias were minimized by assigning the same tasks to different participants in different orders and changing the tool use order. Interpretation bias in the qualitative analysis of Student Answer Completeness Criteria was minimized by multiple experts creating the criteria, yet future studies could involve multiple coders or a consensus-based approach to enhance reliability of the open-ended survey. The main external threat is generalizability beyond our participant set. We recruited students from various CSP sections in three high school CSP courses across grades 9-11 (ages 14-17). Replication to broader demographics is encouraged. We focused on exploratory search where more query reformulation is common; our results may not generalize to other kinds of search. Students may have had previously-held, strong opinions of the exploratory tools Google and ChatGPT. Google is a well-known web search engine that students likely have experience using. ChatGPT is, at the time of writing, one of the most highly publicized and widely recognized AI models available. Its significant media coverage, coupled with the broad popularity of Google’s products, may have influenced and potentially skewed students’ perceptions of its effectiveness in assisting with CSP tasks. This increased visibility and prominence of ChatGPT and Google could lead to an inherent bias in evaluating its utility relative to other tools, such as Aida. It is crucial to account for this 27

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

potential bias, as the overwhelming visibility of ChatGPT and Google might unduly affect judgments regarding the comparative efficacy of various AI tools. Lastly, we acknowledge that the timing of this field study in May 2023, when most CSP topics had likely already been covered, presents a threat to the study’s validity. This may have influenced the students’ interactions with the AI tools, as their prior knowledge could have affected their engagement and the outcomes observed. The findings of this study, while specific to Computer Science Principles (CSP), could have broader implications for other disciplines where generative AI-driven educational tools are utilized. Challenges related to student engagement, response accuracy, and the need for prompt engineering are relevant across fields such as mathematics, biology, and the humanities, where similar technologies are increasingly being explored. Furthermore, the tools selected for this study serve as representatives of broader categories of AI applications, suggesting that the insights gained may inform the design and implementation of similar systems in other educational contexts. 7

CONCLUSIONS AND FUTURE WORK

Our study aimed to investigate the potential of conversational agents in aiding high school students as they acquire knowledge on CSP concepts through exploratory search. We found that conversational agents overall have the potential to help students during the learning process. Specifically, students prefer conversational agents compared to conventional web search. The shorter task duration, less exploratory actions, and higher average extraneous information coupled with the proportionally large amount of copy/paste information using web search promotes the use of conversational agents that are more adept to a pedagogically sound framework for learning CSP at the secondary school level. Furthermore, our results indicate that generative conversational agents are highly effective and interactive, and are preferred over fixed-response conversational agents, yet generative conversational agents pose risks as learning tools. Students are dissatisfied with the interaction and effectiveness of fixed-response conversational agents; however, find them helpful at finding useful information. Based on our findings, future work can focus on bringing the benefits of the fixed-response conversational agent and generative conversational agents together. Specifically, we plan to investigate approaches to customize a hybrid-domain conversational agent for learning CSP concepts. Additionally, we acknowledge that ecological validity and student engagement are critical factors in evaluating the effectiveness of AI-driven learning experiences. Although this study did not include a longitudinal evaluation, we propose it as a future research direction to better understand the long-term effects of such tools on student learning. Our study acknowledges a spectrum of AI design possibilities, and while we focused on specific configurations of generative and fixed AI systems, we recognize that alternative designs could have led to different outcomes. Future work could explore varying system prompts, customization options, and student interaction patterns to further understand the impact of AI design choices on learning experiences. REFERENCES [1] [n. d.]. AP Computer Science Principles COURSE AND EXAM DESCRIPTION Effective Fall 2020 INCLUDES Course framework Instructional section Sample exam questions Create performance task guidelines. https://apcentral.collegeboard.org/media/pdf/ap-computer-science-principlescourse-and-exam-description.pdf [2] [n. d.]. Code.org Curriculum Values. https://code.org/educate/curriculum/values [3] [n. d.]. National Science Foundation - Where Discoveries Begin. https://nsf.gov/events/event_summ.jsp?cntn_id=130646&org=NSF [4] [n. d.]. Professional Learning Values. https://code.org/educate/professional-learning/values [5] 2023. AP ® Computer Science Principles Curricular Requirements CR1. https://apcentral.collegeboard.org/media/pdf/ap-computer-scienceprinciples-sample-syllabus-2.pdf [6] Google . 2023. Bard. https://bard.google.com/ 28

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

[7] OpenAPI . 2023. OpenAI Platform. https://platform.openai.com/docs/guides/gpt/chat-completions-api [8] Rabe Abdalkareem, Emad Shihab, and Juergen Rilling. 2017. What do developers use the crowd for? A study using stack overflow. IEEE Software 34, 2 (2017), 53–60. [9] Shruti Agrawal and Satheesh Abimannan. 2023. Conversational virtual educator using Google’s dialogflow. AIP Conference Proceedings 2755, 1 (09 2023), 020017. https://doi.org/10.1063/5.0150047 arXiv:https://pubs.aip.org/aip/acp/articlepdf/doi/10.1063/5.0150047/18119897/020017_1_5.0150047.pdf [10] Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Mahajan, and Mohsen Dorodchi. 2023. Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaïs Tack, Victoria Yaneva, Zheng Yuan, and Torsten Zesch (Eds.). Association for Computational Linguistics, Toronto, Canada, 709–726. https://doi.org/10.18653/v1/2023.bea1.57 [11] Abdulaziz A. Alfayez and Judy Lambert. 2019. Exploring Saudi Computer Science Teachers’ Conceptual Mastery Level of Computational Thinking Skills. Computers in the Schools 36, 3 (2019), 143–166. https://doi.org/10.1080/07380569.2019.1639593 arXiv:https://doi.org/10.1080/07380569.2019.1639593 [12] Barbara Arfé, Tullio Vardanega, and Lucia Ronconi. 2020. The effects of coding on children’s planning and inhibition skills. Computers & Education 148 (2020), 103807. https://doi.org/10.1016/j.compedu.2020.103807 [13] Kumaripaba Athukorala, Dorota Głowacka, Giulio Jacucci, Antti Oulasvirta, and Jilles Vreeken. 2016. Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks. Journal of the Association for Information Science and Technology 67, 11 (2016), 2635–2651. https://doi.org/10.1002/asi.23617 arXiv:https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.23617 [14] Kumaripaba Athukorala, Dorota Głowacka, Giulio Jacucci, Antti Oulasvirta, and Jilles Vreeken. 2016. Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks. Journal of the Association for Information Science and Technology 67, 11 (2016), 2635–2651. https://doi.org/10.1002/asi.23617 arXiv:https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.23617 [15] Anne Aula, Rehan Khan, and Zhiwei Guan. 2010. How does search behavior change as search becomes more difficult?. In Proceedings of the ACM Conference on Human Factors in Computing Systems - CHI 2010. http://doi.acm.org/10.1145/1753326.1753333 [16] Hannah McLean Babe, Sydney Nguyen, Yangtian Zi, Molly Q Feldman, Carolyn Jane Anderson, and Arjun Guha. 2024. StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code. https://openreview.net/forum?id=o6XxdC4QsX [17] David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Available at SSRN 4337484 (2023). [18] Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023. Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 292–298. https://doi.org/10.1145/3587102.3588852 [19] Nikki Barrington, Nithin Gupta, Basel Musmar, David Doyle, Nicholas Panico, Nikhil Godbole, Taylor Reardon, and Randy D’amico. 2023. A Bibliometric Analysis of the Rise of ChatGPT in Medical Research. Medical Sciences 11 (09 2023). https://doi.org/10.3390/medsci11030061 [20] Luciana Benotti, María Cecilia Martínez, and Fernando Schapachnik. 2014. Engaging High School Students Using Chatbots. In Proceedings of the 2014 Conference on Innovation & Technology in Computer Science Education (Uppsala, Sweden) (ITiCSE ’14). Association for Computing Machinery, New York, NY, USA, 63–68. https://doi.org/10.1145/2591708.2591728 [21] Alex Bernat. 2023. We Need to Fix Secondary School Computer Science. ACM Trans. Comput. Educ. 23, 4, Article 38 (nov 2023), 6 pages. https://doi.org/10.1145/3617599 [22] Virginia Braun and Victoria Clarke. 2021. Thematic Analysis. https://us.sagepub.com/en-us/nam/thematic-analysis/book248481 [23] Robert Brennan and Jonathan Lesage. 2023. Exploring the Implications of OpenAI Codex on Education for Industry 4.0. 254–266. https://doi.org/10. 1007/978-3-031-24291-5_20 [24] Peter Brusilovsky. 2023. The Future of Computing Education Materials. https://csed.acm.org/wp-content/uploads/2023/03/Educational-MaterialsFirst-Draft-1.pdf [25] Christopher Bull and Ahmed Kharrufa. 2023. Generative AI Assistants in Software Development Education: A vision for integrating Generative AI into educational practice, not instinctively defending against it. IEEE Software (2023), 1–9. https://doi.org/10.1109/ms.2023.3300574 [26] Liang Cai, Haoye Wang, Bowen Xu, Qiao Huang, Xin Xia, David Lo, and Zhenchang Xing. 2019. AnswerBot: An Answer Summary Generation Tool Based on Stack Overflow. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 1134–1138. https://doi.org/10.1145/3338906.3341186 [27] Wanling Cai, Yucheng Jin, Xianglin Zhao, and Li Chen. 2023. “Listen to Music, Listen to Yourself”: Design of a Conversational Agent to Support Self-Awareness While Listening to Music. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 119, 19 pages. https://doi.org/10.1145/3544548.3581427 [28] Preetha Chatterjee, Manziba A. Nishi, Kostadin Damevski, Vinay Augustine, Lori Pollock, and Nicholas A. Kraft. 2017. What information about code snippets is available in different software-related documents? An exploratory study. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). 382–386. https://doi.org/10.1109/SANER.2017.7884638 29

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

[29] Michelene T. H. Chi and Ruth Wylie. 2014. The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes. Educational Psychologist 49, 4 (2014), 219–243. https://doi.org/10.1080/00461520.2014.965823 arXiv:https://doi.org/10.1080/00461520.2014.965823 [30] Bruno Pereira Cipriano and Pedro Alves. 2023. GPT-3 vs Object Oriented Programming Assignments: An Experience Report. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 61–67. https://doi.org/10.1145/3587102.3588814 [31] Bruno Pereira Cipriano and Pedro Alves. 2023. GPT-3 vs Object Oriented Programming Assignments: An Experience Report. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 61–67. https://doi.org/10.1145/3587102.3588814 [32] Google Cloud. 2022. https://cloud.google.com/dialogflow [33] Paul A. Crook, Alex Marin, Vipul Agarwal, Samantha Anderson, Ohyoung Jang, Aliasgar Lanewala, Karthik Tangirala, and Imed Zitouni. 2018. Conversational Semantic Search: Looking Beyond Web Search, Q&A and Dialog Systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM ’18). Association for Computing Machinery, New York, NY, USA, 763–766. https://doi.org/10.1145/3159652.3160590 [34] Marian Daun and Jennifer Brings. 2023. How ChatGPT Will Change Software Engineering Education. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 110–116. https://doi.org/10.1145/3587102.3588815 [35] Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto, ON, Canada) (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 1136–1142. https://doi.org/10.1145/3545945.3569823 [36] Paul Denny, James Prather, Brett A. Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N. Reeves, Eddie Antonio Santos, and Sami Sarsa. 2024. Computing Education in the Era of Generative AI. Commun. ACM 67, 2 (jan 2024), 56–67. https://doi.org/10.1145/ 3624720 [37] Richard Devine. 2021. Reflexive Thematic Analysis – The What, Why and How. https://richarddevinesocialwork.com/2021/11/12/reflexivethematic-analysis-the-what-why-and-how/ [38] Dhanraj R. Dhotre, Niraja Jain, Chhaya Mhaske, Nitin Choubey, and Dinesh D. Patil. 2023. ADiTi App: Leveraging Deep Learning and Generative AI for a Chatbot Application with Deep Belief Networks. International Journal of Intelligent Systems and Applications in Engineering 12, 1s (Sep. 2023), 569–576. https://www.ijisae.org/index.php/IJISAE/article/view/3492 [39] Dr. J. T. Dillon. 1981. A Norm against Student Questions. The Clearing House: A Journal of Educational Strategies, Issues and Ideas 55, 3 (1981), 136–139. https://doi.org/10.1080/00098655.1981.10113676 arXiv:https://doi.org/10.1080/00098655.1981.10113676 [40] Felix Dobslaw and Peter Bergh. 2023. Experiences with Remote Examination Formats in Light of GPT-4. In Proceedings of the 5th European Conference on Software Engineering Education (Seeon/Bavaria, Germany) (ECSEE ’23). Association for Computing Machinery, New York, NY, USA, 220–225. https://doi.org/10.1145/3593663.3593695 [41] Hugging Face. 2023. Hugging Face – On a mission to solve NLP, one commit at a time. https://huggingface.co/ [42] Victor Farinango and Santiago Castro Arias. 2023. Pedagogical Support Through the Implementation of a Virtual Assistant to Personalize Education. In Proceedings of the 18th Latin American Conference on Learning Technologies (LACLO 2023), Santiago Berrezueta (Ed.). Springer Nature Singapore, Singapore, 343–356. [43] Donghui Feng, Erin Shaw, Jihie Kim, and Eduard Hovy. 2006. An Intelligent Discussion-Bot for Answering Student Queries in Threaded Discussions. In Proceedings of the 11th International Conference on Intelligent User Interfaces (Sydney, Australia) (IUI ’06). Association for Computing Machinery, New York, NY, USA, 171–177. https://doi.org/10.1145/1111449.1111488 [44] James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. 2023. My AI Wants to Know If This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference (Melbourne, Australia) (ACE ’23). Association for Computing Machinery, New York, NY, USA, 97–104. https://doi.org/10.1145/3576123. 3576134 [45] Maya Flax. 2023. Strategies for enhancing teaching excellence: Providing real-world examples and building rapport. New Vistas 9, 1 (2023). [46] Fiona French, David Levi, Csaba Maczo, Aiste Simonaityte, Stefanos Triantafyllidis, and Gergo Varda. 2023. Creative Use of OpenAI in Education: Case Studies from Game Development. Multimodal Technologies and Interaction 7, 8 (2023). https://doi.org/10.3390/mti7080081 [47] Yasemin Gülbahar and Filiz Kalelioğlu. 2017. Competencies of High School Teachers and Training Needs for Computer Science Education. In Proceedings of the 6th Computer Science Education Research Conference (Helsinki, Finland) (CSERC ’17). Association for Computing Machinery, New York, NY, USA, 26–31. https://doi.org/10.1145/3162087.3162092 [48] Karla Hamlen, Nigamanth Sridhar, Lisa Bievenue, Debbie K. Jackson, and Anil Lalwani. 2018. Effects of Teacher Training in a Computer Science Principles Curriculum on Teacher and Student Skills, Confidence, and Beliefs. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Baltimore, Maryland, USA) (SIGCSE ’18). Association for Computing Machinery, New York, NY, USA, 741–746. https://doi.org/10.1145/3159450.3159496 [49] Karla Hamlen Mansour, Debbie K. Jackson, Lisa Bievenue, Adam Voight, and Nigamanth Sridhar. 2023. Understanding the Impact of Peer Instruction in CS Principles Teacher Professional Development. ACM Trans. Comput. Educ. 23, 2, Article 24 (apr 2023), 21 pages. https://doi.org/10.1145/3585077 30

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

[50] Qiang Hao, Brad Barnes, Robert Branch, and Ewan Wright. 2017. Predicting Computer Science Students’ Online Help-Seeking Tendencies. Knowledge Management & E-Learning: An International Journal 9 (03 2017), 19–32. [51] Christina N. Harrington and Lisa Egede. 2023. Trust, Comfort and Relatability: Understanding Black Older Adults’ Perceptions of Chatbot Design for Health Information Seeking. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 120, 18 pages. https://doi.org/10.1145/3544548.3580719 [52] Arto Hellas, Juho Leinonen, Sami Sarsa, Charles Koutcheme, Lilja Kujanpää, and Juha Sorva. 2023. Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (ICER 2023). ACM. https://doi.org/10.1145/3568813.3600139 [53] Irene Hou, Sophia Mettille, Owen Man, Zhuo Li, Cynthia Zastudil, and Stephen MacNeil. 2024. The Effects of Generative AI on Computing Students’ Help-Seeking Preferences. In Proceedings of the 26th Australasian Computing Education Conference (Sydney, NSW, Australia) (ACE ’24). Association for Computing Machinery, New York, NY, USA, 39–48. https://doi.org/10.1145/3636243.3636248 [54] https://rasa.com/. [n. d.]. Rasa: Open source conversational AI. visited, December 1, 2020. [55] Joy Idialu and Deborah Etsenake. [n. d.]. Whodunnit: Human or AI? https://plg.uwaterloo.ca/~migod/846/current/projects/07-Norhan-DeborahJoy-report.pdf [56] Cruz Izu and Claudio Mirolo. 2020. Comparing Small Programs for Equivalence: A Code Comprehension Task for Novice Programmers. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 466–472. https://doi.org/10.1145/3341525.3387425 [57] Sajed Jalil, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE. https://doi.org/10.1109/icstw58534.2023.00078 [58] Mahmoud Kaddoura. 2013. Think pair share: A teaching learning strategy to enhance students’ critical thinking. Educational Research Quarterly 36, 4 (2013), 3–24. [59] Tyson Kendon, Leanne Wu, and John Aycock. 2023. AI-Generated Code Not Considered Harmful. In Proceedings of the 25th Western Canadian Conference on Computing Education (Vancouver, BC, Canada) (WCCCE ’23). Association for Computing Machinery, New York, NY, USA, Article 3, 7 pages. https://doi.org/10.1145/3593342.3593349 [60] Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias Hagen. 2018. Toward Voice Query Clarification. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). 1257–1260. [61] Natalie Kiesler and Daniel Schiffner. 2023. Large Language Models in Introductory Programming Education: ChatGPT’s Performance and Implications for Assessments. ArXiv abs/2308.08572 (2023). https://api.semanticscholar.org/CorpusID:261030787 [62] Adrian Kirwan. 2023. ChatGPT and university teaching, learning and assessment: some initial reflections on teaching academic integrity in the age of Large Language Models. Irish Educational Studies 0, 0 (2023), 1–18. https://doi.org/10.1080/03323315.2023.2284901 arXiv:https://doi.org/10.1080/03323315.2023.2284901 [63] Emily Kuang, Ehsan Jahangirzadeh Soure, Mingming Fan, Jian Zhao, and Kristen Shinohara. 2023. Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask Them (Voice vs. Text). In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 116, 15 pages. https: //doi.org/10.1145/3544548.3581247 [64] Els Kuiper, Monique Volman, and Jan Terwel. 2005. The Web as an information resource in K–12 education: Strategies for supporting students in searching and processing information. Review of educational research 75, 3 (2005), 285–328. [65] Els Kuiper, Monique Volman, and Jan Terwel. 2009. Developing Web literacy in collaborative inquiry activities. Computers & Education 52, 3 (2009), 668–680. https://doi.org/10.1016/j.compedu.2008.11.010 [66] Sam Lau and Philip Guo. 2023. From "Ban It Till We Understand It" to "Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools Such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 106–121. https://doi.org/10.1145/3568813.3600138 [67] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 124–130. https: //doi.org/10.1145/3587102.3588785 [68] Jacqueline Leonard, Andrea C. Burrows, and Richard Kitchen. 2019. Recruiting, Preparing, and Retaining STEM Teachers for a Global Generation. Brill, Leiden, The Netherlands. https://doi.org/10.1163/9789004399990 [69] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Kevin Knight, Ani Nenkova, and Owen Rambow (Eds.). Association for Computational Linguistics, San Diego, California, 110–119. https://doi.org/10.18653/v1/N16-1014 [70] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol 31

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

Vinyals. 2022. Competition-level code generation with AlphaCode. Science 378, 6624 (Dec. 2022), 1092–1097. https://doi.org/10.1126/science.abq1158 [71] Chao Liu, Xin Xia, David Lo, Cuiyun Gao, Xiaohu Yang, and John Grundy. 2020. Opportunities and Challenges in Code Search Tools. arXiv preprint arXiv:2011.02297 (2020). [72] Qiaoling Liu, Eugene Agichtein, Gideon Dror, Yoelle Maarek, and Idan Szpektor. 2012. When Web Search Fails, Searchers Become Askers: Understanding the Transition. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (Portland, Oregon, USA) (SIGIR ’12). Association for Computing Machinery, New York, NY, USA, 801–810. https://doi.org/10.1145/2348283.2348390 [73] Qianou Ma, Tongshuang Wu, and Kenneth Koedinger. 2023. Is AI the better programming partner? Human-Human pair programming vs. Human-AI pAIr programming. arXiv preprint arXiv:2306.05153 (2023). [74] Hanifah Putri Elisa Mahir Pradana and Syarifuddin Syarifuddin. 2023. Discussing ChatGPT in education: A literature review and bibliometric analysis. Cogent Education 10, 2 (2023), 2243134. https://doi.org/10.1080/2331186X.2023.2243134 arXiv:https://doi.org/10.1080/2331186X.2023.2243134 [75] Kamil Malinka, Martin Peresíni, Anton Firc, Ondrej Hujnák, and Filip Janus. 2023. On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2023). ACM. https://doi.org/10.1145/3587102.3588827 [76] Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46. [77] Wilfred B. W. Martin. 1987. Students’ Perceptions of Causes and Consequences of Embarrassment in the School. Canadian Journal of Education / Revue canadienne de l’éducation 12, 2 (1987), 277–293. http://www.jstor.org/stable/1494907 [78] Mary B. McPherson and Patricia Kearney. 1992. Classroom Embarrassment: Types, Goals, and Face Saving Strategies. [79] Microsoft. 2022. https://www.luis.ai/ [80] Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2023. In-IDE Generation-based Information Support with a Large Language Model. arXiv:2307.08177 [cs.SE] [81] OpenAI. 2023. ChatGPT. https://openai.com/blog/chatgpt. [82] OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] [83] Eng Lieh Ouh, Benjamin Kok Siew Gan, Kyong Jin Shim, and Swavek Wlodkowski. 2023. ChatGPT, Can You Generate Solutions for My Coding Exercises? An Evaluation on Its Effectiveness in an Undergraduate Java Programming Course.. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 54–60. https://doi.org/10.1145/3587102.3588794 [84] Miranda C. Parker. 2023. Barriers and Supports to Offering Computer Science in High Schools: A Case Study of Structures and Agents. ACM Trans. Comput. Educ. 23, 2, Article 20 (mar 2023), 27 pages. https://doi.org/10.1145/3572900 [85] Sara Perez-Soler, Sandra Juarez-Puerta, Esther Guerra, and Juan de Lara. 2021. Choosing a Chatbot Development Tool. IEEE Software 38, 4 (2021), 94–103. https://doi.org/10.1109/MS.2020.3030198 [86] Carrie Anne Philbin. 2023. Exploring the Potential of Artificial Intelligence Program Generators in Computer Programming Education for Students. ACM Inroads 14, 3 (aug 2023), 30–38. https://doi.org/10.1145/3610406 [87] Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 41–42. https://doi.org/10.1145/3568812.3603476 [88] Stephen R. Piccolo, Paul C. Denny, Andrew Luxton-Reilly, Samuel Payne, and Perry G. Ridge. 2023. Many bioinformatics programming tasks can be automated with ChatGPT. arXiv.org abs/2303.13528 (2023). https://doi.org/10.48550/arXiv.2303.13528 [89] Russel A. Poldrack, Thomas Lu, and G. Beguš. 2023. AI-assisted coding: Experiments with GPT-4. ArXiv abs/2304.13187 (2023). https: //api.semanticscholar.org/CorpusID:258331866 [90] James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michael E. Caspersen, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. Transformed by Transformers: Navigating the AI Coding Revolution for Computing Education: An ITiCSE Working Group Conducted by Humans. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 561–562. https://doi.org/10.1145/3587103.3594206 [91] James Prather, Paul Denny, Juho Leinonen, Brett A. Becker, Ibrahim Albluwi, Michelle Craig, Hieke Keuning, Natalie Kiesler, Tobias Kohn, Andrew Luxton-Reilly, Stephen MacNeil, Andrew Petersen, Raymond Pettit, Brent N. Reeves, and Jaromir Savelka. 2023. The Robots Are Here: Navigating the Generative AI Revolution in Computing Education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education (Turku, Finland) (ITiCSE-WGR ’23). Association for Computing Machinery, New York, NY, USA, 108–159. https://doi.org/10.1145/3623762.3633499 [92] Parsa Rajabi, Parnian Taghipour, Diana Cukierman, and Tenzin Doleck. 2023. Exploring ChatGPT’s Impact on Post-Secondary Education: A Qualitative Study. In Proceedings of the 25th Western Canadian Conference on Computing Education (Vancouver, BC, Canada) (WCCCE ’23). Association for Computing Machinery, New York, NY, USA, Article 9, 6 pages. https://doi.org/10.1145/3593342.3593360 [93] Sujeeth Goud Ramagoni and Dennis Brylow. 2023. The Impact of Certified CS Teachers on AP Computer Science Exam Scores: A Study in Wisconsin. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 30–31. https://doi.org/10.1145/3568812.3603486 32

Investigating Conversational Agents to Support Secondary School Students Learning CSP Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

[94] Raghu Raman, Smrithi Venkatasubramanian, Krishnashree Achuthan, and Prema Nedungadi. 2015. Computer Science (CS) Education in Indian Schools: Situation Analysis Using Darmstadt Model. ACM Trans. Comput. Educ. 15, 2, Article 7 (may 2015), 36 pages. https://doi.org/10.1145/2716325 [95] Katherine A Rawson, Ruthann C Thomas, and Larry L Jacoby. 2015. The power of examples: Illustrative examples enhance conceptual learning of declarative concepts. Educational Psychology Review 27 (2015), 483–504. [96] Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 299–305. https://doi.org/10.1145/3587102.3588805 [97] Soo Young Rieh, Kevyn Collins-Thompson, Preben Hansen, and Hye-Jung Lee. 2016. Towards searching as a learning process: A review of current perspectives and future directions. Journal of Information Science 42 (02 2016), 19–34. https://doi.org/10.1177/0165551515615841 [98] Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-Driven Response Generation in Social Media. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, Regina Barzilay and Mark Johnson (Eds.). Association for Computational Linguistics, Edinburgh, Scotland, UK., 583–593. https://aclanthology.org/D11-1054 [99] Steven I. Ross, Michael J. Muller, Fernando Martinez, Stephanie Houde, and Justin D. Weisz. 2023. A Case Study in Engineering a Conversational Programming Assistant’s Persona 121-129. In Joint Proceedings of the IUI 2023 Workshops: HAI-GEN, ITAH, MILC, SHAI, SketchRec, SOCIALIZE co-located with the ACM International Conference on Intelligent User Interfaces (IUI 2023), Sydney, Australia, March 27-31, 2023 (CEUR Workshop Proceedings, Vol. 3359), Alison Smith-Renner and Paul Taele (Eds.). CEUR-WS.org, 121–129. https://ceur-ws.org/Vol-3359/paper13.pdf [100] Corbin Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020. Leading Conversational Search by Suggesting Useful Questions. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 1160–1170. https://doi.org/10.1145/3366423.3380193 [101] Caitlin Sadowski, Kathryn T Stolee, and Sebastian Elbaum. 2015. How developers search for code: a case study. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. 191–201. [102] Shima Salehi, Karen D. Wang, Ruqayya Toorawa, and Carl Wieman. 2020. Can Majoring in Computer Science Improve General Problem-Solving Skills?. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE ’20). Association for Computing Machinery, New York, NY, USA, 156–161. https://doi.org/10.1145/3328778.3366808 [103] Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr. 2023. Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. In Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 (ICER 2023). ACM. https://doi.org/10.1145/3568813.3600142 [104] Anuschka Schmitt, Thiemo Wambsganss, and Jan Marco Leimeister. 2022. Conversational Agents for Information Retrieval in the Education Domain: A User-Centered Design Investigation. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 486 (nov 2022), 22 pages. https://doi.org/10.1145/3555587 [105] Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI’16). AAAI Press, 3776–3783. [106] Amazon Web Services. 2022. https://aws.amazon.com/lex/ [107] Esther Shein. 2019. The CS Teacher Shortage. Commun. ACM 62, 10 (sep 2019), 17–18. https://doi.org/10.1145/3355375 [108] Adish Singla. 2023. Evaluating ChatGPT and GPT-4 for Visual Programming. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 (Chicago, IL, USA) (ICER ’23). Association for Computing Machinery, New York, NY, USA, 14–15. https://doi.org/10.1145/3568812.3603474 [109] J.Jinu Sophia and T.Prem Jacob. 2021. EDUBOT-A Chatbot For Education in Covid-19 Pandemic and VQAbot Comparison. In 2021 Second International Conference on Electronics and Sustainable Communication Systems (ICESC). 1707–1714. https://doi.org/10.1109/ICESC51422.2021.9532611 [110] Pragnya Sridhar, Aidan Doyle, Arav Agarwal, Christopher Bogart, Jaromir Savelka, and Majd Sakr. 2023. Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives. arXiv:2306.17459 [cs.AI] [111] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [112] Davis University of California. 2018. Gunrock: Building a human-like social bot by leveraging large scale real user data. In Alexa Prize SocialBot Grand Challenge 2 Proceedings. https://www.amazon.science/alexa-prize/proceedings/building-a-human-like-social-bot-by-leveraging-largescale-real-user-data [113] Oriol Vinyals and Quoc Le. 2015. A Neural Conversational Model. ICML Deep Learning Workshop, 2015 (06 2015). [114] Raven McCrory Wallace, Jeff Kupperman, Joseph Krajcik, and Elliot Soloway. 2000. Science on the Web: Students Online in a Sixth-Grade Classroom. Journal of the Learning Sciences 9, 1 (2000), 75–104. https://doi.org/10.1207/s15327809jls0901_5 [115] Thiemo Wambsganß, Leonie Haas, and Matthias Söllner. 2021. Towards the Design of a Student-Centered Question-Answering System in Educational Settings. [116] Feng Wang, Mable B Kinzie, Patrick McGuire, and Edward Pan. 2010. Applying Technology to Inquiry-Based Learning in Early Childhood Education. Early Childhood Education Journal 37, 5 (2010), 381–389. https://doi.org/10.1007/s10643-009-0364-6 [117] Tianjia Wang, Daniel Vargas Díaz, Chris Brown, and Yan Chen. 2023. Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE. 33

Conference acronym ’XX, June 03–05, 2018, Woodstock, NY

Frazier, et al.

https://doi.org/10.1109/vl-hcc57772.2023.00018 [118] Florian Weber, Thiemo Wambsganß, Dominic Rüttimann, and Matthias Söllner. 2021. Pedagogical Agents for Interactive Learning: A Taxonomy of Conversational Agents in Education Completed Research Paper. [119] Patricia Widjojo and Christoph Treude. 2023. Addressing Compiler Errors: Stack Overflow or Large Language Models? arXiv:2307.10793 [cs.SE] [120] Eric Hsiao-Kuang Wu, Chun-Han Lin, Yu-Yen Ou, Chen-Zhong Liu, Wei-Kai Wang, and Chi-Yun Chao. 2020. Advantages and Constraints of a Hybrid Model K-12 E-Learning Assistant Chatbot. IEEE Access 8 (2020), 77788–77801. https://doi.org/10.1109/ACCESS.2020.2988252 [121] Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo. 2017. AnswerBot: Automated generation of answer summary to developers’ technical questions. In ASE’17 (1st ed.), Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen (Eds.). IEEE, Institute of Electrical and Electronics Engineers, United States of America, 706–716. https://doi.org/10.1109/ASE.2017.8115681 Automated Software Engineering Conference 2017, ASE 2017 ; Conference date: 30-10-2017 Through 03-11-2017. [122] Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan Gašević. 2023. Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology (Aug. 2023). https://doi.org/10.1111/bjet.13370 [123] Wong Sing Yun, Saizal Pinjaman, Debbra Toria Nipo, and Rudy Ansar. 2023. ChatGPT in Education: A Bibliometric Analysis Using R. In e-Proceedings. 62. [124] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. https://doi.org/10.1145/3544548.3581388 [125] C. Zastudil, M. Rogalska, C. Kapp, J. Vaughn, and S. MacNeil. 2023. Generative AI in Computing Education: Perspectives of Students and Instructors. In 2023 IEEE Frontiers in Education Conference (FIE). IEEE Computer Society, Los Alamitos, CA, USA, 1–9. https://doi.org/10.1109/FIE58773.2023. 10343467 [126] Paul Zhang, Brandon Jaipersaud, Jimmy Ba, Andrew Petersen, Lisa Zhang, and Michael R. Zhang. 2023. Classifying Course Discussion Board Questions Using LLMs. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 658. https://doi.org/10.1145/3587103.3594202 [127] Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018. Towards Conversational Search and Recommendation: System Ask, User Respond. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM ’18). New York, NY, USA, 177–186. [128] Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 436, 20 pages. https://doi.org/10.1145/ 3544548.3581318

Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

Investigating Conversational Agents to Support Secondary School Students Learning CSP

Related documents