arXiv CSopen access

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch BENEDIKT FEIN, University of Passau, Germany GORDON FRASER, University of Passau, Germany The trend of embedding source code for machine learning applications also enables new opportunities in learning analytics in programming education, but which code embedding approach is most suitable for learning analytics remains an open question. A common approach to embedding source code lies in treating the code as a token sequence similar to natural language when training large language models (llms). However, in case of visual block-based programming languages like Scratch, this approach cannot be applied directly. While text-based representations of block-based code can be created to apply llms to this problem, other dedicated embedding models could potentially exhibit improved performance by capturing additional structural information. In this paper, we therefore instantiate four llms and five different popular embedding approaches for Scratch programs, create a token-prediction and two different classification tasks with corresponding datasets, and empirically evaluate the models on them. Our experiments demonstrate that a transfer of code embeddings to the educational environment of Scratch is feasible. The embedding models trained on large open Scratch datasets capture relevant structural and semantic information about the code to enable learning analytics like

arXiv:2607.19291v1 [cs.SE] 21 Jul 2026

predicting functional correctness of student programs, in the typically small classroom setting without requiring further task-specific model fine-tuning. CCS Concepts: • Social and professional topics → K-12 education; Software engineering education; • Software and its engineering → Visual languages; • Computing methodologies → Machine learning. Additional Key Words and Phrases: Scratch, code embeddings, programming education. ACM Reference Format: Benedikt Fein and Gordon Fraser. 2026. EmbeddedKittens: An Evaluation of Code Embeddings for Scratch. ACM Trans. Softw. Eng. Methodol. 1, 1 (July 2026), 36 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

The proliferation of code embeddings has led to their adoption for a wide variety of software engineering tasks, especially since large language models (llms) have become widely available [41]. Combining them with integrations into coding agents (e.g. GitHub Copilot, Claude Code) even allows developers to delegate tasks to these agents for semi-autonomous resolution. The llms are trained on large datasets of openly available natural language and source code [71]. Since open-source code datasets consist primarily of commonly used text-based programming languages like Python or Java [45, 49], the llms are performing best on such common languages [42]. Thus, the main target audience of such llm-based tools are professional, hobbyist, or learning developers using these textual programming languages. Code embeddings are also actively used in educational settings where students use the same llm-based tools to solve course exercises [19, 82]. The tools can also assist teachers in generating exercises [72], grading student submissions [8], Authors’ Contact Information: Benedikt Fein, [email protected], University of Passau, Passau, Germany; Gordon Fraser, [email protected], University of Passau, Passau, Germany.

This work is licensed under a Creative Commons Attribution 4.0 International License. © 2026 Copyright held by the owner/author(s). Manuscript submitted to ACM Manuscript submitted to ACM

1

2

Benedikt Fein and Gordon Fraser Sprite Scripts when space

key pressed

if

then

x = 50

when key pressed

space ▼

broadcast message

if

when I receive

Statements then

message ▼

next costume

Statements move

steps

next costume

Statements

=

when I receive message move 10 steps

(a) Scratch code containing two scripts that are triggered by different events.

x

50

10

broadcast

message ▼ (b) The ast contains nodes with a direct counterpart in the assembled program and abstract nodes inserted by LitterBox.

Fig. 1. Example of the Scratch code as it appears inside a sprite and its abstract syntax tree (ast). Each full Scratch program can contain multiple such sprites that each contain a separate independent set of scripts.

or be used to track students’ exercise progress [59]. Again, these approaches mainly focus on applications to textual programming languages. However, especially in early software engineering education, block-based visual programming environments such as Scratch [53] or Snap! [38] are frequently used instead [55]. These environments allow for the creation of programs by arranging pre-defined blocks representing statements and expressions (cf. Fig. 1a) on a two-dimensional working area. In these environments, the assumption that code can be represented similar to natural language text [39] might not hold since the language consists of freely placeable visual blocks rather than a linear sequence of textual tokens. Similarly, due to this different code representation, existing code embedding models cannot be employed without prior adaptation. Some models like for example code2vec [7], code2seq [5], or ASTNN [83] do not use the text sequence directly, however, but use inputs based on the abstract syntax tree (ast) of the code. Scratch code can also be parsed into such a format (cf. Fig. 1b), which suggests that the adaptation of existing models to the new domain is at least technically feasible. In addition to these foundational differences in language structure, the type of programs found in Scratch also differs. Scratch is mainly designed to implement small animations or interactive games rather than data-manipulating algorithms. For this purpose, Scratch heavily relies on the concurrent execution of code in multiple animated figures (called sprites). Figure 1a shows a typical example where the upper script executes whenever the user presses the space key, and triggers the execution of further code by broadcasting a message event. Since programs written in regular programming languages rarely use such event-handling-based code, common code embedding models rely on a program’s syntactical structure to represent the code [5, 7, 83, 1]. However, this might not be sufficient to capture such common Scratch language constructs, since the relations between event emitters and receivers are lost. Thus, models additionally capturing control- and data-flow information (e.g. GGNN [4]) might exhibit better performance on Scratch-based tasks. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

3

The Scratch environment imposes additional natural-language challenges not faced in most text-based programming scenarios. The Scratch user-base consists of large international communities for example in Brazil (≈ 6.42 % of all users), Spain (≈ 3.33 %), Poland (≈ 3.31 %), or China (≈ 2.55 %).1 The English-default assumption for keywords and method names in the libraries of many programming languages can be difficult for international learners since they will have to understand new-to-them programming concepts in an unfamiliar natural language [75]. The Scratch environment actively tries to mitigate this by providing translations for the user interface and all blocks. Additionally, custom variables, methods, and sprite names are allowed to contain arbitrary Unicode characters rather than being limited to alphanumeric characters. This is stark contrast to the publicly shared open-source code found for example on GitHub. In the Stack code dataset mined from open sources and created for the training of code-llms, 94 % of the analysed Python files use English identifiers, string constants, and documentation [45]. Existing code embedding model approaches might therefore not only have to be adapted to the different program structure of Scratch, but also need to take the differences in the natural languages used by programmers into account. While other work has already started using llms for Scratch-related tasks [20, 27, 23], to the best of our knowledge there so far exists no structured comparison of the performance of different code embedding models for Scratch. This might be a result of many established code embedding comparison tasks not being applicable to Scratch. For example, the commonly used benchmark of method-level code documentation generation or summarisation [5, 1, 81, 74] is not feasible in Scratch, since the projects do not contain any documentation and only rarely contain code comments. Thus, creating suitable training datasets is difficult. Similar challenges arise for the commonly used method or variable naming tasks [4, 5, 7, 51], since both programming concepts are only rarely used in publicly available Scratch projects [2]. However, the structure of Scratch projects allows for the construction of an alternative task. Scratch projects consist of multiple figures (sprites) which can be animated using the code contained within. Similar to classes in object-oriented programming languages, each sprite therefore encapsulates its own state and behaviour. Since all sprites in Scratch must be named by the programmer, large model training datasets can be created by mining publicly shared Scratch projects. To facilitate a baseline comparison of code embedding models for Scratch, we thus introduce the ‘sprite naming’ task in which the models must suggest suitable sprite names given the code contained within. Based on this foundational evaluation investigating whether established code embedding models can be applied to the Scratch context, we evaluate potential applications of the models for education-focussed applications, such as the prediction of program correctness or estimation of exercise progression. This paper extends our prior work [24] which evaluated the applicability of only the code2vec embedding model [7] to Scratch. In detail, the contributions of this paper are as follows: • We train five different code embedding models using various amounts of structural or control-flow information about the program on the sprite naming task to evaluate their applicability to Scratch. We evaluate their performance in comparison to four different pre-trained llms. • To evaluate how the natural language used by the programmer for identifier names and strings impacts the model performance in, we collect 10 Scratch program datasets each using a different primary natural language. We compare the models’ performance on the sprite naming task. • We evaluate fine-tuning strategies that adapt pre-trained sprite naming models to be used as classifiers for whole programs. This evaluates whether the models can be adapted to scenarios where insufficient training data for dedicated training is available. 1 https://scratch.mit.edu/statistics/, accessed 2026-05-19

Manuscript submitted to ACM

4

Benedikt Fein and Gordon Fraser

• To investigate the expressiveness of the embeddings for educational tasks, we evaluate whether pre-trained spritenaming embedding models and embedding-llms can be used to estimate program correctness and student exercise progression. • We make our preprocessing tool, EmbeddedKittens2 , publicly available under an open-source licence to support further research in this area, and to support the development of tools making use of Scratch code embeddings. Our experiments show that models including more structural information about the program outperform ones using a simpler program representation on the sprite naming task. As expected, additionally including control- and data-flow information results in further improvements. In our experiments, all specifically trained Scratch embedding models outperform the pre-trained llms. All models perform considerably worse when evaluated on programs that use non Latin-based character sets which demonstrates that the internationality of the Scratch community needs to be taken into account when developing models for the Scratch programming language. The pre-trained sprite-embedding models can be used estimate the program correctness and student exercise progression without requiring further fine-tuning, thus highlighting their applicability to classroom scenarios with limited availability of training data. 2

Background

2.1

Scratch

Scratch [53] having over 135 million registered users3 highlights its popularity as a programming language amongst teachers and learners. Its block-based concept allows students to construct programs by arranging blocks that represent statements and expressions. The block shapes clearly indicate in which positions they can be placed, and the Scratch interface subsequently prevents invalid block combinations. Scratch contains many standard programming constructs such as loops, conditions (cf. Fig. 1a), or variables. However, most blocks focus on the manipulation of attributes of sprites, the interaction of different sprites, or reactions to user inputs. The sprites can thus be animated to program small interactive games. Since Scratch is fully web-based, it can be used without installation through its official website. This website actively encourages the sharing of programs to allow other users to play the games and inspect their code to gather ideas for their own projects. The website also provides a rest-api through which the publicly shared programs can be programmatically accessed. Large datasets of programs as required for the training of machine learning models can be created by sampling from the over 164 million publicly shared programs. 2.2

Code Embeddings

Code embedding models map source code into a dense vector space representation that aims to capture both semantic and syntactic aspects of the code. This information can then be decoded by the models to assist in code-related tasks. Earlier code embedding models were trained to solve specific tasks like for example suggesting method [7] or variable names [4] or generating suitable documentation [1]. Since source code follows similar patterns as natural language [39], large language models (llms) can be trained on large datasets of mixed natural language and code. More recently, the focus has thus shifted mostly away from task-specific models and instead aims to use the general-purpose nature of large models (e.g. GPT-5 [71], Claude Opus 4.7 4 ) to solve code-related tasks [41]. For example, these models allow for the generation of new code from natural language descriptions or specifications [52, 76], the summarisation or explanation 2 https://github.com/se2p/LitterBox, version 1.12, licenced GPL-3.0-or-later 3 https://scratch.mit.edu/statistics/, accessed 2026-05-19 4 https://www.anthropic.com/system-cards, accessed 2026-05-19

Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

5

of code in natural language [36], or code-code-transformations like the generation of test suites for existing code [65]. To further improve the llms’ performance for code-related tasks, various fine-tuned models like StarCoder 2 [49], DeepSeek Coder v2 [16], or Devstral 2 5 have been developed. Both pre-llm code embedding models and llm-based approaches have also been used in educational settings. Since semantically similar programs should be mapped to similar vectors in embedding space, these vectors can be used directly to find clusters of students with similar solution approaches and help educators find outliers that did not yet solve the task [10, 59]. Using the code embeddings to determine similarity, this approach can be extended to apply human feedback given for one student’s solution also automatically to a similar program of another student [61]. The main focus of llm-based code embeddings in education lies on undergraduate-level Python and Java tasks [63]. llm-based tools are supporting teachers when generating exercises [72], assessing student submissions [8, 32], or to generate next-step hints [64]. Students are actively using llms to solve exercises [19, 82]. While llms have been used in the Scratch context to generate exercise material [33], the main focus seems to be on supporting students in open-ended tasks with chat-based interfaces integrated into the Scratch environment. These can be used to find inspirations for additional features of the programs [21, 20], or to receive help in locating and fixing issues in the program by receiving an llm-generated natural language description of the underlying misconceptions and the issue’s impact for the specific program [27, 25]. Apart from our previous study applying the code2vec model to Scratch [24], applying dedicated non-llm-based code embeddings to Scratch programs to the best of our knowledge remains an unexplored area. 3

Transforming Scratch Code Into the Model Input Formats

Code embedding machine learning models for regular programming languages assume that the code exists in a textual format to which further processing steps can be applied to obtain the model-specific input format. Thus, before being able to apply the models to a visual programming language, additional steps converting the visual representation into a textual one are necessary first. Specifically, we describe our preprocessing for the models used in our experiments: Transformer models [79] and transformer-based llms use flat sequences of tokens to represent the program, code2vec [7] and code2seq [5] use an abstraction derived from the abstract syntax tree (ast) of the program as input, ASTNN [83] uses a sequence of subtrees of the ast, and GGNN [4] additionally requires an extension of the ast into a graph containing control and data flow information. 3.1

The Scratch Data Format

The code of a Scratch project is saved in the form of a single json file. This file contains a list of all sprites of the project. For each sprite its name, ‘costumes’ (i.e. images representing what the figure looks like), sounds, and the actual code it contains, are stored. Within sprites, the code is structured into several scripts (cf. Fig. 1a shows two scripts). However, all code of a sprite is saved as a single flat list of all its blocks. Each block is represented by a unique identifier and an ‘opcode’ which is a unique name that identifies which type of block it is. Since this storage format loses all information about the actual syntactical nesting of the program, each block also contains references to relevant related blocks using the unique block identifier: its parent block, its input blocks (e.g. the left and right operands of an addition block), and for statements also a pointer to the next block attached to the bottom of it. For example, the if contains references to the

=

then

block in Fig. 1a

, the broadcast , and the next costume blocks. Those pointers allow for the recreation of the

5 https://mistral.ai/news/devstral-2-vibe-cli, accessed 2026-05-19

Manuscript submitted to ACM

6

Benedikt Fein and Gordon Fraser

BEGIN_SCRIPT event_whenkeypressed key ( 32 ) control_if < ( x ) operator_equals ( 50 ) > BEGIN_SUBSTACK event_broadcast event_message ( message ) END_SUBSTACK looks_nextcostume END_SCRIPT

when [ space v] key pressed if <(x) = (50) > then broadcast [ message v] end next costume

BEGIN_SCRIPT event_whenbroadcastreceived event_message ( message ) motion_movesteps ( 10 ) END_SCRIPT

when I receive [ message v] move (10) steps

(a) NeuralCodeSum transformer: Flat sequence of abstract tokens.

(b) ScratchBlocks format used for llms.

x (eq if stmts broadcast) message space (whenkeypressed stmts) next-costume ... ⇓ hash x (-1654711692) message space (307684711) next-costume ... (c) code2seq (top) and code2vec (bottom): Paths between leaves of the ast. For code2vec the sequence of non-leaf tokens on the path is hashed.

if

then

when key pressed

space ▼ BLOCK

next costume

BLOCK message ▼

= x

broadcast

when I receive

message ▼ BLOCK

move

steps

10

50

(d) ASTNN: ast split into a sequence of statement-asts. The BLOCK marker nodes indicate that a nested statement list was cut off from the statement-tree at this location. Fig. 2. Processing steps for the program from Fig. 1b to be usable as input to code2seq, code2vec, ASTNN, NeuralCodeSum, and llms.

nested block structure when parsing the project’s json file. The nesting of the visual blocks behaves similar to the use of parentheses and braces in a textual programming language. We make use of LitterBox [28] to parse Scratch programs into their abstract syntax tree (ast) representation (cf. Fig. 1b). LitterBox supports parsing all basic Scratch programs and can also parse programs using additional blocks provided by some of the officially supported extensions available in the Scratch user interface (pen to draw on the game canvas, music, text to speech, and translation). Since Scratch is a block-based language, it does not allow for the introduction of syntactical errors. Therefore, LitterBox can construct a valid ast for all projects that only contain supported blocks and have not been manually tampered with outside the Scratch user interface. All the following model-specific processing steps are then implemented on top of this LitterBox-generated ast as part of our EmbeddedKittens extension to LitterBox. Both tools are available publicly under an open-source licence at https://github.com/se2p/LitterBox. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch 3.2

7

Dedicated Token-Based Models

For this category of models, represented by the transformer-based NeuralCodeSum [1] in our experiments, the code is represented in a flat textual format. For regular programming languages, this simplifies the application of such models since the source code already is in the required format. For the visual language Scratch, neither the format as shown to the user, nor the ast can be directly used. Instead, we designed a translation step from the ast into a textual representation. The translation process is implemented by walking over the ast in depth-first traversal and appending a textual representation for each encountered node to the output text stream. While the Scratch user interface imposes no fixed ordering between scripts placed on the two-dimensional workspace, the ast traversal implicitly serialises the scripts in the into the order in which their first blocks are appearing in the project’s json. For each sprite, the list of blocks in the json represents the order in which the blocks were added to the project. We retain some structural information in the text similar to the braces in some programming languages (e.g. Java, C). We mark the beginning of scripts and indented block stacks inside control structures with special BEGIN_SCRIPT and BEGIN_SUBSTACK tokens respectively and also insert corresponding END_. . . markers (cf. Fig. 2a). Scratch visually categorises statements of different categories by colouring them. For example, all statements relating to the movement of a sprite are coloured blue. In the textual representation we retain this information by not using the English block text as shown to the user, but by instead converting it to an abstract identifier using the group as prefix and a single word for the concrete block name. Expressions used as inputs to blocks are enclosed with parentheses to mimic function calls of regular programming languages. For example, the textual representation of a move 10 steps statement is motion_movesteps (10). For boolean inputs, angles (<, >) are used instead. Both cases mimic the shape used in the visual representation and potentially aid the model in learning from the surrounding context. In case blocks have multiple inputs, we tokenise them in the left-to-right reading order as they appear in the English variant of the block. For example, letter 1 of apple results in the tokenisation operator_letterof (1) (apple). However, Scratch adapts the order of the inputs to embed them as far as possible into the natural grammatical structure of the block text in the user’s language settings. When changing the language to Hungarian for the previous example, the block is shown as apple 1 betűje . In cases where the model will be trained for example on a dataset obtained in a classroom setting where it is known that most students use the local language in the Scratch user interface, the tokensation order might have to be adapted accordingly. Contrary to this tokenisation of statements in Polish notation (i.e. using the operator as prefix with following operands), we tokenise mathematical and logical operators like +, =, or and in their natural-language-independent infix order as also used by most regular textual programming languages. Finally, variable names, string literals, and numbers are kept as-is without changes. This allows the model-specific text-encoder to treat them as required without losing information beforehand during the transformation from ast to text. 3.3

Large Language Models

Since llms are Transformer-based, they also receive flat token sequences as inputs. However, unlike dedicated models, they have been pre-trained on a large corpus of textual resources already. The unseen specially crafted token representation described in the previous section might therefore not be suitable for llms. Instead, the ScratchBlocks representation6 as shown in Fig. 2b has likely been part of the training dataset of llms due to its use for example on

6 https://en.scratch-wiki.info/wiki/Block_Plugin/Syntax, accessed 2026-06-05

Manuscript submitted to ACM

8

Benedikt Fein and Gordon Fraser

the Scratch community forums.7 Prior research confirms that this format works better than the native json program representation when prompting llms [23], and the ScratchBlocks representation has also been adopted successfully in other research prompting llms with Scratch code [27, 33]. Thus, we use the Scratch code to ScratchBlocks conversion as already built into LitterBox and adopt this text representation when prompting llms in our experiments. Since the conversion to ScratchBlocks is internally implemented as an ast traversal, the scripts are again serialised in the order they were added to the project (cf. Section 3.2). 3.4

Code2vec and Code2seq

The code2vec [7] and code2seq [5] embedding models are based around the idea of connecting leaf nodes of the abstract syntax tree and representing a code snippet as its set of all possible such connections [7, 5]. This means that a piece of code is represented by its set of possible connections between leaves of its ast. To limit the computational effort, a length threshold model hyperparameter can be used to prune long connections. During model training an embedding vector for each such connection is learned, and the final code is represented as a weighted average over these embeddings [7]. The code2seq model retains the sequence of individual nodes that are visited when walking the tree between the two chosen leaves (cf. Fig. 2c). In case of code2vec, the path between the leaves is saved as only a hash value of its string representation. After obtaining the ast from LitterBox, the set of connections can be constructed like on the ast of a regular programming language by traversing the ast. A difficulty for this representation of the code as paths comes from the structure of sprites. Since they consist of multiple independent scripts, each of those scripts results in a subtree of the ast that is only connected to the other scripts at the top via a virtual ‘ScriptList’ node which acts as a parent for all scripts of the sprite. Therefore, choosing a short path length during preprocessing results in few connections between scripts but instead results in most paths representing connections within scripts. A longer path length would allow the model to capture more information across scripts later-on. However, such an increase of the path length also has downsides: Firstly, it considerably increases the time required to preprocess the code. Additionally, since it is not possible to give the full set of paths to the model during training due to gpu memory constraints, a fixed number of random samples (200 in case of the original code2vec experiments on Java code [7]) has to be selected for each sprite. This therefore decreases the chance of selecting the ones connecting leaves significant to the sprite’s functionality (e.g. the ‘when this sprite clicked’ event for a button). 3.5

ASTNN: AST-based Neural Network

The ASTNN model [83] incrementally combines information from child nodes into their parents starting from the leaves of the ast. To make this process more efficient and reduce the amount of lost information for deeply nested asts [6], the ast is split into a sequence of separate per-statement subtrees. The individual subtree embeddings are combined into a single embedding using a bidirectional gated recurrent unit (gru) encoder [83]. The per-statement subtrees of the original ast (cf. Fig. 1b) are shown in Fig. 2d. In Scratch, each script starts with an initial event statement at the top (e.g. the ‘when I receive message’ block as shown in Fig. 1a). Its bottom has an indented marker onto which a further regular statement can be attached. A special case are statements representing loops and conditional control structures. These do not only contain a marker at the bottom, but also allow for the connection of additional statements inside (e.g. the if

then

in Fig. 1a). The control structure in this case extends around the

blocks inside the scope of the loop or condition. In the subtree given to ASTNN , such inner blocks are represented by 7 https://scratch.mit.edu/discuss/, accessed 2026-06-05

Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

9

Sprite

MessagePassing

Scripts set var if

var > -1

when I receive

then

by 1

message ▼

Statements

broadcast message change var

GuardedBy

Never

to 0

set

var ▼

if

to

0

Statements

then

LastRead

say

Statements

>

LastWrite

Hello

when I receive message say Hello

var

-1

broadcast

message ▼

change

var ▼

by

1

Fig. 3. Example code and its representation for GGNN as graph based on the ast with additional edges for control and data flow information. LastLexicalUse edges are not shown since the relevant connections are already made by LastWrite and LastRead edges. Edges are shown directionally to highlight their construction.

an abstract BLOCK node. Inputs to statements, i.e. expressions, have a distinctly different shape, being either rounded for numeric and textual inputs, or angled for boolean conditions. This clear visual separations of statements, their grouping into enclosed blocks, and expressions is also retained in the ast as easily separable subtrees per statement with all input expressions being part of the same subtree. All statements that are part of the same indentation level are always children of the same virtual ‘StmtList’ node, thus also clearly marked. Therefore, the splitting of the ast into subtrees per statement can be performed similar to how it can be done for a textual programming language that uses semicolons as statement separator and braces for the scoping of statements into control structures. 3.6

GGNN: Gated Graph Neural Network

As a graph neural network, the GGNN model [4] receives a graph as input. This graph structure is retained internally while training the model. The graph is based on the ast and labels each node with a semantic embedding. During training, multiple rounds of information exchange between nodes along the graph edges propagate this information and adapt the embedding vectors of neighbouring nodes accordingly. A special node is added to the graph such that its final embedding is used as representation for the whole input [4]. Thus, the preprocessing step aims to construct a program graph with semantically relevant edges between nodes. We construct the context graph that contains the syntactic and semantic relationships between the ast nodes following the original approach used on Java code [4]. The node set of the graph and its initial edge set are given by the ast. Each node is labelled by the type of the node (e.g. IfThen), or in case of nodes that contain user-defined values (e.g. literal values, or the name of a variable) a normalised string representation of this value. We also connect each node to its syntactical successor. In case of a parameterless statement, this is the statement attached to the bottom of it. Similar to the tokenisation described in Section 3.2, we again follow the left-to-right reading order in the English variants of the blocks for blocks that accept multiple inputs. For example, letter 1 of apple results in the connections letter of → 1 → apple . To construct further semantic edges, we again use available LitterBox features to extract the control and data flow dependencies within the program. For the data flow analysis, we consider both user-defined variables and sprite attributes that can be changed by the user (e.g. size, x/y-position) since both can be accessed via expression blocks. Manuscript submitted to ACM

10

Benedikt Fein and Gordon Fraser

When referring to variables in the following definitions we consider both variables and attributes. Following the edge nomenclature of Allamanis et al. [4] we construct the semantic connections of the program graph (cf. Fig. 3) as follows: LastRead, LastWrite Such edges are added between each variable use and the previous time it was accessed. These connections can be directly extracted from the data dependency graph. LastLexicalUse This type of edge connects variables to the last position in the program ‘text’ where they appeared previously independent of control flow. We only consider last uses within scripts or procedures since there is no strict order between different scripts. In case the same variable is used multiple times inside a single block, we again use the block order as it appears in the English variant. ComputedFrom These edges connect all variables appearing on the right side of an assignment to the variable on the left side. GuardedBy, GuardedByNegation This type of edge connects each variable appearing in the condition of a control statement (if, if-else, loops) to all uses of the same variable in blocks which are enclosed by the control statement. FormalArgName Custom procedures in Scratch can have parameters. We connect their declaration to all their uses within the custom procedure. ReturnsTo Custom procedures in Scratch cannot return values. Therefore, we only connect the last statement in a procedure back to its declaration. Additionally, we construct semantic edges specific to Scratch programs. Each sprite in Scratch can contain multiple scripts that are executed concurrently. A script can either start executing directly on program start, or alternatively be triggered later by user inputs or other state changes in the program. Scratch also allows to programmatically trigger the execution of other scripts by broadcasting messages (cf. ‘broadcast’ block in Fig. 1a). For these cases we add additional edges to the GGNN context graph for all corresponding sending and receiving block pairs. Similarly, for custom procedures we connect the calling block to the definition. In case the procedure has parameters, we connect each of the values passed in the caller to their corresponding definition in the procedure instead. Finally, we add all backwards edges for all edge types to the constructed graph. This speeds up the propagation of information through the network [4]. 4

Experiments

To provide a better understanding of code embeddings for Scratch programs, we first need to evaluate whether existing code embedding models can be applied to the new domain. However, common model evaluation tasks such as documentation generation [5, 1, 81, 74] or method naming [5, 7, 51] are not possible in Scratch due to lack of suitable training data [2]. Instead, we introduce the ‘sprite naming’ task as an alternative code summarisation task. Similar to method naming, the models are tasked to suggest a suitable sprite name based on the code contained within. To establish a baseline, we compare multiple code embedding models using this task as part of our first research question: RQ1 Can code embeddings be used to summarise sprites, i.e. be used to predict their name? The Scratch user base predominantly consists of an international community of young learners. Since these learners might not yet be familiar with English, the Scratch user interface can be switched to alternative languages and allows arbitrary Unicode characters for sprite names. Consequently, the programmers likely use their familiar natural language to name variables or sprites, which directly elicits our second research question: RQ2 How does the internationality of Scratch influence the sprite name prediction? Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

11

The internal code embeddings of the models trained on the sprite naming task might be overly specific for this application. To investigate whether the embeddings are sufficiently expressive and contain generalising semantic information, we evaluate whether the pre-trained models can be fine-tuned for program classification tasks: RQ3 What is the best method to fine-tune the sprite embeddings for program classification? The sprite naming task mainly serves as a baseline comparison for the model performance, but it is unlikely to be directly relevant for educational scenarios. In our final two research questions, we therefore investigate whether the code embeddings are sufficiently expressive to support more education-focussed tasks, such as predicting program correctness or exercise progress: RQ4 Can Scratch program embeddings be used as surrogate model for program correctness? RQ5 Can Scratch program embeddings be used to determine student exercise progress? 4.1

Dataset Collection

To create sufficiently sized datasets for model training, we used datasets consisting of projects which have been publicly shared on the Scratch website. As per the terms of use,8 all such projects are shared under a Creative Commons CC-BY-SA-2.0 licence explicitly allowing the re-use of the programs under the same conditions as long as attribution to the original author is given.9 The Scratch website actively encourages such re-use and adaptation by providing a ‘remix’ functionality similar to the ‘fork’ mechanism employed by the open-source community on GitHub. The feature allows users to start a new project by using any existing project as starting template instead of starting from an empty one. The Scratch api provides the metadata required to extract the ancestor relationship between base and remixed projects. We ensured that our datasets do not contain both parent and remix projects to avoid a potential data leakage between training and evaluation splits when creating datasets for the purpose of training our machine learning models. 4.2

Model Implementations

We trained most models specifically for our experiments. For llms, we used their pre-trained versions available via the respective apis without further fine-tuning instead. 4.2.1 Task-Specific Training: NeuralCodeSum, code2vec, code2seq, ASTNN, GGNN. For all models we trained ourselves, we used the same model implementations that can be found in the replication package of a prior publication [25]. The replication package contains implementations for the code2vec [7], code2seq [5], ASTNN [83], GGNN [4] models, and for the NeuralCodeSum [1] transformer. These models were used to predict method names in a Java dataset, which closely matches the sprite naming task of our RQ1. The NeuralCodeSum transformer was originally designed for the summarisation of code into a brief natural language sentence [1]. Unlike other general-purpose transformer models, it has therefore been designed specifically for code-related tasks. While newer or larger transformer-based models are available, most Scratch projects in our training datasets are fairly small and do not require large input context window sizes. The generation of long output sequences is also not required for our sprite naming task. We designed our data preprocessing tool to output data in the same format as already used by the model implementations. Thus, no further code changes to the models themselves were required. The ASTNN and code2vec models represent the output as a single label rather than a sequence of subtokens. For the other models, a beam search decoder is used when predicting textual sequences. 8 https://scratch.mit.edu/terms_of_use Section 4.3, accessed 2026-01-15 9 https://creativecommons.org/licenses/by-sa/2.0/, accessed 2026-01-15

Manuscript submitted to ACM

12

Benedikt Fein and Gordon Fraser

4.2.2 Pre-Trained: Large Language Models. We used GPT-5-mini (version: gpt-5-mini-2025-08-07) as available via the OpenAI api. While the full GPT-5 model would provide more advanced reasoning features [71], we did not require these for our experiments. All prompts have been submitted to the api in February 2026. Additionally, we set up a self-hosted instance of Ollama to host three additional text-generating llms: DeepSeek Coder v2 [16], Mistral Small 3.1,10 and Gemma 3 [31]. For the later research questions (RQs 3–5) that require embeddings rather than generated text, we also self-hosted the Qwen 3 Embedding [85] embedding llm.11 We chose DeepSeek Coder v2, Gemma 3 and Mistral Small 3.1 to compare one dedicated code model and two state-of-the-art general purpose models that are self-hostable on a single gpu. Prior research has shown that Gemma 3 is suitable for Scratch-related tasks [23]. While for Gemma 3 the dedicated embedding variant EmbeddingGemma [80] exists, its short context window of only 2 048 tokens would truncate many ScratchBlocks input sequences. We therefore chose Qwen 3 Embedding as alternative due to its performance in the mteb benchmark [22]. By setting the maximum input context size of Ollama to 16 384 tokens, only 32 samples were truncated when prompting the llms for the sprite naming task. Since we could self-host most of the llms used for our experiments and thus did not have to pay to access their apis, only the usage of the OpenAI api resulted in total costs of approximately 100 USD. Using a larger GPT model (e.g. GPT-5.4) would have increased costs by a factor of seven. Even though all these llms are likely pre-trained on data crawled from the official Scratch community forums, we believe the risk of a data leakage to be small since users in the forum likely refer to smaller code snippets in ScratchBlocks format as part of discussions or questions about concrete parts of the code rather than putting large code snippets into their forum posts. Whole Scratch projects are also unlikely to be part of the training data, since downloading the original program-json from the Scratch api is a rate-limited two-step process, the result of which would then have to be converted to ScratchBlocks format (e.g. using LitterBox) to be similar to our inputs. This is unlikely to be implemented in a general-purpose web crawler. 4.3

RQ1: Using Code Embeddings for Code Summarisation

4.3.1 Dataset. To train the models for the task of naming sprites, we randomly sampled 656 164 publicly shared Scratch projects. After removing projects which LitterBox could not process, we split the projects in an 80:10:10 split into training, validation, and testing subsets. Since we want the model to learn to generate user-chosen names, all sprites having a default name (e.g. ‘Sprite 1’ for English- or ‘Hahmo1’ for Finnish-speaking users) were ignored. Similarly, while the background image is modelled as ‘Stage’ and can contain code, it cannot be renamed by users and was therefore also excluded. We retained sprites with names also present in the sprite catalogue, since the user still actively selected a suitable image and thereby its corresponding predefined name (e.g. ‘Hedgehog’, ‘Rocketship’). Finally, sprites not containing any code were removed. This resulted in 1 824 786 sprites in the training, 167 713 in the validation, and 169 187 in the test dataset. As shown in Fig. 4, most programs contain between 1 and 4 sprites (median MD = 2, interquartile range IQR = 3) sprites relevant for the sprite-naming task. These sprites usually have short names (MD = 7, IQR = 4 characters) and are each organised into few separate scripts (MD = 2, IQR = 3). Nevertheless, most sprites contain a meaningful number of blocks (MD = 36, IQR = 22). As the complexity metrics computed by LitterBox show, most sprites contain actual logic making use of branching constructs and operators rather than being a linear sequence of statements (cyclomatic complexity [54]: MD = 11, IQR = 23, Halstead difficulty [37]: MD = 9.38, IQR = 17.2). 10 https://mistral.ai/news/mistral-small-3-1, accessed 2026-06-05 11 DeepSeek Coder v2: https://ollama.com/library/deepseek- coder- v2:16b, hash 63fb193b3a9b; Mistral Small 3.1: https://ollama.com/library /mistral- small3.1:24b, hash b9aaf0c2586a; Gemma 3: https://ollama.com/library/gemma3:27b- it- qat, hash 29eb0b9aeda3; Qwen 3 Embedding https://ollama.com/library/qwen3-embedding:8b, hash 64b933495768

Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

100

101

Sprites per Program

100

100

101

Scripts per Sprite

101

102

Interscript Cyclomatic Complexity

102

100

101

102

103

13

102

102

100

103

Blocks per Sprite

101

104

103

Sprite Name Length

102

Halstead Difficulty

103

Fig. 4. Sprite naming dataset structure.

The code below is written in the Scratch programming language. It is in the ScratchBlocks format which you know for example from the Scratch community forums. The code belongs to a single sprite in the program. I want to find a suitable name for this sprite. Suggest 10 names for the sprite. Only return the list of names with one name per line. ``` {{ source_code }} ```

Fig. 5. llm prompt used for the sprite naming task. source_code is replaced with the sprite’s code in ScratchBlocks format.

Unlike identifiers in most programming languages, Scratch sprite names can contain arbitrary Unicode text. To reduce the vocabulary sizes required in the machine learning models, we subtokenised and normalised the sprite names. We subtokenised the names by splitting on camel case boundaries, boundaries between letters and digits, whitespace, and punctuation characters. The subtokens were then normalised by removing punctuation and converting the characters to lowercase. Characters of writing systems which do not differentiate between upper- and lowercase variants remained unchanged. For embedding models that do not model the labels as subtokens but as single whole tokens (i.e. for ASTNN and code2vec), we concatenated the normalised subtokens back together with ‘|’ characters in between. 4.3.2 Model Training and Hyperparameter Search. We trained all non-pretrained models (cf. Section 4.2) on our sprite naming dataset. For each model, we used the training checkpoint with the lowest validation loss as the model for later evaluation. Since the llms are already pre-trained, no further training was required. Instead, we prompted the models (cf. Fig. 5) via the official api or the self-hosted Ollama endpoint and requested them to return ten suggestions for suitable names for the sprite. The Scratch code was integrated into the prompt in the ScratchBlocks format. In most cases the models responded with an itemised list we could extract the first ten names from. When the model generated fewer than ten suggestions, we excluded the sample from our evaluation to later be able to compute the top-10-accuracy. This filtering step removed between 4 (GPT-5-mini) and 1 269 (0.8 %, Mistral Small 3.1) responses. We focussed our hyperparameter search mainly around parameters that influence the structure of the input or the model itself. Since a full grid-search would have been too computationally expensive, we iteratively fixed all but one hyperparameter in each training run to narrow down the search space. For ASTNN , code2vec, and GGNN , an embedding size of 128 performed best while larger embedding sizes of 256 and 512 yielded the best results for code2seq and NeuralCodeSum, respectively (search space for all models: 128, 256, and 512). For code2vec and code2seq, increasing the ast path length (cf. Section 3.4) to larger values than 8 did not improve the results. We included path lengths up to Manuscript submitted to ACM

14

Benedikt Fein and Gordon Fraser

12 in our parameter search. Increasing the lengths further would have been too computationally expensive in the data preprocessing stage. The same length of 8 was also originally used on Java code [5, 7, 25]. Since using all possible ast paths as model input would be too computationally expensive, 200 random ones are sampled for each sprite. Neither halving nor doubling the number of paths improved the results, even when an increased maximum path length resulted in a larger number of paths per sprite. For ASTNN , we excluded all programs with a tree depth greater than 30 from the training dataset to limit the required computational effort during model training. We configured NeuralCodeSum to use 6 Transformer layers (search space: 4–12). Our GGNN model uses 12 graph convolution layers as encoder (search space: 4, 8, 12, 16). GGNN uses four lstm layers as decoder from embeddings into token sequences while code2seq uses a single lstm decoder layer (search space: 1, 2, 4, 6). 4.3.3 Evaluation. To evaluate the prediction performance, we used established machine learning metrics. We computed the top-𝑘-accuracy (i.e. one perfectly matching name suggestion within the top-𝑘 predicted names) and the F1score (macro-F1 for perfectly matching names) to compare whole predicted to original names. For a more fine-granular evaluation estimating how close the model prediction are to the original name even when not being a perfect match, we also considered multiple sub-token-based metrics. We report their mean values over the whole dataset as the respective overall score. The bleu score [60] takes into account how many of the subtokens of the original name also appear in the prediction. For rouge [48], we report the rouge-2-F1-score to additionally take into account how often bigrams of subtokens are correctly predicted and thus estimate how frequently the correct subtokens not only appear in the predicted name but also are in the correct order. Finally, we compute the meteor score [9] as implemented by the NLTK library.12 Since this implementation assumes an English text, advanced meteor score features like stemming and ignoring synonyms as defined by the WordNet [57] corpus might not be accurate for all data-points in our multilingual Scratch dataset. 4.4

RQ2: Influence of the Internationality of Scratch

4.4.1 Dataset. To investigate the influence of the natural language for RQ2, we made use of two datasets. Our first dataset is a subset of the sprites used for the evaluation of RQ1. This subset aims to simulate a setup similar to the method-naming machine learning task when applied to regular text-based programming languages. We only retained sprites the name of which exclusively contains spaces, digits, and characters of the core Latin alphabet (i.e. a to z in lowerand uppercase, excluding letters with diacritic marks like ä, and other letter variants like æ). This filtering removed approximately 5 % of the sprites for each of the three dataset splits. We applied the same sprite name normalisation steps as for the full sprite naming dataset of RQ1. To allow for a more fine-granular analysis, we created a second dataset by crawling additional projects not already part of the training dataset of RQ1. We required the projects to contain at least one sprite that contains more than 20 user-defined strings (i.e. variable names or string constants). Based on those user-defined strings and the sprite name we then employed the lingua-rs [73] tool to determine the natural language used by the programmer. The tool supports the detection of most languages also supported by the Scratch user interface.13 We only considered sprites where lingua-rs reported at least 90 % confidence in detecting the correct language and excluded the stage and sprites using the default name (cf. Section 4.3.1). Finally, the sprite name normalisation was applied as for our previously described datasets. We limit our evaluation to the ten most frequently occurring languages for which we could randomly sample 5 000 sprites 12 https://www.nltk.org/, accessed 2026-05-19 13 https://github.com/pemistahl/lingua-rs/tree/v1.7.2?tab=readme-ov-file#3-which-languages-are-supported, accessed 2026-01-15

Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

animations

Base Project ID

Project Category

games art music stories

Category

primary secondary

tutorials 0

2000

4000

6000

8000

Number of Projects

10000

15

63958014 40065008 63473670 37836116 114984085 45264192 259029179 109833197 113732481 114953272 37835434 109832775 78692754 171181805 89748886 169974487 171260950 37837664 88534923 37836992 135910010 58877462 65658138 183523664

12000

(a) Project Category

101

102

103

Number of Remix Projects

104

(b) Project Remixes

Fig. 6. Project classification dataset label distributions.

matching these criteria. While six of these languages are using the Latin alphabet, also Russian written in Cyrillic script, and the Chinese, Japanese, and Korean writing systems are represented in the resulting dataset. 4.4.2 Methodology. We evaluated all models used in RQ1 again on the two new datasets. When prompting the llms, we used the same prompt template as for RQ1 (cf. Fig. 5). Since the Latin-only subset of the sprite naming dataset is large enough to train the non-llm models specifically for this dataset, trained all models accordingly and report the performance of models trained on the latin-only subset. We used the same hyperparameters as in RQ1 to train the models. Since the language-specific datasets are too small for dedicated training, we used the models pre-trained for RQ1 and evaluated them without further fine-tuning. We limited our evaluation metrics on the language-specific datasets to the F1, bleu, and rouge-2 scores (cf. Section 4.3.3). Since the computation of the meteor metric assumes an English vocabulary, we excluded it for this experiment. 4.4.3 Statistical Analysis. Following the recommendations of Demšar [18], we used the Friedman test [29] with a significance level of 𝛼 = 0.05 to compare the models on the ten language-specific datasets. A Shapiro-Wilk test [68] showed that some per-model scores are non-normal. Thus, we report the median (md) and interquartile range (iqr) rather than mean values when comparing the results of different models. We used the Nemenyi post-hoc test [58] to declare any pair of models whose mean rank difference exceeds the critical distance (cd) as significantly different. To compute the effect size of differences, we used Vargha-Delaney’s 𝐴ˆ [78]. 4.5

RQ3: Fine-Tuning Sprite Embeddings for Program Classification

4.5.1 Datasets. Instead of labelling the programs per-sprite, the datasets used to answer this research question assign one or more labels to the overall program. Project Category. While many Scratch projects represent interactive games, there also exist projects that instead focus on presenting a story shown as non-interactive animations. Other projects showcase visual or musical arts, for example using the ‘Pen’ or ‘Music’ extensions provided by the Scratch website. To evaluate the models’ ability to classify whole projects according to such categories, we used a dataset of prior research [24] after excluding all projects that were already part of the training dataset of our RQ1 from the validation and test splits of this dataset. The dataset was created Manuscript submitted to ACM

16

Benedikt Fein and Gordon Fraser

by labelling 50 560 projects according to six different base categories (games, animations, art, music, stories, tutorials). Projects of the animations, games, and art categories are the most common ones with similar numbers of projects per category (cf. Fig. 6a). Since programs can fit into multiple categories, some of them have been assigned up to three labels. However, most projects (47 361) are labelled only with a single primary category. Project Remixes. As a second dataset, we used 24 starter projects provided by the RaspberryPi Foundation14 which are specifically designed to be used as a starting point for own projects when learning to program in Scratch. Each starting project is accompanied by a tutorial explaining steps to implement additional functionality. However, these tutorials also encourage users to be creative and extend the projects with additional or different features. Scratch users frequently share such ‘remixes’ publicly (cf. Section 4.1). Using the Scratch-api which links remixes to the base project, we collected a total of 34 856 remixes of the 24 starter projects (median md = 617 remixes per project). As Fig. 6b shows, some base projects are more popular than others, resulting in dataset imbalance. The most popular base project is ‘Boat Race’.15 During collection, we excluded projects containing identical code to the respective starter project. We again ensured that no projects are contained in both the training dataset split of RQ1 and the validation and test splits of this dataset. 4.5.2 Methodology. We used the ‘Project Category’ dataset for a multi-label classification task since each project has been assigned one or more category labels. For the ‘Project Remixes’, each remix was labelled by the starter project it was derived from to construct a multi-class classification task with a single label per project. In both cases, we use common machine learning metrics like the accuracy, precision, recall, and F1 score to quantify the prediction performance. Due to the dataset imbalance, we consider both the macro-average and class-weighted F1 scores. Since the project remixes of the second dataset should still have a somewhat similar structure to the starter project, we assume that the project embedding vectors as generated by the model should form clusters around the embeddings of the starter project. To evaluate whether clusters are formed in embedding space, we computed the Calinski-Harabasz Index [11]. To limit the required implementation and computational effort for the hyperparameter-tuning of fine-tuning the models used for different approaches, we only used the best-performing dedicated model of the previous research questions (GGNN ) and also the Qwen 3 Embedding [85] llm for the evaluation of this research question. As baseline evaluation, we specifically trained GGNN for the two project classification tasks. For the further evaluation aiming to find a suitable fine-tuning approach, we used GGNN as pre-trained for the sprite naming classification and Qwen 3 Embedding in its downloadable pre-trained state. To fine-tune GGNN , it was first converted into an embedding generation model by removing its decoder layers. By aggregating the individual sprite embeddings, we could construct an embedding for the whole program. Unlike for the sprite naming task, we included the stage and sprites independent of their name (i.e. even when they had a default name). Qwen 3 Embedding can be prompted with the sprite code in ScratchBlocks format (cf. Section 3.3) to directly obtain the embedding on which a decoder can be trained. For Qwen 3 Embedding, we derived two main variants: The first variant, ‘Qwen-per-sprite’, is similar to our GGNN approach and receives the ScratchBlocks input per sprite to generate sprite embeddings which can be aggregated. The second variant, ‘Qwen-whole’, receives the code of all sprites together as a single input to directly generate a program embedding. We constructed several approaches with which these base models can be fine-tuned for the classification task. For each approach, we tested various hyperparameters (113 configurations per classification task, of those 80 for GGNN 14 https://projects.raspberrypi.org/en/collections/scratch, Scratch user CodeClubRik https://scratch.mit.edu/users/CodeClubRik/, accessed 2026-06-05 15 https://scratch.mit.edu/projects/63958014, accessed 2026-07-16

Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

17

and 33 for Qwen 3 Embedding). We chose the configurations performing best according to their F1 score as the overall best ones. Dedicated Classification Decoder. This approach computes the individual sprite-embeddings of the project and applies a max-pooling to obtain the project representation. This overall project representation is then fed as input into a small trainable dedicated network that learns to predict the overall project label. We used a Multi-layer Perceptron (mlp) and a Support Vector Machine as implemented by the scikit-learn library16 as two alternative approaches to implement this prediction network. In case of the mlp, we tried configurations without hidden layers that map directly from model embedding size to the number of output neurons (e.g. configuration [128, 6] in case of GGNN for the project category task) up to configurations that contain hidden layers with sizes following decrementing powers of 2 (e.g. [128, 64, . . . , 8, 6]). Training and Fine-Tuning: mlp-Decoder. Instead of using the sprite embedding generator as a fixed model with frozen model weights, we additionally fine-tuned the encoder part of the overall new model. To fine-tune the model, we replaced the original decoder of the model with max-pooling and the mlp as described above. We then either trained the full model without freezing any layers or alternatively kept the encoding layers frozen for one training epoch to force the decoding layers to adapt to the encoder without causing the encoder to unlearn prior information. After this epoch of fine-tuning with the original learning rate (0.002), we reduced the learning rate by a factor of 10 and unfreeze the encoder weights to slowly fine-tune the whole model for the specific task. Finally, instead of fine-tuning the base model, we also used the alternative model structure with the replaced decoder to train new task-specific models. Due to the required computational capacity of tuning Qwen 3 Embedding, we applied this fine-tuning approach only to the GGNN base model. Training and Fine-Tuning: Sequential Sprite Embedding Aggregation. In the Scratch user interface, the user can explicitly sort the sprites into their preferred order. This order is then also reflected in the project json (cf. Section 3.1). The maxpooling step used by the fine-tuning approaches described above dismisses this information. To include it in the model, this approach instead applies recurrent bidirectional long short-term memory (lstm) model layers [40] to aggregate the individual sprite embeddings into the overall project embedding before feeding it into a fully-connected layer mapping the embedding into the predicted labels. Since the number of sprites per project is usually small (cf. Section 4.3.1 and Fig. 4), an lstm should be able to sufficiently retain the sequential information. During hyperparameter tuning we explored configurations having between one and four lstm layers. Optional dropout layers in between had a dropout probability of up to 50 %. Again, we used this model structure both for fine-tuning and dedicated training of GGNN model variants but did not fine-tune the Qwen 3 Embedding llm. We dismissed using a convolutional neural network (cnn) as an alternative for the lstm. Due to the typically low number of sprites per project, the matrices obtained by stacking the individual sprite embeddings would be very narrow in one dimension and thus often require padding to be able to apply the convolution. This would likely reduce the expressiveness of the resulting project embedding in these cases. 4.6

RQ4: Program Embeddings as Surrogate Model for Program Correctness

4.6.1 Dataset. To answer this research question, we used a dataset from prior work [66] containing three Scratch projects from a classroom setting (BoatRace,17 FruitCatching, and Spaceship). For each of the projects there is at least one 16 https://scikit-learn.org/1.8/index.html 17 https://scratch.mit.edu/projects/63957956/, accessed 2026-05-20

Manuscript submitted to ACM

Benedikt Fein and Gordon Fraser

Base Project

18

BoatRace FruitCatching Spaceship 0.0

0.2

0.4

0.6

0.8

#passed test cases / #test cases

1.0

Fig. 7. Whisker test results of the student projects.

bug-free model solution and multiple incomplete or faulty student solution attempts. Each base project is accompanied by a Whisker [17] test suite that verifies whether the required functionality is implemented correctly in the student projects. Most students implemented a meaningful proportion of the functionality (cf. Fig. 7). 4.6.2 Methodology. Using the Whisker test suites, we define the test fitness 𝑓test as measurement of the correctness of a project 𝑝. This measurement is given by the proportion of passed test cases in the test suite: 𝑓test (𝑝) =

|passed test cases for project 𝑝 | |all test cases of the test suite|

(1)

Alternatively, given a set of student projects 𝑃 for the same task (e.g. BoatRace) we estimate the project correctness for a student project 𝑝 ∈ 𝑃 using the model embedding embed(𝑝) ∈ R𝑛 and the embedding embed(𝑠) of the corresponding model solution project 𝑠 as 𝑓embed (𝑝) = 1 −

∥embed(𝑠) − embed(𝑝)∥ max𝑞 ∈𝑃 ∥embed(𝑠) − embed(𝑞)∥

(2)

to obtain the embedding fitness 𝑓embed that measures how far away a project 𝑝 is from the model solution 𝑠. The Euclidean distance between 𝑠 and 𝑝 in embedding space is normalised to be in the same interval [0, 1] as the test fitness. By subtracting this distance from 1, a fitness of 1 represents a program equivalent to the solution for both 𝑓test and 𝑓embed . The program out of the dataset 𝑃 furthest away from the solution receives an embedding fitness of 0. To obtain the model embeddings embed(𝑝), we used GGNN (𝑛 = 128) as the best model of our evaluation of RQ1 and RQ2. As a comparison to large pre-trained llms, we also computed embed(𝑝) with 𝑛 = 4096 using Qwen 3 Embedding [85]. Since the dataset of 41 student projects per model solution is too small to fine-tune the models to create task-specific whole-project embeddings, we used the same approach as in RQ3 and computed individual sprite embeddings using the GGNN (pre-trained on the sprite naming task) and Qwen-per-sprite models and aggregated the sprite embeddings into project embeddings using max-pooling (cf. Section 4.5.2). We again also used the Qwenwhole variant of Qwen 3 Embedding which receives the whole program as single input to directly generate a program embedding. Under the assumption that the model embeddings can be used as surrogate models to estimate program correctness, we should be able to observe a direct linear correlation between the actual correctness 𝑓test and the embedding distance 𝑓embed . To verify this, we computed the linear least-squares regression and Pearson’s 𝑟 for these two sets of measurements. 4.7

RQ5: Program Embeddings as Student Progress Indicator

4.7.1 Dataset. We used ScratchLog [12] to track the changes students made to their Scratch projects during an exercise class session (45 minutes including an introductory explanation). In this exercise, the students started from the basic BoatRace project solution (cf. Section 4.6.1) and should implement four additional features that change the gameplay. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

19

They were tasked with adding a rotating wooden plank the boat can crash against. The game should also be extended such that a randomly placed crab can be collected to obtain bonus points which are tracked in a newly introduced variable. On the game ending screen, the plank should be hidden and the crab should either also be hidden or it should congratulate the player when they reached at least five bonus points. These features thus are non-trivial and require more complex programming constructs such as loops, conditionals, and variables. We could verify the successful implementation of the requirements using a Whisker [17] test suite containing 11 test cases. Saving the state of the projects in 1 minute intervals, we collected a total of 1 088 projects across 48 students. 4.7.2 Methodology. To track the progress of students in a course using code embeddings, Paassen et al. proposed the progress-variance-projection [59]. Given the embeddings of the starter program and a model solution, it can map the student programs into two-dimensional space between these two states. By construction of the projection, the starting program is represented by the coordinates (0, 0) and the solution by (1, 0). Thus, the x-coordinate of a student program’s projection represents the current progress. The y-axis represents the variance within the student program population. I.e. if two student programs successfully implemented the same two features with a different approach, the projection should have an approximately identical x-coordinate but different y-coordinates. While the projection was originally used for code embeddings of Python programs [59], its computation is independent of the underlying embedding generator. Thus, we again used GGNN and the two Qwen 3 Embedding variants as in RQ4 (cf. Section 4.6.2) as pre-trained models without further fine-tuning to generate the Scratch project embeddings. Assuming the resulting projection still reflects the student progress in our Scratch setting like it did in the original projection evaluation on Python code [59], even though we now also use other embedding models, we should be able to observe a direct linear correlation between the test fitness 𝑓test as defined in RQ4 and the projected student progress on the x-axis. In both cases values close to 1 represent programs close to the solution. We verify whether this expected correlation exists using a linear least-squares regression and Pearson’s 𝑟 . 4.7.3 Statistical Analysis. Repeating the linear least-squares regression separately for the programs of each student, we obtained a population of 𝑟 2 coefficients we used for a statistical comparison between the models. We applied the same statistical tests as described in Section 4.4.3. 4.8

Threats to Validity

External. Threats to external validity arise from the sampling of projects for the model training datasets. Since the Scratch community tends to create programs reflecting recent events or popular media [34], existing datasets (e.g. [3]) as well as ours may be biased towards topics trending in the community during the dataset crawling. We mitigate this by sampling projects created over multiple years (between May 2021 and April 2023). A further threat to external validity may be the choice of machine learning models. We partially mitigate this by using a variety of model designs using different levels of abstraction from the program’s ast to estimate the influence of structural program information and by comparing to multiple open or proprietary llms. A threat to external validity remains, since all models used in our experiment assume a text- or ast-based input rather than using visual code information. The design of the Scratch workspace results in multiple related challenges for the application of visual models (e.g. scaling of the code area, overlapping blocks) which warrant dedicated future research on the applicability of such models to Scratch. The choice of sprite naming as an evaluation task highlights another threat to external validity, since by itself it is of limited relevance to the educational community. However, to train the machine learning models a suitably large dataset is required, which is not available for most educational tasks due to the usually small classroom sizes. We therefore use Manuscript submitted to ACM

20

Benedikt Fein and Gordon Fraser

Table 1. Model performance for the sprite naming task on the full dataset and on the dataset containing only names consisting of characters from the Latin alphabet. Best scores per dataset highlighted in bold. Dataset

Model

F1

top-10 acc.

bleu

rouge-2

meteor

Full

ASTNN code2seq code2vec DeepSeek Gemma GGNN GPT Mistral NeuralCodeSum

0.342 0.259 0.282 0.004 0.081 0.550 0.076 0.015 0.159

0.480 0.517 0.524 0.004 0.105 0.581 0.086 0.020 0.263

0.567 0.403 0.416 0.048 0.099 0.783 0.141 0.041 0.230

0.116 0.093 0.105 0.004 0.008 0.162 0.023 0.003 0.042

0.171 0.145 0.151 0.045 0.067 0.470 0.110 0.035 0.138

Latin Only

ASTNN code2seq code2vec DeepSeek Gemma GGNN GPT Mistral NeuralCodeSum

0.344 0.259 0.280 0.004 0.094 0.584 0.081 0.017 0.165

0.494 0.521 0.523 0.004 0.115 0.617 0.091 0.022 0.276

0.583 0.405 0.414 0.051 0.120 0.844 0.149 0.044 0.243

0.129 0.094 0.109 0.004 0.010 0.222 0.024 0.003 0.044

0.174 0.145 0.149 0.047 0.082 0.546 0.116 0.037 0.146

sprite naming as a pre-training task for which we can estimate the initial model performance, before applying the pre-trained models to education-focussed tasks (RQ4 and RQ5). Internal. A threat to internal validity may arise due to bugs in the model implementations. We try to mitigate this by re-using the implementations from prior work [25]. To adapt the models for our Scratch evaluations, only the preprocessing steps have been changed. By implementing the preprocessing as part of a shared tool for all models, we ensure a fair comparison between models. Construct. We mitigate a threat to construct validity by using existing model implementations used in prior research [25], thus reducing the risk of introducing additional bugs. Since the Scratch environment allows arbitrary text as identifiers for the sprites rather than the limited set of alphanumeric characters allowed in most programming languages (cf. Section 4.3.1), the subtokenisation approaches of the existing model approaches might not be optimal for Scratch. Instead, subword tokenisation using byte-pair encodings [67] or the SentencePiece tokeniser [46] might achieve better results. By choosing the existing model implementations we prioritise construct validity and accept this possible threat to external validity, since this paper aims to provide a first overarching evaluation of the applicability of code embeddings for Scratch rather than optimising any specific approach. Most tasks in our evaluation (RQ3–RQ5) also use the model embedding directly and are thus more independent of the chosen token en- and decoder approach. 5 5.1

Results RQ1: Using Code Embeddings to Name Sprites

When naming sprites based on the code within on the full dataset containing Unicode sprite names, the GGNN model works best (F1 = 0.550, cf. Table 1). Our results show a clear gap to the next best model ASTNN (F1 = 0.342), followed by a similar performance of the code2vec and code2seq models. The best model not using structural code information Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

21

is NeuralCodeSum (F1 = 0.159). None of the llms used in our experiment (DeepSeek Coder v2, Gemma 3, GPT-5-mini, Mistral Small 3.1) can reliably suggest the correct sprite name (F1 ≤ 0.081, top-10 accuracy ≤ 0.105). The low performance of all llms is somewhat surprising since llms have been successfully applied to Scratch, for example to offer coding support to students [15, 25, 27]. Like our experiments, these other approaches used the ScratchBlocks format or similar ones when prompting the llms. This suggests that the ScratchBlocks code might be similar enough to regular code (cf. Fig. 2b) to allow the llm to transfer learned information into the new domain. However, the coding support tasks generating natural language descriptions are by design more open-ended whereas the sprite-naming task requires the generation of a specific string. This might be more challenging for the llms, since unlike regular code, Scratch rarely uses custom procedures or variables. Instead, the most common user-defined textual program elements are string constants used for program events or exclamations of the animated sprites. Since the llms treat the code as a flat token sequence, these models might rely more on the names of variables and called methods when processing the code of regular programming languages. Since this information is rarely available on Scratch, the structure of the code becomes more important instead. Thus, we observe the gaps in performances between models which use a flat token-based representation of the code (llms, NeuralCodeSum), the models using an abstracted representation of the abstract syntax tree (code2seq, code2seq, ASTNN ), and GGNN which additionally uses control- and data-flow information. While the F1-score and top-10 accuracy only take perfectly matching full names into account, the bleu, rouge-2, and meteor metrics evaluate how frequently the predicted names are at least partially correct. The GGNN model performs best according to all evaluated subtoken-based metrics, again followed by ASTNN as second-best model. However, we observe a discrepancy in performance between high bleu scores and low rouge-2 scores for all models (e.g. GGNN : bleu = 0.783, rouge-2 = 0.162, ASTNN : bleu = 0.567, rouge-2 = 0.116). The bleu score only checks whether correct individual subtokens appear in the predict name, the rouge-2 score also checks for matching bigrams of tokens to evaluate whether they appear in the correct order. Since ASTNN and code2vec predict names as a whole instead of generating them as a sequence of subtokens, the low rouge-2 scores might be a result of the models generating an incorrect name which however generally matches the correct theme and thus shares subtokens with the original (i.e. resulting in a higher bleu score). The sprite names in Scratch can consist of arbitrary Unicode characters. Since this makes them potentially more complex than the method names of text-based languages, the sequence-generating decoders of the code2seq and GGNN models might not be sufficient for the new task. Thus, in case of these models the low rouge-2 scores might be a result of the decoder layers generating correct individual subtokens which are however in the wrong order or interspersed with incorrect tokens. A more advanced subtokenisation (e.g. byte-pair-encoding [67]) rather than the used natural splitting used in our implementation (cf. Section 4.3.1) could improve performance. Since even the best-performing GGNN model often suggested incorrect names and to further investigate the reasons for the bleu and rouge-2 score discrepancies, we manually inspected a subset of 100 random samples where GGNN predicted an incorrect name. In 55 cases the model predicted a name completely distinct from the original one. For 40 of those 55 samples the original name consists of a single token. In case the prediction was partially correct, reasons for incorrect predictions are missing subtokens (4 samples), or the prediction containing one or more incorrect subtokens (19 samples). However, the most common failure cause for partially correct predictions are repeated subtokens (22 samples). Since these repeated subtokens appear at least once in the original name, its first instance increases the bleu score whereas the repetition does not improve the rouge-2 scores due to non-matching bigrams. Our findings align with prior work on Java method naming using the same model implementation [25], highlighting that the decoder part of the GGNN model likely offers an opportunity for future improvements. Manuscript submitted to ACM

22

Benedikt Fein and Gordon Fraser While for some metrics the other model can achieve similar performance as GGNN , the model clearly outperforms the

others when considering the meteor score (GGNN : 0.470, next best: ASTNN with meteor = 0.171). Since the meteor metric mainly uses unigrams like bleu, the repeated subtokens of GGNN have a limited impact, especially since the meteor metric is weighted to favour recall over precision [9]. Since GGNN also can frequently produce a valid sequence of multiple subtokens before starting the repetition, the alignment-penalty of the meteor metric likely remains small for many samples. Since ASTNN predicts full names rather than sequences of subtokens, the alignment-penalty might be higher in case of non-perfect matches. As indicated by its bleu score of 0.403, the second-best sequence-generating model, code2seq, likely does not achieve a high meteor score (0.145) since it less often suggests relevant subtokens. Summary RQ1: Models integrating more structural information about the code achieve a better performance. Since Scratch code rarely uses user-defined identifiers, models using flat code representations may have more difficulty inferring the code’s intention. Further improvements can likely be achieved by more advanced name subtokenisation or decoder approaches. 5.2

RQ2: Importance of Natural Language for the Sprite Naming Task

Reducing the model training and evaluation datasets to sprites with names that only contain characters of the core Latin alphabet shows no clear difference in model performance compared to the full Unicode dataset (cf. Table 1). The GGNN model still performs best and also the pattern of increasing model performance with increasing structural information used in the model remains unchanged. While there is a slight improvement for all metrics compared to the Unicode dataset (e.g. GGNN F1 = 0.550 vs. 0.584 and bleu = 0.783 vs. 0.844), this can be most likely attributed to the smaller vocabulary size. The subtokenisation process and model decoder designs (cf. Section 5.1) likely remain as the main factors limiting model performance. Evaluating the models on dedicated datasets per natural language used by the programmer can give more insights into the impact of the natural language on model performance. As Table 2 shows, again ASTNN and GGNN perform best in nearly all cases. Only for the programs identified as Chinese, the code2seq model achieves the highest F1 score (0.207, ASTNN : 0.195). Even though the projects of the per-language datasets were specifically chosen for containing multiple user-defined natural language strings and identifiers, the llms can only rarely predict accurate names. Using the 10 languages as population, a Friedman test [29] shows there are significant differences between the metrics of the nine models (𝑝 < 10−11 for all three metrics). We then apply a post-hoc Nemenyi test [58] with cd = 3.80. For all three metrics, the GGNN , ASTNN , code2seq, GPT , and code2seq models form the best-performing group without significant differences in between (listed by increasing mean rank, lower rank is better; same ranking across all three metrics). The other models (DeepSeek Coder v2, Gemma 3, Mistral Small 3.1, and NeuralCodeSum) form a second group with mean rank orders varying by metric. Since the GGNN and ASTNN models achieve the highest scores in nearly all cases, and since they are consistently have the best mean ranks, the following evaluation focusses on these two best-performing models. Both ASTNN and GGNN perform best on English-language programs, which is to be expected since it likely is the most common natural language in the model training dataset. Nevertheless, both models perform worse than on the general sprite naming dataset used for RQ1. While the English-only dataset consists of sprites with fewer blocks than the general dataset (MD = 24, IQR = 43; cf. Section 4.3.1), these blocks are combined into more complex programs (cyclomatic complexity: MD = 19, IQR = 32). This might be a result of our dataset creation process specifically selecting sprites that contain multiple user-defined strings as part of string constants, messages, or variable names. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

23

Table 2. Model performance for projects using different natural language strings. Best scores per language highlighted in bold.

Language

Metric

ASTNN

code2seq code2vec

DeepSeek

Gemma

GGNN

GPT

Mistral

NeuralCodeSum

Chinese

F1 bleu rouge-2

0.195 0.274 0.012

0.207 0.221 0.002

0.192 0.199 0.001

0.004 0.006 < 10−3

0.015 0.022 < 10−3

0.155 0.178 0.010

0.069 0.078 0.001

0.011 0.015 < 10−3

0.171 0.169 < 10−3

English

F1 bleu rouge-2

0.250 0.455 0.102

0.147 0.308 0.083

0.117 0.224 0.076

0.004 0.071 0.007

0.064 0.140 0.018

0.370 0.659 0.150

0.079 0.208 0.054

0.013 0.062 0.007

0.051 0.106 0.025

French

F1 bleu rouge-2

0.143 0.277 0.038

0.073 0.133 0.024

0.047 0.079 0.024

0.002 0.046 0.005

0.038 0.072 0.005

0.262 0.397 0.035

0.056 0.139 0.032

0.010 0.034 0.005

0.011 0.026 0.004

German

F1 bleu rouge-2

0.156 0.268 0.039

0.066 0.124 0.013

0.045 0.062 0.006

0.003 0.044 0.004

0.051 0.099 0.009

0.243 0.397 0.067

0.068 0.153 0.030

0.013 0.039 0.003

0.015 0.027 0.003

Japanese

F1 bleu rouge-2

0.194 0.271 0.013

0.116 0.159 0.008

0.115 0.135 0.006

0.003 0.009 < 10−3

0.015 0.023 0.001

0.187 0.237 0.019

0.049 0.070 0.004

0.004 0.010 < 10 −3

0.039 0.051 0.003

Korean

F1 bleu rouge-2

0.103 0.161 0.013

0.026 0.050 0.005

0.019 0.027 0.003

0.005 0.018 < 10−3

0.013 0.021 < 10−3

0.124 0.169 0.011

0.032 0.003 0.063 0.009 < 10−3 < 10−3

0.006 0.007 < 10−3

Portuguese F1 bleu rouge-2

0.207 0.320 0.067

0.109 0.167 0.028

0.076 0.106 0.030

0.005 0.051 0.009

0.080 0.104 0.009

0.259 0.402 0.075

0.107 0.180 0.045

0.022 0.043 0.004

0.019 0.044 0.014

Russian

F1 bleu rouge-2

0.110 0.216 0.014

0.077 0.100 0.004

0.056 0.073 0.002

0.005 0.022 < 10−3

0.013 0.020 < 10−3

0.142 0.192 0.032

0.055 0.080 0.001

0.005 0.012 < 10−3

0.030 0.038 < 10−3

Spanish

F1 bleu rouge-2

0.190 0.321 0.066

0.071 0.138 0.024

0.052 0.081 0.021

0.003 0.045 0.003

0.057 0.087 0.005

0.286 0.455 0.077

0.083 0.163 0.034

0.013 0.034 0.003

0.016 0.029 0.003

Turkish

F1 bleu rouge-2

0.126 0.263 0.055

0.046 0.092 0.014

0.035 0.043 0.008

0.003 0.034 0.006

0.032 0.045 0.004

0.192 0.290 0.055

0.053 0.099 0.031

0.006 0.021 0.003

0.012 0.019 0.001

Especially variables are likely used by more advanced programmers or in more complex programs. The increase in complexity is also reflected in an increase in the Halstead difficulty metric (MD = 13.7, IQR = 21.1), which is computed based on the number of operators and operators used in the code [37]. When comparing the F1 scores (i.e. considering perfect name suggestions) both ASTNN and GGNN perform worst for Korean and Russian programs. These two languages likely appear as part of the 10 languages sampled for this research question, since their fairly unique writing systems allow the lingua-rs tool [73] to confidently recognise them. However, the models likely exhibit low performance since there are comparatively few Scratch users living in the Manuscript submitted to ACM

24

Benedikt Fein and Gordon Fraser

Table 3. Model performance for the program category and remix classification tasks. The Calinski–Harabasz Index (chi) is computed on the respective program embeddings. Best scores highlighted in bold.

Task

Model

Model Variant

Aggregation Classifier

Acc. Prec. Rec.

F1

F1 (weighted)

CHI

Category GGNN dedicated max pool GGNN fine-tuned lstm GGNN sprite embedding max pool Qwen 3 sprite embedding max pool Qwen 3 program embedding none

MLP 0.498 0.696 0.388 0.440 MLP 0.509 0.671 0.411 0.475 Random Forest 0.297 0.720 0.252 0.353 MLP 0.493 0.610 0.523 0.562 MLP 0.517 0.713 0.460 0.538

0.555 0.574 0.416 0.613 0.613

35.774 25.232 10.346 23.140 13.548

Remixes GGNN dedicated max pool GGNN fine-tuned max pool GGNN sprite embedding max pool Qwen 3 sprite embedding max pool Qwen 3 program embedding none

MLP 0.980 0.844 0.830 0.837 MLP 0.986 0.861 0.843 0.849 Random Forest 0.972 0.930 0.899 0.913 MLP 0.988 0.890 0.885 0.887 MLP 0.986 0.888 0.877 0.881

0.980 0.985 0.970 0.986 0.984

61.838 55.743 199.910 148.006 272.566

respective countries (Russia: 0.78 % of all users, North Korea: 0.11 %, South Korea: 0.03 %18 ) and thus the model training dataset only contains few samples in these languages. GGNN performs best on the six languages using a Latin-based writing system, whereas for ASTNN the bestperforming English and Portuguese datasets are followed by Chinese and Japanese. In case of GGNN , the Latin-based languages are likely the most compatible with our subtokenisation process (cf. Section 4.3.1). For the Chinese, Japanese, and Korean writing systems these whitespace- and capitalisation-based subtokenisation rules may often not apply. Subtokens might also be more diverse due to a larger number of available characters. Consequently, there may be only few examples of each subtoken in the training dataset and thus the model cannot produce valid subtoken sequences. This is highlighted by the difference in bleu score for Chinese programs between GGNN (0.178) and ASTNN (0.274). Using a byte-pair-encoding [67] or SentencePiece tokenizer [46] to subtokenise the sprite names instead could likely improve the performance. ASTNN on the other hand uses the whole name as a single token and might thus require fewer samples in the training dataset to exhibit the better performance for Chinese and Japanese. Summary RQ2: As expected, the models perform better on more common natural languages. A more advanced subtoken encoding should be used to improve the performance for languages not using Latin-based writing systems. 5.3

RQ3: Sprite Embeddings for Program Representation

5.3.1 Results. For this research question we not only fine-tune a sprite-embedding model to evaluate the classification performance, but also investigate the expressiveness of the underlying code embeddings. To limit computational effort, we continue the evaluation only with the best-performing sprite-naming model (GGNN ) and two Qwen 3 Embedding-based variants. Program Classification. For the project category classification task, fine-tuning the GGNN sprite embedding model by adding an lstm aggregation layer followed by a Multi-layer Perceptron (mlp) is the best approach. However, as Table 3 shows, the GGNN model which was specifically trained from scratch for this task performs nearly as well (GGNN tuned: F1 = 0.475, from scratch: F1 = 0.440). When using the encoder layers of the sprite naming model as fixed sprite embedding generators without further fine-tuning, a Random Forest rather than an mlp performs best, but still achieves 18 https://scratch.mit.edu/statistics/, accessed 2026-05-19. The Scratch website allows users to select their country of residence rather than using automatic

geolocation. The actual number of users from North Korea is likely lower. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

25

an overall lower performance (F1 = 0.353) than the other two GGNN variants. Surprisingly, even though the llms were not able to suggest useful sprite names, in this task both Qwen 3 Embedding model variants outperform all GGNN variants. When using the whole program text as single input to the embedding model, it achieves an F1 score of 0.538 and using separate per-sprites inputs followed by max-pooling the sprite embedding even performs slightly better (F1 = 0.562). This performance difference is equalised by considering the class-weighted F1 score (both variants 0.613). The accuracy of the Qwen 3 Embedding-based classifiers (0.493 and 0.517) is similar to that of the best GGNN variant (0.509), but the Qwen 3 Embedding-based model shows a better balance between precision and recall and thus achieves an overall better F1 score. We conjecture that the llm-based classifiers work well for this task since in some program categories the programs are dominated by specific kinds of blocks. For example, ‘animations’ frequently feature blocks relating to the sprite looks but rarely contain blocks handling key or mouse input events. ‘Games’ on the other hand often feature the latter kind of blocks. The information about the kinds of used blocks is also part of the flat ScratchBlocks token sequence given to the llm. Thus embedding structural code information might be less important for this task. For the project remix classification task again using a max-pooling sprite embedding aggregation followed by an mlp classifier works best for most base-model and tuning approaches. All model variants can accurately classify which base project a remixed project was derived from, resulting in minimal differences in weighted F1 scores between the models (range 0.970 to 0.986). Due to the overall high scores and closeness of the scores, no clear best model can be determined. However, the GGNN -based classifier with fixed sprite embeddings and a Random Forest classifier seems to perform best also for classes with fewer samples since it outperforms the other model variants according to the non-weighted F1 score. Since the models can correctly classify most remixes, these results show that all model variants are robust against small changes in the program code. For both the program classification and remix tasks, the relatively simple max-pooling sprite embedding aggregation is the best configuration. We hypothesise that the more complex lstm aggregation does not result in better model performance, since the number of sprites in the Scratch programs is generally quite small (median: 2, cf. Section 4.3.1), and thus the benefits from a sequential lstm-based aggregation are limited. In some cases the majority of the code might also be placed in few of the sprites whereas the other sprites act as static background decoration without own logic. Embedding Expressiveness. Since the classifiers seem to be robust against small changes in the inputs, their embedding vectors potentially also form clusters of close vectors for similar inputs. However, in case of the project category task the embeddings of the specifically trained GGNN classifier achieve the highest, i.e. best, Calinski-Harabasz-Index (chi) [11] even though the Qwen 3 Embedding-based models performed best for the actual classification task. In case of the project remix task, the Qwen 3 Embedding program embeddings achieve the highest scores followed by the aggregated but not fine-tuned GGNN sprite-embeddings. For the project remix task, the Qwen-based model likely produces better clusters since the main purpose of such a pre-trained embedding model is to create similar embeddings for semantically similar inputs. The remixed projects are likely also often syntactically similar to the original project. In case of the project categorisation, the clusters of the Qwen 3 Embedding might not represent the categories relevant for the task but instead others learned from the training data. For example, programs relating to characters in the Nintendo universe (e.g. Mario, Zelda, Donkey Kong) might be clustered due to similar user-defined strings appearing in the project, even though some of these projects are games while others are non-interactive animations. The GGNN Manuscript submitted to ACM

26

Benedikt Fein and Gordon Fraser

100

50

50

0

0

y

y

100

50

50

100

100 100

animations

art

50

0

50

100

games

music

stories

tutorials

x Main Project Category

(a) GGNN

100 animations

art

50

0

50

100

games

music

stories

tutorials

x Main Project Category

(b) Qwen 3 Embedding

Fig. 8. Project Category: Visualisation of the embedding space after t-SNE dimensionality reduction.

classifier on the other hand is specifically trained on this task, so its clusters are more likely to align with the labels learned from the training dataset. By using a t-SNE projection [77] to reduce the high-dimensional embedding vectors to two dimensions, the model embedding spaces can be visualised. Figure 8 shows the embedding projection for the project category task for the— according to the chi—best-performing GGNN and Qwen 3 Embedding model variants. Both figures show the same number of projects, but in case of GGNN the reduction to two dimensions results in many overlapping data points whereas they are more spread out for Qwen 3 Embedding. As the visualisation of both model embedding spaces shows, this multi-class classification task might be inherently more difficult for a classifier model since many categories overlap. For example, it might be hard to discern whether a program is mainly a game that features some animations, or whether it primarily focusses on the animation aspect and contains some interactive elements. This difficulty is reflected by the low chi clustering scores for all model variants (cf. Table 3). Even though the project categories do not form clear clusters, there seem to be differences in the representations in the project categories. For example, in case of Qwen (cf. Fig. 8b), the projects of the ‘animations’ and ‘games’ categories are similarly distributed on the 𝑥-axis, but clearly different on the 𝑦-axis. For the project remix task on the other hand, the projection of the Qwen 3 Embedding embedding space (cf. Fig. 9b) shows a clear separation of clusters. For the GGNN model with a lower chi (199.910 compared to 272.566 for Qwen), most of the clusters remain compact, but some are more spread out, e.g. the green one on the left-hand side (𝑥 ≊ [−70, . . . , −30], 𝑦 ≊ [−20, . . . , 60], cf. Fig. 9a). Additionally, the clusters are no longer as clearly separated. Nevertheless, the visualisations for both models highlight that the overall classifier models are not only robust against small changes to the inputs, but also that similar inputs yield similar embedding vector representations. At the same time, they are sufficiently expressive to map unrelated code into separate clusters. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

27

80

60

60

40

40

20

20

y

y

0 20

0 20

40

40

60

60

80 80

60

40

20

x

0

20

40

(a) GGNN

60

80

60

40

20

0

x

20

40

60

80

(b) Qwen 3 Embedding

Fig. 9. Project remixes: Visualisation of the embedding space after t-SNE dimensionality reduction. Colouring based on the base project.

Summary RQ3: Fine-tuning the pre-trained sprite-embedding models for a new classification task can achieve similar performance as task-specific models. Both the overall classifier models and the embedding vectors are robust against small changes in the program code. The embeddings can be used to form clusters of related programs. 5.3.2 Discussion. Our results show that especially approaches combining the pre-trained sprite embeddings as input for a small task-specific classifier model allows for a computationally efficient adaption of the models to a new task. This lowers the required computational effort when designing new embedding-supported tools. This confirms that such models are well suited for downstream educational tasks. For example, an immediately possible educational application for the clustering of similar projects could be the recommendation of Scratch projects based on the learner’s skills or preferences to encourage them to iteratively explore more challenging programming concepts [62]. 5.4

RQ4: Program Embeddings as Surrogate Model for Program Correctness

5.4.1 Results. The previous research question demonstrated that the embeddings encode relevant information about the programs. In this research question we now evaluate whether this can be used to predict the correctness of a program, even when the models are not fine-tuned for the specific task due to the limited amount of available training data. In our evaluation we observe a statistically significant linear correlation (Pearson’s 𝑟 ) between the test fitness (i.e. true correctness as measured by the test suite) and the embedding fitness for most base-project and embedding model combinations. The slopes < 1 in most cases (cf. Fig. 10) suggest that the embedding fitness tends to underestimate the true functional correctness of the programs. The test fitness correlates strongest with the embedding fitness of the GGNN model for the BoatRace project (𝑟 2 = 0.45, 𝑝 < 10−3 , cf. Fig. 10a). For the Spaceship project on the other hand, the Qwen-per-sprite embeddings perform best (𝑟 2 = 0.49, 𝑝 < 10−3 , cf. Fig. 10c). While the GGNN model performs best for FruitCatching and still exhibits the Manuscript submitted to ACM

28

Benedikt Fein and Gordon Fraser FruitCatching

1.0 y = 0.84x - 0.08

r² = 0.13, p = 0.018

Spaceship

r² = 0.33, p = 0.010

embedding fitness

0.9 0.8 0.7 0.6

0.8 0.6 0.4

0.5 0.8

test fitness BoatRace

r² = 0.37, p = 0.003

0.9

1.0

0.5

0.6

0.7

0.8

test fitness

0.9

1.0

0.6

0.7

0.8

test fitness

0.9

1.0

0.9

1.0

0.9

1.0

(a) GGNN per-sprite embeddings followed by max-pooling aggregation. FruitCatching Spaceship 1.0 y = 0.43x + 0.08 1.0 y = 0.82x - 0.10 r² = 0.10, p = 0.039 r² = 0.44, p = 0.002 0.9

embedding fitness

embedding fitness

1.0

1.0 y = 0.29x + 0.47

0.9 0.8 0.7

0.8

embedding fitness

1.1 y = 0.68x + 0.31

BoatRace

embedding fitness

embedding fitness

y = 1.11x - 0.20

1.0 r² = 0.45, p < 1e-3 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.6 0.7

0.6 0.4

0.8 0.7 0.6 0.5 0.4

0.6 0.6

0.7

0.8

test fitness

0.9

0.2

1.0

0.5

0.6

0.7

0.8

test fitness

0.9

0.3

1.0

0.6

0.7

0.8

test fitness

(b) Qwen 3 Embedding with the whole projects’ ScratchBlocks code as single input. BoatRace FruitCatching Spaceship 1.1 y = 0.62x + 0.32 1.0 y = 0.32x + 0.26 1.0 y = 0.84x - 0.00 r² = 0.21, p = 0.033 r² = 0.08, p = 0.069 r² = 0.49, p < 1e-3 0.9 0.8 0.7

0.9

0.8

embedding fitness

embedding fitness

embedding fitness

1.0

0.6 0.4

0.7

0.8

test fitness

0.9

1.0

0.7 0.6 0.5

0.6 0.6

0.8

0.5

0.6

0.7

0.8

test fitness

0.9

1.0

0.4

0.6

0.7

0.8

test fitness

(c) Qwen 3 Embedding with the individual sprites’ code as inputs followed by max-pooling aggregation. Fig. 10. Linear least-squares regression between test and embedding space distances between student projects and example solution.

expected linear correlation between embedding and test fitness (𝑝 = 0.018), the linear relation between these two fitnesses is weaker (𝑟 2 = 0.13). The correlation for the FruitCatching project may be weaker since the project allows for alternative solutions. Therefore, while some student programs may be nearly correct (i.e. high test fitness), they may still be structurally different to the example solution (i.e. low embedding fitness). For the BoatRace model, the GGNN embeddings may outperform the two Qwen 3 Embedding variants due to the sprite naming pre-training task. Since the BoatRace project is based on a publicly available project published by the CodeClubRik Scratch user (cf. Section 4.5.1), our sprite naming training dataset likely contains similar programs. Spaceship and FruitCatching on the other hand are not based on publicly shared base-projects. Therefore, our GGNN model may be already somewhat familiar with the structure of BoatRace-like programs whereas it is unlikely to Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

29

have seen the other two projects before. Nevertheless, the embedding fitness of the GGNN model still correlates with statistical significance for the unseen projects (cf. Fig. 10a), demonstrating that the embeddings captured generalising semantic and structural information about the programs during the pre-training on the sprite-naming task. The difference in results between BoatRace and Spaceship may be explained by their program structure. In principle, in both projects a vehicle follows the mouse pointer through a labyrinth-like scene towards a finish. However, for BoatRace the majority of the program’s code is part of the ‘Boat’ sprite. In Spaceship a similar amount of code is spread out over the three sprites. For the per-sprite embedding aggregation of BoatRace, the nearly empty sprites might result in embeddings containing little information which however overrides the information from the Boat sprite in the max-pooling step. Therefore, Qwen-whole performs better than the aggregated per-sprite embeddings (whole program embedding: 𝑟 2 = 0.37, per sprite: 𝑟 2 = 0.21, cf. Figs. 10b and 10c). Due to the better distribution of the code in the Spaceship project, the individual sprite embeddings likely contain similar amounts of information and thus retain more of it during the aggregation, resulting in a similar performance for the two Qwen 3 Embedding variants (whole program: 𝑟 2 = 0.44, per sprite: 𝑟 2 = 0.49). Summary RQ4: We observe a statistically significant correlation between true test fitness and embedding fitness for most project/embedding model combinations. This demonstrates that the embeddings capture semantic and structural information about the programs in the pre-training task. This information is applicable to other tasks without further fine-tuning. 5.4.2 Discussion. The results for this research question demonstrate that the pre-trained sprite naming embeddings can be applied to tasks unrelated to the pre-training even in cases where task-specific fine-tuning is not feasible, for example due to a limited dataset size. This suggests that the development of future embedding-supported educational tools could be eased by using pre-trained embeddings rather than having to train task-specific models. For example, the specific task of the embedding-based program correctness estimation could also be integrated into part of automatic program repair tools since the repeated test suite execution tends to represent a considerable proportion of the overall repair time [66]. Similarly, combination of semantic and structural similarity between projects could allow for the integration as part of a next-step hint system guiding students towards an exercise solution. Current hint systems tend to rely on only the structural differences between student program and example solution [26]. 5.5

RQ5: Program Embeddings as Student Progress Indicator

5.5.1 Results. As demonstrated in RQ4, the embeddings can be used to estimate whether a program is functionally similar to an example solution. While this already could give some indication of the progress in an exercise, the progressvariance-projection [59] (cf. Section 4.7.2) is specifically designed to map the embeddings into a two-dimensional representation where the x-axis represents the progress between the starting project at point (0, 0) and the example solution at (1, 0). A linear least-squares regression between the progress component of the projection and the test fitness of the programs confirms that the projection indeed shows the actual functional progress (𝑝 values for all models ≪ 0.05). The linear model fits worst for the Qwen 3 Embedding with max-pooled sprite embeddings (‘Qwen-per-sprite’, 𝑟 2 = 0.436), but considerably better for Qwen 3 Embedding with the full program as input (‘Qwen-whole’, 𝑟 2 = 0.714) and for GGNN (𝑟 2 = 0.735). Repeating the linear least-squares regression separately for the programs of each student, we obtain a population of 𝑟 2 coefficients we can use for a statistical comparison between models. A Friedman test [29] shows Manuscript submitted to ACM

30

Benedikt Fein and Gordon Fraser

that there is a significant difference between populations (𝑝 = 9.01 · 10−9 ). A post-hoc Nemenyi test [58] shows no significant difference (cd = 0.524) between GGNN and Qwen-whole. However, both models perform significantly better than Qwen-per-sprite with large Vargha-Delaney effect size [78] (GGNN compared to Qwen-per-sprite: 𝐴ˆ = 0.813, Qwen-whole compared to Qwen-per-sprite: 𝐴ˆ = 0.809). The correlation between test fitness and progress is highlighted in Figure 11. The scatter plot of all student programs shows that the number of passed tests generally increases as the programs progress closer towards the solution. The large jump in progression between projects close to the start to progress values > 0.5 for Qwen-per-sprite might explain the lower 𝑟 2 coefficient for this model. This gap also makes the projection less useful in practice, since the progressions for both shown example students jump from the start state to a point already fairly close to the solution. For the Qwen-whole variant and GGNN , both students progress through multiple intermediate steps. The y-axis of the projection shows the variance and is intended to highlight differences in programs with similar progression. Outliers on the y-axis can indicate students no longer following the task. A useful embedding projection should therefore map to a sufficient range of variance values. The projection based on the GGNN model exhibits the highest interquartile range (iqr = 0.021 3) for the variance axis. For both Qwen 3 Embedding variants, the variance values are an order of magnitude smaller (whole program: iqr = 0.003 23, aggregated: iqr = 0.002 91). In practice this is not relevant however, since a visualisation similar to Fig. 11 can be scaled accordingly. More relevant in practice is the overall spread of values on the variance axis. For GGNN the range is 9.56 · iqr and Qwen-whole-program has a similar range of 8.74 · iqr. When defining outliers as all points positioned 1.5 · iqr below/above the first/third quantile, the projection of GGNN embeddings results in 88 outliers, whereas for Qwen-whole 50 outliers exist. Summary RQ5: Our results show that the pre-trained GGNN sprite naming embeddings capture relevant transferrable program information that can be used to visualise the student progress without requiring further fine-tuning. In case no specifically trained Scratch model is available, an embedding-llm can be used instead with comparable performance. 5.5.2 Discussion. In practice, not only the quality of the projection may be relevant but also the computational effort required to generate it. Once the GGNN model has been pre-trained, embeddings can be computed locally on a machine equipped only with a cpu. Due to the large model size, computing the llm-based embeddings on the other hand is computationally more expensive and requires a gpu to support the near-real-time computations required to enable in-classroom support for teachers. While hosted llms accessible via a remote api are available, such apis incur costs based on the usage and data protection restrictions might apply. Our results demonstrate that both approaches of using either a dedicated small model or a pre-trained llm can be used for the task with similar results, making the approach more widely accessible by allowing alternative implementations based on the individual compute resource availability. 6

Related Work

The main related areas of research are either machine learning based on visual aspects of code written in textual or visual programming languages, or code embeddings specifically designed for visual programming languages. 6.1

Machine Learning Based on Visual Aspects of Code

Most code embedding models designed for textual programming languages receive the code in various textual (i.e. flat token sequence) or structural formats (i.e. ast, graphs). To understand whether such established models can be applied Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch

GGNN

variance

0.1 0.0 0.1 0.2

0.0

0.5

progress

variance

0.02

1.0

0.0

0.5

1.0

progress

0.0

Qwen 3 Embedding (full project #passed tests as single input)

0

1

2

3

4

5

6

7

8

9

0.5

1.0

0.5

1.0

0.5

1.0

progress

10

0.01 0.00 0.01 0.02

0.0

0.5

progress

0.02

variance

31

1.0

0

0.0

0.5

1.0

progress

0.0

tests Qwen 3 Embedding#passed (sprite embedding aggregation) 1

2

3

4

5

6

7

8

9

progress

10

0.00 0.02 0.0

0.5

progress

1.0

0

0.0

1

2

0.5

1.0

progress 3

#passed tests 4

5

6

7

0.0

8

9

progress

10

Fig. 11. Progress-variance-projection of student program snapshots for the whole class captured every minute. The example student shown in the middle followed a direct path towards the solution with some trial-and-error at the beginning, the middle, and towards the end. The student shown on the right got stuck and did not finish the tasks.

in the Scratch context, we therefore ignored the visual aspects of the block-based programming language in this paper. Nevertheless, some embedding models for textual programming languages include visual information, either as addition to the textual representation [56] or as replacement [14, 43, 44]. Using the notion of a grid-like layout of a source code file, the bounding-boxes of potentially vulnerable code areas can be determined [47]. The grid-like representation has also been used to transform the source code into images by replacing the characters with pixel colours based on the ascii value [14]. To further abstract from the concrete syntax, the inputs for the CodeGrid [43] and WySiSym [44] code clone and vulnerability detection models are image with colours based on the kind of the token (e.g. identifier, keyword, literal). A similar colouring approach has also been employed for automated code readability analysis [56]. Even though the kinds of blocks in Scratch already determine a natural colouring of the pixels (e.g. green for expressions, blue for blocks relating to the sprite movement), the grid-like image generation approaches cannot be directly applied to Scratch since its workspace allows users to freely place blocks in two-dimensional space independent Manuscript submitted to ACM

32

Benedikt Fein and Gordon Fraser

of a grid. While screenshots of the code could be used instead, additional challenges like image scaling and the possibility of overlapping blocks need to be solved. Due to the additional challenges, we did not investigate the impact of including the visual aspects of Scratch code for code embedding models in this paper, but future research should explore this area.

6.2

Code Embeddings for Visual Programming Languages

Other initial work on code embeddings for Scratch focusses on a purely textual or structural representation of the code, even though they do not yet use it to prompt llms. For example, the ScratchRec tool [62] uses a tokenised representation of the program to create a program embedding which can later be used to suggest similar projects to learners. A tokenisation approach similar to our NeuralCodeSum input format (cf. Section 3.2) has been used to create a code completion model for Scratch [35]. One publication hints at using screenshots of Scratch code to detect bug patterns [13], but offers little detail about processing steps or the used machine learning model. All of these existing works use a specific code embedding model to solve the task at hand, rather than comparing the performance of different models. The main focus of recent work on code embeddings for block-based programming languages seems to be the conversion of the code into a flat textual representation so that it can be fed into an llm. For example, the ChatScratch tool [15] allows students to get coding guidance by automatically including the code in a syntax similar to ScratchBlocks in the prompts. The llm ’s output can be parsed back into Scratch blocks by heuristically matching similar block types [15]. The Scratch Copilot [20] extends the Scratch user interface with a similar llm-based coding help, but the publication does not specify which code representation was used. A similar llm-based extension to the Scratch user interface allows for the generation of quizzes to encourage students to think about their code [21]. While the latter two publications hint at including the code in textual form into the prompt, they do not specify the representation further. Other works fully adopted the ScratchBlocks format both as input and expected output format to or from the llm [27, 23] to ask the llm to explain current issues in the code and suggest possible fixes. Subsequently, we adopted the ScratchBlocks format when prompting llms in this paper (cf. Section 3.3). The Stitch conversely uses the Scratch-internal json representation as inputs for an llm-based next-step hint generator, but does not require the llm to produce valid jsons as output [70]. Since some llms are multi-modal, i.e. they can accept text and images as inputs, some recent work uses this feature to create education-focussed tools for Scratch. For example, ScratchEval [30], uses images of Scratch code and a multiplechoice question that requires reasoning about maths, logic, or graphics and use the llm to determine the solution. Similar to our RQ2, they evaluated the impact of the natural language and found that images with descriptions in Chinese resulted in roughly 2–3 % worse performance compared to English [30]. The VisionScratch tool [69] aims to generate feedback for learners by feeding the code and a video recording of the gameplay rather than the code into an llm. Scratch seems to be the most common block-based programming language which machine learning approaches are applies to. While the similar Snap! language is popular in educational contexts, no code embeddings have been designed for it to the best of our knowledge. Embedding models have also been applied to the Simulink language by transforming the graphs into flat textual representations to generate the inputs for an llm [50, 84]. Even though Simulink is a visual domain-specific language, it is considerably different from Scratch since the graph primarily represents data flow between physical components and system states rather than iterative statements. Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch 7

33

Conclusions

In this paper we explored in an initial study how established code embedding models can be applied to Scratch. Our results for the sprite-naming task (RQs 1 and 2) clearly indicate that more structural information about the program results in an improved classification performance. However, when applying the models to tasks requiring embeddings for the whole project (RQs 3–5), a pre-trained embedding-llm performs similarly to our best-performing dedicated Scratch embedding model. This indicates that the two kinds of models could be used in complementary roles. llms might be suitable for higherlevel tasks where overall similarities between projects are relevant, like for example the progress estimation (RQ5) or estimating the difficulty of exercises. As prior research shows, the llms’ large pre-training datasets allow them to be used for tasks combining code context with natural language explanations. The Scratch-specific models on the other hand might be more suitable for programming assistance tools where structural insights are required, for example next-step hint generators or bug-detectors. Since they can be trained for specific tasks, the creation of dedicated small models allow for less resource intensive deployments. This might be relevant in both large- and small-scale deployments where the required resources to host an llm are not available or externally hosted tools cannot be used due to the special care required when working with children’s data. Since this is the first study comparatively exploring code embeddings for Scratch, many opportunities for future improvements and extensions arise. The next steps to improve the models’ performance for the sprite naming task could be an improved tokenisation approach. Similarly, the current models could be extended with suitable changes to the input format and model structure to work on whole programs rather than individual sprites. Based on related research for code embeddings on regular textual languages and due to the multi-modality of llms, including more Scratch-specific information like for example block positions or visual information seems like a promising direction for future research. More generally, when considering the event-based and concurrent execution flows commonly found in Scratch, an integration of these concepts into the structure given to code embedding models might also be of relevance beyond Scratch when embedding the code of other event-driven systems like games and game frameworks, mobile apps, or distributed services. Data Availability A replication package is provided online at doi:10.5281/zenodo.21273159. It contains the sampled Scratch projects used for model training and evaluation, our model implementations, the raw model inference results, and our evaluation scripts. References [1] [2] [3] [4] [5] [6]

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. In Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.449. Efthimia Aivaloglou and Felienne Hermans. 2016. How kids code and how we know: an exploratory study on the Scratch repository. In Conference on International Computing Education Research (ICER). ACM. doi:10.1145/2960310.2960325. Efthimia Aivaloglou, Felienne Hermans, Jesus Moreno-Leon, and Gregorio Robles. 2017. A dataset of Scratch programs: scraped, shaped and scored. In International Conference on Mining Software Repositories (MSR). IEEE. doi:10.1109/msr.2017.45. Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs. In International Conference on Learning Representations (ICLR). arXiv. arXiv: 1711.00740 [cs.LG]. Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018. Code2seq: generating sequences from structured representations of code. In International Conference on Learning Representations (ICLR). arXiv. arXiv: 1808.01400 [cs.LG]. Uri Alon and Eran Yahav. 2021. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations (ICLR). arXiv. arXiv: 2006.05205 [cs.LG]. Manuscript submitted to ACM

34

Benedikt Fein and Gordon Fraser

[7]

Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. Code2vec: learning distributed representations of code. Proceedings of the ACM on Programming Languages, 3. doi:10.1145/3290353. Rishabh Balse, Bharath Valaboju, Shreya Singhal, Jayakrishnan Madathil Warriem, and Prajish Prasad. 2023. Investigating the potential of GPT-3 in providing feedback for programming assessments. In Conference on Innovation and Technology in Computer Science Education (ITiCSE). ACM. doi:10.1145/3587102.3588852. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Association for Computational Linguistics. https://aclanthology.org/W05-0909/. Robert Bazzocchi, Micah Flemming, and Lisa Zhang. 2020. Analyzing CS1 student code using code embeddings. In ACM Technical Symposium on Computer Science Education (SIGCSE). ACM. doi:10.1145/3328778.3372601. Tadeusz Caliński and Jerzy Harabasz. 1974. A dendrite method for cluster analysis. Communications in Statistics, 3, 1, (Jan. 1974). doi:10.1080/03610 927408827101. Laura Caspari, Luisa Greifenstein, Ute Heuer, and Gordon Fraser. 2023. Scratchlog: live learning analytics for scratch. In Conference on Innovation and Technology in Computer Science Education (ITiCSE). ACM, (June 2023). doi:10.1145/3587102.3588836. Xiaolin Chai, Yan Sun, and Yan Gao. 2023. Towards data-driving multi-view evaluation framework for Scratch. Tsinghua Science and Technology, 29, 2, (Sept. 2023). doi:10.26599/tst.2023.9010016. Jinyin Chen, Keke Hu, Yue Yu, Zhuangzhi Chen, Qi Xuan, Yi Liu, and Vladimir Filkov. 2020. Software visualization and deep transfer learning for effective software defect prediction. In International Conference on Software Engineering (ICSE). ACM. doi:10.1145/3377811.3380389. Liuqing Chen, Shuhong Xiao, Yunnong Chen, Yaxuan Song, Ruoyu Wu, and Lingyun Sun. 2024. ChatScratch: an AI-augmented system toward autonomous visual programming learning for children aged 6-12. In Conference on Human Factors in Computing Systems (CHI). ACM. doi:10.1145 /3613904.3642229. DeepSeek-AI et al. DeepSeek-Coder-V2: breaking the barrier of closed-source models in code intelligence. (June 2024). arXiv: 2406.11931 [cs.SE]. Adina Deiner, Patric Feldmeier, Gordon Fraser, Sebastian Schweikl, and Wengran Wang. 2023. Automated test generation for Scratch programs. Empirical Software Engineering, 28, 3, (May 2023). doi:10.1007/s10664-022-10255-x. Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1. http://jmlr.org/papers /v7/demsar06a.html. Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with Copilot: exploring prompt engineering for solving CS1 problems using natural language. In ACM Technical Symposium on Computer Science Education (SIGCSE). ACM. doi:10.1145/3545945.3569823. Stefania Druga and Amy J Ko. 2025. Scratch Copilot: supporting youth creative coding with AI. In Interaction Design and Children (IDC). ACM. doi:10.1145/3713043.3727051. Stefania Druga and Nancy Otero. Scratch Copilot evaluation: assessing AI-assisted creative coding for families. (May 2023). arXiv: 2305.10417 [cs.HC]. Kenneth Enevoldsen et al. MMTEB: massive multilingual text embedding benchmark. (Feb. 2025). arXiv: 2502.13595 [cs.CL]. Benedikt Fein, Patric Feldmeier, Gordon Fraser, and Florian Obermüller. 2026. Reasoning about bugs in learners’ Scratch programs using large language models. In International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). ACM. doi:10.1145/3786580.3786949. Benedikt Fein, Isabella Graßl, Florian Beck, and Gordon Fraser. 2022. An evaluation of code2vec embeddings for Scratch. In International Conference on Educational Data Mining (EDM). International Educational Data Mining Society. doi:10.5281/zenodo.6853103. Benedikt Fein, Maximilian Jungwirth, Gordon Fraser, and Florian Kandlinger. 2026. Challenges of deploying code embeddings: an industrial case study on method name generation. Automated Software Engineering, 33, 2, (Feb. 2026). doi:10.1007/s10515-026-00592-2. Benedikt Fein, Florian Obermüller, and Gordon Fraser. 2022. Catnip: an automated hint generation tool for Scratch. In Conference on Innovation and Technology in Computer Science Education (ITiCSE). ACM. doi:10.1145/3502718.3524820. Benedikt Fein, Florian Obermüller, and Gordon Fraser. 2025. LitterBox+: an extensible framework for LLM-enhanced Scratch static code analysis. In International Conference on Automated Software Engineering (ASE): Tool Demonstration Track. IEEE. doi:10.1109/ASE63991.2025.00357. Gordon Fraser, Ute Heuer, Nina Körber, Florian Obermüller, and Ewald Wasmeier. 2021. LitterBox: a linter for Scratch programs. In International Conference on Software Engineering: Joint Track on Software Engineering Education and Training (ICSE-JSEET). IEEE. doi:10.1109/icse-seet52601.202 1.00028. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32, 200, (Dec. 1937). doi:10.1080/01621459.1937.10503522. Rao Fu, Ziyang Luo, Hongzhan Lin, Zhen Ye, and Jing Ma. 2025. ScratchEval: are GPT-4o smarter than my child? evaluating large multimodal models with visual programming challenges. In Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. doi:10.18653/v1/2025.naacl-short.57. Gemma Team et al. Gemma 3 technical report. (Mar. 2025). arXiv: 2503.19786 [cs.CL]. Víctor González-Calatayud, Paz Prendes-Espinosa, and Rosabel Roig-Vila. 2021. Artificial intelligence for student assessment: a systematic review. Applied Sciences, 11, 12, (June 2021). doi:10.3390/app11125467.

[8]

[9]

[10] [11] [12] [13] [14] [15]

[16] [17] [18] [19] [20] [21] [22] [23]

[24] [25] [26] [27] [28]

[29] [30]

[31] [32]

Manuscript submitted to ACM

EmbeddedKittens: An Evaluation of Code Embeddings for Scratch [33] [34] [35] [36]

[37] [38] [39] [40] [41] [42] [43] [44]

[45] [46] [47] [48] [49] [50] [51]

[52]

[53] [54] [55] [56] [57] [58] [59] [60] [61]

35

Isabella Graßl, Benedikt Fein, and Gordon Fraser. 2025. Detecting gender stereotypes in Scratch programming tutorials. In Koli Calling. ACM. doi:10.1145/3769994.3770019. Isabella Graßl and Gordon Fraser. 2022. Scratch as social network: topic modeling and sentiment analysis in Scratch projects. In International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). ACM. doi:10.1145/3510458.3513021. Elisabeth Griebl, Benedikt Fein, Florian Obermüller, Gordon Fraser, and René Just. 2023. On the applicability of language models to block-based programs. In International Conference on Software Engineering (ICSE). IEEE. doi:10.1109/icse48619.2023.00199. Rajarshi Haldar and Julia Hockenmaier. 2024. Analyzing the performance of large language models on code summarization. In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING). ELRA and ICCL. https://aclanthology.org/2024.lrecmain.89/. Maurice Howard Halstead. 1979. Elements of software science. Elsevier. isbn: 0444002154. Brian Harvey and Jens Mönig. 2010. Bringing “no ceiling” to Scratch: can one language serve kids and computer scientists? In Constructionism. Abram Hindle, Earl T. Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software. Communications of the ACM, 59, 5, (Apr. 2016). doi:10.1145/2902362. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9, 8, (Nov. 1997). doi:10.1162/neco.1997.9.8.1735. Xinyi Hou et al. 2024. Large language models for software engineering: a systematic literature review. ACM Transactions on Software Engineering and Methodology, 33, 8, (Nov. 2024). doi:10.1145/3695988. Sathvik Joel, Jie Wu, and Fatemeh Fard. 2025. A survey on LLM-based code generation for low-resource and domain-specific programming languages. ACM Transactions on Software Engineering and Methodology, (Oct. 2025). doi:10.1145/3770084. Abdoul Kader Kaboré, Earl T. Barr, Jacques Klein, and Tegawendé F. Bissyandé. 2023. Codegrid: a grid representation of code. In International Symposium on Software Testing and Analysis (ISSTA). ACM. doi:10.1145/3597926.3598141. Patrick Keller, Abdoul Kader Kaboré, Laura Plein, Jacques Klein, Yves Le Traon, and Tegawendé F. Bissyandé. 2021. What you see is what it means! semantic representation learning of code based on visualization and transfer learning. ACM Transactions on Software Engineering and Methodology, 31, 2, (Dec. 2021). doi:10.1145/3485135. Denis Kocetkov et al. The Stack: 3 TB of permissively licensed source code. (Nov. 2022). arXiv: 2211.15533 [cs.CL]. Taku Kudo and John Richardson. Sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. (Aug. 2018). arXiv: 1808.06226 [cs.CL]. Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2022. SySeVR: a framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing, 19, 4, (July 2022). doi:10.1109/tdsc.2021.3051525. Chin-Yew Lin. 2004. ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, (July 2004). https://aclanthology.org/W04-1013/. Anton Lozhkov et al. StarCoder 2 and The Stack v2: The Next Generation. (Feb. 2024). arXiv: 2402.19173 [cs.SE]. Dipeeka Luitel, Shiva Nejati, and Mehrdad Sabetzadeh. 2024. Requirements-driven slicing of Simulink models using LLMs. In International Requirements Engineering Conference Workshops (REW). IEEE. doi:10.1109/rew61692.2024.00014. Wei Ma, Mengjie Zhao, Ezekiel Soremekun, Qiang Hu, Jie M. Zhang, Mike Papadakis, Maxime Cordy, Xiaofei Xie, and Yves Le Traon. 2022. GraphCode2Vec: generic code embedding via lexical and program dependence analyses. In International Conference on Mining Software Repositories (MSR). ACM. doi:10.1145/3524842.3528456. Xiangyue Ma, Xiaoting Du, Chenglong Li, Jiangtao Meng, Xiaoke Fang, Wenjie Ding, and Zheng Zheng. 2026. How do large language models perform in deep learning code generation? an empirical study. ACM Transactions on Software Engineering and Methodology, (May 2026). doi:10.114 5/3816024. John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch programming language and environment. ACM Transactions on Computing Education, 10, 4, (Nov. 2010). doi:10.1145/1868358.1868363. Thomas J. McCabe. 1976. A complexity measure. IEEE Transactions on Software Engineering, SE-2, 4, (Dec. 1976). doi:10.1109/tse.1976.233837. Monica M. McGill and Adrienne Decker. 2020. Tools, languages, and environments used in primary and secondary computing education. In Conference on Innovation and Technology in Computer Science Education (ITiCSE). ACM. doi:10.1145/3341525.3387365. Qing Mi, Yiqun Hao, Liwei Ou, and Wei Ma. 2022. Towards using visual, semantic and structural features to improve code readability classification. Journal of Systems and Software, 193, (Nov. 2022). doi:10.1016/j.jss.2022.111454. George A. Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38, 11, (Nov. 1995). doi:10.1145/219717.219748. Peter Bjorn Nemenyi. 1963. Distribution-free multiple comparisons. Ph.D. Dissertation. Princeton University. Benjamin Paassen, Jessica McBroom, Bryn Jeffries, Irena Koprinska, and Kalina Yacef. 2021. Mapping Python programs to vectors using recursive neural encodings. Journal of Educational Data Mining, 13, 3, (Oct. 2021). doi:10.5281/zenodo.5634224. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. BLEU: a method for automatic evaluation of machine translation. In Annual Meeting on Association for Computational Linguistics (ACL). Association for Computational Linguistics. doi:10.3115/1073083.1073135. Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. 2015. Learning program embeddings to propagate feedback on student code. In International Conference on Machine Learning (ICML). PMLR. https://proceedings.mlr.press/v37/piech15.ht ml. Manuscript submitted to ACM

36

Benedikt Fein and Gordon Fraser

[62]

Peng Qi, Yan Sun, Hong Luo, and Mohsen Guizani. 2022. Scratch-Rec: a novel Scratch recommendation approach adapting user preference and programming skill for enhancing learning to program. Applied Intelligence, 52, 8, (Jan. 2022). doi:10.1007/s10489-021-02970-7. Nishat Raihan, Mohammed Latif Siddiq, Joanna C. S. Santos, and Marcos Zampieri. 2025. Large language models in computer science education: a systematic literature review. In ACM Technical Symposium on Computer Science Education (SIGCSE). ACM. doi:10.1145/3641554.3701863. Lianne Roest, Hieke Keuning, and Johan Jeuring. 2024. Next-step hint generation for introductory programming using large language models. In Australasian Computing Education Conference (ACE). ACM. doi:10.1145/3636243.3636259. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50, 1, (Jan. 2024). doi:10.1109/tse.2023.3334955. Sebastian Schweikl and Gordon Fraser. 2025. RePurr: automated repair of block-based learners’ programs. In International Conference on the Foundations of Software Engineering (FSE). ACM. doi:10.1145/3715786. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. (Aug. 2015). arXiv: 1508.07909 [cs.CL]. Samuel Sanford Shapiro and Martin Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52, 3-4, (Dec. 1965). doi:10.1093/biomet/52.3-4.591. Yuan Si, Daming Li, Hanyuan Shi, and Jialu Zhang. 2026. VisionScratch: LLM-based automated feedback generation using code-produced videos for Scratch programs. In International Conference on the Foundations of Software Engineering (FSE). ACM. doi:10.1145/3808163. Yuan Si, Kyle Qi, Daming Li, Hanyuan Shi, and Jialu Zhang. Stitch: step-by-step LLM guided tutoring for Scratch. (Oct. 2025). arXiv: 2510.26634 [cs.SE]. Aaditya Singh et al. OpenAI GPT-5 system card. (Dec. 2025). arXiv: 2601.03267 [cs.CL]. Sandro Speth, Niklas Meißner, and Steffen Becker. 2023. Investigating the use of AI-generated exercises for beginner and intermediate programming courses: a chatgpt case study. In Conference on Software Engineering Education & Training (CSEE&T). IEEE. doi:10.1109/cseet58097.2023.00030. [SW] Peter M. Stahl, lingua-rs version 1.7.2, 2025. url: https://github.com/pemistahl/lingua-rs, swhid: ⟨ swh:1:rel:1a486dd6ad25a610d9f181855c5 92234adf7c31f;origin=https://github.com/pemistahl/lingua-rs;visit=swh:1:snp:f055241154533f746f30d7a9fd85fb8f7371234d ⟩ . Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2025. Source code summarization in the era of large language models. In International Conference on Software Engineering (ICSE). IEEE. doi:10.1109/icse55347.2025.00 034. Alaaeddin Swidan and Felienne Hermans. 2023. A framework for the localization of programming languages. In International Conference on Systems, Programming, Languages and Applications: Software for Humanity (SPLASH). ACM. doi:10.1145/3622780.3623645. Zhao Tian and Junjie Chen. 2026. Aligning requirement for large language model’s codegeneration. In International Conference on Software Engineering (ICSE). ACM. doi:10.1145/3744916.3764572. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 86. http://jmlr.org/pape rs/v9/vandermaaten08a.html. András Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25, 2, (June 2000). doi:10.3102/10769986025002101. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS). https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee2435 47dee91fbd053c1c4a845aa-Paper.pdf. Henrique Schechter Vera et al. EmbeddingGemma: powerful and lightweight text representations. (Sept. 2025). arXiv: 2509.20354 [cs.CL]. Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. doi:10.18653/v1/2021.emnlp-main.685. Mounika Yabaku and Sofia Ouhbi. 2024. University students’ perception and expectations of generative ai tools for software engineering. In Conference on Software Engineering Education & Training (CSEE&T). IEEE. doi:10.1109/cseet62301.2024.10663035. Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In International Conference on Software Engineering (ICSE). IEEE. doi:10.1109/icse.2019.00086. Jingfan Zhang, Delaram Ghobari, Mehrdad Sabetzadeh, and Shiva Nejati. 2025. Simulink mutation testing using CodeBERT. In International Conference on Automation of Software Test (AST). IEEE. doi:10.1109/ast66626.2025.00009. Yanzhao Zhang et al. Qwen3 embedding: advancing text embedding and reranking through foundation models. (June 2025). arXiv: 2506.05176 [cs.CL].

[63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74]

[75] [76] [77] [78] [79]

[80] [81]

[82] [83] [84] [85]

Manuscript submitted to ACM