JupOtter: Cell-Level Bug Detection in Jupyter Notebooks

arXiv:2606.23877v1 [cs.SE] 22 Jun 2026

JupOtter: Cell-Level Bug Detection in Jupyter Notebooks Lukas Ottenhof

Thibaud Lutellier

University of Alberta Edmonton, AB, Canada [email protected]

University of Alberta Camrose, AB, Canada [email protected]

Abstract—Jupyter Notebooks are an increasingly popular coding environment used across many domains, especially in Python-based data science and scientific computing. Originally used for prototyping and interactive exploration, notebooks are increasingly used to develop more complex programs, leading to a rapid rise in buggy notebooks on platforms like GitHub. To address this trend, we present JupOtter, a bug detection system designed specifically for Jupyter Notebooks. JupOtter features three novel contributions: (1) a notebook-specific tokenization strategy that preserves cell structure, (2) a cell-level bug prediction technique, and (3) a new labeled dataset, OtterDataset, containing over 21,000 notebooks annotated for fine-grained celllevel bug detection. JupOtter achieves cell-level bug detection F1 scores that surpass static analyzers and large language models in two out of three evaluation datasets. Index Terms—Jupyter Notebooks, bug detection, defect prediction, cell-level bug localization, transformer models, software quality

I. I NTRODUCTION Jupyter Notebooks have become the standard environment in data science due to their unique interactive structure [1]. Unlike traditional code files, notebooks consist of executable cells that can contain either code or markdown, and can be executed in any order. This cell-based execution model allows users to run time-consuming operations (e.g., data loading) once and iteratively modify downstream code without re-executing the entire notebook. While this flexibility has accelerated development and experimentation, it has also introduced new challenges in code reliability. As notebooks evolve from prototyping tools into complex, multi-stage software artifacts, the risk of implementation bugs grows. This shift has contributed to a rising number of buggy notebooks [2]. Recent studies show that the number of implementation bugs in Jupyter Notebooks found on Stack Overflow is increasing by 48% annually [2], suggesting developers are struggling to manage notebook quality. This problem is especially concerning in domains such as scientific research, finance, and artificial intelligence, where high accuracy and reliability are critical. This paper presents contributions focused on detecting bugs related to code implementation because they represent a significant and growing portion of real-world notebook bugs. Unlike environment-related bugs that depend on external This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

dependencies or configurations, implementation bugs are selfcontained within notebooks and can be identified by learning patterns in code. Detecting such bugs prior to execution can reduce debugging time and resource costs. Jupyter Notebooks present unique challenges for bug detection due to their non-linear, cell-based execution model. Unlike traditional scripts, notebooks allow cells to be run in arbitrary order, which can introduce implementation bugs tied to execution state and cell dependencies. As a result, conventional debugging techniques that work on sequential scripts are often ineffective. Machine learning tools have shown potential in code understanding tasks such as bug detection across programming languages [3]–[7]. However, many existing machine learningbased bug detectors operate at the file or function level, which is ill-suited to the notebook format. Simply knowing that a bug exists in a notebook file is often not actionable, and focusing on a single function may miss cross-cell dependencies. Despite recent advances, existing machine learning tools lack support for the structural and execution-specific characteristics of Jupyter Notebooks. As a result, the software engineering community still lacks effective approaches to automatically detect notebook bugs [8], [9]. To address the unique challenges of analyzing Jupyter Notebooks, we propose JupOtter: a novel machine learningbased bug detection system tailored to the notebook format. JupOtter introduces three key contributions: (1) A new tokenization strategy which enables fine-grained predictions by inserting special tokens that wrap each cell and act as explicit cell boundaries. These boundary tokens allow models to know exactly where notebook cells start and end after tokenization, which is necessary for making precise, localized predictions. Unlike prior code tokenization approaches, which either flatten notebooks into plain scripts or truncate long inputs [10]–[13], our strategy preserves the original cell structure while enabling long-context processing without losing global notebook coverage. This makes finegrained, cell-level reasoning possible at scale. (2) A new cell-level bug prediction technique which allows models to make separate predictions for each cell in a notebook. Unlike previous models built for linear, traditional source code [14]–[17], our technique assumes no fixed execu-

tion order. This matches the naturally non-linear, cell-by-cell execution of computational notebooks. This differs from standard pretrained-model adaptation for sequence classification, which produces a single label per input sequence rather than per-cell representations. (3) OtterDataset, a new dataset of 21,303 Python Jupyter Notebooks labeled for bug detection. This dataset is designed to support cell-level and file-level evaluation and training, while reflecting real-world notebook usage. To the best of our knowledge, this is the first publicly available large-scale dataset of Jupyter Notebooks with cell-level labels. JupOtter was evaluated using three datasets excluded from its training data (OtterDataset, Jupyter Errors [18], and CodeParrot [19]). We aimed to answer four research questions: RQ1: How well does JupOtter detect bugs at the cell level, RQ2: How does JupOtter perform when detecting specific error types, RQ3: How does JupOtter compare to related tools for file-level bug detection, and RQ4: How does each contribution affect performance? II. A PPROACH At a high level, JupOtter processes Jupyter Notebooks by extracting code cells, converting them into a tokenized format that preserves notebook structure, and predicting whether each cell contains an implementation bug. Our approach is designed to handle the unique characteristics of notebooks, including variable-length cells and interleaved code and markdown content. Our approach consists of four main components: (1) a dataset construction phase, (2) a notebook-specific tokenization strategy, (3) a cell-level defect prediction model, and (4) a multi-segment training strategy (see Figure 1). A. The OtterDataset To address the lack of publicly available datasets for supervised cell-level bug detection in Jupyter Notebooks, we constructed OtterDataset: a new labeled dataset of Python notebooks designed for this task. We systematically extracted Jupyter Notebook files from GitHub using GitHub Search API and a set of 32 search queries to ensure broad coverage of topics and error types. After cleaning, labeling, and removing duplicate notebooks, this process resulted in 9,937 buggy notebooks and 11,366 non-buggy notebooks (47% buggy ratio, 21,303 total notebooks). At the cell-level, 43,789 cells are buggy out of 661,909 cells (7% buggy cell ratio), with an average of 2.1 buggy cells per notebook. To prevent overrepresentation of any particular query, we limited the collection to a maximum of 1,000 notebooks per query. The search queries were designed to capture a broad range of domains such as statistics and machine learning, and common bug types. To target common implementation errors, we incorporated bug-related keywords including Python exceptions such as “IndexError” and “TypeError”, based on the list of built-in exceptions in the official Python documentation [20]. When constructing OtterDataset, we targeted environment-related errors and implementation-related errors since they are the most common categories of errors in

notebooks [2]. We provided the complete list of queries used in our supplementary material, along with the number of notebooks they contributed to OtterDataset [21]. Each notebook collected was assigned a unique identifier based on its author name, repository name, and file name to trace its origin. We used this identifier to remove duplicate notebooks that appeared in different queries to limit data leakage in our train-test split. We further validated separation between training and test sets at the author-repository level. 83.9% of test notebooks in OtterDataset come from authorrepository combinations not present in the training set. To assess whether OtterDataset contains notebooks intentionally designed to demonstrate bugs (which would not be representative of real bugs), we randomly sampled and manually inspected 100 notebooks. None were explicitly focused on teaching or demonstrating specific bugs. While some notebooks were educational (e.g., demonstrating library usage), they did not intentionally inject errors. Although such bugfocused notebooks may exist, this inspection suggests they do not represent a substantial portion of the dataset. 1) Notebook Bug Identification: Previous studies have shown that reproducing notebook executions at scale is often unreliable due to missing dependencies, non-trivial environments, and non-linear cell execution [22]. To avoid these issues when creating labels, we did not attempt to re-execute the notebooks. Instead, we analyzed the output field of each executed code cell already present in the notebooks. Prior work has used stored cell outputs to detect and analyze bugs or errors in notebooks [18], [23], and has shown that over 85% of notebooks contain executed cells with output data [22]. From these saved outputs, we extracted and labeled all bugrelated information, including the error type, the cell and line number where the error occurred, and the error message. This information was used to label each cell as containing a bug or not, as well as to record the type of bug encountered. After analyzing all saved cell outputs, we parsed the code cells of each notebook to identify syntax errors that might not be visible in notebook outputs (e.g., if a cell was never executed). Each code cell was parsed individually using Python’s ast module, which raises an exception when a syntax error is encountered. We parsed each cell in addition to examining saved cell outputs as a data cleaning step, to avoid having trivial syntax errors incorrectly labeled as correct code. To handle notebook-specific commands such as %magic and !shell, we first transformed each cell using IPython’s TransformerManager. This preprocessing step converts Jupyter-specific syntax into valid Python, ensuring that such cells are not incorrectly flagged as buggy by the parser. When constructing OtterDataset, we targeted a wide variety of bug types to create a dataset representative of notebooks found online. However, during our labeling process, we excluded errors caused by the execution environment rather than the notebook code: ImportError, FileNotFoundError, KeyboardInterrupt, SystemExit, and ConnectionError. These were ignored to avoid introducing spurious labels. For example, two notebooks with

1.

Dataset Extraction

2. Bug Identification

3. Notebook-aware Tokenization

4. Model Training and Evaluation

AST parsing Notebook Split

Topic filter

CodeT5 Architecture Output cell checker

Bug filter Notebooks repositories

Cell Delimitation Duplicate removal

Chunk 1

Chunk 2

Chunk …

Tokenized Notebooks

Fine-tuned model: JupOtter

Labelled Cells

21,303 Notebooks

Test Set (20%)

Training Set (80%)

Cell-level bug prediction

Fig. 1. Overview of JupOtter, a bug detection system for Jupyter Notebooks. Our approach begins with source notebooks whose code cells are first extracted and preprocessed. The processed code cells are then tokenized into non-overlapping chunks using our tokenization strategy. These chunks are then processed by a pre-trained encoder, and predictions are generated.

identical code might differ only due to a user interruption, which would otherwise be incorrectly flagged as a runtime error. Additionally, we considered excluding memory errors, but we still labeled them as they can also be caused by bugs in the code (e.g., memory leaks or poor implementation). Although error types and messages are stored in the dataset, they were not used during model training, only during evaluation to assess performance on specific bug types. Since the goal of JupOtter is to detect the buggy cell prior to execution, such information would not be available in a real-world scenario. This metadata is included solely to support dataset analysis, evaluation, and future research. Based on the identified errors, we generated a binary label list for each notebook, where each element corresponds to a code cell. A value of 1 indicates that the cell contains a bug, and 0 otherwise. Additionally, we assigned a file-level label marked as buggy if at least one of its code cells contained a bug. These labels were created to enable supervised learning at the file and cell-level and to address the lack of publicly available datasets for bug detection in Jupyter Notebooks. To validate the accuracy of our labeling process, we randomly selected 30 notebooks from OtterDataset for manual label verification. 14 of the sampled notebooks contained at least one cell that could not be executed due to missing dependencies or unavailable data. For these cells, labels were assigned through manual inspection of the code, and when possible, they were executed using dummy input data to confirm expected behavior. These dependency issues differ from the excluded environment categories because the sampled notebooks contain cells whose dependencies were unavailable at verification time, but their bugs when present, are implementation-related. 24 of the 30 notebooks had identical manual and automated labels. In total, these notebooks contained 970 cells, of which 22 (2.3%) of the automated labels differed from their manual

counterparts. With a sample size of 970 cells, this corresponds to a margin of error of ±0.94% at a 95% confidence level (Wald interval). We compared the manual and automated labels using a bootstrap McNemar test with 10,000 resamples and a sample size of 100 to assess statistical agreement. Out of the 10,000 resamples, only 139 (1.39%) were significant at p < 0.05, indicating strong agreement between the automated and manual labels. The median p-value was 0.5000, with a 5th percentile of 0.1250 and a 95th percentile of 1.0000. These results suggest that the automated labeling procedure is largely consistent with manually verified labels, providing confidence in its reliability for large-scale notebook analysis. 2) Dataset Overview: Each notebook entry in OtterDataset includes the following seven features: notebook name: A unique identifier composed of the author’s username, repository, and file name. • notebook structure: Concatenated code cells only, with cell boundary markers. • is buggy: A binary label indicating whether the notebook contains at least one bug. • error locations: A list of cell and line numbers where errors occurred. • error type: The type(s) of detected error(s). • error message: error message(s) detected. • buggy cells: A binary list containing a label for each cell in a given notebook (e.g., [1, 0, 0, 1]). •

Since a notebook can contain multiple errors (e.g., an indentation error in cell 1 and an index error in cell 7), we tracked all errors using lists for both error types and error messages. These are linked through the error_locations column, which combines information from multiple fields to give a detailed description of each buggy cell, including the error type, cell number, and specific line.

B. JupOtter Defect Prediction System In this section, we introduce three key contributions designed specifically for cell-level bug detection in Jupyter Notebooks: (1) a notebook-aware tokenization strategy that preserves cell boundaries and efficiently handles large notebook files; (2) a novel prediction method that identifies individual buggy cells within notebooks; and (3) a mixed-precision, persample training loop that accommodates the multi-segment structure of tokenized notebook content. Together, these contributions form JupOtter, a cell-level bug detection system for Jupyter Notebooks. 1) Notebook-aware tokenization strategy: To tokenize input for our cell-level prediction model, we developed a chunkingbased strategy that segments notebooks into multiple nonoverlapping sequences of cells. This method builds upon prior work that handles large inputs by treating files as nonoverlapping pages [24]. Our approach preserves notebook structure and content without truncation and represents each notebook with three two-dimensional tensors: one for input tokens, one for attention masks, and one for labels. Each row of these tensors contains a chunk of notebook cells whose combined token length does not exceed a user-defined limit. Additionally, we insert special tokens to mark cell boundaries within tokenized sequences, enabling precise identification of each cell’s position after tokenization. We divided notebooks into multiple chunks to ensure that notebooks exceeding the token limit of encoder models could still be fully tokenized and utilized for training without truncation. Special tokens marking cell boundaries allow precise celllevel predictions directly from tokenized outputs. By enabling full processing of large notebooks, our strategy significantly enhances real-world applicability, removing constraints associated with processing only smaller notebooks. Without our tokenization strategy, standard tokenization methods would truncate or split longer notebooks, discarding important syntactic and semantic information at the end of these sequences. This truncation would bias models towards shorter notebooks, limiting their effectiveness on realistic data. By training models with our strategy, we preserve the full context of larger notebooks, enabling models to learn from complete cell sequences and effectively handle realistic, largescale notebook inputs. To support cell-level predictions, it is critical to preserve cell boundaries during tokenization. We achieve this by adding special start and end cell markers. Each notebook is tokenized individually, processing cells sequentially and organizing them into chunks. Cells are added to the current chunk until including another cell would exceed the model’s input constraints. Once the limit is reached, the tokens and attention masks are padded to match the chunk length and stored as rows in their respective two-dimensional tensors. Corresponding labels for each chunk are recorded separately in a label tensor. The process continues iteratively until the entire notebook is tokenized. If a notebook’s total token length never exceeds the chunk length, it is stored as a single padded chunk comprising

Fig. 2. Example of a cell-level prediction output. Each cell is processed and assigned a binary label: 0 for non-buggy cells, 1 for buggy cells.

tensors for tokens, attention masks, and labels. The only case where truncation occurs in our tokenization strategy is when a single cell exceeds the maximum chunk token length. Cells exceeding the chunk length are rare and resemble standalone scripts more than typical notebook cells, and are thus outside the intended scope of our model’s learning strategy. 987 of 661,909 cells in OtterDataset (0.15%) exceed our chunk length of 2,500 tokens. 2) Prediction Model’s Architecture: Our approach is built on top of a pretrained transformer-based encoder (e.g., CodeBERT or CodeT5) which is well-suited for code understanding tasks. JupOtter predicts bugs at the cell-level by using boundary markers to identify cell boundaries within the model’s input. These markers allow the model to generate a binary prediction for each cell in a notebook chunk, indicating whether that cell is buggy (1) or not (0). Unlike traditional file-level models that produce a single label per notebook, our approach provides fine-grained predictions aligned with the notebook’s structure (Figure 2). File-level models cannot produce such localized predictions because they are architecturally constrained to aggregate information into a single global representation and output a single label per file. In contrast, our method explicitly preserves cell boundaries in the input representation and associates predictions with specific structural units of the notebook. Having cell-level predictions is more actionable and self-explanatory than a file-level prediction. For example, while informing a user that a file contains a bug provides guidance, identifying a buggy cell (mean lines of code per cell in OtterDataset is 6.43) provides more precise information and may allow users to take action without additional explanation. To perform cell-level prediction, each tokenized chunk (which may contain several cells) is first passed through a transformer-based encoder to obtain contextualized hidden states. We then identify cell boundaries within the chunk using the special start and end markers and group the hidden states accordingly. For each cell, we compute a single representation by averaging the hidden state vectors of all tokens within

its boundaries, including the boundary markers. This celllevel embedding is passed through a classifier (a linear layer) that outputs a single logit indicating the likelihood that the cell is buggy. Repeating this process for all cells in a chunk yields a vector of cell-level predictions. When processing large notebooks spanning multiple chunks, predictions from each chunk are concatenated to produce a final prediction. Passing tokenized chunks through a transformer-based encoder prior to making predictions allows the model to capture inter-cell relationships without assuming a particular execution order. This helps address the challenge of non-linear execution in notebooks, avoiding false positives that dynamic tools may introduce when guessing execution order, and that static analysis tools may introduce when attempting to parse code sequentially. We investigate the impact of the encoder in our ablation study (RQ4). We compute loss across notebook chunks using BCEWithLogitsLoss, which combines a sigmoid activation with binary cross-entropy loss. Logits and ground-truth labels from all chunks of a notebook are concatenated into a single prediction and label vector. Loss is then computed over these vectors, ensuring that each cell contributes equally to the notebook’s total loss. 3) Training strategy: Our training strategy is designed to effectively handle large notebooks represented as multi-chunk inputs produced by our tokenization strategy. To accommodate this structure, we implemented a training loop that processes the chunks of each notebook individually within each batch. During each epoch, we iterate through batches of notebooks using a user-defined batch size. During training, we use mixed precision training with torch.cuda.amp.autocast and GradScaler. In training, each sample’s loss is scaled by dividing by batch size to normalize gradient contributions across the batch. We perform the backward pass on each sample’s scaled loss individually, accumulating gradients across the entire batch. This per-sample backward pass strategy is necessary due to memory constraints caused by large notebooks, and is repeated until all samples in the batch have been processed, accumulating loss and gradients across samples. After all samples in a batch are processed, we perform a single optimizer step and gradient scaler update using the accumulated gradients from the persample backward passes. To track how models being trained perform on unseen data, we implemented a validation loop within each epoch where our model runs on our OtterDataset testing set. During validation, we calculate loss using the same loss strategy as the training loop. However, no model parameter updates are done based on this loss; it is strictly for monitoring model performance on unseen data. III. E XPERIMENTAL S ETTINGS A. Model and Tokenizer Settings We trained two variants of our prediction model using the CodeT5 encoder family: CodeT5-small (60 million parameters) and CodeT5-base (220 million parameters). This allowed

TABLE I S UMMARY OF DATASETS USED FOR EVALUATION AFTER TOKENIZATION . Dataset

# Files

# Buggy files

# Cells

# Buggy Cells

OtterDataset Test Jupyter Errors CodeParrot

4,113 9,313 4,769

2,006 (49%) 8,740 (94%) 1,108 (23%)

124,364 282,371 78,313

8,598 (7%) 15,960 (6%) 3,797 (5%)

us to explore how model scale affects cell-level bug detection performance. Additionally, because CodeT5 was not pretrained on Jupyter Notebooks, there is no data contamination between our training sets and the data used to train CodeT5. Input tokenization was performed using our custom tokenization strategy in conjunction with the RobertaTokenizer associated with each encoder. While the original CodeT5 models were pre-trained on sequences up to 512 tokens, we increased the maximum sequence length to 2,500 tokens to accommodate longer notebooks and preserve cell-level context. B. Training Settings During training, we used up to the first 4 chunks from each notebook, which allowed us to include all code cells from 19,728 of the notebooks in our dataset (93%). We considered raising the number of chunks, but didn’t because training on additional chunks would increase GPU memory consumption and per-epoch training time (already 2.09 hours per epoch for JupOtter-base on an A100). This isn’t a limitation of the technique, as we were able to tokenize more than 4 chunks; it is a hardware constraint during training. We used a batch size of 4 and 10 epochs. A random seed of 42 was used to split our dataset into a training (80% of total data, 46.6% buggy) and a testing set (20% of total data, 49% buggy). Optimization was performed using the AdamW optimizer with a learning rate of 5e-5. We enabled mixed precision training to improve training speed and reduce GPU memory consumption. Training was conducted on an NVIDIA V100 (32GB) GPU for JupOtter-small and on an NVIDIA A100 GPU (40GB) for JupOtter-base. Training time was 2.09 hours per epoch for JupOtter-base, and 2.15 hours for JupOtter-small. C. Evaluation Benchmark Settings To evaluate the generalization ability of JupOtter beyond its training distribution, we curated and re-labeled two external datasets of Jupyter Notebooks: the Jupyter Errors Dataset [18] and a subset of the CodeParrot GitHub Jupyter dataset [19], both publicly available on Hugging Face. Because no existing benchmarks offered cell-level bug labels, we applied the same annotation strategy used for our OtterDataset to enable consistent evaluation. Table I displays a summary of our three evaluation datasets after tokenization. Files are considered buggy if they contain an error within the first four 2,500 token chunks. Using the author, file, and repository name of each file, no overlap was found in the datasets.

1) Jupyter Errors Dataset: The Jupyter Errors Dataset contains 10,000 real-world Jupyter Notebooks, each known to contain at least one runtime error [18]. We labeled the notebook cells in this dataset as buggy or not-buggy, similarly to OtterDataset’s labeling process. All notebooks were saved with a file name comprised of the author’s name, repository name, and original file name. 660 notebooks were not included due to having duplicate author, file, and repository names, and another 19 were removed due to structural issues, leaving us with a final testing set of 9,321 notebooks. During tokenization, a further 8 were removed due to containing only cells that exceeded our chunk length limit. 2) CodeParrot dataset: The CodeParrot GitHub Jupyter contains 164,619 Jupyter Notebooks collected from GitHub [19]. For our evaluation, we selected the first 5,000 valid Python notebooks from this dataset. Similar to our other datasets, notebooks were saved with a file name comprised of the author’s name, repository name, and file name found on GitHub. A total of 108 notebooks were excluded due to duplicate author, file, and repository names, resulting in a final testing set of 4,892 notebooks. A further 123 notebooks were removed during tokenization due to token and chunk size constraints. We labeled this subset using the same strategy applied to the previous datasets. IV. E VALUATION & R ESULTS We evaluate JupOtter using three datasets: a held-out portion of our OtterDataset, a 4,769-notebook subset of CodeParrot [19], and the Jupyter Errors Dataset (9,313 notebooks) [18]. We compare JupOtter against Flake8, GPT-4o-mini, and Gemini 3 Flash in RQ1 and RQ3 where we evaluate cell-level and file-level defect prediction. We further evaluate JupOtter in RQ2 where we explore its ability to detect specific bug types, and perform an ablation study in RQ4. A. RQ1: How well does JupOtter detect bugs at the cell level? This research question evaluates how well JupOtter generalizes to new data, including both held-out notebooks from OtterDataset and two external benchmark datasets. We evaluate two versions of JupOtter, JupOtter-small and JupOtter-base, trained on CodeT5-small and CodeT5-base, respectively. Both versions are trained on 80% of OtterDataset. 20% of the instances are held out and used as a test set. We also evaluate Flake8, GPT-4o-mini, and Gemini 3 Flash, configured for cell-level bug detection, to establish a baseline of performance. We use three test sets for evaluation: OtterDataset Test Set, CodeParrot GitHub Jupyter subset [19], and the Jupyter Errors Dataset [18]. We report both cell-aggregated and file-aggregated metrics (F1, precision, recall, accuracy). Accuracy figures are included in results tables for completeness and should be interpreted with caution given the class imbalance (approximately 7% buggy cell rate). In cell-aggregated evaluation, we accumulated true positives, false positives, true negatives, and false negatives across all notebook cells in the test set to compute overall precision, recall, F1 score, and accuracy. This metric does not

distinguish between notebooks, each cell being treated as an independent unit. This method gives larger notebooks with more cells, more influence on the evaluation metrics since each cell contributes equally to the final evaluation metrics. These cell-aggregated metrics measure how many cells are correctly identified by our model. The file-aggregated metrics are computed for each notebook in our testing set individually. We then averaged these metrics across all notebooks and used the averaged precision and recall to compute a final averaged F1 score. These file-aggregated metrics reduce the potential bias created by large notebooks. Since RQ1 focuses on cell-level performance across multiple files and datasets, we only report file-aggregated metrics in this paper, while cell-aggregated results are available in supplementary material [21]. Flake8 was adapted for cell-level bug detection by first mapping each notebook cell to its corresponding line numbers, preserving the line numbering across cells. The notebooks were then converted into Python scripts, allowing Flake8 to analyze the entire file. We considered running Flake8 on each cell individually, but this approach caused many false positives. We also only consider errors raised by Flake8 and ignore warnings and style suggestions. When Flake8 detected an error, we mapped the error’s line number back to the corresponding cell and created a prediction array where the buggy cell was marked with a 1, and all preceding cells were marked with 0. Since Flake8 only reports up to the first fatal error it encounters without reporting subsequent errors, subsequent cells were not considered. We trimmed the labels to match the number of cells analyzed by Flake8. If no errors were detected, the prediction array consisted entirely of 0s, indicating no buggy cells. Additionally, we configured GPT-4o-mini and Gemini 3 Flash for cell-level bug detection. Each notebook was presented to the models incrementally: at each step, the models were given the current cell together with all previously seen cells and generated predictions, and a prediction was recorded for the current cell. Thus, the models precisely labeled each cell while having access to the full prefix of the notebook. Due to the large number of API calls required to generate predictions for all cells (485,048), we used random sampling and selected 100 notebooks from each dataset (7,628 cells, 6% cell-level bug rate) to construct a manageable evaluation set for Gemini 3 Flash and GPT-4o-mini. Gemini 3 Flash averaged 4.8 minutes per notebook, making evaluation on the full test set impractical. The initial prompt was: “You are analyzing a Jupyter Notebook cell-by-cell. You must remember previous cells. For each cell, answer ONLY with YES or NO indicating whether THIS CELL contains a bug.” Each subsequent cell and the model’s previous responses were appended to this prompt, forming the incremental prompting scheme. We only accepted outputs that matched exactly YES or NO after removing punctuation, white spaces, and converting characters to uppercase. Once a model produced an output that did not conform to this format, evaluation of the current notebook was terminated. Notebooks for which the model

TABLE II C ELL - LEVEL RESULTS FOR J UP OTTER , F LAKE 8, GPT-4 O - MINI , AND G EMINI 3 F LASH ON DIFFERENT BENCHMARKS . B EST RESULTS ARE IN BOLD . Tech. Flake8 JupOttersmall JupOtterbase GPT4o-mini Gemini3-F lash

Pre. 0.75 0.93 0.88 0.26 0.45

OtterDataset Test Rec. F1 Acc. 0.69 0.69 0.75 0.85 0.88

0.72 0.79 0.81 0.40 0.59

0.80 0.95 0.95 0.74 0.85

Pre.

CodeParrot Rec. F1

Acc.

Pre.

0.62 0.94 0.86 0.16 0.32

0.90 0.92 0.91 0.95 0.95

0.69 0.97 0.96 0.70 0.78

0.63 0.93 0.82 0.16 0.32

failed to produce a complete sequence of valid predictions were evaluated by truncating the label sequence to the number of cells successfully processed. Gemini 3 Flash was able to generate predictions for 7,626 cells (99.99%), while GPT-4omini was able to produce predictions for 7,561 cells (99.12%). Table II summarizes results for JupOtter-base, JupOttersmall, Flake8, GPT-4o-mini, and Gemini 3 Flash when evaluated across all three datasets. On the held-out OtterDataset test set, JupOtter-base achieves a precision, recall, and F1 of 0.88, 0.75, and 0.81, respectively, while JupOtter-small achieves 0.93, 0.69, and 0.79. JupOtter-base and JupOttersmall achieved their best cell-level bug detection performance on our labeled subset of the CodeParrot GitHub Jupyter dataset, with JupOtter-base achieving an F1 score of 0.89 and an accuracy of 0.96, and JupOtter-small achieving an F1 score of 0.93 and an accuracy of 0.97. Their strong performance on both datasets suggests JupOtter is able to generalize well and did not overfit to its training set. However, on the Jupyter Errors dataset, JupOtter maintains high precision but achieves a lower F1 score, indicating that it misses a subset of runtime errors rather than overpredicting bugs. This may be due to the limited quantity of some runtime errors in the OtterDataset, which we used for training, when compared to other error types. Additionally, cells containing runtime bugs that were never executed are labeled non-buggy in our ground truth, meaning the training signal for runtime errors is inherently partial. Consequently, reported recall figures should be interpreted as lower bounds on true performance. RQ2 investigates in more detail which types of error JupOtter can detect well and which ones it struggles with. We measured the statistical significance of the performance difference between the two JupOtter models and found them significant (Wilcoxon Signed-Rank Test, p < 0.01 on all three benchmarks). This was not done between Flake8, GPT-4omini, and Gemini 3 Flash due to notebooks for which they were unable to generate complete predictions. Moving from a smaller encoder (CodeT5-small, 60 million parameters) to a larger one (CodeT5-base, 220 million parameters) yielded only marginal improvements in two benchmarks (OtterDataset Test and Jupyter Errors) while slightly decreasing performance in CodeParrot. The cost trade-off of JupOtterbase and JupOtter-small is discussed in Section III-B. JupOtter demonstrates strong generalization to new notebooks, particularly those containing typical implementation

0.74 0.93 0.89 0.28 0.48

Jupyter Errors Rec. F1 Acc. 0.43 0.17 0.22 0.68 0.72

0.51 0.28 0.34 0.26 0.44

0.69 0.91 0.91 0.66 0.76

errors. Its performance drops slightly on notebooks dominated by rarer runtime errors, which may be addressed in future dataset augmentation. These results suggest that JupOtter can be deployed in real-world notebook development environments to flag buggy cells with high accuracy, particularly when errors follow common Python patterns. We additionally analyzed whether JupOtter’s predictions are influenced by cell size rather than code content by measuring token counts per cell in OtterDataset. We compared the token distributions of cells predicted as buggy and non-buggy using both Cliff’s Delta and the Mann–Whitney U test. While the difference was statistically significant (p < 0.001), the effect size was negligible (Cliff’s Delta = 0.081), and the difference in median token count between the two groups was only 4 tokens. This indicates that cell length alone does not meaningfully explain JupOtter’s predictions. This observation is consistent with prior work showing that bugs in notebooks are primarily correlated with developer behavior rather than superficial code properties [9]. RQ1 Summary: JupOtter achieves strong cell-level bug detection performance on multiple unseen datasets, with best F1 of 0.93 and accuracy of 0.97, indicating good generalization. B. RQ2: How does JupOtter perform when detecting specific error types? This research question evaluates whether JupOtter’s performance varies across different categories of bugs, and whether certain error types are easier or harder for the model to detect. We selected the five most frequent error categories from OtterDataset: TypeError, NameError, SyntaxError, ValueError, and AttributeError. For each category, we evaluated JupOtterbase on notebooks containing at least one cell with the corresponding error. These notebooks were drawn from the labeled subset of the CodeParrot GitHub Jupyter dataset to ensure that the evaluation was performed on external data. Table III shows the cell-aggregated results of JupOtter-base across the five error types. JupOtter achieves the strongest performance (F1: 0.87) for SyntaxError; performs moderately well on TypeError and ValueError with an F1 score of 0.64 and 0.63, respectively; and struggles on AttributeError (F1: 0.38) and NameError (F1: 0.34).

TABLE III C ELL - LEVEL PERFORMANCES OF J UP OTTER - BASE ON THE FIVE MOST COMMON ERROR TYPES . Error Type

Recall

Precision

F1

Accuracy

Attribute errors Name errors Syntax errors Type errors Value errors

0.28 0.23 0.85 0.51 0.51

0.63 0.65 0.90 0.84 0.82

0.38 0.34 0.87 0.64 0.63

0.92 0.92 0.96 0.94 0.95

RQ2 Summary: JupOtter performed well for Type, Value, and Syntax errors, but struggled for Attribute and Name errors. High accuracy is maintained in all cases, but recall is significantly reduced for hard-to-catch error types. C. RQ3: How does JupOtter compare to related tools for filelevel bug detection? RQ3 evaluates JupOtter’s performance on file-level bug detection tasks and compares it to traditional static analysis tools and prior work using language models. Since no prior work has focused on cell-level bug detection, we cannot directly compare JupOtter to existing cell-level approaches. To evaluate our baselines in a setting most similar to related tools, we evaluate JupOtter’s file-level bug detection ability, enabling a direct comparison with Flake8, GPT-4omini, and Gemini 3 Flash. This file-level evaluation provides a more direct comparison to JupOtter and related work than the cell-level analysis in Section IV-A, where baselines were adapted for cell-level analysis. Because file-level detection does not require multiple API calls per notebook, whereas celllevel detection requires a separate API call for each cell, GPT4o-mini and Gemini 3 Flash can be evaluated on the entirety of the test datasets, enabling a substantially larger comparison without random sampling. We considered comparing JupOtter against existing filelevel defect prediction models, such as DeepLineDP [14], and LineVul [25], but did not include them in our evaluation. Existing defect prediction models are designed for traditional source code files and are not trained to process Jupyter Notebook-specific syntax, dynamic execution, or cell structure. Including existing defect prediction models would unfairly showcase baselines outside their intended domain. The large language models we evaluate are general-purpose, and Flake8 can be configured to analyze notebook code, making these baselines more appropriate for comparison. To evaluate Flake8, we transformed notebook files into Python scripts since Flake8 doesn’t work directly on notebooks. Using GPT-4o-mini and Gemini 3 Flash for bug detection at scale requires automatically scanning model outputs and determining if a potential bug has been flagged. To achieve this, we prompted the models to respond with only binary answers (“YES” or “NO”) using the instruction: “You are performing file-level bug detection. Respond YES if there is a bug in the file; otherwise, respond NO. Only ever respond with YES or NO.” Capitalization and punctuation in the model

outputs were removed for consistency. Out of 18,195 answers generated by GPT-4o-mini, 1,055 (6%) were non-binary, while Gemini 3 Flash produced 3,646 (20%) non-binary answers. These responses were omitted from our evaluation, as their large number made manual verification impractical. Table IV reports the F1 scores, precision, and accuracy of file-level predictions. JupOtter-base outperformed GPT-4omini and Gemini 3 Flash on two of the three test datasets in terms of F1 score. JupOtter achieved F1 scores of 0.76 and 0.68 on the OtterDataset test set and CodeParrot respectively, while Gemini 3 Flash had an F1 score of 0.72 on OtterDataset test set, and all 3 baselines achieved an F1 score of 0.48 or less on the CodeParrot dataset. GPT-4o-mini performed best on the Jupyter Errors dataset, with an F1 score of 0.87. GPT-4o-mini, Gemini 3 Flash, and Flake8 perform strongly on the Jupyter Errors dataset, where every untruncated notebook contains at least one runtime error (94% of notebooks in our truncated testing set). This strong performance on the Jupyter Errors dataset is likely due to the baselines being biased towards classifying notebooks as buggy, shown by their lower precision on the two other datasets. These results indicate that JupOtter complements existing static analysis and language model-based tools. While Flake8, Gemini 3 Flash, and GPT-4o-mini tend to classify notebooks as buggy, resulting in higher false positive rates, JupOtter-base is more conservative, with higher precision and performing strongly on datasets containing error-free notebooks, such as OtterDataset and CodeParrot. This suggests JupOtter may be particularly useful in file-level bug detection scenarios where minimizing false positives is important, whereas Flake8, Gemini 3 Flash, or GPT-4o-mini may be more suitable when maximizing recall is the priority. The weakness of JupOtter-base on the Jupyter Error benchmark may suggest that it struggles when detecting runtime errors present in this dataset. Overall, these results suggest that although file-level bug detection is not the intended use for JupOtter-base, it still outperforms traditional static analysis tools and general large language models at file-level bug detection tasks, though runtime errors remain a challenge. To measure whether there is a statistically significant difference in the performance of JupOtter-base and Flake8, we used the McNemar test, measuring the significance of the difference in performance on each dataset. We found a significant statistical difference with p values < 0.01 on all three testing sets. We did not perform paired statistical testing between GPT-4omini, Gemini 3 Flash, and JupOtter because GPT-4o-mini and Gemini 3 Flash were unable to generate predictions for every notebook, leaving many samples unpaired. RQ3 Summary: JupOtter-base outperformed the baselines in terms of F1 score on two of three test datasets, achieving F1 scores of 0.76 vs. 0.72 and 0.68 vs. 0.48 relative to the strongest performing baseline.

TABLE IV F ILE - LEVEL RESULTS FOR J UP OTTER - BASE , F LAKE 8, GPT-4 O - MINI , AND G EMINI 3 F LASH ON DIFFERENT BENCHMARKS . B EST RESULTS ARE IN BOLD . Tech. Flake8 JupOtterbase GPT4o-mini Gemini3-F lash

Pre. 0.61 0.88 0.55 0.61

OtterDataset Test Rec. F1 Acc. 0.77 0.67 0.87 0.88

0.68 0.76 0.67 0.72

0.64 0.80 0.58 0.66

Pre.

CodeParrot Rec. F1

Acc.

Pre.

0.31 0.62 0.29 0.33

0.94 0.76 0.91 0.87

0.51 0.83 0.46 0.58

0.92 0.94 0.92 0.93

0.47 0.68 0.44 0.48

Jupyter Errors Rec. F1 Acc. 0.72 0.35 0.83 0.78

0.81 0.51 0.87 0.85

0.69 0.38 0.78 0.75

D. RQ4: How does each contribution affect performance?

V. T HREATS TO VALIDITY

In this section, we present an ablation study to quantify the contribution of JupOtter’s components, in particular our notebook-aware tokenization strategy and our use of multisegment processing for long notebooks. Models in this section were tested using the OtterDataset test set. 1) Impact of the new tokenization strategy: Since CodeT5 cannot perform bug detection out of the box, we first trained a baseline model using the original CodeT5 tokenizer with a classification head that outputs a single logit for file-level prediction. This model achieves an F1 score of 0.73, compared to 0.76 for JupOtter-base. More importantly, this minimal baseline cannot localize bugs at the cell level and fails to fully tokenize 84% of notebooks due to context length limitations, compared to only 7% with JupOtter using our tokenization strategy. This demonstrates that our new tokenization strategy is not only necessary for localization but also crucial for processing the majority of notebooks. While our tokenization strategy has a limited impact on filelevel prediction (+3% F1), it is required for predictions at the cell-level, as it enables distinguishing between individual cells and allows processing of 77% of notebooks that would otherwise exceed token limits. Additionally, we attempted to train a model that directly classifies token embeddings without first using an encoder. This model failed to learn and predicted all samples as non-buggy (the majority class), confirming that contextual encoding is required prior to classification. 2) Impact of the multi-segment tokenization: We trained a cell-level model that uses special boundary tokens to mark cells, but without splitting long notebooks into multiple chunks. This model achieves an F1 score of 0.78, compared to 0.81 for the full JupOtter model. While the performance drop is moderate, this baseline suffers from the same problem as our minimal baseline: it is unable to fully process 84% of notebooks due to context length limitations. In contrast, JupOtter’s tokenization strategy allows the model to scale to much larger notebooks using the same encoder size, where each chunk used effectively adds an entire encoder’s worth of tokens which can be processed.

Data Leakage: A potential threat to the validity of our results is data leakage, specifically, the possibility that the models may have access to indirect information about the bugs under evaluation. We mitigate this threat in three ways. First, the pre-trained model we use, CodeT5, was trained on the CodeSearchNet dataset [26] and additional data from C/C# repositories, none of which include Jupyter Notebooks. Therefore, there is no overlap between the pre-training corpus and our evaluation data. Second, for fine-tuning, we ensured that all duplicate files were removed to prevent any instance of the same notebook appearing in both the training and testing subsets of OtterDataset. Third, we addressed a common threat in defect prediction studies, where a test set includes buggy files whose fixed versions appear in the training set. This can introduce unrealistic “future knowledge” into the model. To avoid this, we did not use version history or commit data during the construction of OtterDataset, and ensured that multiple revisions of the same notebook could not appear across training and test sets. We cannot guarantee the absence of data leakage for the Gemini 3 Flash and GPT-4o-mini baselines. However, any such leakage would advantage these baselines rather than affect the performance of JupOtter.

RQ4 Summary: While JupOtter’s components provide performance improvements, their primary benefit is a substantial increase in scalability to large notebook lengths, while also enabling precise cell-level localization.

Generalization of our Results: The scope of our dataset may limit generalization. We trained and evaluated the model on a specific collection of notebooks that may not fully capture the diversity of coding styles, bug types, or domains. To mitigate this, we collected notebooks using a variety of queries to target diverse domains and error types and used two additional external test sets collected by other researchers. Incorrect Labeling: The quality of our labeled dataset depends on the accuracy and consistency of the bug annotations, which were derived through automated inspection. Any errors, inconsistencies, or biases in labeling could affect the model’s training and evaluation, potentially leading to underperformance on certain bug types. To assess the quality of our automatic labelling, we manually labeled a sample of our data and only observed minimal inconsistency (see section II-A1). VI. R ELATED W ORK Bugs in Jupyter Notebooks: Prior work examined Jupyter Notebooks and their associated bugs on GitHub and Stack Overflow. Researchers found that the most common types of bugs are environment errors and implementation errors [2]. Environment errors account for 35.6% of .ipynb errors on

GitHub and 43.2% on Stack Overflow, while implementation errors make up 44.2% of errors on GitHub and 22% on Stack Overflow. Additionally, both error types are growing rapidly on Stack Overflow, with implementation errors increasing at a rate of 48% per year and environment errors at 38%. Prior research has analyzed bugs and software across various contexts [8], [22], [23], [27]–[29]. Notably, Wang et al. [8] focus specifically on Jupyter Notebooks and highlight that although they are widely adopted by students, researchers, and data scientists, they often suffer from poor coding practices. Their analysis found common issues such as unused variables, lack of adherence to Python style conventions, and inconsistent code structure. The authors argue that these issues reduce the reproducibility and quality of research and advocate for automated tools to improve code quality in notebooks. Our work builds on this motivation by developing an automated technique to detect implementation bugs in Jupyter Notebooks, addressing an aspect of notebook reliability that goes beyond style and formatting issues. Other Notebook Studies: A growing body of research has explored the structure, quality, and usage patterns of Jupyter Notebooks. Several studies investigated the notebook ecosystem and detailed notebook characteristics [30], common cleaning activities [31], [32], code reuse [33], bugs [34], reproducibility [35], [36], or proposed support for debugging [37]. Others compared notebooks with traditional Python scripts [38], [39] or detailed best practices [40] and challenges in the notebook ecosystem [41], [42]. Machine learning techniques have also been explored for various tasks in notebook environments such as code generation [43], [44]. Defect Prediction and Bug Detection: To our knowledge, no prior work has explored cell-level bug detection in Jupyter Notebooks using language models. Previous work [45] is the closest related work and investigates file-level bug detection using GPT-4. The authors synthetically generated notebooks by splitting Python scripts into cells, then applied GPT-4 as a static analyzer to detect bugs, with and without runtime information. While informative, their approach does not operate at the cell-level and lacks real-world notebook structure. Outside of notebooks, numerous studies have explored defect prediction in traditional source code [7], [46]–[49]. ChatDBG [50] integrates LLMs with traditional debuggers to improve the usability and capabilities of traditional debuggers, and LOVA [5] leverages self-attention mechanisms within LLMs to localize vulnerabilities in source code. BUGLAB [3] presents a self-supervised learning approach for bug detection and repair, while DefectHunter [4] employs the Conformer architecture to identify vulnerabilities. Toggle [6] uses LLMs to predict bugs at the token level. Due to the lack of ML baselines for cell-level detection, we evaluated JupOtter against a static analyzer. Many Python static analyzers exist [51]–[55]. Among these, Flake8 [56] was selected as a baseline for its speed and configurability. Defect prediction approaches have used semantic representations to improve prediction [57]. Additionally, work has

evaluated the consistency and reliability of model-agnostic explainable techniques across different defect prediction settings, emphasizing that explainability remains a challenge even for well-trained models [58]. Prior work has also highlighted that most benchmark datasets for software bug prediction use randomly selected historical versions, which may not reflect real-world continuous software development [59]. Machine Learning Models for Code: The encoder we use to build JupOtter is CodeT5, a Transformer model pretrained on large-scale code datasets for program understanding tasks [11]. We chose it as a relatively lightweight model compared to more recent code language models. Its followup, CodeT5+, has demonstrated improved performance across benchmarks [60]. Other pre-trained models in this space include CodeBERT [10], CodeLlama [61], GraphCodeBERT [12], CuBERT [62], and PLBART [63]. These models were pre-trained on large amounts of code text data for code understanding tasks such as code summarization, generation, and defect prediction. While we chose CodeT5 as our base model, our approach is not bound to a specific model and could be applied to many pre-trained code models. We also drew on previous work [24] to address the challenge of processing large notebooks. This work processes full source documents as a series of non-overlapping pages rather than as a continuous sequence. Following this idea, we tokenize notebooks into multiple non-overlapping cell sequences. Notebook Datasets: We found no existing datasets with cell-level bug annotations for Python Jupyter Notebooks, which motivated the creation of OtterDataset (detailed in Section II-A). While prior datasets [18], [19], [43], [64]–[66] exist for analyzing notebooks, most are unlabeled or focus on usage patterns and code quality rather than fine-grained bug detection. We also re-labeled two datasets introduced in previous work that included buggy notebooks [18], [19]. VII. I MPLICATIONS AND F UTURE W ORK Practical Deployment: JupOtter is designed with practical deployment in mind. By supporting long notebooks and generating cell-level predictions, it offers a finer level of granularity than traditional file-level or function-level bug detectors. This enables developers to pinpoint specific faulty cells rather than being presented with vague file-wide warnings. This capability is especially valuable in educational and research contexts, where feedback needs to be actionable and precise. In contrast, function- or file-level tools may fail to provide applicable predictions for notebooks. By offering predictions at the celllevel, JupOtter better aligns with real-world notebook usage and has the potential to improve both developer productivity and code quality. Additionally, JupOtter’s prediction system can be modified to output a vector of predictions per cell rather than a single logit, enabling the model to indicate not only whether a cell is buggy, but also the type of error present. However, this would require a more extensive labeling process. A promising direction for future work is integrating JupOtter into popular notebook environments such as JupyterLab

or VS Code as a live assistant. This would allow users to receive real-time bug predictions while writing code, much like linting tools for scripts. To support this, future research could explore optimizing the model for latency and memory usage, investigating incremental inference strategies (e.g., predicting only changed cells), and providing confidence scores or explanations alongside predictions. Deployment pipelines could also include automatic triaging of predictions (e.g., flagging only high-confidence bugs) to reduce noise and improve user trust. By addressing these engineering and UX challenges, JupOtter could serve as a practical assistant that helps developers catch errors earlier and code with greater confidence. Extension to Markdown Cells: While effective, our tokenization strategy only considers code cells and treats chunking strictly in terms of token length, which excludes context from markdown cells. One future direction is to extend our tokenization strategy to create chunks of tokens based on context boundaries indicated within markdown headings. Incorporating this structure may help models better understand code in context and improve detection. Summary: More broadly, JupOtter opens the door to cell-level predictions in notebook environments. Notebooks are widely used in education, research, and data science, yet tools for analyzing and improving their reliability are still limited. Our work provides a foundation for future tools that integrate with notebook workflows and help authors write more correct and maintainable code. VIII. C ONCLUSION In this paper, we introduced JupOtter, a machine learningbased system designed specifically for cell-level bug detection in Jupyter Notebooks. Our approach incorporates a novel tokenization strategy that preserves notebook structure, a prediction method that operates at the cell-level, and a training strategy tailored for multi-segment inputs. We also released OtterDataset, a large-scale labeled dataset for supervised bug detection in Jupyter Notebooks, comprising 21,303 notebook files. Through extensive evaluation across three benchmarks, we demonstrated that JupOtter achieves strong performance on both cell-level and file-level bug detection tasks, outperforming static analysis tools as well as large language models. DATA AVAILABILITY Training and evaluation code, trained model parameters, configuration for baselines, error examples, datasets, and notebook labeling code are available on Zenodo [21]. ACKNOWLEDGMENTS ChatGPT was used for table formatting, text editing (e.g., grammar and clarity), and writing routine portions of code. AI was not used to generate results, figures, datasets, or core contributions. This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

R EFERENCES [1] J. M. Perkel, “Why jupyter is data scientists’ computational notebook of choice,” Nature, vol. 563, no. 7732, pp. 145–147, 2018. [2] T. L. De Santana, P. A. d. M. S. Neto, E. S. De Almeida, and I. Ahmed, “Bug analysis in jupyter notebook projects: an empirical study,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 4, pp. 1–34, 2024. [3] M. Allamanis, H. Jackson-Flux, and M. Brockschmidt, “Self-supervised bug detection and repair,” Advances in Neural Information Processing Systems, vol. 34, pp. 27 865–27 876, 2021. [4] J. Wang, Z. Huang, H. Liu, N. Yang, and Y. Xiao, “Defecthunter: A novel llm-driven boosted-conformer-based code vulnerability detection mechanism,” arXiv preprint arXiv:2309.15324, 2023. [5] Y. Li, X. Li, H. Wu, Y. Zhang, X. Cheng, Y. Liu, F. Xu, and S. Zhong, “If llms would just look: Simple line-by-line checking improves vulnerability localization,” arXiv preprint arXiv:2410.15288, 2024. [6] S. B. Hossain, N. Jiang, Q. Zhou, X. Li, W.-H. Chiang, Y. Lyu, H. Nguyen, and O. Tripp, “A deep dive into large language models for automated bug localization and repair,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1471–1493, 2024. [7] M. Pradel and K. Sen, “Deepbugs: A learning approach to name-based bug detection,” Proceedings of the ACM on Programming Languages, vol. 2, no. OOPSLA, pp. 1–25, 2018. [8] J. Wang, L. Li, and A. Zeller, “Better code, better sharing: on the need of analyzing jupyter notebooks,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering: new ideas and emerging results, 2020, pp. 53–56. [9] W. Jiang, D. Pressato, H. Darji, and T. Lutellier, “Exploring the jupyter ecosystem: An empirical study of bugs and vulnerabilities,” in 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 2025, pp. 92–103. [10] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” in Findings of the association for computational linguistics: EMNLP 2020, 2020, pp. 1536–1547. [11] Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 8696–8708. [12] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al., “Graphcodebert: Pre-training code representations with data flow,” arXiv preprint arXiv:2009.08366, 2020. [13] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 7212–7225. [14] C. Pornprasit and C. K. Tantithamthavorn, “Deeplinedp: Towards a deep learning approach for line-level defect prediction,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 84–98, 2022. [15] J. Li, P. He, J. Zhu, and M. R. Lyu, “Software defect prediction via convolutional neural network,” in 2017 IEEE international conference on software quality, reliability and security (QRS). IEEE, 2017, pp. 318–328. [16] H. K. Dam, T. Pham, S. W. Ng, T. Tran, J. Grundy, A. Ghose, T. Kim, and C.-J. Kim, “A deep tree-based model for software defect prediction,” arXiv preprint arXiv:1802.00921, 2018. [17] D. Hin, A. Kan, H. Chen, and M. A. Babar, “Linevd: Statement-level vulnerability detection using graph neural networks,” in Proceedings of the 19th international conference on mining software repositories, 2022, pp. 596–607. [18] K. Grotov, S. Titov, Y. Zharov, and T. Bryksin, “Untangling knots: Leveraging llm for error resolution in computational notebooks,” arXiv preprint arXiv:2405.01559, 2024. [19] CodeParrot, “Github jupyter notebooks dataset,” https://huggingface.co/ datasets/codeparrot/github-jupyter, 2022. [20] P. S. Foundation, Built-in Exceptions, 2024, accessed: February 26, 2025. [Online]. Available: https://docs.python.org/3/library/exceptions.html [21] L. Ottenhof and T. Lutellier, “Jupotter: Cell-level bug detection in jupyter notebooks, supplementary material,” 2026. [Online]. Available: https://zenodo.org/records/20706524 [22] J. F. Pimentel, L. Murta, V. Braganholo, and J. Freire, “A largescale study about quality and reproducibility of jupyter notebooks,”

in 2019 IEEE/ACM 16th international conference on mining software repositories (MSR). IEEE, 2019, pp. 507–517. [23] Y. Wang, W. Meijer, J. A. H. Lopez, U. Nilsson, and D. Varro, “Why do machine learning notebooks crash? an empirical study on public python jupyter notebooks,” IEEE Transactions on Software Engineering, 2025. [24] Y. Liu, A. Ni, L. Nan, B. Deb, C. Zhu, A. Hassan, and D. Radev, “Leveraging locality in abstractive text summarization,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 6081–6093. [25] M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based linelevel vulnerability prediction,” in Proceedings of the 19th International Conference on Mining Software Repositories, 2022, pp. 608–620. [26] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436, 2019. [27] Y. Tang, H. Cao, Y. Chen, and D. Lo, “Characterising bugs in jupyter platform,” in Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, 2025, pp. 136– 146. [28] L. Tan, C. Liu, Z. Li, X. Wang, Y. Zhou, and C. Zhai, “Bug characteristics in open source software,” Empirical software engineering, vol. 19, pp. 1665–1705, 2014. [29] D. Cotroneo, R. Pietrantuono, S. Russo, and K. Trivedi, “How do bugs surface? a comprehensive study on the characteristics of software bugs manifestation,” Journal of Systems and Software, vol. 113, pp. 27–43, 2016. [30] M. Källén and T. Wrigstad, “Jupyter notebooks on github: Characteristics and code clones,” The Art, Science, and Engineering of Programming, vol. 5, no. 3, pp. 15–1, 2021. [31] H. Dong, S. Zhou, J. L. Guo, and C. Kästner, “Splitting, renaming, removing: a study of common cleaning activities in jupyter notebooks,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). IEEE, 2021, pp. 114–119. [32] R. L. Cunha, L. C. V. Real, R. Souza, B. Silva, and M. A. Netto, “Context-aware execution migration tool for data science jupyter notebooks on hybrid clouds,” in 2021 IEEE 17th international conference on eScience (eScience). IEEE, 2021, pp. 30–39. [33] N. Ritta, T. Settewong, R. G. Kula, C. Ragkhitwetsagul, T. Sunetnanta, and K. Matsumoto, “Reusing my own code: Preliminary results for competitive coding in jupyter notebooks,” in 2022 29th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 2022, pp. 457–461. [34] J. Patra and M. Pradel, “Nalin: learning from runtime behavior to find name-value inconsistencies in jupyter notebooks,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 1469– 1481. [35] J. Wang, T.-y. Kuo, L. Li, and A. Zeller, “Assessing and restoring reproducibility of jupyter notebooks,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 138–149. [36] ——, “Restoring reproducibility of jupyter notebooks,” in Proceedings of the ACM/IEEE 42nd international conference on software engineering: Companion proceedings, 2020, pp. 288–289. [37] M. V. Merino, L. T. van Binsbergen, and M. Seraj, “Making the invisible visible in computational notebooks,” in 2022 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 2022, pp. 1–3. [38] K. Grotov, S. Titov, V. Sotnikov, Y. Golubev, and T. Bryksin, “A largescale comparison of python code in jupyter notebooks and scripts,” in Proceedings of the 19th international conference on mining software repositories, 2022, pp. 353–364. [39] K. Adams, A. Vilkomir, and M. Hills, “A comparison of machine learning code quality in python scripts and jupyter notebooks,” Journal of Computing Sciences in Colleges, vol. 39, no. 5, pp. 96–108, 2023. [40] L. Quaranta, F. Calefato, and F. Lanubile, “Eliciting best practices for collaboration with computational notebooks,” Proceedings of the ACM on Human-Computer Interaction, vol. 6, no. CSCW1, pp. 1–41, 2022. [41] T. Settewong, N. Ritta, R. G. Kula, C. Ragkhitwetsagul, T. Sunetnanta, and K. Matsumoto, “Why visualize data when coding? preliminary categories for coding in jupyter notebooks,” in 2022 29th Asia-Pacific Software Engineering Conference (APSEC). IEEE, 2022, pp. 462–466. [42] S. Chattopadhyay, I. Prasad, A. Z. Henley, A. Sarma, and T. Barik, “What’s wrong with computational notebooks? pain points, needs, and design opportunities,” in Proceedings of the 2020 CHI conference on human factors in computing systems, 2020, pp. 1–12.

[43] S. Chandel, C. B. Clement, G. Serrato, and N. Sundaresan, “Training and evaluating a jupyter notebook data science assistant,” arXiv preprint arXiv:2201.12901, 2022. [44] P. Yin, W.-D. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski et al., “Natural language to code generation in interactive data science notebooks,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 126–173. [45] Y. Wang, J. A. H. López, U. Nilsson, and D. Varro, “Using run-time information to enhance static analysis of machine learning code in notebooks,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 497– 501. [46] K. Gaddamwar, Y. Srivastava, J. Parashar, R. A. Parmar, A. K. Pandey, and V. S. Kushwah, “Deep learning for contextual bug detection and automated fixes in software systems,” in 2024 2nd International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT), vol. 1. IEEE, 2024, pp. 624–629. [47] N. Jiang and Y. Wu, “Repaircat: Applying large language model to fix bugs in ai-generated programs,” in Proceedings of the 5th ACM/IEEE International Workshop on Automated Program Repair, 2024, pp. 58– 60. [48] E. Azizi and L. Zaman, “Astrobug: Automatic game bug detection using deep learning,” IEEE Transactions on Games, 2024. [49] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” in Proceedings of the Network and Distributed System Security Symposium. The Internet Society, 2018. [50] K. H. Levin, N. van Kempen, E. D. Berger, and S. N. Freund, “Chatdbg: Augmenting debugging with large language models,” Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 1892–1913, 2025. [51] L. Quaranta, F. Calefato, and F. Lanubile, “Pynblint: a static analyzer for python jupyter notebooks,” in Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI, 2022, pp. 48–49. [52] PyCQA, “Pylint - static code analyzer for python,” https://pylint.pycqa. org, 2025, version 3.1.0, accessed July 2025. [53] P. Frost and contributors, “Pyflakes - static code analyzer for python,” https://github.com/PyCQA/pyflakes, 2025, version 3.2.0, accessed July 2025. [54] SonarSource, “Sonarqube,” https://www.sonarqube.org, 2024, version 10.4, accessed July 2025. [55] P. Subotić, L. Milikić, and M. Stojić, “A static analysis framework for data science notebooks,” in Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice, 2022, pp. 13–22. [56] PyCQA, “Flake8,” https://flake8.pycqa.org/, 2025. [57] S. Wang, T. Liu, J. Nam, and L. Tan, “Deep semantic feature learning for software defect prediction,” IEEE Transactions on Software Engineering, vol. 46, no. 12, pp. 1267–1293, 2018. [58] J. Shin, R. Aleithan, J. Nam, J. Wang, and S. Wang, “Explainable software defect prediction: Are we there yet?” arXiv preprint arXiv:2111.10901, 2021. [59] S. Wang, J. Wang, J. Nam, and N. Nagappan, “Continuous software bug prediction,” in Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2021, pp. 1–12. [60] Y. Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi, “Codet5+: Open code large language models for code understanding and generation,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 1069–1088. [61] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023. [62] A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in International conference on machine learning. PMLR, 2020, pp. 5110–5121. [63] W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pretraining for program understanding and generation,” in Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies, 2021, pp. 2655–2668. [64] M. Mostafavi Ghahfarokhi, A. Asgari, M. Abolnejadian, and A. Heydarnoori, “Distilkaggle: A distilled dataset of kaggle jupyter notebooks,”

in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 647–651. [65] L. Quaranta, F. Calefato, and F. Lanubile, “Kgtorrent: A dataset of python jupyter notebooks from kaggle,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 550–554. [66] V. Paruchuri, “vikp clean notebooks filtered,” https://huggingface.co/ datasets/vikp/clean notebooks filtered, 2023.