PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns

PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns Syrym Abdikhan

Mazhar Hameed

Syrym.Abdikhan@gisma-student.com Gisma University of Applied Sciences Potsdam, Germany

mazhar.hameed@gisma.com Gisma University of Applied Sciences Potsdam, Germany

arXiv:2608.14742v1 [cs.SE] 13 Aug 2026

ABSTRACT

our empirical understanding of how Pandas-based workflows are constructed and evolve within notebooks remains limited. Prior research has analyzed Jupyter notebooks at scale, focusing on general aspects, such as code structure, reproducibility, and coding practices [9, 22, 27]. Similarly, other efforts have examined computational notebooks to understand machine learning practices, reproducibility challenges, and coding habits [27–29]. Moreover, some studies have focused on specific libraries, such as NumPy [3], which is widely used for numerical computations [10, 11, 30]. While these efforts have provided valuable insights into notebook-based development, they do not focus on Pandas workflows specifically, nor do they examine how Pandas operations are composed and evolve in real-world notebook-based data analysis. Analyzing Pandas-based workflows is critical for understanding how data manipulation tasks are structured in practice and for informing the design of data analysis tools and systems [8, 19]. Although Pandas is widely adopted, its Pythonic execution model is not always optimized for performance, motivating extensive efforts to accelerate it through parallelization, distribution, or alternative execution backends, as seen in systems, such as Xorbits [16], CuDF [1], Modin [20, 21], and Dias [6]. Such efforts depend on representative, real-world Pandas code to guide system design and optimization. Beyond performance, an empirical understanding of Pandas operation usage and composition patterns is also essential for emerging applications, including automated data pipeline generation, benchmarking of data-processing systems, and the integration of large language models for code generation and transformation [6, 14, 20, 26]. However, the absence of a dedicated resource capturing realworld Pandas workflows creates a significant gap. While platforms, such as GitHub host a vast number of Jupyter notebooks, these notebooks have not yet been systematically analyzed to provide Pandas-specific insights at scale. This limits our ability to answer fundamental questions about how Pandas is used in practice, which operations dominate real workloads, and how workflows evolve over time [19, 21]. To address this gap, we introduce PandasCorpus, a resource of real-world Pandas workflows collected from GitHub repositories spanning the period from 2015 to 2025. The resource comprises 139k Jupyter notebooks from approximately 100k distinct repositories, organized into structured corpora and stored in Parquet format for efficient access. We characterize notebooks using general structural characteristics and Pandas-specific workflow characteristics, and conduct empirical analyses of notebook evolution and Pandas API usage, including temporal trends, code executability, and recurring operation sequences observed in practice.

Pandas has emerged as the de facto library for data processing and machine learning, widely used for tasks, such as data loading, transformation, and analysis. Despite its ubiquity, there has been limited systematic investigation into how Pandas is used in real-world projects and how typical workflows are composed in practice. To address this gap, we introduce PandasCorpus, a dataset curated from GitHub repositories that captures real-world Pandas workflows at scale. In this work, a workflow refers to Pandas-based code contained in Jupyter notebooks, a prevalent medium for writing, executing, and sharing data analysis code. The dataset comprises 139k notebooks from approximately 100k repositories and captures more than 4M Pandas API calls spanning 136 distinct operations. Beyond dataset construction, we characterize workflows using structural and Pandas-specific features and analyze notebook evolution between 2015 and 2025. Our study examines code executability, notebook size, and recurring sequences of Pandas operations, providing empirical insights into how Pandas is used in practice. The resulting corpus offers a reusable resource for studying data analysis workflows, Pandas usage patterns, and library-aware code composition. Both the dataset and the extraction pipeline are publicly available via GitHub1 and Zenodo2 .

KEYWORDS Pandas, Jupyter Notebooks, Data Analysis Workflows, Software Repository Mining, Code Analysis

1

PANDAS AS THE DE-FACTO LIBRARY

In recent years, data science and software engineering have increasingly relied on libraries that simplify and accelerate data manipulation [15, 17, 24]. Among these, Pandas has become the de-facto standard for many data-driven workflows, offering a rich set of operations for data loading, transformation, and analysis [12, 13, 18, 23]. While this flexibility empowers users, it also shifts substantial responsibility onto them, including how to compose operations effectively and efficiently. As a result, understanding how Pandas is used in real-world settings has become increasingly important. This need is particularly pronounced in the context of computational notebooks, such as Jupyter [4], which have emerged as the dominant medium for experimentation, teaching, and practical data science work. Jupyter notebooks are lightweight, easy to share, and support a wide ecosystem of libraries, making them widely adopted across academia and industry [5, 9, 24, 25]. Despite their popularity, 1 https://github.com/SyrymAbdikhan/PandasCorpus 2 https://doi.org/10.5281/zenodo.17367371

1

Syrym Abdikhan and Mazhar Hameed

Table 1: PandasCorpus Statistics Name

Count

Pandas operations used for notebook crawling Total repositories crawled Jupyter notebooks in crawled repositories (25 GB) Repositories containing Pandas-based workflows Jupyter notebooks with Pandas-based workflows (18 GB) Repositories containing datasets (e.g., CSV files) Code cells extracted Markdown cells extracted

136 137.1k 197.9k 99.9k 139.0k 62.2k 3.8M 1.4M

with indicators of Pandas usage, such as import statements and file-reading operations. Queries are executed in both ascending and descending order to partially mitigate GitHub API limitations. The resulting notebook URLs are deduplicated, yielding approximately 198k candidate notebooks from 137k repositories. In the second stage, candidate notebooks are downloaded and filtered using the same relevance checks applied during discovery. This process results in 139k notebooks from approximately 100k repositories. Finally, the notebooks are organized into separate code-cell and markdown-cell corpora containing approximately 3.8M and 1.4M cells, respectively. While both corpora are retained, the analyses in this study focus on code cells. The dataset is stored in compressed Parquet format for efficient storage and access. Code Parsing and Characterization. We analyze individual code cells to characterize Pandas usage within notebooks. Before parsing, code is sanitized by removing IPython magic and shell commands and resolving minor syntax inconsistencies. The cleaned code is then parsed into an Abstract Syntax Tree (AST)3 , enabling precise identification of function calls and syntactic constructs. We traverse the AST to extract notebook- and workflow-level characteristics. Compared to substring-based matching, AST-based analysis reduces false positives by distinguishing executable code from comments and string literals, although occasional overlaps with non-Pandas objects may still occur.

Our paper makes the following main contributions: • PandasCorpus, a resource of real-world Pandas workflows collected from GitHub repositories, covering Jupyter notebooks created between 2015 and 2025. • A reproducible pipeline for collecting, filtering, and organizing Jupyter notebooks containing Pandas-based workflows, enabling structured analysis at the notebook level. • An empirical study of Pandas usage in practice, including dataset characterization, temporal trends in notebook evolution, and recurring operation and workflow patterns. The remainder of this paper is organized as follows. Section 2 introduces PandasCorpus and describes the data collection and processing pipeline. Section 3 presents empirical insights into the dataset, focusing on notebook characteristics and Pandas usage patterns observed in practice. Section 4 discusses potential use cases enabled by PandasCorpus and concludes the paper.

2

2.2

We organize the extracted information into two groups of workflow characteristics: general notebook-level characteristics and Pandasspecific workflow characteristics. The first group captures structural properties of notebooks, including code size, organization, and complexity, while the second focuses on Pandas API usage and workflow composition patterns. General Notebook-Level Characteristics. The general notebooklevel characteristics summarized in Table 2 are commonly used in prior studies on computational notebooks [7, 9, 22, 27]. We retain these established characteristics to provide a structural baseline and enable comparison with existing notebook corpora, rather than as Pandas-specific descriptors. The results show substantial variation in notebook size and structure. The median notebook contains 139 lines of code and exhibits a nested block depth of two, suggesting moderate structural complexity. The distributions are right-skewed, with a subset of notebooks containing thousands of lines or statements, reflecting the heterogeneity of coding practices across Jupyter notebooks. Building on this general characterization, PandasCorpus additionally captures Pandas-specific usage and composition characteristics that are not represented by these conventional notebook-level measures. Pandas-Specific Workflow Characteristics. While the general characteristics describe the structure of computational notebooks, they do not capture how data-manipulation operations are used and composed within them. To provide a Pandas-oriented representation of real-world workflows, PandasCorpus therefore augments these conventional notebook characteristics with a set of Pandas-specific characteristics capturing API usage frequency, chained operations, chained indexing, visualization activity, model-training

PandasCorpus

This section introduces PandasCorpus and describes how the resource was constructed from GitHub repositories. We first outline the data collection and filtering process used to identify Jupyter notebooks containing Pandas-based workflows, and then describe how the resulting notebooks are organized and characterized.

2.1

Workflow Characterization Dimensions

Process and Background

Data Source. We collected data from public GitHub repositories using GitHub’s REST API [2]. The collection targeted Jupyter notebooks containing references to the Pandas API and was conducted in October 2025, covering repositories created between 2015 and 2025. To guide the data collection process, we selected 136 distinct Pandas API operations as search targets. These operations represent a comprehensive set of DataFrame-centric methods and functions that form the backbone of typical Pandas workflows. They include core manipulation operations (e.g., drop, join, groupby), data-cleaning steps (e.g., fillna, replace, transform), aggregation functions (e.g., agg, sum, mean), reshaping transformations, time-series operations, and common type-conversion utilities. Table 1 summarizes the overall statistics of PandasCorpus. The collection complied with GitHub’s Terms of Service and excluded personally identifiable information. Collection Pipeline. The collection pipeline consists of two stages: notebook discovery and content retrieval. We first generate query templates for 136 selected Pandas API operations combined

3 https://docs.python.org/3/library/ast.html

2

PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns

Table 2: Statistics of General Notebook-Level and PandasSpecific Workflow Characteristics #

Characteristic

#Notebooks

Med

Max

139,032 127,215 139,032 77,225 78,272 107,522 139,030 139,027 139,030 139,027 139,032 138,932

139 17 86 1 1 2 85 51 13 4 134 8

6,764 1,229 3,909 1,254 325 18 9,615 6,176 3,421 30 5,207 608

139,032 139,032 139,032 139,032 136,960 136,960 136,960 59,542 57,322 39,738

18 2 16 18 1 24 25 0 0 0

2,918 95 2,112 2,918 13 4,169 5,129 234 91 261

3.1

Notebook-Level Characteristics 1 2 3 4 5 6 7 8 9 10 11 12

Lines of Code Number of Blank Lines of Code Number of Statements Number of Parameters Number of User-Defined Functions Nested Block Depth Number of Operands Number of Operators Number of Unique Operands Number of Unique Operators Number of Identifiers Number of Imports

Notebook Executability Over Time

Figure 1 shows the percentage of notebooks containing at least one syntax-level error, as detected through AST parsing, over the period 2015–2025. Errors were identified when the Python ast module raised a SyntaxError during parsing. Since the AST parser halts at the first syntax violation, the reported values reflect notebooks containing at least one syntactic inconsistency. The results indicate a substantial decrease in syntax-level errors over time. In 2015, approximately 74.6% of notebooks contained syntax errors. This percentage decreases steadily over time, reaching 10.7% in 2024, followed by a slight increase to 11.6% in 2025. Overall, the long-term trajectory indicates substantial improvement in notebook syntactic validity. At the same time, the number of notebooks grows dramatically, particularly after 2022, increasing from fewer than 5k notebooks per year before 2021 to 30k in 2024 and nearly 40k in 2025. Despite this rapid growth, the relative error rate continues to decline, suggesting that increased volume does not correspond to increased syntactic instability.

Pandas-Specific Characteristics 1 2 3 4 5 6 7 8 9 10

Number of Pandas Operations Longest Chained Operation Number of Chained Operations Total Length of Chained Operations Longest Chained Indexing Number of Chained Indexing Total Length of Chained Indexing Number of Visualizations Number of Training Calls Number of Shell Commands

calls, and the use of IPython magic and shell commands. Chained operations describe compositions of multiple Pandas calls within a single expression (e.g., df.groupby().agg().reset_index()), while chained indexing captures successive indexing operations (e.g., df[mask][col]). Table 2 summarizes the resulting characteristics and their statistics. Notebook-level statistics for these characteristics provide an overview of how Pandas is used in practice. Each notebook contains at least one Pandas operation, with a median of 18 operations and an interquartile range between 9 and 35. The usage distribution is highly skewed, with a small number of notebooks containing nearly 3,000 Pandas operations. However, the median longest chained operation is 2, indicating that multi-step operation chaining is common across notebooks. These characteristics provide a descriptive framework for analyzing notebook structure and Pandas usage. In the following section, we examine notebook executability, temporal trends, Pandas operation usage, and workflow composition patterns.

3

Figure 1: Notebooks Containing Syntax Errors Over Time

A pronounced drop is observed between 2016 and 2017, where the error rate decreases from 63.3% to 36.4%. While the dataset does not allow us to attribute this shift to a single cause, it may reflect broader ecosystem changes, such as maturation of notebook tooling, evolution of Python versions, or improved development practices. Similarly, the continued decline after 2022 coincides with a period of rapid growth in notebook creation, though causal interpretation requires further investigation. To better understand the nature of syntax failures, Table 3 summarizes the distribution of error categories extracted from parsing errors. The majority of errors are structural syntax issues, including unmatched or unclosed brackets (42.07%), unterminated strings (21.05%), and generic syntax errors (16.51%). Indentation-related errors account for an additional 10.4%. Since errors are detected statically at parse time, these categories reflect syntactic inconsistencies rather than runtime or semantic failures. The dominance of bracket- and string-related errors suggests that many notebooks contain incomplete or partially written code fragments, which is consistent with the iterative and exploratory nature of real-world notebook-based development.

ANALYSIS AND FINDINGS

Building on the characterization of PandasCorpus presented in Section 2, we now examine how Pandas workflows manifest in practice. This section investigates notebook executability over time, common error types, the evolution of Pandas API usage, and recurring operation sequences observed across the dataset. 3

Syrym Abdikhan and Mazhar Hameed

Table 3: Syntax Error Categories Error Category

Percentage

Unmatched or unclosed brackets Unterminated string literal Generic syntax error Indentation-related error Missing comma in expression Invalid decimal literal Invalid or unexpected character Other

3.2

42.07% 21.05% 16.51% 10.4% 3.87% 3.32% 1.99% 0.77%

Figure 2: Temporal evolution of the top 10 Pandas operations (ranked by overall usage).

Pandas Operations Coverage

The Pandas API provides a broad collection of operations for data manipulation, aggregation, reshaping, and transformation. Identifying which of these operations are most frequently used helps characterize practical usage patterns and uncover the core building blocks of real-world workflows. To examine how Pandas is used in practice, we analyze the frequency of 136 DataFrame-centric operations across all notebooks in the corpus. Table 4 presents all 136 Pandas DataFrame operations ranked in descending order by usage frequency. Figure 2 illustrates the usage trends of the most frequently invoked operations from 2015 to 2025 (log-scaled counts). The distribution is highly skewed. A small subset of core operations dominates usage throughout the entire period. In total, the top 15 operations account for more than half of all recorded Pandas calls, while the least-used operations contribute negligibly to overall usage. Core manipulation and aggregation operations, such as head, mean, sum, groupby, and drop consistently rank among the most frequent across all years. Their steady growth over time reflects both the increasing volume of notebooks and the central role of these operations in typical workflows. Less frequently used operations, including specialized reshaping or time-related transformations, appear sporadically and at much lower magnitudes. This long-tail pattern suggests that real-world Pandas workflows are built around a compact set of recurring transformation primitives rather than the full breadth of the API. Figure 2 illustrates the distribution of selected operations, highlighting the dominance of a small subset of functions, while Table 4 provides the complete ranked list of all 136 Pandas DataFrame operations.

3.3

(2) sequences must appear in at least 1% of notebooks (≈1,400 notebooks); (3) trivial repetitions of the same operation (e.g., repeated mean or apply) are excluded. The resulting sequences, summarized in Table 5, are grouped into four categories reflecting common workflow stages: integration and construction, exploration and validation, cleaning and transformation, and aggregation and statistics. Overall, the identified sequences reveal that Pandas workflows exhibit recurring structural patterns rather than arbitrary compositions. This consistency suggests that real-world Pandas usage is organized around a limited set of stable workflow motifs. Integration & Construction. Sequences in this category are characterized by repeated occurrences of read_csv, join, merge, and concat, indicating that data loading and integration form a central stage of many workflows. Patterns, such as read_csv → concat and read_csv → merge suggest that analysts frequently combine multiple external data sources early in the workflow. The interleaving of inspection operations (e.g., head → merge → head) indicates that integration steps are often validated immediately after execution. In addition, sequences, such as DataFrame → to_csv → read_csv reveal intermediate materialization and reloading of data, possibly for debugging or restructuring purposes. Exploration & Validation. Sequences in this category highlights the importance of quick visual inspection in notebook-based workflows. Typical patterns, such as read_csv → head → describe and read_csv → head → info indicate that data exploration commonly begins immediately after loading a dataset. Validationrelated sequences, including head → isnull → sum and sum → duplicated → sum, suggest systematic checks for missing values and duplicate records. These patterns reflect standard data quality verification steps embedded directly into exploratory workflows. The frequent alternation between sorting and inspection (e.g., sort_values → head) further illustrates the iterative nature of exploration, where users repeatedly refine views of the data. Cleaning & Transformation. Sequences in this category reveal structured preprocessing behavior preceding analysis. Patterns, such as isnull → sum → dropna and duplicated → sum → drop_duplicates indicate that cleaning operations are typically preceded by explicit validation steps, suggesting that users first

Composition Patterns of Pandas Operations

While frequency analysis reveals which operations dominate individually, it does not capture how operations are composed in practice. To better understand workflow structure, we analyze recurring sequences of Pandas API calls observed across notebooks. These sequences reflect higher-level composition patterns that characterize real-world Pandas workflows. To identify such patterns, we extract sequences of executable Pandas API calls by analyzing control-flow paths within notebooks. Only operations reachable through actual execution paths are considered. To focus on meaningful patterns, we apply three filtering criteria: (1) sequences must have a minimum length of two operations; 4

PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns

Table 4: Frequency of 136 Pandas operations across PandasCorpus, ranked by usage. Counts represent the number of observed operation calls, and percentages indicate their share of all recorded Pandas calls. #

Operation

Count

%

#

Operation

Count

%

#

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46

head mean sum groupby drop apply astype reset_index join replace value_counts sort_values copy unique concat fillna merge rename dropna max isnull get isin transform set_index min describe map std isna where count agg round drop_duplicates shift median sample nunique to_numpy quantile rolling query tail abs corr

300877 223797 202306 198799 156530 144117 135149 132761 115077 102480 100374 99509 88413 87775 87511 85858 84685 72700 72697 70321 59656 52841 50553 49427 48745 48635 48367 47434 42621 40075 35487 35448 35098 32721 29131 28018 26052 23959 23825 23540 23406 22915 22896 22221 21237 20878

7.50 5.58 5.04 4.95 3.90 3.59 3.37 3.31 2.87 2.55 2.50 2.48 2.20 2.19 2.18 2.14 2.11 1.81 1.81 1.75 1.49 1.32 1.26 1.23 1.21 1.21 1.21 1.18 1.06 1.00 0.88 0.88 0.87 0.82 0.73 0.70 0.65 0.60 0.59 0.59 0.58 0.57 0.57 0.55 0.53 0.52

47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92

sort_index get_dummies to_numeric resample duplicated assign unstack pivot_table update cumsum squeeze mode diff stack transpose filter eval insert idxmax notna reindex notnull pct_change var pop cut crosstab nlargest skew clip melt pivot to_period rank explode ffill idxmin ewm pipe interpolate cov asfreq tz_localize qcut mask xs

19860 18046 17354 16508 16004 15432 14286 14092 13372 13175 12551 12540 12519 12295 12145 11734 11062 10980 10972 10936 9695 9380 9286 8907 8709 8654 8591 8131 7991 7850 7690 7489 6146 5797 5756 5683 5642 5455 5049 4609 4346 4256 4165 4149 4118 4073

0.49 0.45 0.43 0.41 0.40 0.38 0.36 0.35 0.33 0.33 0.31 0.31 0.31 0.31 0.30 0.29 0.28 0.27 0.27 0.27 0.24 0.23 0.23 0.22 0.22 0.22 0.21 0.20 0.20 0.20 0.19 0.19 0.15 0.14 0.14 0.14 0.14 0.14 0.13 0.11 0.11 0.11 0.10 0.10 0.10 0.10

93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136

quantify data quality issues before modifying the dataset. The sequence fillna → mean → fillna reflects staged imputation strategies, where missing values are first partially addressed, summary statistics are computed, and then used to perform statistically

Operation cumprod take aggregate rename_axis droplevel add_prefix factorize tz_convert between_time sem product prod kurtosis add_suffix nsmallest corrwith combine_first kurt to_timestamp expanding bfill set_axis swaplevel merge_asof convert_dtypes cummax combine align truncate compare to_xarray at_time first_valid_index last_valid_index reorder_levels wide_to_long merge_ordered reindex_like cummin infer_objects asof T from_dummies lreshape

Count

%

3960 3898 3752 3710 3594 3537 3305 3207 3158 3083 2958 2838 2760 2656 2616 2590 2575 2502 2480 2414 2395 2223 1962 1855 1825 1773 1674 1461 1412 1310 1308 1285 1136 822 762 733 610 510 490 457 424 51 50 19

0.10 0.10 0.09 0.09 0.09 0.09 0.08 0.08 0.08 0.08 0.07 0.07 0.07 0.07 0.07 0.06 0.06 0.06 0.06 0.06 0.06 0.06 0.05 0.05 0.05 0.04 0.04 0.04 0.04 0.03 0.03 0.03 0.03 0.02 0.02 0.02 0.02 0.01 0.01 0.01 0.01 <0.01 <0.01 <0.01

informed imputation on remaining fields. Additionally, the pattern read_csv → to_datetime → set_index reflects a canonical preprocessing pipeline for time-series data, where temporal columns are parsed and promoted to an index for subsequent analysis. 5

Syrym Abdikhan and Mazhar Hameed

Table 5: Most Frequent Pandas Operation Sequences #

Operation Sequence

multi-stage aggregation, suggesting that users often refine summaries across multiple grouping dimensions. Taken together, these categories indicate that Pandas workflows often exhibit a recurring progression. While not strictly linear in every notebook, this layered structure reflects a compositional regularity in real-world Pandas usage and provides a foundation for future workflow modeling, optimization, and automated assistance.

%

Integration & Construction 1 2 3 4 5 6 7 8 9

join → read_csv → join DataFrame → DataFrame → concat read_csv → read_csv → concat head → merge → head read_csv → read_csv → merge read_csv → join → read_csv DataFrame → DataFrame → merge read_csv → join → read_csv → join DataFrame → to_csv → read_csv

2.7% 2.17% 2.04% 2.03% 1.92% 1.84% 1.58% 1.57% 1.43%

4

In this paper, we introduced PandasCorpus, a resource of realworld Pandas workflows collected from GitHub repositories spanning 2015 to 2025. The corpus comprises 139k Jupyter notebooks from approximately 100k repositories and provides a structured basis for studying how Pandas is used in practice. Through empirical analyses, we examined notebook executability trends over time, the frequency of commonly used Pandas operations, and recurring operation compositions observed in real-world workflows. PandasCorpus enables a range of research directions centered around real-world data manipulation workflows. First, the corpus facilitates code and workflow retrieval. Unlike traditional code datasets that focus on isolated functions, PandasCorpus preserves workflow structure, enabling retrieval models to consider sequences of operations and their contextual dependencies. This supports code search, workflow recommendation, and next-step prediction in interactive data analysis environments. Second, the dataset enables mining and modeling of recurring workflow patterns. The extracted operation sequences reveal compositional regularities that can inform automated pipeline construction, workflow abstraction, and intent modeling. Finally, PandasCorpus supports program analysis and optimization research. The corpus provides a large-scale collection of real-world Pandas workflows suitable for studying refactoring strategies, detecting anti-patterns, improving performance, and evaluating library-aware tooling.

Exploration & Validation 1 2 3 4 5 6 7 8 9 10 11 12 13 14

head → isnull → sum head → sort_values → head head → drop → head read_csv → head → describe sort_values → head → sort_values read_csv → head → isnull read_csv → head → tail read_csv → head → info read_csv → head → drop sum → duplicated → sum read_csv → head → isnull → sum sort_values → head → sort_values → head read_csv → head → value_counts head → unique → unique

4.25% 3.29% 3.22% 2.6% 2.26% 2.29% 2.16% 2.12% 2.12% 2.09% 1.86% 1.71% 1.4% 1.32%

Cleaning & Transformation 1 2 3 4

fillna → mean → fillna isnull → sum → dropna duplicated → sum → drop_duplicates read_csv → to_datetime → set_index

1.95% 1.91% 1.08% 1.12%

REFERENCES [1] 2025. cuDF: GPU DataFrame Library for Python (RAPIDS). https://rapids.ai/cudfpandas/. Accessed: 2025-10-20. [2] 2025. GitHub REST API Documentation. https://docs.github.com/en/rest? apiVersion=2022-11-28. Accessed date: 2025-07-28. [3] 2025. NumPy: The fundamental package for scientific computing with Python. https://numpy.org/. Accessed: 2025-10-19. [4] 2025. Project Jupyter: Jupyter Notebooks. https://jupyter.org/. Accessed: 202510-19. [5] Zeynep Akkalyoncu Yilmaz, Charles LA Clarke, and Jimmy Lin. 2020. A lightweight environment for learning experimental IR research practices. In Proceedings of the 43rd International Conference on Research and Development in Information Retrieval (SIGIR). 2113–2116. [6] Stefanos Baziotis, Daniel Kang, and Charith Mendis. 2024. Dias: Dynamic Rewriting of Pandas Code. Proceedings of the ACM on Management of Data 2, 1 (2024), 1–27. [7] Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N Nguyen. 2013. Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In 35th International Conference on Software Engineering (ICSE). IEEE, 422–431. [8] Itay Elyashiv, Amir Gilad, Edna Isakov, Tal Tikochinsky, and Amit Somech. 2024. PD-Explain: A Unified Python-native Framework for Query Explanations Over DataFrames. Proceedings of the VLDB Endowment 17, 12 (2024), 4473–4476. [9] Mojtaba Mostafavi Ghahfarokhi, Arash Asgari, Mohammad Abolnejadian, and Abbas Heydarnoori. 2024. DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks. Proceedings of the 21st International Conference on Mining Software Repositories (MSR) (2024), 647–651. [10] Serge Guelton. 2021. numpy-benchmarks. https://github.com/serge-sans-paille/ numpy-benchmarks. [11] Serge Guelton. 2025. Pythran tests. https://github.com/serge-sans-paille/ pythran/tree/master/pythran/tests.

Aggregation & Statistics 1 2 3 4 5 6 7 8 9

groupby → agg → reset_index groupby → mean → groupby rolling → mean → rolling mean → rolling → mean mean → groupby → mean sum → sum → mean groupby → sum → reset_index groupby → mean → reset_index groupby → apply → reset_index

IMPLICATIONS AND CONCLUSION

4.86% 3.12% 3.11% 2.58% 2.46% 1.45% 1.42% 1.35% 1.31%

Aggregation & Statistics. Sequences in this category are strongly centered around groupby, which appears in the majority of the most frequent patterns. The canonical structure groupby → agg → reset_index (4.86%) highlights a standard workflow in which grouped summaries are materialized back into flat tabular form. Several additional patterns, such as groupby → mean → groupby and mean → groupby → mean, indicate iterative re-grouping and 6

PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns

VLDB Endowment 13, 12 (2020). [21] Devin Petersohn, Dixin Tang, Rehan Durrani, Areg Melik-Adamyan, Joseph E Gonzalez, Anthony D Joseph, and Aditya G Parameswaran. 2021. Flexible rulebased decomposition and metadata independence in modin: a parallel dataframe system. Proceedings of the VLDB Endowment 15, 3 (2021). [22] Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. 2021. Kgtorrent: A dataset of python jupyter notebooks from kaggle. In 18th International Conference on Mining Software Repositories (MSR) (2021), 550–554. [23] Matthew Rocklin et al. 2015. Dask: Parallel computation with blocked algorithms and task scheduling.. In SciPy. 126–132. [24] San Diego State University. 2022. Python for Social Scientists, Linguistics/BDA 572. https://gawron.sdsu.edu/python_for_ss. [25] Xinhui Tu, Jimmy Huang, Jing Luo, Runjie Zhu, and Tingting He. 2019. Parrot: a python-based interactive platform for information retrieval research. In Proceedings of the 42nd International Conference on Research and Development in Information Retrieval (SIGIR). 1289–1292. [26] Gerardo Vitagliano, Mazhar Hameed, Lan Jiang, Lucas Reisener, Eugene Wu, and Felix Naumann. 2023. Pollock: A data loading benchmark. Proceedings of the VLDB Endowment 16, 8 (2023), 1870–1882. [27] Jiawei Wang, Li Li, and Andreas Zeller. 2020. Better code, better sharing: on the need of analyzing jupyter notebooks. In 42nd International Conference on Software Engineering (ICSE-NIER). 53–56. [28] Cong Yan and Yeye He. 2020. Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks. Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (2020). [29] Junwen Yang, Yeye He, and Surajit Chaudhuri. 2021. Auto-pipeline: synthesizing complex data pipelines by-target using reinforcement learning and search. Proceedings of the VLDB Endowment 14, 11 (2021), 2563–2575. [30] Alexandros Nikolaos Ziogas, Tal Ben-Nun, Timo Schneider, and Torsten Hoefler. 2021. NPBench: A benchmarking suite for high-performance NumPy. In Proceedings of the 35th ACM International Conference on Supercomputing. 63–74.

[12] Mazhar Hameed, Gerardo Vitagliano, Fabian Panse, and Felix Naumann. 2024. Tasheeh: Repairing row-structure in raw csv files. International Conference on Extending Database Technology (EDBT) (2024), 426–439. [13] Mazhar Hameed, Gerardo Vitagliano, Fabian Panse, and Felix Naumann. 2025. Repairing Raw Data Files with TASHEEH. ACM SIGMOD Record 54, 1 (2025), 90–99. [14] Alekh Jindal, K Venkatesh Emani, Maureen Daum, Olga Poppe, Brandon Haynes, Anna Pavlenko, Ayushi Gupta, Karthik Ramachandra, Carlo Curino, Andreas Mueller, et al. 2021. Magpie: Python at Speed and Scale using Cloud Backends. In Conference on Innovative Data Systems Research (CIDR). [15] Li-Ting Liao. 2020. Solving Real-World Business Questions with Python Pandas. https://medium.com/li-ting-liao-tiffany/solving-real-world-businessquestions-with-pandas-70ef8ef02675. [16] Weizheng Lu, Kaisheng He, Xuye Qin, Chengjie Li, Zhong Wang, Tao Yuan, Xia Liao, Feng Zhang, Yueguo Chen, and Xiaoyong Du. 2024. Xorbits: Automating Operator Tiling for Distributed Data Science. In 2024 IEEE 40th International Conference on Data Engineering (ICDE). 5211–5223. [17] Sean MacAvaney and Craig Macdonald. 2022. A Python Interface to PISA!. In Proceedings of the 45th International Conference on Research and Development in Information Retrieval (SIGIR). 3339–3344. [18] Craig Macdonald. 2018. Combining terrier with Apache Spark to create agile experimental information retrieval pipelines. In Proceedings of the 41st International Conference on Research and Development in Information Retrieval (SIGIR). 1309–1312. [19] Angelo Mozzillo, Luca Zecchini, Luca Gagliardelli, Adeel Aslam, Sonia Bergamaschi, and Giovanni Simonini. 2025. Evaluation of Dataframe Libraries for Data Preparation on a Single Machine. International Conference on Extending Database Technology (EDBT) (2025), 337–349. [20] Devin Petersohn, Stephen Macke, Doris Xin, William Ma, Doris Lee, Xiangxi Mo, Joseph E. Gonzalez, Joseph M. Hellerstein, Anthony D. Joseph, and Aditya Parameswaran. 2020. Towards scalable dataframe systems. Proceedings of the

7

Related documents

• PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns#609173
• AgRefactor: Self-Evolving Agentic Workflow for HLS Compatibility and Performance#652372
• AgRefactor: Self-Evolving Agentic Workflow for HLS Compatibility and Performance#650919
• Scikit-fingerprints: Python library for scikit-learn compatible molecular fingerprints and chemoinformatics#1050922
• The Other Half of Workflow Portability: Evidence-Backed HPC Site Profiles with Agentic Discovery#1160978