CC4Q Workbench: From Quantum Repositories to Analyzable Code Comments

CC4Q W ORKBENCH: From Quantum Repositories to Analyzable Code Comments Zenghui Zhou, Yuechen Li* , Zheng Zheng

arXiv:2610.32651v1 [cs.SE] 23 Oct 2026

Beihang University, Beijing, China {zhouzenghui, liyuechen, zhengz}@buaa.edu.cn * Corresponding author

Abstract—Quantum software development kits (SDKs) expose high-level APIs whose use requires an understanding of underlying mathematical principles, circuit semantics, and hardware properties. Code comments can provide natural-language or structural explanations that are not explicit in API calls. However, studying comments in a new quantum SDK still requires researchers to coordinate separate steps for comment extraction, quantum-specific classification, and manual verification, resulting in repeated integration effort. To address this problem, we present CC4Q W ORKBENCH, an open-source workbench that integrates these steps into a reusable workflow. Given a Python repository at a specified revision, CC4Q W ORKBENCH automatically extracts code–comment pairs, segments comments into sentence-level units, and applies two trained classifiers to identify quantumspecific content and assign quantum-knowledge categories. Its web interface supports exploration of bundled datasets, inspection of code snippets, corpus construction from new repositories, manual annotation with model predictions hidden during binary labeling, and classifier fine-tuning using exported annotations. The release provides aggregate statistics for six SDK datasets comprising 39,593 sentence-level code comment units (SCCUs), of which 17,517 are browsable in the web interface. Applied to three additional application-layer Qiskit repositories outside the original CC4Q dataset, the pipeline ran without code changes and produced 11,517 SCCUs. Manual verification of 379 SCCUs showed 85.0% agreement between human annotations and the binary classifier’s predictions. Index Terms—Quantum software engineering, Code comments, Software tool

I. I NTRODUCTION Quantum SDKs provide high-level APIs that make quantum operations easier to express, but these abstractions do not eliminate the need to understand the concepts and execution semantics behind them. In practice, a short API call can hide quantum concepts and implicit behaviors. For example, a call such as qc.cx(0, 1) does not explain which qubit controls the operation and which qubit is changed, potentially making it harder for developers to understand and debug the circuit’s behavior. Code comments often fill this gap by explaining quantum behavior in natural language next to quantum code [1], [2]. In this context, an existing study [1] released CC4Q, a dataset of 12,812 extracted code–comment pairs (CCPs) and 28,076 labeled sentence-level code comment units (SCCUs) from three Qiskit repositories, together with comment taxonomies associated with the quantum-specific semantics. As quantum software ecosystems grow, reusable tooling for code-

comment analysis across SDKs is increasingly important. Although the research scripts of [1] are reusable, applying them to another Qiskit repository still requires coordinating separate steps for extracting and linking comments, obtaining labels, and checking their reliability. Existing comment-analysis tools for classical software, such as CPC [3], target Java API documentation and do not identify quantum-specific knowledge in comments. Researchers therefore lack an integrated workflow for constructing, checking, and adapting quantum comment corpora across repositories. To fill this gap, we introduce CC4Q W ORKBENCH, an open-source workbench for constructing, exploring, and validating quantum code-comment corpora. While the companion study [1] focused on comment taxonomies and empirical findings, this paper focuses on the tool itself: how it is built for reuse, and how users work with it through visual interfaces and developer APIs. This paper makes five contributions: An integrated workbench: The web interface brings together five components to support corpus exploration, code inspection, repository processing, manual labeling, and model training, respectively. • Extensible corpus construction: The CLI and web workflow process Python repositories by extracting source-linked CCPs and segmenting comments into SCCUs, with support for labeling units using the quantum-relatedness and quantum-category classifiers. • Reusable and trainable models: Users can load released checkpoints for prediction and fine-tune BERT classifiers with built-in or user-provided labeled data. • Visual exploration and annotation: Corpus statistics are visualized to support cross-source comparison, and the annotation interface facilitates manual labeling. • Transfer evaluation: We apply the pipeline without code changes to three previously unstudied applicationlayer repositories (i.e., qiskit-machine-learning, qiskit-nature, and qiskit-algorithms) yielding 5,502 CCPs and 11,517 SCCUs, while manual checking of 379 SCCUs found 85.0% agreement between human labels and the binary model’s predictions. •

Paper structure: Section II describes CC4Q W ORKBENCH’s architecture. Section III presents three practical use cases, while Section IV evaluates the pipeline’s transferability. Section V concludes with future work.

II. T OOL A RCHITECTURE Fig. 1 summarizes how CC4Q W ORKBENCH turns a Python repository into a labeled corpus of source-linked comments. Users can explore and review the corpus through the web interface or access the pipeline and related functions through the developer API. In the following subsections, we describe the end-to-end builder first (Section II-A), then introduce the other four components of the user interface (Section II-B), and finally present details about the developer API (Section II-C). A. End-to-end Builder Corpus Builder, a core component of CC4Q W ORK BENCH, provides an end-to-end path from a repository to a

source-linked corpus. It accepts either a local Python directory or a GitHub repository and can carry the result through comment processing and model labeling in the same workflow. This removes the manual handoffs between scripts and intermediate files that otherwise make it difficult to repeat the analysis for another SDK. Corpus Builder automatically proceeds through four stages. First, users provide a local Python project or a GitHub repository and may specify the version to process. The builder then scans the repository’s Python files to extract docstrings, inline comments, and consecutive comment lines, associating each with its source file, line range, and enclosing code entity types (i.e., module, class, or function). In this stage, LibCST supports this extraction and code-context recovery [4]. Next, the builder converts longer comments into sentencelevel units while preserving their source links; docutils [5] and spaCy [6] support the processing of structured docstrings and sentence boundaries. Finally, when labeling is enabled, two classifiers perform the two labeling tasks of CC4Q: (1) a binary task labels each unit as quantum-specific or nonquantum, and (2) a six-class task assigns each quantumspecific unit to mathematics for quantum information, quantum algorithms, the quantum circuit model, quantum hardware, quantum software framework, or others. The builder saves the labeled corpus in the workspace, where it can be opened in Corpus Explorer for browsing and comparison. Listing 1. Worked example: a class docstring processed by Live Inspector and stored as a labeled corpus record. # source: entanglement.py, class EntanglementBuilder (L7 -11) { "comment_text": "Constructs maximally entangled multi-qubit states on a quantum register.", "comment_type": "docstring", "code_entity_type": "class" , "quantum_related": true, "quantum_category": "Quantum circuit model" }

Listing 1 gives a concrete example of the output. The class docstring is stored with its comment type, enclosing class, source snippet, line information, and predicted labels. Keeping these fields together lets a researcher inspect the code behind a label and move from corpus summaries back to the original source.

B. Components in User Interface The previous subsection described how Corpus Builder turns a repository into a source-linked corpus. The other four components support work with that corpus: researchers can explore its contents, inspect predictions on code snippets, check labels manually, and adapt a classifier with labeled data. The web interface is launched with cc4q demo and uses a Starlette/Uvicorn JSON API [7] and an Apache ECharts frontend [8]. Corpus Explorer supports browsing and comparing bundled and newly built corpora. It summarizes quantum-specific shares and category profiles, and uses heatmaps to show how comments are distributed across comment forms and code scopes. Selecting a chart value filters the linked comment records, so users can move from corpus-level patterns to source-linked examples. Live Inspector lets users paste Python code or choose a built-in example for on-demand analysis. After extraction, it displays the comments split into sentence-level units and their predictions from the currently active classifiers. For units predicted as quantum-specific, it also shows the prediction confidence and assigned knowledge category. This provides a quick way to inspect model behavior on code without building a full corpus. Annotation Studio supports manual review of model labels. Users can open the supplied verification sample or upload a JSONL file, then label each unit as quantum-specific or non-quantum and assign a category to quantum units. Model predictions are hidden during the binary decision; the predicted category is offered only as an editable suggestion after a unit is marked quantum. Labels are saved as users work, and completed sessions can be exported for evaluation or training. Model Trainer uses labeled data to adapt either the binary quantum-relatedness classifier or the six-class category classifier. Users can select a built-in dataset or upload their own labeled records, then monitor training loss and validation metrics while training proceeds, where CC4Q W ORKBENCH locally registers completed models that can be selected for subsequent predictions. C. Developer API and Command-line Interface Besides the web interface for interactive use, CC4Q W ORK BENCH offers a developer API so researchers can integrate corpus analysis into automated scripts or extend the tool. The API exposes the same core capabilities as the workbench, so programmatic use can fit into an existing research workflow. The Python API provides functions to extract code– comment pairs, segment comments, and apply the released classifiers to snippets or corpora. It also supports sampling records for annotation, training classifiers from labeled data, and evaluating their predictions. Workspace functions persist corpora, annotation sessions, and evaluation results, while the model registry stores trained models for reuse. Listing 2. CC4Q command-line entry points. # Commands are run separately; arguments are illustrative. cc4q download-models

User Interface Corpus Explorer

Corpus Builder

Live Inspector

• Statistical visualization • Cross-source comparison

① Acquire Source

Annotation Studio

• End-to-end builder • Support for both local sources and GitHub links

• Illustrative example • Live analysis and SCCU classification

② LibCST Extraction

• Visualized interface for human annotation

③ Sphinx Segmentation

Model Trainer

• Configurable data and parameters • Model training for BERT

④ Quantum Labeling

Developer API Corpus Pipeline

Model Hub

Inference

• cc4q.pipeline

• cc4q.models

• cc4q.predict

Training & Evaluation

• cc4q.train • cc4q.evaluate

Sampling & Annotation

• cc4q.sample • cc4q.annotate

Workspace & Registry

• cc4q.workspace • cc4q.registry

Web Workbench • cc4q.web

Fig. 1. Architecture of CC4Q W ORKBENCH. cc4q build --repo REPO --output-dir DIR cc4q predict --input SCCUS --output LABELED cc4q sample --input DATA --output SAMPLE cc4q annotate --data SAMPLE --output-dir ANNOTATIONS cc4q train --task TASK --data LABELS --output-dir MODEL cc4q evaluate --annotations ANNOTATIONS --key KEY cc4q demo cc4q serve --host HOST --port PORT

The cc4q command-line interface provides a direct way to run these operations from a terminal; Listing 2 summarizes the available commands. A researcher can build a corpus, write predictions to a file, prepare samples for annotation, train or evaluate a classifier, and launch the web workbench with cc4q demo. Explicit input and output paths make these commands suitable for repeatable batch processing and automated scripts. III. U SE C ASES To illustrate how CC4Q W ORKBENCH supports practical research tasks, we present three use cases that demonstrate its application to quantum-repository exploration, codebase analysis, and domain adaptation. Use case 1: Exploring a new quantum repository. Beyond the three infrastructure-layer repositories in the original CC4Q dataset, real-world quantum development ecosystems usually include repositories serving specific application purposes. With CC4Q W ORKBENCH, for example, a researcher studying the documentation of qiskit-nature, a Qiskit repository designed for quantum chemistry and other natural-science applications, can directly select the repository from the bundled list or provide its GitHub URL to Corpus Builder. The tool can then generate a labeled dataset of 4,842 sentences, with a predicted quantum share of 28.5%, a comment density of 0.127 (comment lines divided by code lines), and a category profile. The profile shows that 28.9% of comments concern mathematics, consistent with the SDK’s quantum-chemistry domain. The researcher can export the dataset in JSONL format for further analysis.

Use case 2: Analyzing an existing quantum repository. A library maintainer can run the Docker image, point Corpus Builder to a local repository, and inspect the resulting heatmap to examine how quantum knowledge is distributed across code comments. For example, the maintainer can investigate whether quantum-related knowledge is concentrated in class docstrings rather than inline comments. The maintainer can also use Live Inspector to analyze a newly developed module and create a labeling session in Annotation Studio to validate the dashboard’s predictions on a manually labeled sample. Use case 3: Adapting the classifiers to a new domain. We acknowledge that existing classifiers reused from CC4Q may not be generalizable to all quantum repositories, as they were trained on a limited set of infrastructure-layer repositories. When an SDK combines classical numerical code with quantum primitives, the default classifier may misclassify some comments, potentially degrading data quality. To improve domain adaptation, a research team can manually label a few hundred domain-specific units in Annotation Studio, export the annotations as a JSONL file, and use Model Trainer to finetune the classifier from the BERT base model. The adapted model is then registered locally and becomes available for subsequent corpus builds and inspections. IV. T RANSFER VALIDATION The preceding sections described the functionality and use cases of CC4Q W ORKBENCH, while this section evaluates whether those capabilities remain dependable in use. We therefore address two questions: whether the binary predictions are sufficiently reliable for corpus checking, and whether the complete pipeline transfers to repositories that were not used to develop the models. A. Using the Classifier on New Qiskit Repositories The earlier study evaluated the BERT-based classifiers on test data from three infrastructure-level repositories. Here we

TABLE I DATASETS BUILT BY CC4Q W ORKBENCH FOR THREE ADDITIONAL REPOSITORIES , UNMODIFIED . T HE ORIGINAL THREE REPOSITORIES ARE SHOWN FOR REFERENCE [1]. Repository

Version Layer

CCPs SCCUs Q.% Density

qiskit qiskit-aer qiskit-ibm-runtime

2.1.0 infra. 0.17.2 infra. 0.45.1 infra.

9,677 21,970 35.4 1,153 2,368 40.6 1,982 3,738 23.5

0.126 0.114 0.210

qiskit-machine-learning qiskit-nature qiskit-algorithms

0.9.1 0.8 0.4.0

1,663 2,037 1,802

0.172 0.127 0.155

infrastructure total application total

appl. appl. appl.

3,140 22.4 4,842 28.5 3,535 25.9

12,812 28,076 34.3 5,502 11,517 26.0

ask whether these classifiers still work on another Qiskit repository outside the training set. We then sampled 385 units across the three new repositories (qiskit-machine-learning, qiskit-nature, qiskit-algorithms), stratified by SDK, code-entity type, and the model’s predicted label. Six items were excluded because their model labels had been shown to the annotator during calibration, leaving 379 scored units as the benchmark. One author then checked the binary labels following the annotation guideline from the earlier study. The model agreed with the human labels on 85.0% of the units. The exact numbers reproduce with: cc4q evaluate \ --annotations exp/spotcheck_annotation.csv \ --key exp/spotcheck_answer_key.jsonl \ --contaminated 1,8,16,25,62,132

The 57 disagreements contain 32 false positives and 25 false negatives. There are two possible reasons: (1) the model sometimes marks classical numerical parts of variational workflows as quantum, such as ODE (ordinary differential equation) solvers and quasi-Newton optimizers used in numerical computations; (2) short units that use specialized abbreviations are misjudged, such as MP2 (second-order Møller–Plesset perturbation theory), VSCF (vibrational self-consistent field), and FCIDump (a file format for one- and two-electron integrals). Not all labels were checked by humans, and the classifiers are not perfect despite their test-set accuracy of 0.936 (binary) and 0.847 (six-class) reported in [1]. Accordingly, CC4Q W ORKBENCH lets users train or fine-tune the classifiers on their own labeled data. The web release bundles 361 of these 379 units as an unlabeled labeling demo; the scored benchmark for evaluation remains the full 379-unit set in exp/. B. Using the Pipeline on New Qiskit Repositories To assess whether the pipeline transfers without code changes, we ran it on the three repositories listed in Table I, where Q.% is the share of SCCUs that the binary classifier predicts as quantum-specific and Density is the ratio of comment lines to code lines. Across the six repositories, applicationlayer repositories show a smaller share of comments predicted as quantum than infrastructure-layer SDKs (26.0% vs. 34.3%). We treat this as a model-labeled observation, not a manually verified finding. Even so, it suggests that different SDK layers may need different documentation guidance.

For reproducibility, all SDK release tags, dataset statistics, and validation files are pinned in the repository. Python dependencies are pinned or lower-bounded in the packaged requirements. The Docker image and the pytest suite are verified to build and run in CI. The demo can also start without model checkpoints by turning off model-dependent actions gracefully. V. C ONCLUSION AND F UTURE W ORK We present CC4Q W ORKBENCH, an open-source tool that turns the methodology of CC4Q construction into a reusable workflow. Starting from a Python quantum repository, the tool builds a labeled comment dataset, shows where quantum knowledge appears, lets users check labels without seeing model predictions during labeling, and supports fine-tuning the classifiers on exported labels. On three repositories not used in the earlier study, the tool ran without modification, and manual checking found 85.0% agreement between human labels and the binary model’s predictions. However, CC4Q W ORKBENCH currently only supports Python-based quantum SDKs such as Qiskit, because stages like automatic comment extraction and segmentation are often language-specific. Future work includes parsers for nonPython SDKs to support more quantum ecosystems. DATA AVAILABILITY CC4Q W ORKBENCH is open source under the MIT license at https://github.com/MaxwelsDonc/cc4q-workbench, where the repository includes the Docker image, CLI and Python APIs, SHA256-verified model assets, six-dataset demo data, the 361-unit labeling demo, and all validation files. Also, we offer the documentation of CC4Q W ORKBENCH at https:// zhouzenghui.site/cc4q-workbench/. The original CC4Q dataset is archived on Figshare [9]. R EFERENCES [1] Z. Zhou, Y. Li, Y. Cai, J. Wen, X. Yu, Z. Zheng, and B. Yin, “Code comments for quantum software development kits: An empirical study on qiskit,” ACM Transactions on Software Engineering and Methodology, 2026, accepted for publication. [2] X. Yu, Y. Li, J. Wen, K.-Y. Cai, and B. Yin, “Quainth: A code comment approach for application-oriented quantum programs via n-version llms,” in 2025 25th International Conference on Software Quality, Reliability and Security (QRS). IEEE, 2025, pp. 13–24. [3] J. Zhai, X. Xu, Y. Shi, G. Tao, M. Pan, S. Ma, L. Xu, W. Zhang, L. Tan, and X. Zhang, “CPC: Automatically classifying and propagating natural language comments via program analysis,” in Proceedings of the 42nd International Conference on Software Engineering (ICSE), 2020. [4] Meta, “LibCST: A concrete syntax tree parser and serializer library for python,” https://github.com/Instagram/LibCST, 2024. [5] Docutils contributors, “Docutils: Documentation utilities,” https://docutils. sourceforge.io, 2024. [6] M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength natural language processing in python,” https://spacy. io, 2020. [7] Encode OSS, “Starlette: A lightweight asgi framework/toolkit,” https:// www.starlette.io, 2024. [8] Apache Software Foundation, “Apache ECharts: A powerful, interactive charting and data visualization library for the browser,” https://echarts. apache.org, 2024. [9] Z. Zhou, Y. Li, Y. Cai, J. Wen, and X. Yu, “Artifact of code comments for quantum software development kits: An empirical study on qiskit,” Figshare, 2025. [Online]. Available: https://doi.org/10.6084/m9.figshare. 30085657.v1

Related documents

• CC4Q Workbench: From Quantum Repositories to Analyzable Code Comments#1448167
• NxM-Version Programming for Quantum Software: High-Level Components across Frameworks and Engines#1117967
• QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging#1062922
• Dataset for <i>Model-Driven for Quantum Software Engineering: A Systematic Mapping Study</i>#380003
• Automated Quantum Software and AI Engineering#124121