CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams Simon Scholz

University of Cologne Cologne, Germany sschol24@smail.uni-koeln.de

arXiv:2608.07036v1 [cs.SE] 7 Aug 2026

Abstract Automated UML generation from sketches and images is gaining renewed attention with the rise of large language models and multimodal AI. However, reproducible evaluation remains difficult due to the lack of public datasets with executable groundtruth models. We present CAS2UML, a public dataset of 557 handdrawn UML diagrams, including 271 class diagrams and 286 activity diagrams, each paired with manually validated PlantUML code. We also provide a PlantUML-based validation tool and reusable scripts for checking the syntactic correctness and renderability of generated UML artifacts, enabling reproducible benchmarking of sketch-to-UML approaches. The dataset, validation tool, processing scripts, documentation, and demonstration video are publicly available at: Dataset: https://huggingf ace.co/datasets/Seym0n /cas2uml_hand-drawn_to_plantuml_dataset; Tool and Scripts: https://github.com/Seym0n/handwritten-uml-dataset; Video: https://www.youtube.com/watch?v=KQrYeGgT3hs.

CCS Concepts • Software and its engineering → Unified Modeling Language (UML); • Computing methodologies → Machine learning.

Keywords dataset, UML, handwritten UML, PlantUML, benchmark, sketch recognition, software modeling, large language models ACM Reference Format: Simon Scholz and Mersedeh Sadeghi. 2026. CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), October 12–16, 2026, Munich, Germany. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3832783.3834613

1

Introduction

The Unified Modeling Language (UML) remains a central notation for software specification, analysis, design, and communication among stakeholders [10, 11]. Among its diagram types, class and activity diagrams are particularly important: class diagrams are consistently reported as the most frequently used UML notation, while activity diagrams are among the most common behavioral notations used for workflow and process modeling [9, 14]. Together,

This work is licensed under a Creative Commons Attribution 4.0 International License. ASE ’26, Munich, Germany © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2882-2/2026/10 https://doi.org/10.1145/3832783.3834613

Mersedeh Sadeghi✉

University of Cologne Cologne, Germany sadeghi@cs.uni-koeln.de they cover complementary structural and behavioral views of software systems. The automatic creation of UML models from informal inputs has therefore been a long-standing goal in software engineering. Earlier work explored model generation from textual requirements, rendered diagrams, and scanned documentation, while recent advances in multimodal AI and large language models have renewed interest in generating UML from prompts, screenshots, photographs, and hand-drawn sketches [2–4, 8, 18]. These developments suggest that sketch-to-UML and image-to-model pipelines are becoming increasingly feasible. However, progress is limited by the lack of public benchmark datasets that pair hand-drawn UML diagrams with executable, machine-readable ground truth. Existing resources are either imageonly, restricted to visual annotations, limited to small in-house benchmarks, or focused on a single diagram type [3, 12]. This makes it difficult to evaluate whether generated UML artifacts are not only visually plausible, but also syntactically valid, editable, and reusable in modeling toolchains. To address this gap, we introduce CAS2UML, a public dataset of 557 hand-drawn UML diagrams, consisting of 271 class diagrams and 286 activity diagrams. Each sample is paired with validated PlantUML1 code, providing an executable textual representation of the corresponding sketch. PlantUML was selected because it is human-readable, editable, renderable, and already used as an output format in recent UML-generation studies [1, 3]. In addition, we provide a validation tool and scripts that support the creation, rendering, and syntactic checking of sketch–PlantUML pairs. Together, the dataset and tool enable reproducible evaluation of sketch-recognition, image-to-UML, multimodal AI, and automated model-generation approaches. In summary, this paper contributes (i) the CAS2UML dataset of validated sketch–PlantUML pairs, and (ii) a validation tool and reusable scripts for checking PlantUML renderability, syntactic correctness, and dataset integrity.

2 Background and Related Works 2.1 Class and Activity Diagrams Class diagrams and activity diagrams were selected because they represent two of the most widely used UML notations. Class diagrams capture the static structure of a system, including classes, attributes, operations, and relationships, and are often used for domain modeling, architectural design, and implementation-oriented documentation [2, 8]. They are consistently reported as the most 1 https://plantuml.com/

ASE ’26, October 12–16, 2026, Munich, Germany

Scholz et al.

Table 1: Comparison of UML sketch datasets. Dataset / Study

Source

Diagram type(s)

# samples

Ground truth

Validated

Public

Conrardy & Cabot [3] Piucco [12] Ražinskas et al. [13]

Hand-drawn Hand-drawn Hand/tool-drawn

UML class UML class UML use case

4 70 346

PlantUML Element masks Bounding boxes

Partial – –

No Yes Yes

Ours

Hand-drawn

Class + activity

557

PlantUML (+XMI for class)

Yes

Yes

frequently studied and most broadly covered UML artifact across the research literature, books, tools, and courses [9, 14]. Activity diagrams complement this structural view by modeling behavior, workflows, decisions, parallel flows, and process logic. Their broad set of control-flow constructs makes them widely used for business-process modeling, requirements analysis, and serviceoriented and workflow-based systems [14]. By covering both class and activity diagrams, CAS2UML supports evaluation across structural and behavioral UML modeling tasks.

2.2

Limitations of Existing Datasets

Despite growing interest in generating UML models from sketches and diagram images, reproducible evaluation remains a major challenge. Existing studies on image-based or sketch-based UML creation rely on very small evaluation sets. Related LLM-based UML studies also typically evaluate on small, task-specific benchmarks, such as ten domain descriptions in Chen et al. [2] and twenty class-diagram exercises in De Bari et al. [4]. Also Eklund and Jonsson [5] benchmark LLMs on UML generation from informal notations, but evaluate model behaviour rather than releasing paired sketch–ground-truth data; CAS2UML is complementary in providing the resource such benchmarks require. Table 1 focuses on resources that are closest to our setting: handdrawn UML sketches. Large mined repositories such as Lindholmen [6] contain many UML images, but they primarily represent rendered or collected diagram images rather than hand-drawn sketches. They therefore address an adjacent image-based UML setting rather than the sketch-to-UML setting targeted in this paper. Existing hand-drawn UML resources are much smaller and are usually limited to a single diagram type. Conrardy and Cabot [3] used only four hand-drawn class-diagram examples, while Piucco [12] provides 70 handwritten class-diagram images with visual annotations, but no PlantUML or XMI ground truth. Ražinskas et al. [13] provide a public collection of UML use-case sketches with element-level annotations; however, the released dataset is not paired with executable UML ground truth for each sketch. In contrast, CAS2UML provides 557 hand-drawn diagrams across class and activity diagrams, each paired with validated executable PlantUML code. In summary, three observations emerge from the literature review. First, in neighboring modeling domains such as BPMN, public hand-drawn datasets already support evaluation at the scale of several hundred diagrams [16, 17], whereas comparable UML resources remain limited in size and often provide only visual annotations. Second, recent UML-generation studies typically rely on small, task-specific, and often unreleased evaluation sets. Third, no prior dataset combines hand-drawn UML inputs with editable, machine-readable, and end-to-end validated ground truth at a scale suitable for systematic benchmarking. CAS2UML is designed to

fill this gap by enabling sketch-recognition, sketch-to-UML, and automated model-generation approaches to be evaluated against the same executable reference.

3

CAS2UML Data Creation

The dataset was constructed to provide paired examples of handdrawn UML diagrams and executable PlantUML specifications. The creation workflow followed four main steps: (i) defining diagramcontent specifications, (ii) generating or collecting initial PlantUML representations, (iii) creating handwritten diagram images from validated references, and (iv) verifying the one-to-one correspondence between each sketch and its machine-readable ground truth. The first author performed all manual steps.

3.1

Class Diagram

The class-diagram subset was created through a two-source process. First, we reused the 70 handwritten class-diagram images released by Piucco [12]. Since this dataset provides visual annotations but no executable UML representation, we created corresponding PlantUML code for these samples and validated the resulting sketch– PlantUML pairs using the procedure described in Section 3.2. Second, we created 208 additional class-diagram samples to increase the size, diversity, and construct coverage of the dataset. For this purpose, we defined a coverage guideline specifying the UML elements and modeling situations to be represented. The guideline includes common class-diagram constructs such as associations, aggregation, composition, inheritance, interfaces, dependencies, multiplicities, and abstract classes. It also incorporates examples inspired by the Gang of Four design patterns in order to obtain realistic object-oriented structures with varying complexity [7]. Table 2 summarizes the resulting coverage guideline.

3.2

Dataset Processing, Validation, and Release

For each additional class-diagram sample, we first created a reference PlantUML specification using an LLM-assisted authoring workflow with ChatGPT 5.3, Claude Sonnet 4.6 and Gemini 3 Flash. The generated code was manually inspected, corrected where necessary, and rendered using PlantText2 to verify that the resulting diagram matched the intended class-diagram structure. The validated reference diagrams were then manually redrawn on different media, including tablets, plain paper, and checkered paper. This process produced hand-drawn images with variation in writing style, line quality, background, and acquisition conditions. Each final sample, therefore, consists of a handwritten classdiagram image paired with validated executable PlantUML code. 2 https://www.planttext.com/

CAS2UML: A Handwritten Sketch-to-PlantUML Dataset for Class and Activity Diagrams

ASE ’26, October 12–16, 2026, Munich, Germany

Table 2: Class-diagram coverage guideline: UML constructs and GoF-inspired patterns used during sample creation. Numbers in parentheses indicate how often a pattern was redrawn in a distinct context. Category

Covered elements / patterns (Instances)

Creational patterns Structural patterns Behavioral patterns

Singleton (3), Factory Method (3), Abstract Factory (2), Builder (1), Prototype (1) Adapter (3), Bridge (2), Composite (2), Decorator (3), Facade (3), Flyweight (1), Proxy (2) Observer (2), Strategy (3), Command (2), State (3), Template Method (2), Chain of Responsibility (2), Iterator (2), Mediator (2), Memento (2), Visitor (2) Unidirectional association, bidirectional association, self-association, ternary association Simple aggregation, composition, mixed aggregation/composition Single inheritance with 2–3 levels, multiple inheritance, interface implementation, abstract classes with concrete subclasses, mixed inheritance and composition One-to-one, one-to-many, many-to-many, optional, bounded ranges Dependency relation using dashed arrows

Association types Aggregation and composition Inheritance hierarchies Cardinality variations Dependencies

Figure 1: The CAS2UML validation tool with side-by-side sketch input, PlantUML annotation, and rendered preview. This process resulted in 278 class-diagram pairs before XMI filtering and 271 released class-diagram samples after excluding seven diagrams whose PlantUML code could not be serialized to XMI due to unsupported association-class constructs.

3.3

Activity Diagram

For the activity-diagram subset, we constructed the data from scratch, since no comparable public hand-drawn activity-diagram dataset was available. The goal was to obtain a diverse set of activity diagrams that covers common control-flow constructs while remaining suitable for manual redrawing and end-to-end validation. We used two sources for creating reference diagrams. First, we considered the Lindholmen dataset [6], a large collection of UML diagram images mined from GitHub repositories. To identify candidate activity diagrams, we filtered the dataset metadata using URL-path keywords, in particular occurrences of “activity” together with common image-file extensions. This filtering step yielded approximately 1,000 candidate activity-diagram images. From these candidates, around 80 diagrams were selected as reference templates based on readability, relevance, and suitability for redrawing. Second, to increase construct coverage beyond the selected repository mined examples, we created additional activity-diagram specifications covering common UML activity constructs, including actions, initial and final nodes, decision and merge nodes, fork and join nodes, loops, and swimlanes. PlantUML code for these diagrams was produced through an LLM-assisted authoring workflow using Claude Sonnet 4.6 and Gemini 3 Flash, and rendered

using PlantText. The generated diagrams were then manually inspected and corrected to ensure that the resulting activity flows were syntactically valid and semantically coherent. The final handwritten images were created by manually redrawing the validated reference diagrams on different media, including tablets, blank paper, and checkered paper. This process introduced variation in handwriting style, layout, line quality, background, and acquisition conditions. Each handwritten activity diagram was then paired with its corresponding validated PlantUML specification. In total, the activity-diagram subset contains 286 paired samples, each consisting of a handwritten activity-diagram image and an executable PlantUML representation.

3.4

CAS2UML Validator Tool

To support the creation and verification of the sketch–PlantUML pairs, we developed a custom web-based validation tool. The tool displays each handwritten UML image next to an editable PlantUML panel and a rendered preview of the corresponding diagram. This setup allows one to iteratively edit the PlantUML code, render it locally using PlantUML, and check whether the executable representation matches the handwritten input. Existing annotation tools did not provide the required combination of sketch visualization, editable PlantUML, and live UML rendering; therefore, we implemented a lightweight interface using Gradio. Figure 1 shows the validation interface. For each sample, the tool records the image path, diagram type, and validated PlantUML code. The diagram type is specified as either class diagram or activity diagram. After saving an annotation, the tool stores the pair in a JSON file and allows navigation through the remaining samples. This workflow ensures that each released sample contains both a handwritten image and an executable PlantUML representation. After annotation, the dataset was processed for public release. All image paths were verified against the annotation file, non-image files were excluded, and all images were resized so that the longest side did not exceed 1000 pixels. This step reduced variation caused by different acquisition conditions, such as scanned paper drawings and tablet-generated images, while preserving visual readability. To ensure that the released PlantUML annotations are executable, we applied an additional validation procedure. Each annotation was checked using the PlantUML JAR with the -syntax flag to

ASE ’26, October 12–16, 2026, Munich, Germany

Scholz et al.

verify that the code is parseable and free of compilation errors. For class diagrams, we also generated StarUML-compatible3 XMI files from the validated PlantUML code using the PlantUML server. This step was applied only to class diagrams, since PlantUML does not support XMI export for activity diagrams. To document the visual diversity of the released dataset, we also report the drawing media used for the handwritten samples in Table 3. The dataset includes tablet drawings, plain-paper sketches, checkered-paper sketches, photographed paper diagrams, and one lined-paper sample. This variation supports evaluation under different visual conditions and acquisition settings. Table 3: Distribution of drawing media in CAS2UML. Medium

Class

Activity

Total

Tablet Plain paper Checkered paper Photographed paper∗ Lined paper

149 43 8 70 1

120 52 114 0 0

269 95 122 70 1

Total

271

286

557

∗ Sourced from Piucco [12].

Finally, the processed images, PlantUML code, diagram-type labels, and available XMI files were uploaded to the HuggingFace Hub4 . We also release the validation script, which produces machine-readable error reports and can be reused to check PlantUML outputs generated by sketch-to-UML systems. The overall processing and release workflow is summarized in Figure 2. As preliminary evidence of utility, we use CAS2UML in followup work to fine-tune a handwritten-UML-to-PlantUML model that, evaluated via automatic metrics and a human ranking study, becomes competitive with proprietary vision-language baselines [15]. Load images from folder

Keep as-is

Activity Diagram

Check missing annotations and exclude non images Load annotations from JSON

Diagram

Resize longest side of each image to 1000px

Generate Star XMI Code

Non images

Diagram

Class Diagram

PlantUML Code

Star XMI Code

PlantUML Server

Figure 2: CAS2UML processing and validation pipeline

4

Conclusion

This paper introduced CAS2UML, a public dataset of 557 handdrawn class and activity UML diagrams paired with validated executable PlantUML code, together with a validation tool and processing pipeline for creating and checking sketch–PlantUML pairs. By 3 https://staruml.io/

4 https://huggingface.co/

providing an executable ground-truth reference, CAS2UML enables reproducible benchmarking of sketch-recognition, image-to-UML, and automated model-generation approaches, and supports future research on AI-assisted software modeling.

References [1] Bilal Al-Ahmad, Anas Alsobeh, Omar Meqdadi, and Nazimuddin Shaikh. 2025. A Student-Centric Evaluation Survey to Explore the Impact of LLMs on UML Modeling. Information 16, 7 (2025). doi:10.3390/info16070565 [2] Kua Chen, Yujing Yang, et al. 2023. Automated domain modeling with large language models: A comparative study. In 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems. IEEE, 162–172. doi:10.1109/MODELS58315.2023.00037 [3] Aaron Conrardy and Jordi Cabot. 2024. From image to UML: First results of image based UML diagram generation using LLMs. (2024). doi:10.48550/arXiv.2 404.11376 [4] Daniele De Bari, Giacomo Garaccione, Riccardo Coppola, Marco Torchiano, and Luca Ardito. 2024. Evaluating large language models in exercises of uml class diagram modeling. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 393–399. doi:10.1145/3674 805.3690741 [5] Cecilia Eklund and Tom Jonsson. 2025. BENCHMARKING LARGE LANGUAGE MODELS IN UML DIAGRAM GENERATION FROM INFORMAL NOTATIONS. [6] Regina Hebig, Truong Ho Quang, et al. 2016. The quest for open source projects that use UML: mining GitHub. In Proceedings of the ACM/IEEE 19th international conference on model driven engineering languages and systems. 173–183. doi:10.1 145/2976767.2976778 [7] John Hunt. 2013. Gang of four design patterns. In Scala Design Patterns: Patterns for Practical Reuse and Design. Springer, 135–136. doi:10.1007/978-3-319-02192-8 [8] Bilal Karasneh and Michel RV Chaudron. 2013. Extracting UML models from images. In 5th Int. Conf. on Computer Science and Information Technology. IEEE. doi:10.1109/CSIT.2013.6588776 [9] Hatice Koc, Ali Mert Erdoğan, Yousef Barjakly, and Serhat Peker. 2021. UML diagrams in software engineering research: a systematic literature review. In Proceedings, Vol. 74. MDPI, 13. doi:10.3390/proceedings2021074013 [10] Parastoo Mohagheghi, Vegard Dehlen, and Tor Neple. 2009. Definitions and approaches to model quality in model-based software development–A review of literature. Information and software technology 51, 12 (2009), 1646–1669. doi:10.1 016/j.infsof.2009.04.004 [11] Mert Ozkaya and Ferhat Erata. 2020. A survey on the practical use of UML for different software architecture viewpoints. Info. and Software Tech. (2020). doi:10.1016/j.infsof.2020.106275 [12] Leticia Piucco. 2024. Handwritten UML Class Diagrams. Kaggle dataset. https: //www.kaggle.com/datasets/leticiapiucco/handwritten-uml-class-diagrams Accessed: 7 May 2026. [13] Mantas Ražinskas, Benas Miliūnas, Mantas Jurgelaitis, Lina Čeponienė, and Lina Bisikirskienė. 2024. Transforming sketches of UML use case diagrams to models. IEEE Access 12 (2024), 185826–185837. doi:10.1109/ACCESS.2024.3514455 [14] Gianna Reggio, Maurizio Leotta, Filippo Ricca, and Diego Clerissi. 2014. What are the used activity diagram constructs? a survey. In 2014 2nd International Conference on Model-Driven Engineering and Software Development. 87–98. doi:10 .5220/0004712000870098 [15] Mersedeh Sadeghi, Simon Scholz, and Adrian Psoch-Bajraktari. 2026. A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation. In Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS). doi:10.48550/arXiv.2607.28987 arXiv:2607.28987. [16] Bernhard Schäfer, Han van der Aa, et al. 2021. Sketch2BPMN: Automatic recognition of hand-drawn BPMN models. In International Conference on Advanced Information Systems Engineering. Springer, 344–360. doi:10.1007/978- 3- 03079382-1_21 [17] Bernhard Schäfer, Han van der Aa, Henrik Leopold, and Heiner Stuckenschmidt. 2023. Sketch2Process: End-to-End BPMN Sketch Recognition Based on Neural Networks. IEEE Transactions on Software Engineering 49, 4 (2023), 2621–2641. doi:10.1109/TSE.2022.3228308 [18] Liping Zhao, Waad Alhoshan, et al. 2021. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study. Comput. Surveys 54, 3 (2021), 1–41. doi:10.1145/3444689

Received 2026-05-12; accepted 2026-06-19

Related documents

• Simulating Validity: Modal Decoupling in MLLM Generated Feedback on Science Drawings#999228
• DisciplineGen-1M: A Large-Scale Dataset for Multidisciplinary Visual Generation and Editing#793763
• DisciplineGen-1M: A Large-Scale Dataset for Multidisciplinary Visual Generation and Editing#789758
• FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning#1037499
• Biological illustration#1067608