arXiv:2607.28987v1 [cs.SE] 31 Jul 2026
A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation Mersedeh Sadeghi
Simon Scholz
Adrian Psoch-Bajraktari
[email protected] University of Cologne Cologne, Germany
[email protected] University of Cologne Cologne, Germany
[email protected] University of Cologne Cologne, Germany
Abstract
1
Handwritten UML sketches are common in early software design, but turning them into structured, analysable modelling artefacts still requires manual reconstruction. Vision-language models can generate PlantUML from diagram images, but prompt-based use treats this as image-to-text generation rather than structured model generation. We investigate formalism-aware rewards: feedback signals derived from analysable model representations rather than surface text. In a worked example, we adapt a vision-language model for handwritten UML-to-PlantUML generation using supervised fine-tuning followed by Group Relative Policy Optimisation. Generated PlantUML is compared against target representations, using XMI for class diagrams and control-flow graphs for activity diagrams. Emerging results show that the adapted model improves compilability and conversion quality over the untuned open model and one proprietary baseline, while remaining competitive with a stronger proprietary baseline on class diagrams. The added benefit of the reward-guided stage remains open on the current held-out set. Error analysis and metric-validity results show that modelling acceptability is only partially captured, motivating rewards and evaluations that combine model analysis with human judgement.
Software models are often first created informally: on whiteboards, in notebooks, or as quick sketches during design discussions [3]. These sketches are useful for communication, but remain disconnected from model-driven engineering workflows until manually reconstructed in a modelling tool. This reconstruction step is tedious and error-prone, especially when the intended output is not merely a visual diagram but an analysable model representation [18, 19]. Recent vision-language models offer an appealing shortcut: given a diagram image, they can be prompted to generate textual modelling code such as PlantUML. Yet treating sketch-to-model conversion as generic image-to-text generation misses a central property of the task: the output is a model artefact whose quality depends on syntactic validity, structural fidelity, and modelling meaning. This distinction matters because plausible text is not necessarily a valid or useful model. A generated PlantUML file may look close to a reference while failing to compile; it may compile while omitting relationships, attributes, or control-flow branches; or may express a meaning-preserving alternative differing from the reference representation. Prompt-only use of general-purpose vision-language models has little access to modelling-specific constraints during generation, and standard textual or visual similarity metrics capture only part of the problem: they do not directly ask whether the generated artefact can be parsed, analysed, and used as a model. We therefore explore a different view: model analysis itself can provide feedback for sketch-to-model generation. We call this idea formalism-aware rewards. Rather than scoring generated PlantUML only as text, we convert it into an analysable representation and compute feedback in terms of the target modelling formalism. The same representation-based machinery then serves both as an automatic evaluation metric and as training feedback (Section 3). We study this idea on handwritten UML-to-PlantUML generation. Our pipeline first uses supervised fine-tuning to teach a small open vision-language model, Qwen3.5-4B, the image-to-PlantUML task, then applies group-relative policy optimisation (GRPO) with formalism-aware reward components, using a dataset of around 500 handwritten class and activity diagrams paired with PlantUML [1]. We compare the resulting model against its untuned base and two proprietary baselines (Gemini 3 Flash and GPT-4.1 Mini) on a heldout set, and conduct a human ranking study to test whether automatic model-level scores align with perceived modelling quality. The goal is not a definitive benchmark but to examine whether formalism-aware model analysis can become part of the training and evaluation loop. We pursue three research objectives. • RO1 is to test whether the full adaptation pipeline can make a small open model competitive with proprietary models on
CCS Concepts • Software and its engineering → Software system models; Unified Modeling Language (UML); • Theory of computation → Reinforcement learning; • Computing methodologies → Natural language processing. ACM Reference Format: Mersedeh Sadeghi, Simon Scholz, and Adrian Psoch-Bajraktari. 2026. A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation. In Proceedings of NIER Track- Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS) (MODELS 2026). ACM, New York, NY, USA, 7 pages. https: //doi.org/XXXXXXX.XXXXXXX Accepted for publication in the Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems (MODELS 2026). This is the accepted author manuscript and may differ from the final published version. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. MODELS 2026, Málaga, Spain © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
Introduction
MODELS 2026, October 04–09, 2026, Málaga, Spain
handwritten UML-to-PlantUML conversion, in both first-attempt compilation and conversion quality. • RO2 is to examine how far formalism-aware automatic metrics agree with human judgements of model quality, and where they diverge. • RO3 is to isolate the reward-guided stage and study when and how formalism-aware rewards contribute beyond supervised adaptation. Our emerging results are encouraging but deliberately qualified. The adapted model moves from last to second overall in both automatic and human evaluation, making a small open model competitive with substantially larger proprietary systems. At the same time, an SFT-only ablation shows that, on our small held-out set, the specific added effect of the reward-guided stage cannot yet be separated from supervised adaptation: formalism-aware rewards are well-defined model-level signals, but how to make them reliably improve beyond supervised fine-tuning remains open. Our contribution is threefold: a formalism-aware reward and evaluation scheme that derives feedback from analysable model representations rather than text; emerging evidence, across automatic metrics, a human study, and error analysis, that the resulting adapted model is competitive on the task; and an ablation that clarifies that the added benefit of the reward-guided stage is not yet established on the held-out set.
2
Related Work and Positioning
UML diagram recognition has long been studied as a vision and parsing problem. Early systems relied on deterministic image-processing or trained object-detection pipelines to detect diagram elements and export structured representations such as XMI or PlantUML [2, 8, 15]. These pipelines work under controlled conditions but are brittle under handwriting variation, overlapping elements, and noisy layouts. Recent work therefore turns to multimodal language models prompted directly for UML-to-code conversion. Conrardy et al. [5] show that proprietary vision-language models often recover the coarse structure of handwritten UML diagrams yet still produce syntax errors, omissions, and hallucinated elements as complexity grows. Similar patterns recur for other PlantUML and Mermaid generation tasks, where models capture high-level structure but struggle with auxiliary constructs and strict compilability [7, 14]. A smaller body of work studies task-specific fine-tuning. Bates et al. [4] fine-tune LLaVA variants for UML-to-code generation and report improved syntax-error rates over untuned baselines, but evaluate mainly with textual or visual similarity and coarse, categorical error counts. Naboichenko et al. [13] fine-tune a visionlanguage model for question answering over UML class diagrams, showing that targeted adaptation of a smaller model can outperform larger general-purpose ones on diagram-specific tasks. These studies support the value of domain adaptation but do not use the analysable structure of the generated UML artefact as a training signal. Related evaluation work, for example [5, 6], relies on mistake counts, similarity measures, or manually assigned quality dimensions; these metrics remain either underspecified, surface-oriented, or hard to scale. We instead evaluate generated PlantUML through representations derived from the modelling formalism itself. In doing so we build on model-comparison and semantic-differencing
Mersedeh Sadeghi, Simon Scholz, and Adrian Psoch-Bajraktari
research in model-driven engineering, where tools such as SiDiff/UMLDiff align and difference models structurally rather than as text [20]. Our reward can be read as a lightweight, task-specific model-comparison metric turned into a training and evaluation signal for sketch-to-model generation. Outside UML, reinforcement learning has been used for structured multimodal generation: table-to-LaTeX and floorplan-to-JSON systems combine supervised fine-tuning with GRPO-style optimisation using rewards over syntax, structure, or spatial overlap [10, 12]. In those domains the reward stage added value after supervised finetuning, whereas our SFT-only ablation shows that such added value does not arise automatically in UML sketch-to-model generation. To our knowledge, prior sketch-to-model work has not used formalismaware model comparison, over UML-specific representations such as XMI and control-flow graphs, as a reinforcement-learning reward for sketch-to-model generation.
3
Approach: A Formalism-Aware Reward Loop
We treat generated PlantUML as a model artifact, not as text. It can be compiled, parsed into a structured representation, and compared against a reference model; prompt-based uses of vision-language models often leave this structure implicit, evaluating outputs mainly by textual similarity or by whether the generated code compiles. We compute the training reward instead from the structure of the target modelling formalism, so that it measures whether the generated model is correct rather than whether its text matches. We call these formalism-aware rewards. Both stages train on a dataset of ∼500 handwritten UML diagrams spanning class and activity diagrams, each paired with human-written PlantUML code [17]. Each sample provides the image, the diagram type, and a reference, which we use as ground truth, already in analysable form: XMI for class diagrams and a control-flow graph for activity diagrams. At reward time the policy is prompted to convert the image; only the generated PlantUML is converted into the matching representation and compared against the stored reference. Two-stage adaptation. Adaptation proceeds in two stages, shown in Figure 1. The first stage is supervised fine-tuning, which teaches the model the basic task: given a handwritten UML image, produce syntactically valid PlantUML that matches the reference. This stage establishes competence, but it optimises only for surface agreement with the reference text, not for the structure of the resulting model. The second stage addresses exactly that gap. We continue training with reinforcement learning, using Group Relative Policy Optimization (GRPO), so that the model is rewarded for the structural correctness of the model it produces. For each input, the model generates a group of candidate PlantUML outputs. Each candidate is converted into an analysable representation and scored by the formalism-aware rewards, giving every candidate a scalar reward. GRPO then compares the candidates within the group: those scoring above the group average are reinforced and those below it are discouraged, so the model gradually shifts probability toward generations whose underlying model structure matches the reference. Crucially, the learning signal comes not from the generated text but from the model extracted from it.
A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation Prompt Handwritten UML
Supervised fine-tuning (SFT) Stage 1
Initialises
Policy model (VLM)
Generated PlantUML
GRPO update (Stage 2)
Reinforcement Learning
Activity diagram Class diagram
Model Extraction ANTLR4 parser & CFG Builder PlantUML server
Group Relative Policy Optimization (GRPO)
Formalismaware reward
Structural comparison
XMI
CFG
Ground-Truth XMI/CFG
Figure 1: The two-stage, formalism-aware reward loop. To our knowledge, no prior work on UML generation closes the training loop in this way, deriving the reinforcement signal from analysable representations of the generated models, such as XMI and control-flow graphs, rather than from textual similarity. Reward instantiation. We instantiate the loop for two diagram types. For class diagrams, PlantUML exports the generated code to XMI; a generation that fails to compile or yields unparsable XMI scores zero. Valid outputs are compared with the reference over classes, attributes, methods, and relationships, so the reward captures structure that textual similarity cannot, such as whether classes are recovered and whether associations and inheritance relations connect the correct endpoints. Activity diagrams have no comparable export, and textual comparison misleads because one control flow admits many syntactic forms. We parse the generated diagram into a control-flow graph (CFG), whose nodes are actions and control-flow constructs and whose edges are execution paths, and compare it with the stored reference CFG. The reward combines control-flow-graph structure with label similarity, separating control-flow recovery from wording: a diagram with the right structure but imperfect labels, or the reverse, still earns partial credit. Each diagram type’s reward sums to 20. A shared format reward (0–1) applies to both. The class-diagram reward adds a compilation reward (0–1) and structural rewards over classes (0–3), relationships (0–5), methods (0–5), and attributes (0–5); the activitydiagram reward adds a compilation reward (0–1), a control-flowgraph structural reward (0–9), and a label reward (0–9). Because every component scores a generated model against its reference in the formalism’s own terms, these scores also serve as an automatic, model-level measure of conversion quality, whose agreement with human judgement we examine in our evaluation. Class-diagram reward. For class diagrams, generated PlantUML is compiled to XMI through the PlantUML server; if compilation fails or yields no parseable XMI, all content rewards are zero. Otherwise the class-level reward compares the recovered classes by name and type, and attribute and method rewards are computed per class by greedily matching generated members to target members on case-insensitive name, with partial credit for matching signatures and modifiers. The relationship reward covers six kinds (association, aggregation, composition, generalisation, realisation, dependency), matching a generated relationship to a target only
MODELS 2026, October 04–09, 2026, Málaga, Spain
when both its kind and its connected classes agree, with partial credit for multiplicities and role names. Activity-diagram reward. The activity-diagram structural reward is computed over the extracted CFG and does not use labels, which are scored separately. It averages two sub-scores: a weighted Jaccard similarity over node-kind multisets, penalising missing or spurious constructs, and an assignment-based topology score inspired by graph edit distance. The latter matches target and generated CFG nodes by minimum-cost bipartite assignment using the Hungarian algorithm; substitution costs distinguish same-kind from differentkind matches, include a swimlane-aware penalty, and compare the outgoing-edge kinds of matched nodes. Label similarity is then computed over matched nodes and edges using TF-IDF cosine similarity rather than exact string matching, so that distinctive domain terms count more than common function words. The artifact provides exact weights, edit costs, and class-diagram sub-scores. A reusable pattern. The reward design follows a reusable pattern: parse the model into the formalism’s canonical structure and compare it against the reference. The pattern is general, but each formalism must be instantiated with its own representation and comparison, as the class-diagram and activity-diagram rewards above illustrate. We apply it to class and activity diagrams, two of the most widely used and complementary UML notations, capturing static structure and dynamic behaviour respectively [9, 16]; the corresponding analysable representations are XMI for class diagrams and a CFG for activity diagrams. Infrastructure. Reward computation runs inside the training loop, so the reward is available only after each sampled generation is converted into an analysable representation. Three components make this practical. A persistent conversion server keeps repeated PlantUML-to-XMI conversion off the training critical path; a patched PlantUML processing loop turns malformed generations into deterministic zero rewards rather than crashes; and a custom ANTLR4 grammar and CFG builder, which we developed for PlantUML activity diagrams, produce the activity-diagram CFG. These components are enabling infrastructure; the contribution is the loop that turns model analysis into a training signal.
4
Emerging Results
We evaluate the approach from three complementary angles. The automatic evaluation reuses the formalism-aware reward functions of Section 3 as model-level metrics: it measures whether generated PlantUML is syntactically valid and whether its extracted representation preserves the reference structure. The human evaluation assesses perceived modelling quality, which the automatic metrics cannot fully capture. The qualitative error analysis then characterises the mistakes each model makes and explains where automatic and human judgements agree and where they diverge. All three use a held-out test set of 15 handwritten diagrams per type and compare four models: our fine-tuned Qwen3.5-4B, the untuned Qwen3.5-4B base model, and two proprietary baselines, Gemini 3 Flash and GPT-4.1 Mini. Each model was prompted in the form it is expected to handle best: the fine-tuned and base models received the short template used during fine-tuning, while the proprietary baselines received a longer, more detailed instruction
MODELS 2026, October 04–09, 2026, Málaga, Spain
adapted from prior work [5], even though that work found shorter prompts worked at least as well for their models. This gives the proprietary baselines a more explicit instruction prompt, making the comparison conservative with respect to the fine-tuned model.
4.1
Automatic Evaluation
The automatic evaluation reports the combined content score, the sum of the structural reward components (maximum 19); the format reward (Section 3) is excluded, as it checks only the response prefix and not conversion quality. Because the held-out set is intentionally small, we treat these results as emerging evidence rather than a definitive benchmark comparison. Compilation. The most basic requirement is that a generated diagram compiles at all; non-compiling PlantUML yields no model and is unusable in any downstream workflow. Adaptation has its largest effect here. Our model compiles every class diagram (100%) and raises activity-diagram compilation from the base model’s 20.0% to 86.7%, matching Gemini 3 Flash and ahead of GPT-4.1 Mini (66.7%); it is the only model to compile all class diagrams (Table 1). Compilation is consistently lower for activity than for class diagrams across all models, reflecting the tighter syntactic constraints of control-flow logic. Conversion quality. On the combined content score (Table 1), the adapted model moves from last to second on both diagram types. On class diagrams it reaches 17.60, marginally ahead of Gemini 3 Flash (17.36) and well above GPT-4.1 Mini (14.97) and the base model (11.27). On activity diagrams it reaches 13.71, behind Gemini 3 Flash (14.01) but ahead of GPT-4.1 Mini (9.79) and far above the base model, whose very low score (2.81) reflects a model barely functional on this type before adaptation. The absolute gain over the base model is therefore largest on activity diagrams, where adaptation turns near-failure into usable output. Component decomposition. Table 2 breaks down the automatic score by component. Relative to the untuned base, fine-tuning improves every component; the largest class-diagram gain is in relationship recovery, and the largest activity-diagram gain is compilation, followed by structure and labels. Against the proprietary baselines, no single model dominates: on class diagrams our model leads Gemini 3 Flash and GPT-4.1 Mini on compilation, class-level identification, and attributes, but trails Gemini on relationships and methods; on activity diagrams our model is level with Gemini on compilation but trails on structure and labels, while leading GPT-4.1 Mini throughout. The component scores measure recovery of specific modelling properties, rather than imitation of reference text.Wh Ablation. Table 2 also lists the supervised-only checkpoint (SFT) as an ablation reference. On the combined content score, the full SFT-plus-GRPO pipeline is statistically indistinguishable from supervised fine-tuning alone on both class diagrams (𝑝 = 0.94) and activity diagrams (𝑝 = 0.64), using paired Wilcoxon tests. The higher SFT figures on the activity components should therefore not be read as a systematic regression: on this small held-out set the reward-guided stage demonstrates feasibility but not yet a statistically detectable gain over supervised fine-tuning.
Mersedeh Sadeghi, Simon Scholz, and Adrian Psoch-Bajraktari
Table 1: Automatic and human evaluation results. Comp. (%) Model
Content (/19)
Borda
Class Activity Class Activity Overall Class Activity
Gemini 3 Flash 93.3 Ours (fine-tuned) 100.0 GPT-4.1 Mini 86.7 73.3 Base (Qwen)
86.7 86.7 66.7 20.0
17.36 17.60 14.97 11.27
14.01 13.71 9.79 2.81
0.759 0.613 0.442 0.186
0.731 0.662 0.408 0.200
0.787 0.564 0.477 0.172
Table 2: Per-component automatic scores on the held-out set. Component
Gemini 3 Flash
Ours
GPT-4.1 Mini
Base(Qwen)
SFT (no RL)
Class diagrams Compilation Class-level Relationship Attribute Method
93.3 92.2 88.9 93.3 91.0
100.0 98.3 84.2 98.0 90.9
86.7 84.9 64.2 84.0 82.8
73.3 70.3 37.6 65.3 65.6
100.0 98.3 83.6 98.0 91.2
Activity diagrams Compilation Structural Label
86.7 75.1 70.9
86.7 73.4 69.2
66.7 50.9 50.5
20.0 16.0 13.0
100.0 86.0 78.6
4.2
Human Evaluation
Automatic rewards measure whether generated PlantUML is valid and structurally close to the reference, but they cannot fully determine whether a generated diagram is a good interpretation of the sketch. We therefore conducted a human ranking study over the same four models. The 30 held-out sketches were split into three blocks of ten, each block balanced with five class and five activity diagrams; each of the 26 participants was randomly assigned one block and ranked the four models’ outputs for its ten sketches, so all 30 sketches were covered across blocks. We capped each participant at ten sketches because ranking more is cognitively demanding. Each rater saw the original handwritten sketch alongside the four models’ outputs rendered as diagram images, and ranked them from best to worst; an output that failed to compile was shown as a rendering-error image rather than a diagram. The reference model was not shown, so raters judged how well each output captured the sketch rather than how closely it matched the reference. We aggregate the rankings with a Borda count, which converts each ranking into points (three for first place down to zero for last), sums them, and normalises the total to [0, 1] by the maximum attainable score; we relate the result to automatic scores by Spearman correlation. Participants were recruited via SurveyCircle and social media and screened with a UML-knowledge test, retaining only those above a competence threshold to reach the final sample of 26. The sample skewed toward computing backgrounds (21 of 26 in CS/Business Informatics/SE) and early-career education levels (10 current Bachelor’s students, 3 doctoral candidates). Seventeen had hands-on UML experience and nine had encountered it without using it; none reported no exposure, consistent with the screening criteria. Inter-rater agreement within groups was strong (Kendall’s 𝑊 between 0.80 and 0.91), which lends weight to the aggregate ordering.
A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation
Ranking. The human ranking confirms the main trend of the automatic evaluation: the adapted model moves from last to second overall, ahead of GPT-4.1 Mini and behind Gemini 3 Flash in perceived quality (Table 1); its mean rank improves from 3.44 (last of four) to 2.16, behind Gemini’s 1.72. This agreement provides evidence that the adapted model’s gains are visible to human raters, not only to the automatic metrics. The human preference for Gemini is wider on activity diagrams than on class diagrams, mirroring the automatic scores, where our model is closest to Gemini on class diagrams and trails further on activity. Metric validity. The combined content score correlates positively and significantly with the human rankings (𝜌 = 0.565, 𝑝 < 0.001, 𝑛 = 120 model-diagram pairs), and similarly within each diagram type (𝜌 = 0.607 for class, 0.636 for activity). The association is significant but only moderate: a higher automatic score reliably coincides with a better human rank, yet the two do not measure quite the same thing. Two asymmetries make the gap concrete. First, on class diagrams our model and Gemini 3 Flash score almost the same (17.60 and 17.36) yet human raters still prefer Gemini, so the metric misses something the raters weigh. Second, compilation success predicts human preference far more strongly for activity diagrams (𝜌 = 0.689) than for class diagrams (𝜌 = 0.444): activity diagrams are syntactically harder to compile, so compiling at all is a stronger signal of quality there, whereas almost all class diagrams compile and compilation discriminates little.
4.3
Qualitative Error Analysis
To understand the remaining weaknesses of the adapted model, we compared each handwritten diagram, its reference model, and the four generated outputs across all 120 outputs (four models on 30 diagrams), grouping recurring mistakes with an inductively derived taxonomy covering failures of extraction, compilation, recognition, typing, structure, hallucination, omission, and cosmetic fidelity. The taxonomy is an organising scheme rather than a measurement instrument: given the interpretive nature of the labelling and the modest sample, we characterise dominant patterns rather than report per-category frequencies. The dominant patterns differ by diagram type. In class diagrams, remaining errors mostly concern local modelling details, such as missing members, wrong relationship types or endpoints, or details hard to read in dense sketches. In activity diagrams, they often concern control-flow interpretation, such as substituting one branching or looping construct for another or attaching a label to the wrong flow part. Hallucinations and omissions recur across all models, while the untuned base produces the most spurious structure. The main lesson from this analysis is the distinction between meaning-preserving deviations and meaning-altering errors. The former changes the representation while keeping the intended interpretation: representing a multi-way branch as a sequence of conditionals, for example, may preserve the intended control-flow logic even though the parser and the reward functions treat the two forms as different structures. Figure 2 shows a representative case: the model renders the sketch’s decision as a switch-branch where the reference uses an if-branch. Both produce structurally
MODELS 2026, October 04–09, 2026, Málaga, Spain
identical diagrams and preserve the decision logic, yet the controlflow-graph reward scores the switch as a deviation. Such meaningpreserving differences account for part of the gap between the automatic metrics and human raters. A meaning-altering error, by contrast, changes the logic the diagram represents. Representation-based rewards penalise both whenever they differ from the target representation, whereas human raters need not: in optional explanations, participants often tolerated meaning-preserving deviations while penalising meaningaltering ones. This is the gap that the moderate metric-human correlation reflects, separating genuine model limitations from the strictness of the chosen representation. Tellingly, meaning-altering errors cluster where the model must read fine visual detail, such as a multiplicity on a dense diagram, pointing to a limit in visual grounding rather than in PlantUML generation.
4.4
Lessons Learned
Returning to the objectives of Section 1, three lessons emerge. On RO1, the full adaptation pipeline yields a competitive model overall, moving the small open model from last to second among four, ahead of GPT-4.1 Mini and behind Gemini 3 Flash. On RO3, however, the SFT-only ablation shows that the specific contribution of the rewardguided stage remains open: on this small held-out set, supervised fine-tuning and the full pipeline are statistically indistinguishable. Establishing whether and how model-level rewards help beyond supervised adaptation is the question taken up in Section 5. A cross-cutting lesson, beyond the three objectives, is that while the idea of deriving rewards from analysable model representations is generic, the representation must suit each diagram type. XMI fits class diagrams, whose elements (classes, attributes, methods, relationships) can be extracted and compared directly; activity diagrams need a control-flow graph, which compares process structure better than raw PlantUML text but also exposes the difficulty of telling an equivalent control-flow formulation from a genuinely different one. Designing the analysable representation per formalism is therefore part of the method, not a generic text-matching step, and extending it to further notations is a natural next step. On RO2, formalism-aware metrics are scalable but partial proxies for modelling quality. They capture important downstream properties, especially compilability and recoverable structure, but they penalise meaning-preserving variation that human modellers accept and miss the finer cues raters weigh. This argues for evaluation that pairs formalism-aware analysis with human judgement, and for reward designs that distinguish meaning-preserving from meaningaltering differences rather than scoring every structural deviation as an error. Together with the visual-grounding limitation identified above, these lessons frame the agenda we turn to next.
5
Future Plans
Our results suggest that the full adaptation pipeline can yield a small open model competitive on sketch-to-model conversion, while also exposing two limits: automatic metrics correlate with human judgement only moderately, and the added effect of the reward-guided stage remains unresolved on the current held-out set. This points to a research agenda rather than a single missing feature: making the reward, the optimisation procedure, the model, and the evaluation reason about meaning rather than surface structure.
MODELS 2026, October 04–09, 2026, Málaga, Spain
(a) Handwritten input
Mersedeh Sadeghi, Simon Scholz, and Adrian Psoch-Bajraktari
(b) Reference: if-style branch
(c) Generated output: switch-style branch
@startuml [...] if (Attempts > 3?) then (yes) :Lock account for 15 minutes; :Send security notification email; else (no) :Return to login screen; endif [...] @enduml ye s
A tte m p ts > 3 ?
no
@startuml [...] switch (Attempts > 3?) case ( Yes ) :Lock account for 15 minutes; :Send security notification email; case ( No ) :Return to login screen; endswitch [...] A tte m p ts > 3 ? @enduml Yes
Lock account for 15 m inutes
R eturn to login screen
S end security notification em ail
Lock account for 15 m inutes
No
R eturn to login screen
S end security notification em ail
Figure 2: A meaning-preserving deviation: from the handwritten sketch (a), the reference uses an if-branch (b) and the model an equivalent switch-branch (c). The two render to identical diagrams, yet the structural metric penalises the switch. Richer reward optimisation. One direction is to make the reward more meaning-aware. A representation-based reward treats any deviation from the stored target as an error, whereas a modeller accepts semantically equivalent formulations. This matters most during training: an acceptable meaning-preserving generation, such as the switch-branch of Figure 2, can be discouraged by the current reward even though it faithfully captures the sketch. Future rewards should therefore grant credit for semantic equivalence, comparing models up to meaning-preserving transformations rather than by structural identity alone. A second, complementary direction concerns how multiple reward components are optimised. Because our reward sums several subscores (five for class diagrams, three for activity diagrams), group-relative optimisation that normalises the aggregate can let strong components mask weak ones, weakening the training signal for weaker subscores. Decoupling the normalisation per component, as in group reward-decoupled policy optimisation (GDPO) [11], may provide a more targeted signal for the individual constraints our component analysis shows are recovered unevenly. More generally, reward-guided fine-tuning should be treated as an optimisation problem in its own right, including component and diagram-type balancing, regularisation against the supervised policy, and early stopping. Visual grounding. Our error analysis traced many remaining meaning-altering errors not to PlantUML generation but to reading fine visual detail, such as a multiplicity or an arrowhead that distinguishes a generalisation from a directed association. Addressing this calls for stronger visual grounding: spatially aware encodings that localise elements within the image, training data annotated with element positions rather than image-code pairs alone, and agentic zoom-into-image recognition that inspects dense regions at higher resolution. The last is a natural fit for the recognition errors we observed, where a method or attribute name is substituted with a visually similar one. Because this limit is perceptual, it is unlikely to be solved by reward aggregation alone; gains in visual grounding should instead compound with better reward design.
Generalisation to further formalisms. The reward loop is not specific to the two notations we study. Many modelling languages with analysable representations could support the same recipe: parse the generated artefact into that representation and derive feedback from its fidelity to a target representation. Extending it to further UML notations such as state machines and sequence diagrams, and to non-UML languages such as BPMN or SysML, would test how far formalism-aware rewards transfer and what representation each formalism requires. Towards interactive modelling. Finally, the divergence between automatic rewards and human judgement is itself an opportunity. Rather than treating generation as a single shot evaluated in batch, a system could present its output, flag elements its own model analysis finds uncertain or inconsistent, and let the modeller correct them. Those corrections supply exactly the meaning-aware signal a representation-based reward lacks, and could feed back as preference data for further training. Combined with a larger and more diverse evaluation — left to future work given the deliberately small held-out set used here — this would move formalism-aware models from a one-shot converter towards an interactive modelling assistant embedded in real workflows.
6
Conclusion
This paper argued that sketch-to-model generation should be treated as structured model generation rather than image-to-text generation. We explored formalism-aware rewards as one way to turn model analysis into feedback, using XMI comparison for class diagrams and control-flow-graph comparison for activity diagrams. The adapted open model becomes competitive with larger proprietary baselines, but the added benefit of the reward-guided stage beyond supervised fine-tuning remains unresolved on the current held-out set. More broadly, the results show that model-level metrics capture important aspects of compilability and structure, while still falling short of human judgement on modelling meaning.
A Formalism-Aware Reward Loop for Handwritten UML-to-PlantUML Generation
Data and Artifact Availability Code, prompts, reward definitions, and evaluation material are available in an anonymous repository: https://anonymous.4open. science/r/uml-to-plantuml-F938.
References [1] Anonymous. 2025. Dataset Paper Accepted but Not Yet Publicly Available. Anonymised reference. Full citation withheld for double-anonymous review and will be added in the camera-ready version.. [2] Monique Axt. 2023. Transformation of sketchy UML Class Diagrams into formalPlantUML models. [3] Sebastian Baltes and Stephan Diehl. 2014. Sketches and diagrams in practice. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT/FSE14). ACM, 530–541. doi:10.1145/2635868. 2635891 [4] Averi Bates, Ryan Vavricka, Shane Carleton, Ruosi Shao, and Chongle Pan. 2025. Unified modeling language code generation from diagram images using multimodal large language models. Machine Learning with Applications 20 (2025), 100660. doi:10.1016/j.mlwa.2025.100660 [5] Aaron Conrardy and Jordi Cabot. 2024. From Image to UML: First Results of Image Based UML Diagram Generation Using LLMs. arXiv:2404.11376 [cs.SE] https://arxiv.org/abs/2404.11376 [6] Daniele De Bari, Giacomo Garaccione, Riccardo Coppola, Marco Torchiano, and Luca Ardito. 2024. Evaluating Large Language Models in Exercises of UML Class Diagram Modeling. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Barcelona, Spain) (ESEM ’24). Association for Computing Machinery, New York, NY, USA, 393–399. doi:10. 1145/3674805.3690741 [7] Grant Guernsey. 2025. Harnessing Large Language Models for Automated Software Diagram Generation. Master’s thesis. University of Cincinnati. OhioLINK Electronic Theses and Dissertations Center. http://rave.ohiolink.edu/etdc/view? acc_num=ucin1746701542674719 [8] Bilal Karasneh and Michel R.V. Chaudron. 2013. Extracting UML models from images. In 2013 5th International Conference on Computer Science and Information Technology. 169–178. doi:10.1109/CSIT.2013.6588776 [9] Hatice Koç, Ali Mert Erdoğan, Yousef Barjakly, and Serhat Peker. 2021. UML Diagrams in Software Engineering Research: A Systematic Literature Review. Proceedings 74, 1 (2021). doi:10.3390/proceedings2021074013 [10] Jun Ling, Yao Qi, Tao Huang, Shibo Zhou, Yanqin Huang, Jiang Yang, Ziqi Song, Ying Zhou, Yang Yang, Heng Tao Shen, and Peng Wang. 2025. Table2LaTeXRL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models. arXiv:2509.17589 [cs.AI] https://arxiv.org/abs/ 2509.17589 [11] Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, et al. 2026. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242 (2026). [12] Yuanqing Liu, Ziming Yang, Yulong Li, and Yue Yang. 2026. FloorplanVLM: A Vision-Language Model for Floorplan Vectorization. arXiv:2602.06507 [cs.CV] https://arxiv.org/abs/2602.06507 [13] Artem Naboichenko and René Peinl. 2026. Unlocking UML Class Diagram Understanding in Vision Language Models. arXiv:2605.11634 [cs.CV] https: //arxiv.org/abs/2605.11634 [14] HG Ranjani and Rutuja Prabhudesai. 2025. Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs. arXiv:2509.11667 [cs.LG] https://arxiv.org/abs/2509.11667 [15] Mantas Ražinskas, Benas Miliūnas, Mantas Jurgelaitis, Lina Čeponienė, and Lina Bisikirskienė. 2024. Transforming Sketches of UML Use Case Diagrams to Models. IEEE Access 12 (2024), 185826–185837. doi:10.1109/ACCESS.2024.3514455 [16] Gianna Reggio, Maurizio Leotta, Filippo Ricca, and Diego Clerissi. 2014. What are the used activity diagram constructs? a survey. In 2014 2nd International Conference on Model-Driven Engineering and Software Development (MODELSWARD). 87–98. [17] Simon Scholz and Mersedeh Sadeghi. 2026. CAS2UML: A Handwritten Sketchto-PlantUML Dataset for Class and Activity Diagrams. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026). Accepted for publication. [18] Harald Störrle. 2017. How are Conceptual Models used in Industrial Software Development? A Descriptive Survey. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (Karlskrona, Sweden) (EASE ’17). Association for Computing Machinery, New York, NY, USA, 160–169. doi:10.1145/3084226.3084256 [19] Jagoda Walny, Jonathan Haber, Marian Dörk, Jonathan Sillito, and Sheelagh Carpendale. 2011. Follow that sketch: Lifecycles of diagrams and sketches in software development. In 2011 6th International Workshop on Visualizing Software
MODELS 2026, October 04–09, 2026, Málaga, Spain
for Understanding and Analysis (VISSOFT). 1–8. doi:10.1109/VISSOF.2011.6069462 [20] Zhenchang Xing and Eleni Stroulia. 2005. UMLDiff: an algorithm for objectoriented design differencing. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 54–65.