Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM Daniel Amyot
arXiv:2607.28825v1 [cs.SE] 30 Jul 2026
University of Ottawa, Ottawa, Canada [email protected]
Abstract. Enterprise-modeling (EM) tools are often complex and hard to extend. Yet, users may want to explore new EM features and capabilities that currently do not exist. AI coding agents can help here by enabling the development of new capabilities and entire tools, but whether we can trust a modeling-language tool an LLM largely wrote remains a question. This paper reports on the AI-assisted construction of PM4Py-UCM, an open-source tool that mines Use Case Map (UCM) models from event logs. PM4Py-UCM’s capabilities include some expected from process mining tools (e.g., performance heat-maps and dashboards) and distinctive ones (e.g., mined executable scenarios/variants, and model decomposition). We mined the development record itself, composed of 18 agent sessions (374 human turns and 10,328 tool actions over 65 hours), 151 commits, 20 releases, and a test suite grown from 108 to 691 test functions, in order to characterize, in a single in-depth case, how the tool was built with an agent (Claude Code), complemented by an independent static assessment of the resulting code (coverage, complexity, maintainability, security, architecture). We contribute a reproducible, privacy-preserving toolkit and taxonomy that classify human turns and flag cross-cutting consistency work, agent corrections, and retracted requests. Up to version 0.7.4, fixes outnumber features 2.3:1, with ≈18% of turns for correcting agent errors. Feature waves dragged a measurable tail of documentation/test/notebook consistency work, and tests grew lockstep with features. We finally present lessons learned, centered on making model transformations mechanically checkable, and the oraclebased validation strategy that closed the “the agent said it works” gap, for responsibly engineering EM tooling with AI. Keywords: Enterprise modeling · Process mining · AI-assisted development · Claude Code · Use Case Maps · Experience report
1
Introduction
Graphical models have supported enterprise/process modeling and requirements engineering for over a century [22], and for most of that time they were drawn by hand. Process mining (PM) [1] is a particular type of enterprise modeling (EM) approach that enables discovering such models from event logs, turning recorded behavior into models represented with Petri nets [17], BPMN [25], or
2
D. Amyot
(in our case) Use Case Maps [8, 18]. Yet the tools that make this possible are themselves a bottleneck: EM tooling, especially commercial solutions, is typically complex and hard to extend [13, 30], limiting how quickly the community can put new modeling and analysis ideas into practice. The recent rise of artificial intelligence (AI) agents for coding and other long/complex tasks typically performed by humans changes this context radically [7, 29]. This paper explores the development of process-mining tooling using a commercial coding agent based on Large Language Models (LLM), namely Claude Code 1 . This new tool was motivated by the needs of ongoing research projects with Ottawa-based hospitals, where online PM tools (which represent the majority of high-end commercial solutions [24]) cannot be used due to strict privacy policies preventing the storage of patient-related information on the cloud, and where locally-installed solutions do not address all the needs of these projects. Such an endeavor poses a practical research question (RQ) the EM community will increasingly face: RQ: Can we trust a modeling tool an Agentic AI system largely wrote? This paper reports on PM4Py-UCM, an open-source extension of the PM4Py library [6] that mines executable Use Case Map models from event logs and exports them to the jUCMNav tool, the main UCM modeling environment [27]. As an EM-language tool, PM4Py-UCM encodes a standardized metamodel [18], performs a chain of model transformations (process tree → UCM model → jUCMNav interchange file, and a round trip back), and must stay faithful to a reference editor (jUCMNav). Over roughly ten weeks, it was built almost entirely through an AI coding agent, and the entire development is recorded in agent-session transcripts and version control. We exploit that record to measure how the tool was actually built. This paper provides the following contributions: – C1 A new version of PM4Py-UCM that extends previous versions [2, 4]) with new capabilities that include i) filtering and activity renaming, ii) performance overlays, iii) discovering and comparing model families, iv) userdefined dashboards, v) self-contained interactive HTML reports with navigable SVG models, vi) project saving/sharing/resuming, and vii) executablecode export (Sect. 3). – C2 A measured, reproducible account of AI-assisted PM-tool development, including human/agent interaction mix, the balance of features vs. fixes vs. corrections, a developer-identified cross-cutting consistency effort, effort per modeling concern, and how tests grew with features (Sect. 5). – C3 An independent, tool-based assessment of the resulting artifacts (test coverage, code complexity, maintainability, security, and architecture) corroborating the internal, oracle-based validation (Sect. 5.5). – C4 A privacy-preserving, reusable reflexive-mining method and toolkit, plus distilled lessons for engineering PM/EM tooling with agentic AI (Sects. 4, 6). 1
https://code.claude.com/docs/en/overview
Building a Process-Modeling Tool using AI
3
Note that this is an exploratory single-case study; we scope quantitative claims to this one project throughout, and treat the patterns as hypotheses for replication. In addition to the above sections, this paper discusses necessary background (Sect. 2), threats to validity (Sect. 7), related work (Sect. 8), and, finally, conclusions (Sect. 9).
2
Background
The User Requirements Notation (URN) is an international standard (ITU-T Z.151 [18]) that combines the Goal-oriented Requirement Language (GRL) with a process notation, namely Use Case Maps (UCM) [8]. A UCM is a graph of path nodes, including start/end points, responsibilities (or activities in BPMN terms), OR- and AND-forks/joins for choice and concurrency, and stubs containing plug-in sub-maps. These path nodes are optionally bound to a twodimensional hierarchy of components, which describe activity performers (roles, persons, systems, etc.). UCM offers an executable scenario semantics, enabling validation as well as sequence-chart and test generation. The open-source Eclipse tool jUCMNav [27]2 is the de-facto URN editor and defines the .jucm (XMI) file format PM4Py-UCM targets. URN has two decades of applied experience across combined goal/process modeling [5]. Process mining discovers models from event logs [1]. Among existing PM algorithms, the inductive miner yields a block-structured process tree [23] that maps cleanly to BPMN and UCM. PM4Py [6] is the dominant open-source Python library for PM, and represents the start point for extension with PM4Py-UCM. Prior work introduced the UCM discovery pipeline, hierarchical process decomposition, and performer binding [4], as well as concurrency-aware variant clustering with executable scenario synthesis [2]. This paper treats those as given and focuses on the tool’s broader capabilities and, centrally, on how it was built. Contemporary LLM-based coding agents operate largely autonomously over a repository with tools to read, edit, run, and test code, driven by naturallanguage instructions. Their use is spreading in practice faster than it is being studied [11, 16], particularly for specialized software such as modeling tooling. In a recent study, Bhati [7] argues that development based on coding agents has shifted from code generation to delegated execution under human supervision. Here, we indeed treat the agent as the primary implementer and the human as specifier, reviewer, and domain authority, and we measure that division of labor.
3
PM4Py-UCM Capabilities
PM4Py-UCM3 extends PM4Py [6] so that UCMs become a first-class output of
process discovery. Its pipeline reuses the inductive miner [23] to obtain a process 2 3
Available at https://github.com/JUCMNAV/jUCMNavPlus Code and documentation available: https://github.com/ProcessMining-uOttawa/ pm4py-ucm/; Web demo: https://pm4py-ucm.streamlit.app/
4
D. Amyot
Fig. 1. PM4Py-UCM: Streamlit-based Web interface with BPMN rendering, including performance heat-map overlays (activity/edge text with color and thickness encoding time and frequency metrics).
tree, converts it into a Python UCM object model mirroring the URN metamodel, renders it in UCM or BPMN style (with components), and serializes it to a round-trip-safe .jucm file for jUCMNav. The tool is available both through a Web application (using Streamlit [21]) and programmatically (Python API). At the tool’s core is a Python object model implementing the UCM subset of the URN metamodel (Z.151), later extended with a scenario layer (also from Z.151) to make mined models executable [2]. Discovery is a mainly a chain of model transformations: process tree → UCM object model → .jucm file. These transformations are where correctness is hardest and least visible: a misplaced condition or label yields a .jucm file that opens cleanly in jUCMNav yet executes the wrong scenario, e.g., a silent semantic corruption that raises no exception. We distinguish here capabilities expected of a mature process-mining tool [24] from those that are, to our knowledge, distinctive; several of the latter were introduced in prior work [2, 4] while others are NEW ( C1 ). Some of the expected capabilities of PM4Py-UCM include: – NEW Log filtering by specific activities, activity frequency, variant frequency, date ranges (fully inside or intersecting), duration percentile, and top variants by case-coverage %, with export of the filtered log. – NEW Activity renaming, with import/export of the renaming map. – NEW Project/session saving and resuming. – NEW Performance-metric overlays for activities and branches, and visual metric heat-maps on the process models (Fig. 1). – NEW Dashboard creation with multiple representations, including model pinning (Fig. 2).
Building a Process-Modeling Tool using AI
5
Fig. 2. PM4Py-UCM custom analytics dashboard (KPI, gauges, distributions...).
Fig. 3. jUCMNav: scenario execution of a mined UCM and generated sequence diagram.
Among PM4Py-UCM’s distinctive capabilities, we find: – Log-driven UCM discovery: to our knowledge, PM4Py-UCM offers the first pipeline to make Use Case Maps a first-class process-mining output [4]. – Performer-aware component binding: activities bound to a two-dimensional hierarchy of components mined from log roles/resources—the “who” similar to BPMN pools/lanes, which are seldom mined [4]. – Hierarchical decomposition of models using UCM stubs containing sub-processes, to ease the understanding of complex, spaghetti-like models [4].
6
D. Amyot
Fig. 4. Generated HTML side-by-side comparison report for a UCM model family.
– Executable scenario synthesis: one UCM scenario per concurrency-aware variant, with variant-driven or data-driven (decision-tree [26]) branch conditions [2]. These can be executed for visualization and sequence diagram generation in jUCMNav (Fig. 3). – NEW Model families with dynamic-stub variation points: partitioning a log by 1–2 case attributes yields a family that assembles into a configurable umbrella reference model (and a corresponding .jucm file). – NEW Family comparisons and dashboards exportable as single, self-contained interactive HTML files that embed scalable SVG process models (UCM and BPMN) with navigation hyperlinks to sub-processes (Fig. 4). – NEW Performance overlays computed and scaled at three levels: local (one map), per-family-member (across a family member’s decomposed maps, and global (the whole model). – NEW A scriptable Python API : every capability is available programmatically, with no dependence on the Web app. The Web app also supports exporting a whole analysis as executable Python code: as a standalone pipeline program and as a JupyterLab Notebook4 that doubles as an interactive API tutorial.
4
Reflexive-Mining Method
The method used to answer the research question is based on the following pipeline (Fig. 5), itself implemented in Python ( C4 ). 4
https://jupyter.org/
Building a Process-Modeling Tool using AI
7
Fig. 5. Pipeline applied to PM4Py-UCM: From an agent’s logs to trustworthy evidence.
I) Data Inputs. We draw on three artifacts of the development itself: a) 18 AI-agent session transcripts (Claude Code JSONL chat logs with timestamped human turns, agent tool calls, model identifiers, and token usage), b) the project’s version history (151 commits, 20 release tags), and c) the test suite tracked across releases. The corpus is frozen at release v0.7.4 (July 22, 2026). II) Inclusion. Some sessions also resume-copy earlier turns verbatim; we collapse 48 such duplicates to their first occurrence. Removing interrupt (e.g., caused by network loss) and attachment artifacts (some elements are in both categories) leaves 317 substantive human turns, i.e., requests to Claude Code. III) Taxonomy. Each turn receives one primary intent and zero or more cross-cutting flags (Table 4). In particular, the cross_cutting flag itself captures requests to propagate a change across auxiliary artifacts (e.g., save/load schema) so they stay consistent after a feature lands. IV) Labeling. Because flags such as corrects_agent depend on what surrounds a turn, every turn is labeled in context. All 374 turns were first labeled by an LLM reading each turn with its in-session neighbors, and a random sample was reviewed by the author; this is the gold standard used throughout. A separate LLM-assisted classifier (cheap claude-haiku-4-5 model) reproduces this labeling, and a transparent keyword baseline is retained for auditability (Sect. 5.2). The prompt used is as follows (TAXONOMY is from Table 4): You classify a single USER turn from a software-development chat log. Use the taxonomy exactly. Consider the provided previous/next turns only as context for the flags (especially user_incorrect and corrects_agent). Return STRICT JSON: {"primary":"...","flags":[...], "confidence":0.0-1.0, "rationale":"<=15 words"}.\n" + TAXONOMY
V) Outputs and Privacy. We report workflow measures (sessions, toolaction-to-turn ratio, active time as a conservative floor of inter-event gaps ≤10 min, output tokens as an effort proxy), commit/churn and release cadence, test growth, and a keyword mapping of turns to modeling concerns. Only prompt-free, aggre-
8
D. Amyot
Table 1. Interaction taxonomy applied to each human turn: exactly one primary intent, plus zero or more (non-exclusive) cross-cutting flags. Primary intent (chooses exactly one) feature: add new capability / field / view fix: report or repair a defect / wrong behavior refactor: restructure / simplify / rename, no behavior change test_qa: tests, coverage, oracles, test plan, quality assurance design: options / architecture / planning discussion (pre-code)
docs: README, docstrings, tutorials, diagrams, papers release_ops: versioning, packaging, PyPI, deploy, CI, git operations data_analysis: running the tool, mining logs, inspecting outputs, notebooks clarification: short answer / approval / redirect (“yes”, “continue”) meta_process: memory, session setup, plan housekeeping
Cross-cutting flags (zero or more) user_incorrect: the user’s own request/assumption was later shown wrong, or retracted (“you’re right”, “never mind”) scope_change: changes an earlier requirement (“instead”, “no longer”, “drop”) rework: revisiting work previously considered done
cross_cutting: propagate a change across auxiliary artefacts for consistency (README, test counts, notebooks, save/load schema, exporters, class diagram) corrects_agent: correcting an agent mistake, hallucination, or regression
gated results are published; the toolkit lives in a repository5 separate from the measured project to avoid an observer effect, and is archived for reproducibility [3]. The gold labels and all aggregates are released so the coding can be independently audited and re-labeled. The corpus is single-subject (the author’s own development), uses no third-party data, and is shared only as aggregates.
5
Quantitative Development Assessment
We reconstructed the development of PM4Py-UCM from 18 AI-agent sessions (2026-05-13 → 07-23, ≈10 weeks; some sessions spanned more than one day) and the git history, including test cases. All figures below describe this single project ( C2 ): we report counts and proportions of its turns, not inferential statistics. After removing duplicate turns and interrupt/attachment artifacts, 317 substantive human turns remain. For the original 374 turns, 10,328 agent tool actions were observed, with a ≈28:1 tool-action-to-turn ratio. Development spanned three model generations, ≈65 h of active time, 151 commits (+75,204/ − 9,187 lines), and 20 releases (v0.2.0 → v0.7.4). Figure 6 summarizes this timeline. 5.1
Interaction Mix
Table 2 provides the distributions. Fixes dominate (33.8%), outnumbering features (14.8%) at a 2.3:1 ratio. In essence, the work was overwhelmingly iterative refinement rather than first-shot generation. 5
https://github.com/damyot/pm4py-ucm-devmining
Building a Process-Modeling Tool using AI
9
Fig. 6. PM4Py-UCM development timeline: commits/day, releases (orange lines), and test-suite growth. Table 2. Primary-intent and cross-cutting-flag distribution over 317 turns. Primary intent fix clarification feature release_ops docs design meta_process data_analysis refactor test_qa
n
%
107 33.8 51 16.1 47 14.8 34 10.7 26 8.2 24 7.6 21 6.6 4 1.3 2 0.6 1 0.3
Flag (non-excl.) n corrects_agent cross_cutting rework scope_change user_incorrect
%
57 18.0 31 9.8 8 2.5 4 1.3 3 0.9
In terms of correction and cross-cutting effort, corrects_agent fires on 18.0% of turns (57/317; nearly one turn in five repaired an agent mistake), whereas user_incorrect occurs in only ≈1% of turns (three identified instances): friction came overwhelmingly from agent error rather than human error. This is a conservative lower bound; the flag requires an explicit in-transcript retraction. The developer-proposed cross_cutting flag (9.8%) captures consistency work, peaking with the dashboards/families wave in week 29 (Fig. 7). 5.2
Labeling Reliability
We initially attempted a transparent, keyword-based classifier as a cheap baseline. The context-aware LLM-based labels, partially validated by the domain expert, remain our reference gold standard. Hence, the comparison measures the baseline’s inadequacy, not the reference’s reliability. The baseline agrees poorly and, crucially, fails to detect agent corrections at all, in part because correcting the agent is a semantic act, not a lexical one. The baseline reliably captured only lexically-marked categories such as cross-cutting consistency work. We therefore rely on the context-aware gold labels throughout. The full comparison, including per-class precision/recall and Cohen’s κ, is reported in the online appendix [3].
10
D. Amyot
Fig. 7. Weekly interaction mix. Bursts: core W20 (2026) & dashboards/families W29. Table 3. Effort by modeling concern × intent (10 top concerns, 199/245 topic-bearing turns). #Corr = agent-correction turns; #CC = cross-cutting turns. Concern
N #Fixes #Features %Fix #Corr #CC
scenarios & conditions 38 visualization & layout 36 mining & filtering 22 release & CI 21 dashboards 18 web/Streamlit UI 17 docs 15 families & compare 13 performance & heat-maps 13 jUCMNav export/round-trip 6
5.3
18 28 11 3 10 8 1 9 4 5
8 8 6 0 3 4 0 3 5 0
47 78 50 14 56 47 7 69 31 83
16 10 2 0 5 5 0 6 2 5
4 2 1 4 1 2 8 0 3 0
Effort by Modeling Concern
Restricting to the 245 topic-bearing turns (further excluding the clarification and meta_process categories), effort concentrated on scenarios & conditions and on visualization & layout (Table 3). The cross-tabulation reveals that visual/layout work is 78% fixes and jUCMNav export 83% fixes. The coding agent struggled most with external-format and visual fidelity (XMI structure, PNG/SVG rendering, exact condition semantics). Agent corrections cluster on scenarios/conditions (16) and visualization (10), whereas cross-cutting consistency work concentrates on documentation (8). 5.4
Quality Kept Pace with Features
The test suite grew from 108 test functions (v0.2.0) to 691 by v0.7.4 (10→26 test files; parametrization expands these into about 880 actual test cases executed). The test numbers rose in lockstep with feature waves (Fig. 8), indicating that tests were written right away, not retrofitted. Test-line insertions are ≈17% of all insertions. Crucially, the suite does more than exercise code: it closes the “the agent said it works” gap for the transformation chain through oraclebased checks (hand-computed fixtures), algebraic invariants and metamorphic
Building a Process-Modeling Tool using AI
11
Fig. 8. Test-suite growth over the full commit history: 108 test functions on 2026-05-13 to 691 on 2026-07-22 (test files on the right axis; releases marked).
transforms [10] on the mined models, byte-deterministic XMI round-trips, and differential comparison against PM4Py’s own metric functions (e.g., about durations). Such testing was also enabled by embedding sample event logs in the tool and its repository. This validation strategy, not the agent’s assurances or claims, is what helped make this agent-written modeling tool trustworthy, and the 18% correction rate shows why it was necessary.
5.5
Independent Quality, Security, and Architecture
Beyond our own tests, we ran standard third-party analyzers on the agentwritten code, so the trust argument does not rest on our oracles/tests alone (Table 4, C3 ). The library is both well covered and well structured, as assessed by pytest-cov6 : 90.6% line coverage (7,236 of 7,983 lines over ≈880 pytest cases). Static metrics computed with radon7 give an average cyclomatic complexity of 5.6 (69% of blocks rated A; worst case 47, in the log-partitioning routine) and a mean maintainability index of 70 (95% of files rated A). Static security analysis with Bandit8 over ≈25k LOC (≈19.9k Source LOC) reports no medium- or high-severity findings; the twelve low-severity flags are benign (enumeration string literals misread as passwords, shell=False subprocess calls, intentional fallback except blocks). Architectural contracts checked with import-linter9 hold: the object model (metamodel) does not import the discovery algorithms, and reaches the visualization layer only through a single documented layouter reuse. A dependency graph over the 64 code module files shows low coupling (71 inter-module imports). Figure 9 tracks the trend: as the library tripled in size, average complex6
https://pypi.org/project/pytest-cov/ https://github.com/rubik/radon 8 https://github.com/pycqa/bandit 9 https://github.com/seddonym/import-linter 7
12
D. Amyot
Table 4. Independent quality, security, and architecture metrics for the PM4Py-UCM library (from third-party analyzers). Aspect
Metric
Value
Tests (pytest-cov) line coverage / cases 90.6% / ≈880 Complexity (radon) avg (final rel.) / % rank A / max 5.6 / 69% / 47 Maintainability (radon) mean MI / % rank A / trend 70 / 95% / 82→70 (CC flat) Size (radon) SLOC / comment ratio 19.9k / 15% Security (Bandit) high / medium / low 0 / 0 / 12 (benign) Architecture (import-linter) contracts / coupling 2/2 hold / 64 mod., 71 imports Dependencies (pip-audit) known CVEs / packages (OSV) 0 / 59
Fig. 9. Library maintainability index (left) and average cyclomatic complexity (right) across releases (v0.2.0 to v0.7.4): complexity stays flat (≈5.5, rank A–B) while maintainability declines only mildly as the code triples in size.
ity stayed flat at ≈5.5 (rank A–B) while the maintainability index declined only mildly (82→70, remaining maintainable). Dependency-vulnerability scanning (pip-audit10 ) was run against the OSV database11 and reported no known vulnerabilities across the resolved dependency closure (59 packages). The declared version lower bounds of Streamlit and scikit-learn admit older advisories, but the latest versions cause no warnings. Overall, quality was controlled, not sacrificed, under rapid agentdriven code and feature growth. These additional checks also corroborate our internal oracle-based validation with independent, reproducible evidence.
6
Discussion: Lessons for AI-Assisted PM/EM Tooling
Here are important lessons learned while using Claude Code for developing and then analyzing PM4Py-UCM, which will hopefully shed some light on agentic AIassisted PM/EM tool building ( C4 ). 10 11
https://github.com/pypa/pip-audit A distributed vulnerability database for Open Source: https://osv.dev/
Building a Process-Modeling Tool using AI
13
1. Building a PM/EM tool with an agent is a refinement loop , not one-shot generation. Despite the author’s three decades of experience in requirements modeling (including the standardization of URN and the development of jUCMNav) and one decade in process mining, fixes outnumbered features 2.3:1, and the ≈28:1 autonomy ratio is accompanied by an 18% correction rate: the human’s dominant activity is steering and repairing. Developers should budget for iteration and for a human-in-the-loop who can recognize defects, many of which could be subtle. 2. Agents are weakest where PM/EM tools are most demanding. Corrections concentrate on (executable) scenarios/conditions and visualization, and the highest fix rates fall on visualization/layout (78%) and jUCMNav/XMI export (83%). LLMs produce plausible structure but struggle with exact conformance to a standardized metamodel, a reference editor’s format (e.g., jUCMNav’s), and a renderer’s expectations (e.g., PNG resolution quality). The response is to make these boundaries mechanically checkable with roundtrip determinism, golden-file oracles (e.g., valid and representative .jucm files in XMI), visual-diff oracles, and metamodel conformance checks. 3. Model transformations fail silently, so verification must be executable. A logically wrong condition in a UCM model raises no exception; it yields a model that opens cleanly in jUCMNav yet behaves incorrectly. The risk concentrates in the discovery transformation chain and in keeping the Python metamodel faithful to Z.151’s as the scenario layer is added. That user_ incorrect stays near 1% while corrects_agent is 18% shows the human-inthe-loop was the last line of defense. Pairing every feature with an executable oracle [10] (importing generated models back into jUCMNav, replaying scenarios, checking invariants) turns silent semantic errors into visible failures. The test suite growing with features (108→691 functions) is the mechanism that kept an agent-written tool trustworthy. 4. Feature velocity has a consistency tax, and it lands on documentation. The cross_cutting flag (9.8%, concentrated on docs/READMEs) quantifies a familiar software-maintenance pain, amplified when features ship every few days. Agents do this propagation well when asked but rarely initiate it; a “definition of done” enumerating auxiliary artifacts, handed to the agent as a checklist, would recover much of this cost. 5. Externalised memory and parallel worktrees are practical enablers. Multiweek agent development spanned resumed Claude Code sessions (limited to 1M tokens) and parallel git worktrees, stitched by explicit memory/hand-off files; this is a reusable practice pattern that scaled. In this experience report, an under-resourced PM tool reached 20 releases and a five-view analytics (including many new features) application in ten weeks, built largely by an agent. AI agents can lower the barrier to bespoke PM/EM tooling, but only with the verification scaffolding above and a domain expert in the loop. The 18% of turns that corrected the agent were expert corrections about URN and jUCMNav semantics. AI amplified throughput, but did not necessarily replace modeling expertise. Hence, for AI-assisted EM tooling we recommend:
14
D. Amyot
Recommendations 1) Make transformation boundaries mechanically checkable (round-trips, oracles); 2) Pair every feature with an executable oracle; 3) Give the agent a definition-of-done that enumerates cross-cutting artifacts; 4) Involve a domain expert in the loop as the verifier of record.
7
Threats to Validity
Here are important threats categorized along three main validity types. Construct. Labels are interpretive: i) corrects_agent may fold in underspecified prompt, and we did not separate genuine agent error from ambiguous instructions, so 18% is an upper bound on the former; ii) user_incorrect is a lower bound as it requires an explicit in-transcript retraction (and some selfcaught mistakes were actually removed from the conversations by changing the erroneous prompt itself). Moreover, iii) the gold labels are LLM-proposed and expert-verified (through a sample) rather than independently human-coded. Internal. Transcripts capture the chat channel only; out-of-chat reasoning, manual edits, and local testing are not fully captured, so agent effort is an upper bound and human effort an underestimate. The developer and analyst are the same person, and an AI assisted both building and analyzing the PM tool; we mitigate the labeling risk with a published taxonomy, gold labels, a reproducible pipeline, and an independent keyword baseline, but a second human coder remains future work. External. With a single tool, developer (domain expert), and agent family, one domain (PM), and ten weeks, this is a case study, not a controlled experiment. Development also spanned three model generations (Opus 4.7/4.8, Fable 5), so this mix reflects a moving agent rather than a fixed one. What we expect to transfer are the method, the qualitative patterns, and the analysis tool. Whether this generalizes to other EM tools (more complex or on a domain different from PM) and to developers with less domain expertise is for future work.
8
Related Work
There are several related areas worth discussing here. AI-assisted software engineering. A growing literature studies LLM code assistants and their effect on productivity and defects [11, 16, 29]. These works measure task completion or suggestion acceptance on general-purpose code. Our analysis suggests a complementary dimension: silent semantic corruption in model transformations. Bhati [7] also highlights five open problems for agentic development (evaluation, governance, technical debt, skill redistribution, and the economics of attention), but our paper only partially addresses the first one. Building PM/EM tools. LLMs are increasingly used to produce modeling content (e.g., UML [9] or BPMN [15]) or to generate code [12]. Instead, we use an agent to build the modeling tool and measure that construction, which is an
Building a Process-Modeling Tool using AI
15
under-studied topic. PM4Py-UCM itself extends a long line of PM methods [6,20], but differs from metamodeling platforms [13,14,28,30] enabling the specialization of languages and analyses by design. Method and verification. We mine the development record itself (e.g., agent transcripts), as in studies of LLM-generated code in repositories [19], but for a human-agent pair rather than a human team. We recast model-transformation testing (oracles, metamorphic relations, round-trip determinism [10]) as the scaffolding that makes agent-written tooling dependable.
9
Conclusion
The answer to our research question from this case is a qualified yes: trust in PM4Py-UCM came not from the agent but from the scaffolding around it. Making the transformation boundaries mechanically checkable (golden-file oracles, metamorphic and deterministic round-trips against a reference metamodel and editor) and keeping a domain expert as the record verifier are what turned silent semantic corruption into visible failures. The ≈18% of turns spent correcting the agent (with expert corrections about URN and jUCMNav semantics) show why both were necessary. Development was fix-dominated (2.3:1 ratio over features) and its consistency tax landed on documentation, yet quality kept pace: tests grew in lockstep with features and independent analyzers found the resulting code well covered, low in complexity, and free of medium/high-severity issues. For the EM community entering the era of AI, the message is practical: agents can sharply lower the cost of custom modeling tooling, but only when the tool’s outputs are made verifiable at the metamodel, exporter, and renderer boundaries, with a domain expert in the loop (the last point being echoed in [7]. Agentic AI amplified throughput, but it did not replace modeling expertise. The paper is accompanied by a privacy-preserving replication package [3] (including code and derived results) so others can reproduce our results and re-run the analysis on their own agent-assisted projects. Future work follows two tracks. For PM4Py-UCM, we consider the LLM-assisted interpretation and recommendations, and inference of goal models (GRL) traceable to the mined UCM. On the method side, we plan replication on other tools and developers/teams, with automated support for multiple repositories, and the proper generation of event logs that PM tools (e.g., PM4Py-UCM) can mine. Although this paper represents only one data point, we hope that replication by others in different contexts will enable generalizability through meta-analysis. Acknowledgments. This work is supported by the NSERC Discovery grant Requirements-Oriented Process Mining. GenAI (Claude Code with the Fable 5 and Opus 4.7/4.8 models) was used in three roles: i) as the main implementer of PM4Py-UCM, the subject of this report; ii) to mine and analyze the development record and to produce draft figures/tables; and iii) to assist in drafting and revising some of the accompanying text. The author reviewed/verified the AIgenerated content (computations, results, figures, and text) for correctness.
16
D. Amyot
References 1. van der Aalst, W.: Process Mining: Data Science in Action, pp. 3–23. Springer (2016). https://doi.org/10.1007/978-3-662-49851-4_1 2. Amyot, D.: On process mining executable use case maps: Concurrency-aware scenario synthesis with variant- and data-driven conditions. In: 18th System Analysis and Modelling Conference (SAM ’26). ACM (2026) 3. Amyot, D.: PM4Py-UCM reflexive development-mining and quality toolkit (replication package). Zenodo (2026). https://doi.org/10.5281/zenodo.21611544 4. Amyot, D.: Towards process mining use case map models with PM4Py-UCM. In: IEEE 34th Int. Requirements Engineering Conference Workshops (REW). IEEE CS (2026). https://doi.org/10.48550/arXiv.2606.04350 5. Amyot, D., Akhigbe, O., Baslyman, M., Ghanavati, S., Ghasemi, M., Hassine, J., Lessard, L., Mussbacher, G., Shen, K., Yu, E.: Combining goal modelling with business process modelling: Two decades of experience with the user requirements notation standard. Enterprise Modelling and Information Systems Architectures (EMISAJ) 17, 2:1–38 (2022). https://doi.org/10.18417/emisa.17.2 6. Berti, A., van Zelst, S., Schuster, D.: PM4Py: A process mining library for Python. Software Impacts 17 (2023). https://doi.org/10.1016/j.simpa.2023.100556 7. Bhati, H.: Agentic AI in the software development lifecycle: Architecture, empirical evidence, and the reshaping of software engineering (2026), https://arxiv.org/ abs/2604.26275 8. Buhr, R.J.A.: Use case maps as architectural entities for complex systems. IEEE Transactions on Software Engineering 24(12), 1131–1155 (1998). https://doi. org/10.1109/32.738343 9. Cámara, J., Troya, J., Burgueño, L., Vallecillo, A.: On the assessment of generative AI in modeling tasks: An experience report with ChatGPT and UML. Software & Systems Modeling 22(3), 781–793 (2023). https://doi.org/10.1007/ s10270-023-01105-5 10. Chen, T.Y., Kuo, F.C., Liu, H., Poon, P.L., Towey, D., Tse, T.H., Zhou, Z.Q.: Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys 51(1), 4:1–4:27 (2018). https://doi.org/10.1145/3143561 11. Cui, K.Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., Salz, T.: The effects of generative AI on high-skilled work: Evidence from three field experiments with software developers. Management Science (2026). https://doi.org/10.1287/mnsc.2025. 00535 12. Dong, Y., Jiang, X., Qian, J., Wang, T., Zhang, K., Jin, Z., Li, G.: A survey on code generation with LLM-based agents (2025), https://arxiv.org/abs/2508.00083 13. Fill, H.G., Karagiannis, D.: On the conceptualisation of modelling methods using the ADOxx meta modelling platform. Enterprise Modelling and Information Systems Architectures (EMISAJ) 8(1), 4–25 (2013). https://doi.org/10.18417/ emisa.8.1.1 14. Frank, U.: Multi-perspective enterprise modeling: Foundational concepts, prospects and future research challenges. Software & Systems Modeling 13(3), 941–962 (2014). https://doi.org/10.1007/s10270-012-0273-9 15. Hörner, L.F., Möller, M., Reichert, M.: Automatically generating BPMN 2.0 process models from natural language process descriptions: Challenges, framework, quality assessment. Business & Information Systems Engineering 68, 59–83 (2026). https://doi.org/10.1007/s12599-025-00983-x
Building a Process-Modeling Tool using AI
17
16. Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., Wang, H.: Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33(8) (2024). https://doi.org/10.1145/3695988 17. ISO: ISO/IEC 15909-1:2019 – Systems and software engineering – High-level Petri nets – Part 1: Concepts, definitions and graphical notation (2019), https://www. iso.org/obp/ui/#iso:std:iso-iec:15909:-1:en 18. ITU-T: Recommendation Z.151 (10/2018): User requirements notation (URN) — language definition. Tech. rep., International Telecommunication Union (2018), https://www.itu.int/rec/T-REC-Z.151-201810-I/en 19. Ji, Y., Wang, J., Zhou, Y., Chen, F., Yang, H.: An exploratory study on LLMgenerated code and comments in code repositories. Journal of Systems and Software 242, 113026 (2026). https://doi.org/10.1016/j.jss.2026.113026 20. Kesici, C.A., Ozkan, N., Taşkesenlioglu, S., Erdogan, T.G.: A systematic literature review of studies comparing process mining tools. International Journal of Information Technology and Computer Science 14(5), 1–14 (2022). https: //doi.org/10.5815/ijitcs.2022.05.01 21. Khorasani, M., Abdou, M., Fernández Hernández, J.: Streamlit for Web Development: Build and Scale Secure Python-Powered Apps with Streamlit. Apress Berkeley, USA (2025). https://doi.org/10.1007/979-8-8688-1826-4 22. Laue, R., Mayr, H.C., Thalheim, B.: 100 years of graphical business process modelling: Guest editorial. Enterprise Modelling and Information Systems Architectures (EMISAJ) 17, 3–1 (2022). https://doi.org/10.18417/emisa.17.3 23. Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Discovering block-structured process models from event logs - a constructive approach. In: Application and Theory of Petri Nets and Concurrency. LNCS, vol. 7927, pp. 311–329. Springer (2013). https://doi.org/10.1007/978-3-642-38697-8_17 24. Loyola-González, O.: Process mining: software comparison, trends, and challenges. International Journal of Data Science and Analytics 15(4), 407–420 (2023). https: //doi.org/10.1007/s41060-022-00379 25. OMG: Business process model and notation (BPMN), version 2.0.2 (2014), https: //www.omg.org/spec/BPMN/2.0.2 26. Pedregosa, F., Varoquaux, G., Gramfort, A., et al.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011), http: //scikit-learn.sourceforge.net 27. Roy, J.F., Kealey, J., Amyot, D.: Towards integrated tool support for the user requirements notation. In: System Analysis and Modeling (SAM 2006). LNCS, vol. 4320, pp. 198–215. Springer (2006). https://doi.org/10.1007/11951148_13 28. Sandkuhl, K., Stirna, J., Persson, A., Wißotzki, M.: Enterprise Modeling: Tackling Business Challenges with the 4EM Method. The Enterprise Engineering Series, Springer (2014). https://doi.org/10.1007/978-3-662-43725-4 29. Sergeyuk, A., Golubev, Y., Bryksin, T., Ahmed, I.: Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward. Information and Software Technology 178, 107610 (2025). https://doi.org/10.1016/j.infsof. 2024.107610 30. Vernadat, F.: Enterprise modelling: Research review and outlook. Computers in Industry 122, 103265 (2020). https://doi.org/10.1016/j.compind.2020.103265