A Methodology for Investigating AI Patterns Prevalence in Software Repositories Srinath Perera1 , Hasinthaka Piyumal1 , Frank Leymann2 , and Rania Khalaf1
arXiv:2607.00558v1 [cs.SE] 1 Jul 2026
WSO21 , University of Stuttgart2 Santa Clara, CA, USA1 , WSO2, Stuttgart, Germany2 e-mail: {srinath, hasinthaka, rania}@wso2.com 1 , [email protected] Abstract—As Artificial Intelligence(AI)-based applications take off, a clear understanding of AI patterns can uplift the quality of AI applications. Many AI patterns have been proposed in the literature; however, their prevalence in real-life code has not yet been validated. Understanding the actual use of those patterns in practice can clarify our understanding both of the significance of these patterns and their utility. In this paper, we present a methodology to a) identify relevant patterns by mining the literature and then to b) validate their presence and prevalence in actual code repositories using active learning. To that end, we identify 14 AI pattern classes by mining 44 published AI patternrelated sources. Then we use an active learning approach to determine the prevalence of the most common pattern class across 100 GitHub open AI repositories. Using prevalence estimation, we propose bounds on the accuracy of the occurrences. The model achieves 56% accuracy and 55% recall in an 8-way classification task, significantly outperforming the 11% random-chance baseline. Furthermore, the prevalence estimation offers usable bounds for analyzing pattern applications. This methodology provides a robust foundation to start understanding how AI patterns are used in practice, a field that currently lacks empirical data. Keywords-patterns;pattern analysis; software; software engineering
I. I NTRODUCTION Large Language Models (LLMs) [1] now provide generalpurpose, high-quality Artificial Intelligence (AI) capabilities that require little to no user-provided training data. These models have unlocked many previously infeasible use cases. Several patterns and abstractions including RetrievalAugmented Generation (RAG) [2], ReAct [3], and agent-based frameworks [4] now help developers build AI applications. Software Design patterns [5] document recurring problems and solution templates, thereby improving software quality by enabling, communicating, and educating best practices. As we will discuss in the related work section, many AI patterns have been proposed in the literature. Those patterns aim to capture the authors’ observations and experiences. Validating those patterns by understanding their usage frequency in practice can clarify their relative importance, thereby improving the quality of AI-based applications. Furthermore, estimating usage frequency will help us verify candidate patterns using the rule of three (rule of X) [6]. We have identified 769 AI design pattern candidates proposed in the literature. Using word embeddings of those pattern descriptions and clustering, we have grouped them into 78 refined pattern candidates. Then, by manual inspection, we have categorized them into 14 pattern classes. To verify the prevalence of pattern classes in practice, we selected 100 open-source GitHub repositories that implement
real-world AI applications. From those repositories, we have extracted 2442 code communities. (A code community is a group of tightly coupled methods (procedures) in the call graph.) Then, using an active learning-based approach [7], we built a classifier to detect the most common patterns listed above. Active learning builds a model with the help of an (e.g., human) oracle, aiming to minimize the number of data annotations (i.e., oracle queries). We proceed by starting with an initial model and iterative labeling of a few data-points that exhibit higher uncertainty in their predictions under the current model. Then, we rebuild the model with the new data and iterate. Consequently, we obtain an accuracy bound for how many times each of the above pattern classes will occur in the repositories. The paper makes the following contributions. • By combining methods active learning and human-in-theloop annotations, we propose a robust methodology to detect patterns without pre-existing annotated datasets, thereby addressing a significant bottleneck in software engineering research: labeled data for niche or emerging patterns is rare. • We propose a new chunking method to generate code embeddings useful for pattern detection by applying the Louvain method [8] to call graphs to identify "code communities" that serve as chunks. • We propose applying prevalence estimation techniques (matrix inversion and Monte Carlo simulations) to estimate the true frequency of patterns with useful error bounds. • We synthesized 769 pattern candidates into a refined taxonomy of 14 pattern classes. We provided one of the first empirical validations of AI pattern prevalence in real-life code by analyzing 100 open-source repositories. The remainder of the paper is organized as follows. The section II discusses related work, and the following Section III describes the proposed methodology. Section IV describes the implementation details, followed by Section V, which describes the results. The section VI discusses the findings and lessons learned, and Section VII concludes the paper. II. R ELATED WORKS This section discusses previous work on AI patterns and techniques for Mining Patterns from Code. A. AI Patterns Researchers and practitioners have proposed many AI patterns. Among post-LLM patterns, Huang [9] is a book on LLM design patterns. Gullí [10] and Liu et al. [11] discuss agent design patterns. Subramaniam [12] and Jain [13] provide
the practitioner’s perspective. The AWS web page [14] provides Among them, classification techniques such as those of a detailed description of best practices for implementing AI and Uchiyama et al. [32] and Dwivedi et al. [33] trained a classifier ML applications on AWS. Also, Databricks documentation [15] on software metrics such as Depth of Inheritance, Coupling, lists a collection of Agent system design patterns. Furthermore, the number of methods, etc. Chihada et al. [34] use features Arslan et al. [16] and Singh et al. [17] provide detailed surveys extracted from an association graph to train the classifier. of RAG patterns; articles [18] and [12] present practitioners’ Zanoni et al. [35] use a combination of graph matching and perspectives on the same pattern. ML techniques. The downsides of initial ML techniques include the need Among pre-LLM patterns, Lakshmanan et al. [19] is a textbook on design patterns from the ML domain. Nalchigar [20] for feature engineering, loss of information in features, and provides a detailed taxonomy of solution patterns for ML limited availability of training data. Najam et al. [36] build a representation of the source code, tasks such as clustering, classification, and anomaly detection. Washizaki [21] lists 15 design patterns (derived from a literature apply the word2vec algorithm to construct an embedding space, study) for building ML applications. Rodaz et al. [22] propose and then use a classification algorithm. Pandey et al [23] use a mathematical formalism that sketches tasks around ML, but code embeddings (RoBERTa) generated from code with a KNN it is not digestible by developers. The Azure website [23] lists algorithm to achieve an F1 score of 0.91. This approach is very flexible as it can be used with minimal effort to detect best practices for MLOps. You can find a comprehensive list of sources we used to mine new patterns if labels for the data are available. Most of the above approaches work with benchmarks patterns from [24]. These include the sources mentioned above, as well as additional practitioner content, such as articles, blogs, P-MART [37] and DPB [38], which primarily focus on software engineering patterns (e.g., GOF). No suitable dataset and documentation. is available for detecting AI patterns. Furthermore, a common limitation is that most benchmarks B. Mining Pattern from Code focus solely on GOF patterns. One notable exception is When detecting patterns in code, we need to first find a code Fernández et al. [39], which first builds quantum circuits and representation and a pattern representation, and then match then uses state-machine-based pattern-detection methods. them. Calculating code representations involves a trade-off Our work is motivated by the results of Pandey et al. [23], between losing information and maintaining generalizability. which demonstrated the viability of embedding-based apPattern representations closer to code are often suboptimal proaches. By retaining most of the information in code, as they fail to generalize across multiple languages. On the embeddings avoid the need for feature engineering. However, other hand, manually crafted pattern representations require embeddings are highly sensitive to the chunking method (i.e., significant effort and scale poorly to large numbers of patterns. which code is grouped together for calculating embeddings). We Many techniques have been proposed for finding design patterns extended the work of Pandey et al. [23], which uses P-MART from code. with clean code segments each including a single pattern, to Search-based approaches use representations closer to code. detect patterns in real-life code repositories. Also, our work Work such as Kramer et al. [25] and Dabain et al. [26] focused extends beyond well-known patterns (e.g., GOF) and limited on detecting patterns using rules that describe them. These annotated datasets by addressing the lack of training data via rules detect relationships between classes (e.g., inheritance an active learning approach. hierarchies and method invocations). To reduce the overhead III. M ETHODOLOGY of pattern specification, Ghulam et al. [27] break GOF [5] patterns into 44 reusable atomic patterns, detect each using Figure 1 depicts the methodology we used to estimate the SQL, Regular Expressions, or AST parsing, and combine them occurrence of AI patterns in real-world AI applications as a to discover the patterns. Zdun et al. [28] turn microservice flow chart. It has four steps. Each of the following subsections patterns into formal rules and provide a tool that automatically discusses four steps of the methodology in detail. evaluates code in terms of compliance with these patterns. A logical extension of this method is to represent patterns A. Step 1: Extracting pattern candidates from the literature as subgraphs and to search the code graph for them. MayFor mining AI patterns, we used the sources listed in the van [29] is an example of this approach. On the other hand, related work and practitioner articles, blogs, and documentation. Tsantalis [30] used similarity scores between the code graph You can find the detailed list from [24]. For each source, we and the pattern subgraph, enabling more flexible matching. extracted text, and for videos, we used the transcript. GEML [31] evolves a set of human-readable if-then-else rules We query an LLM (using Prompts [24]) to extract pattern using a greedy algorithm based on inexact graph matching to candidates. We receive resulting pattern candidates in the detect a pattern. template specified in the prompt. More recent work often focuses on ML-based statistical To detect similarity between pattern candidates, we genermethods, which can avoid challenges of specifying patterns ated word embeddings for pattern description and clustered using rules or templates. Instead of specifying the pattern, they embeddings using the DBSCAN algorithm. We explored cluslearn a pattern-identifier from sample data. tering algorithms DBSCAN, HDBSCAN, OPTICS, Bayesian
Start
Step 1 AI Pattern Sources
Extract Patterns with LLM
Step 2
AI Pattern Candidates Embeddings + Clustering + summarise with LLM
14 Pattern Classes
Categorize Manually
100 Gitrepos
Label Manually
Training data set
Candidates to label
Code communities
Refined AI Pattern Candidates
Step 4
Step 3 Generate sample code to bootstrap GEN_DATA
Generate call graph + detect communities
Train multiple models by 5-fold cross-validation.
Forecast pattern frequencies
Model
Select N data points to label based on uncertainty
Continue for T steps
Forecasted frequencies with error bounds
Forecasted frequencies
Simulations
End
Figure 1. Proposed Methodology
Gaussian Mixtures, and K-Means. We picked DBSCAN by evaluating the even distribution of cluster sizes via silhouette score. We summarize each resulting cluster using an LLM with the prompt [24] and use them as refined pattern candidates. Then, by manually inspecting pattern candidates, we categorize them under 14 pattern classes using the following criteria. • We started with pattern classes: RAG, "Advanced LLM Prompting," "Forecasting with Classical Models," and "MultiAgent Architecture," which were identified in related work. • We tried to identify a pattern class for each refined-patterncandidate that we identified by clustering. If we failed, we created a new pattern class. Following this approach, we added pattern classes "Evaluating LLM Results," "Multimodal Data Processing and Prompting," "Using Tools with LLMs," "LLM-based User Intent Extraction," "LLM Fine Tuning," "Training & Alignment," "Enabling Reliability, Explainability, or Robustness," "LLM-based Planning, XoT, ReAct, or Reasoning," and "MLOps" using this approach. • We also added 2 more pattern classes: "Preprocessing Text and Numerical Data" and "Model Abstraction," while manually labeling data in step four. The classification of patterns and their refined candidates can be found in [24].
possess between 250 and 1,000 stars. We manually verified each candidate to ensure it was a valid AI-related project. The complete list of selected repositories is provided in [24]. Manual inspection of 50 repositories with over 1,000 stars revealed that the majority were foundational frameworks, such as TensorFlow. Consequently, we targeted the 250–1,000 star range to intentionally isolate the ‘application layer’ of AI software, to understand how typical developers use AI in practice. Embeddings will only work if chunks include full patterns and are reasonably small. For each repository, we generated a procedure call graph and detected communities using the Louvain method [8]. Since all functions within a single community are closely coupled, code within each community is likely to be related. Operating under the hypothesis that such code communities contain complete patterns, we treated all code in each code community as a single chunk and generated code embeddings for each community using gemini-embedding-001. We call this dataset Dunlabeled . C. Step 3: Build a model with Active learning
We built a model to detect known code pattern classes by first creating a bootstrap version, selecting data for manual labeling via active learning criteria, and then iteratively rebuilding the model using five-fold cross-validation [40]. This crossB. Step 2: Collecting Real-world Code Communities validation approach allows us to fully utilize all labeled data A code community is a group of tightly coupled methods points. (procedures) in the call graph. To extract code communities, To bootstrap the model, we prompted the LLM to generate we curated a dataset of 100 repositories hosted on GitHub. multiple samples for pattern classes identified in step 1, which The selection process filtered for Python repositories that we will call Dinit .
The algorithm in Figure 1 explains the active learning based approach. At each step (algorithm 2, line 5), we dropped classes with fewer than 5% of the data and 20 data points. We do not know the true labels; hence, we cannot control the distribution of labeled data without significant effort. In the tradeoff between stable results vs. forecasting more classes, we chose stability. As steps progressed and more data became available, more classes will become available. Input : Dinit (Initial generated data), Dunlabeled (Unlabeled embeddings), T (Total iterations) Output : Mf inal (Trained pattern identifier model) Vorig ← ∅ 2 V ← Dinit 3 Mlist ← None
11 12
for i ← 1 to T do V ← handle_rare_classes(V )
6
// Step 1: Train model using current pool Mlist ← train_hybrid_model_cv(V)
7
// Step 2: Active Learning - Identify uncertain samples Shl ← find_uncertain_topN(Mlist , Dunlabeled )
8 9 10 11 12
2
10
4
5
N ← {"lr", "knn", "svc"} Cf olds ← ∅ // fold configs 3 F ← ∅ 4 Of olds ← ∅ 5 for f ← split_to_folds(V) do 6 Dtrain ← f.train 7 Dval ← f.validation 8 Dtest ← f.test 1
9
// Initialize original verified pool // Initialize current working pool
1
// Pseudocode for train_hybrid_model_cv(..) Input: Verified Data V Output: Models M , F1 scores of models F
13
// use algorithm in Figure 4 to forecast Of olds ← Of olds ∪ forecast(Mlist , Dtest ) Cf olds ← get_configs(Mlist ) F ← F ∪ {F1 } print_results(Of olds , Dtest )
// Aggregate results and train final ensemble 14 Mlist , _ ← train(N , V, ∅, Cf olds ) 15
// Step 3: Human-in-the-loop labeling Dnew ← label_manually(Shl ) // Step 4: Update pools Vorig ← Vorig ∪ Dnew Dunlabeled ← Dunlabeled − Shl V ← Vorig
return {Mlist , F} Algorithm 2: Building a model with Cross-validation // Pseudocode for forecast(...) Input: x Input data set, Models M , F1 scores of models F Output: Y The final predicted class labels
/* Definitions: */ C: The set of all possible class labels th 2 fi ∈ F : The F1 score used as the raw weight for the i model 3 P (y = c | mi , x): The probability that model mi assigns to class c for input x 1
return Mlist Algorithm 1: Active Learning based Pattern Identifier
In find_uncertain_topN()function, we select data points to label from Dunlabeled by forecasting them using current models, normalizing class probabilities of different models, averaging them to create overall predicted class probabilities, and performing margin sampling. Margin is the difference between the highest and second-highest classes in the forecast, and we select data points to label based on the smallest margin. Algorithm in Figure 2 presents the logic for building a crossvalidated model. And algorithm 3 describes how to perform the final forecast as a weighted average of probability distributions, with weights based on F1 scores across folds. D. Step 4: Pattern Occurrence Estimation based on the results Let m denote the model trained in step 3 to detect pattern classes in a code community. We utilize the normalized confusion matrix derived from the model’s cross-validation, denoted as C. Given the set of code communities D (constructed in step 2), we define OD as the observed distribution of pattern classes in D as predicted by m. Assume TD is the estimated true distribution of class in D. Osamp = C · TD =⇒ TD = C−1 Osamp
Mlist , F1 ← train(N , Dtrain , Dval , ∅)
(1)
Given OD , we can find TD by using a linear solver. When estimating TD using our model, we face two sources of uncertainty: model misclassifications and sampling errors from OD . We minimize the first error using equation 1, and
4
// Ensemble Forecast Calculation P | fi Pn Y = argmax |M · P (y = c | mi , x) i=1 fj c∈C
5
j=1
return Y Algorithm 3: Final Forecast
the second by sampling from a multinomial distribution to estimate the true frequencies, then performing a Monte Carlo simulation to obtain error bounds. Algorithm 4 depicts our approach. IV. I MPLEMENTATION D ETAILS At every step, we use Gemini 3 Flash as the LLM. All experiments are done in an Intel Core™ i7-10510U × 8 processor with 16.0 GiB running Ubuntu 24.04.3 LTS We already discussed the choice of clustering algorithm used in step 1 in the methodology. Furthermore, we tried dimension reduction with PCA and UMAP. We received the best results with DBSCAN with UMAP. In step 3, for code embedding, we experiment with embedding methods Jina Embeddings v2 (Code), CodeSage Large v2, CodeRankEmbed, RoBERTa, CodeBERT, Voyage Code2, Gemini Text Embedding 004, and selected Gemini Text Embedding 00,4, which provided the best classification F1score while building a bootstrapping model in active learning.
Input: C (Confusion Matrix), O (Observed counts), α (significance level), N (Iterations) Output: [L, U ] (Confidence interval bounds) R←∅ 2 Cnorm ← normalize(C) P 3 n ← O 4 p ← O/n 1
// Initialize results collection // Normalize columns to sum to 1 // Total sample count // Observed frequencies
for i ← 1 to N do // Sample from multinomial to simulate observation noise 6 Osamp ← sample_multinomial(n, p)
5
7
// Solve linear system using Equation 1 E ← solve(Cnorm , Osamp )
8
R ← R ∪ {clip(E)}
// Store clipped estimates
L ← percentile(R, α/2) U ← percentile(R, 100 − α/2) 11 return [L, U ] Algorithm 4: Inversion-based Estimation via Monte Carlo Simulation 9
10
V. R ESULTS Manually verified data had pattern frequencies as follows: Forecasting with Classical Models (64), None (64), LLM-based Multimodal Generative Prompting (52), Preprocessing Text and Numerical Data (51), Using Tools with LLMs (31), RAG (28), Agent Architecture (20), Model Abstraction(20), Evaluating LLM Results (18), MLOps (18), Advanced LLM Prompting (14), LLM-based Planning, XoT, ReAct, or Reasoning (12), Enabling Reliability, Explainability, or Robustness (8), LLMbased User Intent Extraction (8), and LLM Fine-Tuning, Training & Alignment (4). However, the rest of the results will only have 7 pattern classes because the model will automatically map any class less than 5% and 20 instance to the None class. Table I shows the results after the fourth iteration. The verified data column shows the number of manually labeled data points available. As expected, classes with fewer training samples have lower F1 scores. Table 2 shows the confusion matrix.
C8
C7
C6
True Class C5 C4
C3
C2
C1
Unlike step 1, in step 3, using dimensionality reduction with the classifier model reduced the F1-score; Hence, we did not Confusion Matrix Heatmap use any dimensionality reduction. 50 6 1 0 4 2 1 4 2 While building the model for step 3, we tried Logistic Regression (LR), Neural Networks, Support Vector Classifier 0 50 6 0 2 0 1 4 40 (SVC), Naive Bayes, Random Forest, XGBoost, Gradient 0 6 38 0 3 0 0 5 Boosting, K-Nearest Neighbour (KNN), and Decision Tree. We 30 used LR, SVC, and KNN, which provided the best individual 2 1 0 10 1 1 3 2 F1 scores in the final averaging ensemble. 0 4 8 0 25 3 1 9 20 While building the model in step 3, we achieved about a 4% 0 0 0 1 4 18 2 3 improvement in F1 score using the averaging ensemble method in code listing 3. Building a linear regression stacked model 10 2 0 1 5 1 0 11 11 using outputs from a neural network and a linear regression 2 10 8 2 9 4 4 46 model also yielded good results. Still, we chose the averaging 0 C1 C2 C3 C4 C5 C6 C7 C8 ensemble method because the stacked model is complex and Predicted Class highly susceptible to overfitting. Figure 2. Weighted Vote Confusion Matrix In step 3, we used the following workflow for manually labeling code communities. Given a code community (cc) and a predicted pattern class pc, we use the LLM to generate a code A strong main diagonal indicates the model’s strength. As description for cc, ask the LLM to judge whether the forecasted we discussed in the methodology, most classes are often pattern class pc is correct, and describe its decision. We then misclassified as "None", and the "None" class often gets manually verified that the LLM’s judgment was correct. If the misclassified. Hence, it is a common source of error. Conprediction (pc) is wrong, we read the description of cc and sidering the often misclassified patterns, "Using tools with propose an alternative pattern class. [24] lists Prompts used LLM" is often misclassified as "None", which could be because for this step. "tool use" is not very prominent in code and lacks a strong During active learning, in the third iteration, we tried building signal. Furthermore, classical models and preprocessing both the model both with and without LLM-generated sample data, get confused by "None" in both directions, which may be and it performed better with only verified data. So we only because these patterns include a wide variety of techniques used verified data from the third iteration onward. that are harder to generalize with a small set of samples. Figure 3 depicts our results from the prevalence estimation In step 4, for the simulation, we assume the confusion matrix of the model developed in step 3 is representative. described in step 4. As discussed in the methodology, we model the sampling There are several zeros in bounds for the last two classes, error as independent categorical draws from a multinomial which could occur when a class is affected by high falsedistribution, with the observed category counts as the basis positive noise from other classes or when it is often misclassifor the distribution. We used 20000 as the iteration count, fied as other classes. We conducted a sensitivity test for both increasing it until repeated tests did not change the outputs. cases and found that the probable cause is that "Tool Use
TABLE I. C LASSIFICATION PERFORMANCE METRICS ACROSS CATEGORIES .
Category C1 C2 C3 C4 C5 C6 C7 C8
Precision
Recall
F1-Score
Verified Data
Agent Architecture Forecasting with Classical Models Multimodal Generative Prompting Model Abstraction Preprocessing Text and Numerical Data RAG Using Tools with LLMs None
0.50 0.69 0.62 0.45 0.53 0.67 0.42 0.56
0.30 0.79 0.73 0.50 0.50 0.64 0.35 0.54
0.38 0.74 0.67 0.48 0.52 0.65 0.39 0.55
20 63 52 20 50 28 31 85
Overall Values
0.56
0.55
0.55
Confidence Intervals (P5 to P95) for Class Forecasts C3-Multimodal C2-Classical Pattern Class
through group chat. LLM-based Multimodal Generative Prompting - Predicting object centers, dimensions, and local offsets, inferring an object’s 3D position, bounding box prediction, motion prediction, sequential detections of an object from Lidar data, resizing and padding 3D tensors, generating synthetic training data. • Model Abstraction - abstracting multiple AI backends or local/ cloud producers, routing, adopting Prompts via template, and managing API keys • Preprocessing Text and Numerical Data - anonymizes, binning, type conversions, data dictionary, extracting metadata, text normalization, categorical feature encoding • Retrieval Augmented Generation(RAG) - search with vector database, using custom knowledge bases, caching, and retriever component •
C1-Agent C5-Prepro-TN C6-RAG C4-Abstraction C7-Tools 0
50
100
150 200 Occurrence
250
300
Figure 3. Estimated Pattern Class Prevalence with 95% confidence intervals
for LLMs" is often misclassified as "None" as shown in the confusion matrix in Figure 2. Smaller training samples likely cause wider confidence intervals for the last two classes. When we exclude those, the confidence interval variation is 45%, which provides usable estimates for our problem. Furthermore, confidence levels indicate when estimates are unreliable, and users can address that by running more iterations of the algorithm. Since the algorithm selects labeling data based on uncertainty, there is a good chance that underrepresented classes will be labeled in subsequent iterations. Where are the prevalence results at the current accuracy level useful? First, it can confirm the wider availability of 5/7 pattern classes. Furthermore, it can give us relative availability of different pattern classes. For example, even using given confidence levels, we can argue that pattern classes C2 and C3 are much more common than C1, C5, and C6. Such understanding can help us focus our attention on teaching these patterns. However, the current accuracy level is not enough to identify a specific pattern given a code segment, which is a future direction we plan to explore. The following lists concrete examples organized by pattern class, found based on the model’s forecast for extracted communities. • Agent Architecture - multi-agent debate, multi-agent orchestration, communication through an agent bus, coordination
VI. D ISCUSSION AND L ESSONS L EARNED One code community can house multiple patterns. However, to keep manual data labeling simple, we have only used a single label. However, our output provides a probability distribution across all classes, which can be used to detect multiple patterns. Contrasting our results with those of Pandey et al. [23], which achieved an F1-score of 0.91, it is worth noting that we also tried the same algorithms and embeddings. The likely difference is the curated nature of PMART vs. the complexity of real-life code we used, which often includes additional logic. Furthermore, as we discussed in the results section, users of our methodology can assess the reliability of estimates using error bounds. If they require tighter error bounds, they can likely achieve them by running more iterations of the step three active learning algorithm, which likely yields more training data for those classes and thereby improves the error bounds. Given the limited understanding of design pattern frequency in the real world, even the bounds we reported after four iterations remain usable for 5 out of the 7 classes and represent a significant step forward. In step 3, we folded classes that had fewer than 5% or 20 samples into the None class, which is a practical choice. Some patterns are common, while others are rare, and naturally, there are many potential patterns. To make our approach stable, we had to draw the line somewhere. When the number of
samples in a class is small, when combined with 5-fold crossVII. C ONCLUSION AND F UTURE W ORK validation, the F1-score changed significantly (>10%) due to In this paper, we aim to enable and shift the exploration each misclassified sample, thus becoming unreliable [41]. of AI design patterns toward empirical validation using However, we acknowledge that this practical choice reduces software repositories. To that end, we address three challenges. the granularity of results, as artificial non-class can interfere Embedding-based pattern detection is highly susceptible to with other classes. Because the boundary between what is chunking methods. We proposed a new chunking method based considered a pattern and what is not is fluid (e.g., subjective on call graph-based community detection. The performance in how abstract we want the problem and solution to be), we of the resulting model validates our method. Second, AI has believe our choice is still useful. If needed, practitioners can limited annotated pattern datasets, which we address through often get more granularity by labeling more data points. an active learning-based approach. Third, we address both We extend our work beyond basic software patterns (e.g., sampling and classification errors while estimating pattern GOF) and limited annotated datasets by addressing the lack of prevalence by using a matrix-inversion-based technique and training data via an active learning approach. The methodology Monte Carlo simulations. The model achieves 56% accuracy and 55% recall in an we used is highly independent of AI patterns, and the same approach is likely to work well with other types of patterns. The 8-way classification task, which is 5 times higher than the 11% ability to detect patterns by building a classifier on embeddings random-chance baseline. Prevalence estimation provided useful has been demonstrated by Pandey et al. [23] using the P-MART error bounds for 5 of 7 classes. In other cases, users can detect data set, and by this paper on real-world AI projects. It is likely unreliable estimates through error bounds and discard them, that other domains also contain sufficient information in their or obtain better estimates by running more iterations in step three. embeddings. We synthesized 769 pattern candidates into a refined taxonDetecting patterns via embeddings is highly sensitive to the omy of 14 pattern classes. Most patterns show more than three choice of chunking method. Initially, we generated embeddings examples required to pass the rule of three. Frequency estimates per file in the repository, resulting in poor classification for those classes confirm our understanding in some cases performance in step 3. We needed a chunking method that (e.g., Preprocessing Text and Numerical Data and Classical would keep all relevant code about a pattern in a single chunk. Models) and open new avenues of inquiry in others (Multimodal Given that goal, typical chunking approaches like fixed size, prompting and RAG). pooling, by function, or by class would not work. AST-like This work opens several avenues for future research. First, methods (e.g., CAS [[66]) also focus on code syntax structure applying the techniques in other domains to explore the not the dependacies. In contrast, a call graph naturally captures universality of the approach (e.g., applying the same techniques dependencies similar to code graph-based approaches used in for microservices, quantum computing), second, studying the related work(e.g., Mayvan [29], Tsantalis [30] ). It is much relationships between possible bounds vs. the number of more likely that code from the same pattern, which is more training samples for each class, and third, exploring techniques, closely related, is more connected in the call graph. This is such as SLM fine-tuning and constructive learning, to improve also a natural extension of Codegrag [42], which has used model performance. All information required to recreate the a similar idea for code retrieval. The performance improved above work, including code, labeled data points from step 4, significantly after adopting the call graph-based communities and code, is available in [24]. as chunks. We tried to detect patterns by clustering the code communiR EFERENCES ties collected in step 2 and then interpreting and summarizing [1] Y. Chang et al., “A survey on evaluation of large language models”, ACM Transactions on Intelligent Systems and Technology, the resulting clusters. This approach did not work well. vol. 15, no. 3, pp. 1–45, 2024. DOI: 10.1145/3641289 However, it may be possible to fine-tune the embeddings using [2] P. Lewis et al., “Retrieval-augmented generation for knowledgetechniques such as contrastive learning improve the separation intensive NLP tasks”, in Advances in Neural Information between clusters, which is also a future work we plan to Processing Systems (NeurIPS), vol. 33, 2020, pp. 9459–9474. explore. [3] S. Yao et al., “ReAct: Synergizing reasoning and acting in language models”, arXiv preprint arXiv:2210.03629, 2023. Methodology uses LLM in three ways, and each can Accessed: Jan. 9, 2026. [Online]. Available: https://arxiv.org/ introduce bias. First, pattern candidates were extracted using abs/2210.03629 the LLM and manually curated to identify pattern classes. [4] J. He, C. Treude, and D. Lo, “LLM-based multi-agent systems Manual curation should help reduce bias, but will not address for software engineering: Literature review, vision, and the road ahead”, ACM Transactions on Software Engineering and bias shared between LLMs and humans. Second, although we Methodology, vol. 34, no. 5, pp. 1–30, 2025. DOI: 10.1145/ bootstrapped with LLM-generated data in the first round, after 3702989 3rd round, we dropped LLM-generated data because only using [5] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design labeled data gave better results. Third, LLM-assisted manual Patterns: Elements of Reusable Object-Oriented Software. labeling can also introduce bias. Exploring the use of LLM as Addison-Wesley Professional, 1995. a judge technique to reduce such bias is a useful direction for future research.
[6] [7] [8]
[9] [10]
[11] [12] [13]
[14]
[15] [16] [17]
[18] [19] [20] [21] [22] [23] [24] [25]
[26]
C. Kohls and S. Panke, “Is that true...? thoughts on the epistemology of patterns”, in Proceedings of the 16th Conference on Pattern Languages of Programs, 2009, pp. 1–14. A. Tharwat and W. Schenck, “A survey on active learning: State-of-the-art, practical challenges and research directions”, Mathematics, vol. 11, no. 4, p. 820, 2023. P. De Meo, E. Ferrara, G. Fiumara, and A. Provetti, “Generalized Louvain method for community detection in large networks”, in 2011 11th International Conference on Intelligent Systems Design and Applications (ISDA), IEEE, 2011, pp. 88–93. DOI: 10.1109/ISDA.2011.6121634 K. Huang, LLM Design Patterns: A Practical Guide to Building Robust and Efficient AI Systems. O’Reilly Media, 2025. A. Singh, A. Ehtesham, S. Kumar, and T. T. Khoei, “Agentic retrieval-augmented generation: A survey on agentic rag”, arXiv preprint arXiv:2501.09136, 2025. [Online]. Available: https: //arxiv.org/abs/2501.09136 A. Gullí, Agentic Design Patterns. Packt Publishing, 2024. B. Subramaniam, “Emerging patterns in building GenAI products”, 2024, Accessed: Jan. 22, 2026. [Online]. Available: https://martinfowler.com/articles/gen-ai-patterns/ A. Jain, “Agentic AI architectures and design patterns”, 2024, Accessed: Jan. 22, 2026. [Online]. Available: https://medium. com/@anil.jain.baba/agentic- ai- architectures- and- designpatterns-288ac589179a Amazon Web Services, “AWS prescriptive guidance: Patterns: AI & machine learning”, 2026, Accessed: Jan. 22, 2026. [Online]. Available: https://docs.aws.amazon.com/prescriptiveguidance/latest/patterns/machinelearning-pattern-list.html Databricks, “Agent system design patterns”, 2026, Accessed: Jan. 22, 2026. [Online]. Available: https://docs.databricks.com/ aws/en/generative-ai/guide/agent-system-design-patterns M. Arslan, H. Ghanem, S. Munawar, and C. Cruz, “A survey on RAG with LLMs”, Procedia Computer Science, vol. 246, pp. 3781–3790, 2024. DOI: 10.1016/j.procs.2024.11.123 J. Alammar, “Retrieval-augmented generation (RAG) patterns and best practices”, InfoQ, 2024, Accessed: Jan. 22, 2026. [Online]. Available: https : / / www. youtube . com / watch ? v = eUY9i1CWmUg Neo4j, “GraphRAG field guide: RAG patterns”, 2026, Accessed: Jan. 22, 2026. [Online]. Available: https://neo4j.com/blog/ developer/graphrag-field-guide-rag-patterns/ V. Lakshmanan, S. Robinson, and M. Munn, Machine Learning Design Patterns. O’Reilly Media, Inc., 2020. Accessed: Jan. 22, 2026. N. Soroosh et al., “Solution patterns for machine learning”, in International Conference on Advanced Information Systems Engineering (CAiSE), Springer, 2019, pp. 43–58. H. Washizaki et al., “Software-engineering design patterns for machine learning applications”, Computer, vol. 55, no. 3, pp. 30–39, 2022. DOI: 10.1109/MC.2021.3139049 R. Benjamin et al., “A pattern language for machine learning tasks”, arXiv preprint arXiv:2407.02424v2, 2025. [Online]. Available: https://arxiv.org/abs/2407.02424 S. K. Pandey et al., “Design pattern recognition: A study of large language models”, Empirical Software Engineering, vol. 30, no. 3, p. 69, 2025. “Ai patterns github repository”, 2026, Accessed: Jan. 27, 2026. [Online]. Available: https://github.com/wso2- incubator/aipatterns C. Kramer and L. Prechelt, “Design recovery by automated search for structural design patterns in object-oriented software”, in Proceedings of WCRE’96: 3rd Working Conference on Reverse Engineering, IEEE, 1996, pp. 208–215. DOI: 10. 1109/WCRE.1996.558906 H. Dabain, A. Manzer, and V. Tzerpos, “Design pattern detection using FINDER”, in Proceedings of the 30th Annual
[27]
[28]
[29] [30]
[31]
[32]
[33] [34]
[35]
[36] [37] [38]
[39]
[40] [41] [42]
ACM Symposium on Applied Computing, 2015, pp. 1554–1560. DOI : 10.1145/2695664.2695755 G. Rasool and P. Mäder, “Flexible design pattern detection based on feature types”, Automated Software Engineering, vol. 18, no. 3-4, pp. 339–365, 2011. DOI: 10.1007/s10515-0110084-2 U. Zdun, E. Navarro, and F. Leymann, “Ensuring and assessing architecture conformance to microservice decomposition patterns”, in International Conference on Service-Oriented Computing, Springer, 2017, pp. 411–429. B. B. Mayvan and A. Rasoolzadegan, “Design pattern detection based on the graph theory”, Knowledge-Based Systems, vol. 120, pp. 211–225, 2017. N. Tsantalis, A. Chatzigeorgiou, G. Stephanides, and S. T. Halkidis, “Design pattern detection using similarity scoring”, IEEE Transactions on Software Engineering, vol. 32, no. 11, pp. 896–909, 2006. DOI: 10.1109/TSE.2006.112 R. Barbudo, A. Ramírez, F. Servant, and J. R. Romero, “Geml: A grammar-based evolutionary machine learning approach for design-pattern detection”, Journal of Systems and Software, vol. 175, pp. 110–919, 2021. S. Uchiyama, H. Washizaki, and Y. Fukazawa, “Design pattern detection using software metrics and machine learning”, in First International Workshop on Model-Driven Software Migration (MDSM 2011), 2011, pp. 38–42. A. K. Dwivedi, A. Tirkey, and S. K. Rath, “Software design pattern mining using classification-based techniques”, Frontiers of Computer Science, vol. 12, no. 5, pp. 908–922, 2018. A. Chihada, V. Arnaoudova, L. M. Eshkevari, G. Antoniol, and Y.-G. Gueheneuc, “Source code and design conformance, design pattern detection from source code by classification approach”, Applied Soft Computing, vol. 26, pp. 357–367, 2015. DOI: 10.1016/j.asoc.2014.09.043 M. Zanoni, F. A. Fontana, and F. Stella, “On applying machine learning techniques for design pattern detection”, Journal of Systems and Software, vol. 103, pp. 102–117, 2015. DOI: 10. 1016/j.jss.2015.01.037 N. Nazar, A. Aleti, and Y. Zheng, “Feature-based software design pattern detection”, Journal of Systems and Software, vol. 185, pp. 111–179, 2022. Y.-G. Guéhéneuc, “P-mart: Pattern-like micro architecture repository”, Proceedings of the 1st EuroPLoP Focus Group on pattern repositories, pp. 1–3, 2007. F. A. Fontana, A. Caracciolo, and M. Zanoni, “DPB: A benchmark for design pattern detection tools”, in 2012 16th European Conference on Software Maintenance and Reengineering, IEEE, 2012, pp. 235–244. DOI: 10.1109/CSMR.2012.33 M. Fernández-Osuna, M. A. Pérez-Delgado, M. Rojo-Martínez, and M. Piattini, “Exploring design patterns in quantum software: A case study”, Computing, vol. 107, no. 5, pp. 1–31, 2025. DOI : 10.1007/s00607-024-01365-z R. D. King, O. I. Orhobor, and C. C. Taylor, “Cross-validation is safe to use”, Nature Machine Intelligence, vol. 3, no. 4, pp. 276–276, 2021. C. Beleites, U. Neugebauer, T. Bocklitz, C. Krafft, and J. Popp, “Sample size planning for classification models”, Analytica chimica acta, vol. 760, pp. 25–33, 2013. K. Du et al., “Codegrag: Bridging the gap between natural language and programming language via graphical retrieval augmented generation”, arXiv preprint arXiv:2405.02355, 2024.