arXiv CSopen access

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

arXiv:2606.12231v1 [cs.SE] 10 Jun 2026

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study GUANGZONG CAI, School of Computer Science, Wuhan University, China RUIYIN LI, School of Computer Science, Wuhan University, China PENG LIANG, School of Computer Science, Wuhan University, China ZENGYANG LI, School of Computer Science, Central China Normal University, China MOJTABA SHAHIN, School of Computing Technologies, RMIT University, Australia The adoption of AI-powered Integrated Development Environments (AI IDEs) has introduced “Rules” as a novel software artifact, allowing developers to persistently inject project-specific constraints and architectural guidelines into the context of Large Language Models (LLMs). Despite their role in aligning AI behavior with developer intent, the taxonomy, evolution, and practical impact of these rules remain largely unexplored. To bridge this gap, we conducted a mixed-methods empirical study on AI IDE rules. By mining 83 open-source projects and extracting 7,310 rules, we established a comprehensive taxonomy comprising 5 primary and 25 secondary categories. We then triangulated these artifacts with survey responses from 99 practitioners. Our analysis identified a contrast between developer priorities and actual configurations: while practitioners rate architectural constraints as highly important, rule files in repositories primarily consist of low-level workflow and code formatting constraints. Furthermore, our analysis of 1,540 rule evolution events revealed that rules are updated frequently. Repository data further indicate that rule evolution is primarily driven by constructive context expansions (29.17%) and enrichments (26.59%). In contrast, surveyed developers reported modifying rules primarily to correct AI errors (77.78%), typically by adding new negative constraints rather than editing existing ones. Finally, an artifact compliance assessment of 160 rule evolution events revealed that updating rules significantly improves the adherence of software artifacts, with the average artifact compliance rate increasing by 22.99% (from 49.14% to 72.13%) following an update. Our study provides empirical insights that can help developers optimize prompting strategies and guide tool builders in designing automated conflict-detection and context-management mechanisms for AI IDEs. Authors’ Contact Information: Guangzong Cai, School of Computer Science, Wuhan University, Wuhan, China, [email protected]; Ruiyin Li, School of Computer Science, Wuhan University, Wuhan, China, [email protected]; Peng Liang, School of Computer Science, Wuhan University, Wuhan, China, [email protected]; Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China, [email protected]; Mojtaba Shahin, School of Computing Technologies, RMIT University, Australia, [email protected].

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-7392/2026/6-ART https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

2

Cai et al.

CCS Concepts: • Software and its engineering → Integrated and visual development environments; Software development techniques; Software configuration management and version control systems. Additional Key Words and Phrases: AI IDEs, Rules, Prompt Engineering, Agentic Coding ACM Reference Format: Guangzong Cai, Ruiyin Li, Peng Liang, Zengyang Li, and Mojtaba Shahin. 2026. Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study. ACM Trans. Softw. Eng. Methodol. 1, 1 (June 2026), 52 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

In recent years, the rapid advancement of Large Language Models (LLMs) has reshaped the paradigms of software engineering practice [72, 74, 91]. Evolving from early code completion tools to today’s AI Integrated Development Environments (AI IDEs) such as Cursor [11], Windsurf [30], and Trae [18] - AI has transcended its role as a mere plugin, emerging as a coding agent capable of perceiving project context and executing complex tasks [56]. At the core of this transformation is a deepening interaction between developers and AI: developers no longer settle for one-off Q&A sessions, but instead expect AI to strictly adhere to specific project conventions, architectural patterns, and coding styles [100]. To achieve this, many AI IDEs have introduced a novel software artifact called Rules (e.g., Cursor [10], Windsurf [29], Trae [19], Qoder [4], Kiro [7]). By configuring specific rule files (e.g., .cursor/rules) within a project, developers can persistently inject project-specific knowledge (e.g., “Please use Spring Boot version 4.0 or higher”, “Follow Domain-Driven Design”) into the context of AI IDEs [67, 81]. This mechanism essentially functions as persistent prompt engineering [50, 93], ensuring that the AI IDEs’ behavior not only maintains long-term alignment with developer intent [23] but also explicitly enhances the quality of the generated code [36, 52, 77]. Such a practice is gaining wide attention across the software development landscape [23, 50, 56]. In projects developed with AI IDEs, rule files are emerging as a critical type of configuration file [80, 88]. Despite the growing adoption of AI IDE rules, the understanding of their taxonomic characteristics and evolutionary patterns remains limited. First, as an emerging type of software artifact, the taxonomic characteristics of rules remain ill-defined. While a handful of studies have touched upon rule categorization, they are often limited by small sample sizes [24, 50] or restricted to specific IDEs [23, 83], lacking a comprehensive and generalizable taxonomy. Second, rules are not static artifacts. As software projects iterate, codebases expand, and tech stacks update - compounded by the potential for AI “hallucinations” - developers must continuously maintain and refine these rules [66]. However, the content of rule evolution, the reasons for its evolution, and how the compliance rate of software artifacts changes after rule evolutions remain largely unexplored. Without a deep understanding of their evolutionary lifecycle, rule files may become obsolete due to delayed updates or, worse, provide misleading context to AI IDEs, thereby undermining the effectiveness of human–AI collaboration in AI-assisted software development. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

3

To bridge the aforementioned gaps, this work aims to conduct a large-scale empirical study to investigate the state of practice regarding AI IDE rules in open source software (OSS) projects. Specifically, this study is dedicated to addressing the following two Research Questions (RQs): • RQ1: What are the categories of rules in OSS projects developed by AI IDEs? • RQ2: How do rules in OSS projects developed by AI IDEs evolve? To answer these two RQs, we employed a mixed-methods research design, combining mining software repositories with a subsequent developer survey. First, utilizing the GitHub API combined with manual filtering, we curated a dataset of 83 projects explicitly developed using AI IDEs. From these projects, we collected 325 rule files, extracting a total of 7,310 individual rules. Through Open Coding [86], we constructed a hierarchical taxonomy consisting of 5 major categories and 25 sub-categories. Next, we extracted rules that had undergone changes in their commit history. By analyzing rule-content diffs, co-changed files, and commit messages, we characterized the types of rule evolution, the reasons behind these changes, and the resulting shifts in software artifact compliance (i.e., the extent to which artifacts adhere to the updated rules). Furthermore, we designed and conducted a survey targeting developers with practical experience in configuring AI IDE rules. The survey consisted of 11 questions covering respondents’ demographics and rule usage practices. We finally received 99 valid responses. By combining “objective mining data” with “subjective developer perceptions”, we integrated these complementary perspectives to obtain a comprehensive understanding of rule taxonomy and evolution in AI IDEs. The main contributions of this work are as follows: • We established a taxonomy of AI IDE rules (5 primary, 25 secondary categories) based on 7,310 mined rules. Triangulating this with a 99-participant survey revealed a clear contrast between developers’ perceived architectural priorities and their actual formatting-centric configurations. • We characterized rule evolution by analyzing 1,540 change events, identifying a gap in the perceived reasons for rule evolution between commit data (driven by context expansion) and developer responses (focused on correcting AI errors). • We quantified the impact of rule maintenance on codebase adherence, demonstrating through an assessment of 160 rules that rule updates improved average artifact compliance by 22.99%. Paper organization: Section 2 provides a brief background on AI IDEs and their rules. Section 3 reviews related work on AI IDEs and rules in AI IDEs. Section 4 outlines the research questions and details our data collection procedures and research methodology. Section 5 presents the study results. Section 6 discusses the findings and provides implications for both practitioners and researchers. Section 7 addresses the threats to validity. Section 8 concludes the work with future directions. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

4

2

Cai et al.

Background

This section first introduces the emergence of AI IDEs (Section 2.1) and then discusses the concept of rules in AI IDEs (Section 2.2), which serves as the primary mechanism for customizing their behavior. 2.1

AI IDEs

Unlike traditional IDEs that integrate AI features as optional extensions (e.g., VS Code with the GitHub Copilot plugin [42]), an AI IDE refers to an integrated development environment built with artificial intelligence as a core architectural component [44]. In these IDEs, AI is not an auxiliary feature layered on top of an existing editor; rather, it constitutes the central organizing principle that shapes the IDE’s functionalities, interaction paradigms, and iterative evolution [100]. This architectural distinction leads to fundamental differences in the assistant capability of AI IDEs. Plugin-based solutions operate within the constraints imposed by the host IDE, limiting their access to certain runtime states, internal editor signals, or fine-grained interaction data. In contrast, AI IDEs integrate Large Language Models (LLMs) directly into the core editor infrastructure, enabling deeper access to project-wide context, internal state transitions, and user interaction history. As a result, AI IDEs typically support advanced capabilities beyond code completion, including autonomous task decomposition, multi-file refactoring, architectural reasoning, context-aware code generation, and continuous alignment with project-specific configurations (e.g., rule files) [92]. In this study, we focus on five representative AI IDEs that support user-defined rule configurations: Cursor [11], Windsurf [30], Trae [18], Qoder [5], and Kiro [8]. Table 1 presents information on these AI IDEs, highlighting the locations of their rule files. These AI IDEs share a common characteristic: they allow developers to explicitly define “Rules” that the AI IDEs must follow during code generation and chat interactions. Notably, Kiro refers to its rule mechanism as Steering. By default, Kiro automatically generates three configuration files - product.md, tech.md, and structure.md - under the .kiro/steering/ directory. Users may also define additional rule files within this directory to customize Kiro’ behavior [7]. 2.2

Rules in AI IDEs

In the context of AI IDEs, rules are textual instructions or directives defined by developers to constrain and guide the behavior of the LLM-driven AI assistant for software development [10, 19]. These rules are typically aggregated within specific rule files located in the project’s root or configuration directory. This mechanism functions as a form of persistent prompt engineering. When a user interacts with the AI assistant, the IDE automatically retrieves the relevant content from these rule files and injects them into the LLM’s context window. The primary purpose of rules is to align the AI IDEs’ output with the specific requirements of the project, which often include architectural constraints, coding styles, library preferences, and development workflow guidelines [23, 50]. A typical rule file is written in natural language (often Markdown). Figure 1 shows a snippet ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

5

Table 1. Comparison of AI IDEs and Their Rule Mechanisms AI IDE

Provider

Release

Rule File Location(s)

Cursor

Anysphere

6 Apr 2023 [9]

.cursor/rules/, .cursorrules [10]

Windsurf

Codeium

13 Nov 2024 [28]

.windsurf/rules/, .windsurfrules [29]

Trae

ByteDance

20 Jan 2025 [17]

.trae/rules/ [19]

Qoder

Alibaba

21 Aug 2025 [3]

.qoder/rules/ [4]

Kiro

Amazon

14 Jul 2025 [6]

.kiro/steering/ [7]

Note: The release dates are obtained from official changelogs. For Cursor, the listed date corresponds to the version in which it transitioned to the VSCode-based architecture. For Cursor and Windsurf, legacy rule file formats (e.g., .cursorrules, .windsurfrules) were later superseded by directory-based rule management mechanisms.

of a typical rule file for a React Native project, sourced from the community rule repository cursor.directory1 . As shown in Figure 1, the rule explicitly defines the Role (“expert in TypeScript...”), Style Constraints (“avoid classes”), and Technical Constraints (“Use Expo’s AppLoading”). Without this rule, a generic LLM might generate class-based components or use deprecated libraries, requiring the developer to perform cumbersome manual corrections.

Example of AI IDE Rules You are an expert in TypeScript, React Native, Expo, and Mobile UI development. Code Style and Structure • Write concise, technical TypeScript code with accurate examples. • Use functional and declarative programming patterns; avoid classes. Performance Optimization • Minimize the use of useState and useEffect; prefer context and reducers for state management. • Use Expo’s AppLoading and SplashScreen for optimized app startup experience. ...

Fig. 1. An example of a rule file defining coding standards.

1 https://cursor.directory/rules

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

6

3 3.1

Cai et al.

Related Work Evaluation and Impact of AI IDEs

Current research has extensively investigated AI-assisted programming tools, with a particular focus on AI IDEs [92, 100]. These studies can be broadly categorized into four dimensions: capabilities and benchmarking, impact on development practices, security assessments, and future directions. Capabilities and Benchmarks. Initial studies have surveyed the landscape of AI coding tools, providing comparative analyses of features in AI IDEs such as Windsurf and Cursor [92]. Beyond functional comparisons, researchers have constructed datasets from GitHub to evaluate these AI IDEs [56, 101] and explored their advanced capabilities in specific tasks, such as code refactoring [48], issue resolution [54], end-to-end microservice generation [75], and design issues in large-scale project generation [51]. Due to its strong reasoning capabilities, Cursor is increasingly utilized as a baseline for tasks like codebase question answering [99], doc-to-code synthesis [58], and microservice bug localization [71]. Impact on Development and Education. A significant body of work investigates how AI IDEs reshape the software development lifecycle. While some studies highlight improvements in coding efficiency [1, 15], others, such as He et al. [45], caution that Cursor may trade long-term code quality for short-term speed. To comprehend developer requirements and interaction behaviors, researchers have conducted questionnaire surveys on developers’ experiences using AI coding assistants and CodeLLMs [94] as well as empirical analysis of actual in-IDE human-AI conversation data [90]. In the educational domain, the impact of AI IDEs like Cursor on teaching and learning methodologies has been widely examined [85]. Security and Risk Assessment. As AI IDEs gain widespread adoption, concerns regarding the security risks introduced by AI-assisted code generation have grown significantly. Security evaluations of AI IDEs have covered a wide spectrum of risks, including the vulnerability of generated code [63, 69], and security issues arising from the invocation of external tools (e.g., shell commands, file systems, and APIs) by AI IDEs [98]. More sophisticated threats targeting AI IDEs and their context have also been reported, including prompt injection attacks [61], malicious payloads embedded in configuration files [60], and API key cracking [68]. Emerging Paradigms and Future Directions. With the rise of “Vibe Coding”, a wave of recent research has focused on this novel programming paradigm, discussing the pivotal role of AI IDEs in enabling natural language-driven development [25, 27, 41, 49, 57]. Furthermore, researchers have discussed current challenges - such as limitations in the context window capacity of LLMs and hallucinations in AI-generated code by AI IDEs [79, 95] - while envisioning future directions like multimodal programming and self-evolving agents [100].

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

3.2

7

Rule Configuration for AI Coding Agents

Research on AI coding rules can be categorized into two streams: studies focusing on specific functional aspects and empirical studies characterizing the rule ecosystem itself. Specific Applications and Impacts of Rules. Several studies have investigated the impact of rules in specific domains. In terms of efficiency, Lulla et al. analyzed the impact of AGENTS.md on runtime performance and token consumption [62]. More critically, Gloaguen et al. investigated the effect of repository-level context files (AGENTS.md) on automated agents’ task success rates, revealing that auto-generated contexts yield no improvement, while human-written ones offer only marginal gains [43]. Regarding interaction and format, McMillan et al. examined how file formats (e.g., YAML vs. Markdown) affect agent behavior [65], and further showed that configuration file design choices (e.g., length, structure, and instruction placement) have a negligible impact on instruction compliance. Instead, their findings indicate that an agent’s adherence to rules is primarily affected by the specific type of coding task and the length of the interaction session; specifically, the likelihood of an agent complying with rules progressively degrades as it continuously generates more code (e.g., generating subsequent functions) within a single session [64]. Vaithilingam et al. proposed SemanticCommit, a mixed-initiative interface to facilitate intent integration into rule files [93]. Furthermore, Pimenova et al. discussed the status of rules in the emerging “Vibe Coding” paradigm [77], and Hora et al. provided the best practices for using rules to guide AI coding agents in generating mock objects and handling dependencies during software testing [47]. Empirical Characterization of Rules. A growing body of work has conducted large-scale empirical studies on rule files (e.g., AGENTS.md, CLAUDE.md, .cursorrules) to understand their structure and content. Galster et al. analyzed configuration practices in agentic AI coding tools and observed that among the eight identified configuration mechanisms, static context files (e.g., CLAUDE.md and AGENTS.md) are the most widely adopted by developers in GitHub repositories. They dominate the configuration landscape and often serve as the sole mechanism used, whereas advanced mechanisms (such as Skills and Subagents) remain less prevalent, appearing in fewer than 20% of repositories and primarily in Cursor-based projects [40]. Studies on Claude Code configurations [24, 83] found that these files typically exhibit a shallow hierarchical structure, with content predominantly focused on operational commands (build/run), implementation details, and high-level architecture. Studies on AGENTS.md [23, 66] analyzed thousands of repositories, revealing that these files are not static documentation but evolving artifacts maintained through frequent updates. However, Chatlatanagulchai et al. noted a significant gap: while functional-related rules are prioritized, non-functional rules like security and performance are rarely specified [23]. Jiang et al. qualitatively analyzed 401 Cursor repositories. They constructed a taxonomy comprising five high-level themes (e.g., Norms, Guidelines) and explored how rule usage varies across programming languages and project domains [50].

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

8

3.3

Cai et al.

Conclusive Summary

Existing studies on AI IDEs have primarily examined their capabilities, impact on development practices, and associated security risks [45, 69]. These works typically treat AI IDEs as holistic tools or focus on the code artifacts they generate [48, 71]. In contrast, we focus on the Rule mechanism within AI IDEs, which connects “developer intent” with “AI behavior”. By exploring the categories and evolution of rules, we reveal the lifecycle characteristics of this emerging software artifact, providing empirical evidence for developers to utilize rules more effectively and for AI IDE designers to optimize rule mechanisms. Meanwhile, research on rule configuration has begun to explore the structure and usage of rule files [24, 83] and to propose initial taxonomies [50]. However, these studies are often limited to a single AI IDE and therefore do not capture the diversity of rule usage across different AI IDEs?. Furthermore, significant gaps remain regarding the dynamics of these rules. First, although existing studies (e.g., [23, 66]) observed that rules evolve, their analysis was primarily descriptive, lacking a systematic investigation into the drivers behind these evolutions. Second, no prior work has empirically evaluated how the compliance of software artifacts changes over time following rule evolution. Our study advances this field by not only constructing a comprehensive taxonomy of rules in AI IDEs but, more importantly, by employing multi-perspective data analysis to reveal why rules change and how the adherence of software artifacts shifts after rules are added or modified. 4

Research Methodology

Our study uses a mixed-methods approach consisting of two main phases: (1) analyzing rules collected from open-source projects developed using AI IDEs, and (2) surveying practitioners about their rule usage practices. First, we collected rule files from relevant projects by mining GitHub repositories to answer RQ1 regarding the categories of rules present in these projects. Subsequently, we conducted a practitioner survey to validate the mining results and explore how these rules evolve during development (RQ2). This methodological triangulation [86] strengthens the findings by integrating objective mining data with developers’ perspectives. Figure 2 shows an overview of the research methodology employed in this study. 4.1

Objective and Research Questions

Our study aims to empirically investigate the practices of configuring and maintaining rules in projects developed using AI IDEs. We examine the categories of explicitly defined rules used to govern the behavior of AI IDEs and explore how these rules evolve over time. This includes identifying the reasons for rule evolution and evaluating how the compliance of software artifacts changes after these rule updates. Specifically, we address the following Research Questions (RQs): RQ1: What are the categories of rules in OSS projects developed by AI IDEs? Motivation. Although rules are an emerging software artifact in AI IDEs, their application in software development is not yet well understood. RQ1 addresses this ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

(1) Mining Study Design (RQ1 & RQ2)

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

Phase 1: Data Collection & Filtering

Phase 2: Taxonomy Construction (RQ1)

(Time-Slicing API)

🧩 Rule Extraction

🔍 GitHub Search

Phase 3: Rule Evolution Analysis (RQ2)

⏳

Evolution Extraction & Categorization (RQ2.1)

& Segmentation

⚙️ Keyword & Rule

🛢️ 1,540 Evolved

🏷️ Open Coding

File Filtering

(7,310 Rules)

👁️ Manual Inspection (Verify AI IDE Usage)

9

Rules (Diffs)

🧠

Reason Analysis (LLM Filter + Open Coding)

📊

🗂️ 83 AI IDE Projects

Rule Taxonomy (5 Main, 25 Sub-cats)

Phase 4: Questionnaire Design

Phase 5: Data Collection

(325 Rule Files)

🤖

Compliance Assessment (LLM Filter + LLM-as-a-Judge)

📄

📄

504 Clear Rules (6 Driving Reasons)

160 Verifiable Rules Evaluated

(2) Practitioner Survey

Synthesis & Triangulation

📝

Draft Design (Mapping to RQs)

🔍

GitHub Search API (Commits in AI IDE Repos)

📧 Extract Target Emails

🧪

Pilot Surveys (n=50, 2 rounds)

📄 Final Online Questionnaire

Phase 6: Data Filtering

🧹

Quality Filtering (Screening & Careless)

🔄 Methodological Synthesis

Objective Data + Subjective Perceptions

(2,302 from Commits)

🚀 Localized Dispatch (En & Zh versions)

✅

99 Valid Responses

🛢️ 117 Initial

Comprehensive Understanding of Rule Taxonomy & Evolution

Responses

Fig. 2. Overview of the mixed-methods research process.

by classifying these rules through an empirical study. Constructing a taxonomy provides a structured view of how developers use rules to regulate AI IDE generated projects. Additionally, this taxonomy establishes the basis for our subsequent analysis of evolutionary trends across different rule categories. RQ2: How do rules in OSS projects developed by AI IDEs evolve? Motivation. Rules change throughout a software project’s lifecycle. Understanding these evolutionary characteristics is necessary for evaluating the maintenance and evolution effort required for AI-assisted development. We decompose RQ2 into three sub-RQs. RQ2.1: What categories of rules undergo evolution? Motivation. This RQ maps the changed rules to the taxonomy established in RQ1. By analyzing the distribution of evolved rules across categories and their change types (i.e., additions, modifications, and deletions), we identify which categories of rules developers update most frequently. This analysis describes the current state of rule maintenance and evolution in AI IDEs. RQ2.2: What are the driving reasons for rule evolution? Motivation. Identifying the underlying reasons for rule changes explains how developers adapt rules in software development with AI IDEs. By analyzing commit messages, rule content diffs, and co-changed files, we categorize the drivers of rule evolution, such as fixing AI hallucinations, improving prompt clarity, andsynchronizing with technology stack updates. The answer of this RQ can help developers ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

10

Cai et al.

write more maintainable rules and inform AI IDE designers on how to improve rule management mechanisms. RQ2.3: How does the compliance rate of software artifacts change following rule evolution? Motivation. The primary purpose of maintaining and evolving rules is to align software artifacts with developers’ intent. However, it is unclear whether rule evolution improves this alignment in practice. This RQ examines how the compliance rate of software artifacts changes after rule updates, providing empirical evidence on the effectiveness of rule maintenance and evolution. Understanding these changes helps evaluate whether updating rules improves the consistency of software artifacts and informs practitioners about the practical impact of rule-driven workflows. 4.2

Mining Study Design

4.2.1 Data Collection and Filtering. We mined open-source projects containing rule configuration files from GitHub. To ensure the consistency of our longitudinal analysis, we restricted our data collection to the main/master branch of each repository, with a strict cutoff date for commit histories set to 24 October 2025. The data collection and filtering process was conducted in the following three steps: (1) Initial Repository Search via Time-Slicing. We selected five AI IDEs that support rule mechanisms (Cursor, Windsurf, Trae, Qoder, and Kiro). We queried the GitHub Repository Search API2 to identify projects whose description or README files matched specific inclusion keywords (see Table 2). For instance, when searching for Cursor projects, we used keywords like “cursor IDE” and “by cursor”. Because the GitHub Search API imposes a limit of 1,000 results per query, we addressed this constraint by slicing the search window chronologically. For each AI IDE, the window spanned from its release date (listed in Table 1) to 15 September 2025. This time-slicing approach enabled us to collect a broader set of relevant projects within the search period. (2) Keyword and Rule File Filtering. Because some AI IDE names are ambiguous (e.g., “cursor” often refers to a database cursor or a mouse pointer), we applied a set of exclusion keywords (listed in Table 2) to remove tutorials, toy projects (e.g., test, demo), and repositories related to these ambiguous terms. Next, we queried the GitHub Repository Contents API using the template3 to verify the presence of the target AI IDE’s rule files or directories in the remaining repositories, using the paths defined in Table 1. We excluded any projects that lacked these rule files. Finally, to filter out inactive repositories, we excluded projects with fewer than 10 commits and 0 stars. (3) Manual Inspection. The first author manually reviewed the remaining projects. This step served two purposes: first, to remove any irrelevant or toy projects that were not excluded by the keyword filtering; and second, to verify that the project’s

2 https://api.github.com/search/repositories 3 https://api.github.com/repos/<owner>/<repo>/contents/<rule_file_path>

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

11

README or description explicitly stated it was developed using an AI IDE (e.g., “This project was fully created and is maintained by AI agents”4 ). Following this process, we obtained a final dataset of 83 projects (the complete project list is available in our online replication package [21]). Table 3 presents a breakdown of the project counts for each AI IDE across the different filtering stages. Table 2. Inclusion and Exclusion Keywords for Repository Search Category

Keywords

Inclusion (e.g., Cursor)

cursor IDE, cursor AI, using cursor, cursor Agent, use cursor, with cursor, by cursor, in cursor, through cursor, via cursor

Exclusion

test, demo, learn, practice, practical, rule, rules, mouse, pagination, a cursor, curse, 3D cursor, your cursor, custom cursor, cursor navigation, navigation, cursor move, example, awesome, movement, keyboard, custom, position, animat, pointer, theme, attempt, moving, move, Paginate, Trying, try, simple, quick, oracle, store procedure, mysql, sql server, cursor library, Drawing, draw, prompt, collection, hand gesture, canvas, click, cursor array, SQL, experiment, scrollbar, Portfolio, template, course, tutorial, toy, live cursor, guideline, quiz, small, exploring, realtime, real-time, window, first, homework

Table 3. Project Counts Across Data Filtering Stages AI IDE Cursor Windsurf Qoder Trae Kiro Total

Initial Search

Keyword and Rule File Filtering

Manual Inspection

18,790 3,684 125 1,445 1,543 25,587

132 9 12 152 189 494

46 5 1 6 25 83

4.2.2 Rule Extraction and Taxonomy Construction. To answer RQ1, we first extracted and segmented the rules from the 83 collected projects. By querying the GitHub Repository Contents API, we extracted standalone rule files (e.g., .cursorrules and .windsurfrules) as well as all files residing within specific rule directories (e.g., .cursor/rules/). Most of these artifacts are formatted as Markdown files. The AI IDE Kiro automatically generates three default files under its .kiro/steering/ directory: product.md, tech.md, and structure.md. Because this study focuses on rules that guide and constrain AI IDE coding behaviors, technical specifications, and collaboration patterns, we excluded product.md. This file primarily documents business requirements and product features, which deviate from our focus on development 4 https://github.com/petermodzelewski/yt-database

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

12

Cai et al.

constraints. Following this extraction phase, we obtained 325 valid rule files from the 83 projects. Because a single rule file typically contains multiple independent rules, we established “a single rule” as our fundamental unit of analysis. The first author manually reviewed and segmented all files. To ensure the reliability of this segmentation process, the first and third authors resolved ambiguous boundaries through discussion, while the other co-authors audited the results and provided feedback. We defined a complete “semantic unit” as the smallest indivisible text block that conveys a standalone, actionable constraint for the AI IDE. The boundary of a rule is determined by its contextual independence. In most cases, a single line of text (e.g., an independent bullet point) encapsulates a complete constraint and is therefore treated as a single rule. However, multiple lines are merged into a single cohesive rule if they exhibit structural or logical dependencies that render them incomplete or meaningless if separated. For instance, we preserved multi-line blocks as single units when they comprised a conditional trigger followed by a nested list of steps, or a high-level directive accompanied by explanatory code snippets. As illustrated in Figure 3, Lines 1, 2, 3, and 4 each represent an independent rule because they are contextually self-contained. Each color corresponds to one complete rule. In contrast, Lines 5 through 11 collectively form a single rule because the numbered steps (Lines 6–11) are logically bound to the conditional prerequisite stated in Line 5 (“When suggesting code or solutions:”). Separating these blocks would cause the individual steps to lose their triggering condition. Example of Rule Segmentation 1 2

- Write thorough tests for all major functionality.

3 4

- Always use ’docker compose’ command instead of docker-compose.

5 6 7 8

When suggesting code or solutions: 1. First, analyze the existing code structure and performance implications. 2. Provide a step-by-step plan for implementing changes or new features. 3. Offer code snippets that demonstrate best practices for Pixi.js and TypeScript in a game development context. 4. Always consider the performance impact of suggestions, especially for mobile devices. 5. Provide explanations for why certain approaches are more performant or efficient. 6. Be aware of potential Pixi.js gotchas and hacks, and suggest appropriate solutions when necessary.

9 10 11

- Always think about what other methods and areas of code might be affected by code changes.

- Always setup the docker files to ensure that the layers are cached and the most changed files are last in the docker file.

Fig. 3. An example of segmenting rule text into distinct semantic units (rules).

After completing the segmentation, we extracted 7,310 individual rules from the 325 files. We then employed an open coding approach [86] to categorize these rules (the raw files and coding mapping datasets are publicly available in our replication package [21]). The first author reviewed each rule and assigned an initial code. When disagreements ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

13

or uncertainties arose, the first and third authors held discussions and applied the negotiated agreement approach [22] to reach a consensus. For example, referring back to Figure 3, we coded Lines 1 through 4 as Testing Strategy, Design Principles, Environment Configuration, and Environment Configuration, respectively. Conversely, we coded the entire block from Lines 5 to 11 as AI Behavior & Decision Strategies. Through this process of induction and abstraction, we constructed a hierarchical taxonomy comprising 5 primary categories and 25 secondary categories. 4.2.3 Evolution Rule Extraction and Categorization. We focused on the main/master branch of each repository and extracted the evolutionary history of the 325 rule files collected in Section 4.2.2. To account for file renames or relocations during project iterations, we used Git commands to track the complete lifecycle path of each rule file. We extracted historical commit hashes, timestamps, and corresponding code differences (Diffs) to create chronologically ordered evolution sequences. Statistics show that 55 of the 83 projects contain rule files that underwent evolution. At the file level, 117 of the 325 rule files contain at least one modification (i.e., commit count > 1). We performed fine-grained rule segmentation on the extracted Diffs using the same criteria described in Section 4.2.2. Concurrently, we labeled the change type of each rule as Added, Modified, or Deleted. Figure 4 illustrates these types: Line 3 represents an added rule, Line 5 represents a deleted rule, and Lines 7 and 8 demonstrate a modified rule. We identified and excluded trivial changes limited to punctuation, whitespace, or line breaks (i.e., Chore modifications). Finally, we employed a deductive coding approach [38] to categorize the evolved rules. The first author used the two-level taxonomy from Section 4.2.2 as a codebook to map each evolved rule to a category. We resolved ambiguous mappings through offline discussions between the first and third authors. We extracted and segmented 1,894 evolved rules from the Diffs of the 117 files. After filtering out 354 Chore modifications, we retained 1,540 valid evolved rules for subsequent analysis (the raw Diff blocks and categorized mapping dataset are available in our replication package [21]). 4.2.4 Analysis of Reasons for Rule Evolution. After extracting the 1,540 valid evolved rules in Section 4.2.3, we investigated the reasons behind these changes by analyzing their contextual artifacts, including rule content, change types, commit messages, and co-changed files. Figure 6 outlines the overall procedure for this analysis and the subsequent compliance assessment. Because rules and changes vary in granularity, not every rule evolution provides sufficient context to determine its cause (e.g., changes with uninformative commit messages). Therefore, we first filtered the 1,540 rules based on determinability, assessing whether the rationale for each change could be objectively identified. Because manually reviewing 1,540 rules is labor-intensive, and recent studies have demonstrated the feasibility of applying LLMs to mining software repositories [32, 33, 35], we used LLMs for automated data labeling, complemented by human sampling to verify reliability. Based on Cochran’s sample size formula [2], a population of 1,540 requires a minimum sample of 308 to achieve a 95% confidence level with a 5% margin of error. We randomly sampled 308 rules to establish a human-labeled ground truth. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

14

Cai et al.

Example of Rule Changes (Diff) 1 2 3 4 5 6 7 8

Follow Go best practices: Use go mod for dependency management + Use testify for assertions Follow standard project structure - Write idiomatic Go code ... - The file MUST be named in the format ‘YYYYMMDDHHmmss_short_description.sql‘ with proper casing for months, minutes, and seconds in UTC time: + The file MUST be named in the format ‘YYYYMMDDHHmmss_short_description.sql‘ with proper casing for months, minutes, and seconds in EAT (East Africa Time - Africa/Nairobi) time:

Fig. 4. An example of extracting changes from rule Diffs.

The first author and a software engineering practitioner independently conducted a pilot labeling on 30 initial samples, resulting in a Cohen’s Kappa of 0.76. After resolving disagreements and aligning their criteria, the two human coders independently labeled the full 308-rule subset. The final Cohen’s kappa was 0.86, indicating strong inter-rater agreement. Using the insights from the human labeling process, we iteratively refined prompts to guide the LLMs in assessing the determinability of 1,540 rules. To reduce potential model bias, we independently applied three LLMs (Gemini 3 Flash Preview, GLM 5, and Qwen3 Max). During the prompt refinement, we analyzed cases where the LLMs disagreed with each other or deviated from the human ground truth, and we updated the instructions to handle these edge cases. Figure 5 shows the final prompt used for the LLMs (with minor omissions for brevity; the full version is available in our replication package [21]). This prompt contains an Ultra-Conservative inference protocol, which mandates that the models adhere to the principle of Evidence over Inference. Specifically, a rule change is classified as Clear only if there is a structural match between the rule difference and the co-changed file paths, or a semantic alignment with the commit message (e.g., a commit message stating “refactor(rules): update instruction compliance and conventional commit rules for better enforcement”). Otherwise, vague or generic changes are classified as Unclear. For the full dataset, the Fleiss’ Kappa [37] among the three LLMs was 0.78. For the 308-rule sample, the Cohen’s Kappa [31] between the LLMs’ majority vote and the two human coders was 0.78 and 0.75, respectively. These metrics demonstrate consistency both among the LLMs and between the LLMs and human evaluators. Following this LLM cross-filtering, we extracted a subset of 504 rules that all three models unanimously determined as Clear. The first author then applied open coding to these 504 rules to identify the reasons for their evolution, resolving boundary ambiguities through discussions with the third author. During this process, we developed ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

15

Abridged Prompt for Reason Determinability Filtering

System Prompt # Role: You are an ultra-conservative Researcher in Software Evolution. Your task is to analyze a specific **change event** of an AI IDE Rule and determine the **Primary Driver (Reason)**. # Core Philosophy: "Evidence over Inference." You are strictly biased towards "Unclear". You only assign a specific category if there is **DIRECT, UNDENIABLE evidence** linking the Rule Diff to the Context. You must explicitly punish generic advice, unannotated deletions, and speculative causality. ...(Omitted content) # Inference Logic (The Ultra-Conservative Protocol) ...(Omitted content) **Step 1: The "Direct Link" Check (Structural or Intentional)** * **Structural Match (Absolute Proof):** Is there an **exact keyword, precise filename, or unique path match** between the Rule text and the Co-changed Files list (e.g., Rule mentions ‘extractors/‘ and Co-changed files include ‘src/extractors/‘)? * *YES:* This is definitive evidence of Synchronization or Context Enrichment. **This structural evidence overrides a seemingly unrelated commit message.** * **Intentional Match:** Does the Commit Message perfectly and explicitly describe the exact change made in the Rule Diff? * *YES:* Strong evidence for a specific category. * *NO to both:* Do NOT assume a link just because they use the same framework/language. Proceed to Step 2. **Step 2 ...(Omitted content) # Output Format Return a SINGLE JSON object: { "tier_1": "’Clear’ or ’Unclear’", "confidence": "...", "reasoning": "A single sentence. If Unclear, state why the evidence is insufficient." } # Reference Examples (Learn the Decision Boundary) ...(Omitted content)

User Prompt # Input Data for Analysis ## 1. Rule Info * **Project Name:** {project_name} * **Target Rule File:** {rule_file_path} * **Change Type:** {change_type} * **Rule Content (Diff):** {rule_content} ## 2. Commit Info * **Commit Message:** {commit_message} ## 3. Co-changed Files (Format: change_type file_path added_lines deleted_lines) {co_changed_files_raw_list} # Task Based on the inputs above, categorize the driver of this rule change.

Fig. 5. The prompt structure used for LLM-based reason determinability filtering. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

16

Cai et al.

Analysis of Reasons for Rule Evolution Filtering Phase

Coding Phase

LLM-based Filtering

Reliability Assurance (Sample: 308 Rules)

Evaluate determinability (N = 1,540)

Kappa: 0.86

: 0.78 Kappa

Gemini 3 Flash Preview GLM 5 Qwen 3 Max

Kappa: 0.75

Coder 1

Kappa: 0.78

504 Rules

Open Coding

Taxonomy

(Determinable)

Qualitative Analysis

6 Reason Categories

Coder 2

Coding

Prompt Iteration

Initial Dataset 1,540 Evolved Rules (From RQ 2.1)

Compliance Assessment of Rule Evolution Filtering Phase

Evaluation Phase Window Filtering

Logical Pruning

Reliability Assurance (Sample: 284 Rules)

• Exclude Deleted (-296) • Unverifiable (-167)

• Context Empty (-9) • Multi-change (-26) Retained: N = 240

Kappa: 0.86

Gemini 3 Flash Preview GLM 5 Qwen 3 Max Kappa: 0.75

Prompt Iteration

Compliance Results 160 Rules Evaluated

0. 7 pa :

K ap

Evaluate determinability (N = 1,077)

8

LLM-based Filtering

Coder 1 K

.76 a: 0 app

Coder 2

275 Rules (Assessable)

LLM-as-a-Judge Applicability Check (-80)

Prompt Iteration

Fig. 6. Overview of the rule evolution analysis and compliance assessment process.

guiding criteria to support the classification of the reasons for rule evolution. For example, if a rule addition accompanied the creation of new files, we distinguished between Context Enrichment (providing static documentation or references) and Expansion (imposing new logical constraints). We classified a rule modification synchronized with a file rename or move as Synchronization. We categorized changes that introduced negative constraints (e.g., “Do NoT”) or were associated with fix-related commit messages as Correction, and we classified non-logical adjustments that improved tone or formatting as Refinement. Through this open coding, we inductively derived 6 primary categories of reasons for rule evolution. 4.2.5 Compliance Assessment of Rule Evolution. To investigate the practical impact of rule evolution, we conducted a longitudinal assessment of the evolved rules to measure how the compliance of software artifacts (e.g., code, configuration files, documentation, directory structures, and commit messages) changed before and after rule modifications. As illustrated in Figure 6, this pipeline consists of three phases: preliminary exclusion, verifiability filtering, and compliance evaluation via LLM-as-ajudge [46]. First, during the preprocessing phase, we pruned the initial 1,540 evolved rules. We excluded 296 rules with a deleted change type because tracking compliance for a removed constraint is obsolete. We also filtered out 167 rules from categories that lack statically verifiable artifacts (e.g., soft guidelines for AI interaction behavior or subjective code review instructions without objective metrics). Following these exclusions, 1,077 rules remained for analysis. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

17

Second, similar to the data filtering process for ambiguous rules (Section 4.2.4), not all remaining rules could be objectively assessed for compliance via static artifacts. Because manual verification is labor-intensive, we again used LLMs for data labeling and validated the results through manual sampling. Based on Cochran’s formula [2], a population of 1,077 requires a minimum sample of 284 to achieve a 95% confidence level with a 5% margin of error. Therefore, we randomly selected 284 rules for manual labeling. After resolving discrepancies, the Cohen’s Kappa [31] between the first author and another software engineering practitioner reached 0.86. We then independently applied three LLMs (Gemini 3 Flash Preview, GLM 5, and Qwen3 Max) to evaluate the entire dataset. Figure 7 shows the abridged prompt for this task (the full version is available in our replication package [21]). In this prompt, we applied a static verifiability criterion: a rule is verifiable only if its compliance can theoretically be checked by a simple script (e.g., regular expressions, AST parsing) without false positives. We excluded rules requiring runtime execution, complex control flow tracing, or subjective intent interpretation, as well as rules serving merely as references. Conversely, rules specifying explicit file paths or naming conventions were considered verifiable. The Fleiss’ Kappa [37] among the three LLMs was 0.75, and the Cohen’s Kappa between the LLMs’ majority vote and the two human annotators was 0.78 and 0.76, respectively. Following this step, we extracted 275 rules that all three models unanimously identified as statically verifiable. For these 275 rules, we defined an observation window consisting of up to 5 commits before and 5 commits after the rule change event (or the available commits if fewer than 5 existed). To reduce evaluation noise, we filtered out tangled commits and large files. The filtering criteria excluded: (1) commits modifying more than 50 files; (2) commits modifying more than 2,000 lines; (3) individual files with more than 1,000 modified lines; and (4) individual files whose total length exceeds 1,000 lines. Triggering these criteria only excluded the specific commit or file, rather than immediately discarding the rule. However, if applying these filters eliminated all preceding or all succeeding commits for a specific rule, we discarded that rule (removing 9 rules). Furthermore, to prevent analysis conflicts, we discarded rules that were subsequently modified or deleted in a later commit within the window, or had already been created or modified in a prior commit (removing 26 rules). Consequently, 240 rules remained for the LLM-as-a-judge evaluation. Finally, because recent studies show that using LLMs as judges in software engineering tasks achieves results comparable to human evaluation [46, 96], we constructed an evaluation process based on the LLM-as-a-judge approach. We developed the prompt iteratively: we sampled a small set of records to draft an initial version, manually inspected the LLM outputs, identified errors, summarized their root causes, and refined the prompt accordingly. We repeated this cycle until three consecutive rounds of manual inspection revealed no errors or unreasonable judgments. Figure 8 presents the resulting abridged prompt (the full version is available in our replication package [21]). To reduce hallucinations, the LLM evaluated each commit independently. As shown in the User Prompt of Figure 8, we supplied the LLM with inputs including rule-specific details, target commit information, a snapshot of the project directory tree after the ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

18

Cai et al.

Abridged Prompt for Rule Verifiability Filtering

System Prompt # Role: You are an expert MSR (Mining Software Repositories) Researcher and Automated Testing Architect. Your task is to analyze "AI IDE Rules" and determine if they are **Technically Verifiable** based on concrete software artifacts. ...(Omitted content) # The Core Question (The "Simple Script" Golden Rule) **"Can I write a simple, deterministic script (using ONLY simple Regex, basic AST parsing, or file path checks) to verify this rule across thousands of commits WITHOUT ANY false positives/negatives?" **If the answer requires "understanding developer intent", "tracing complex control flows", or "verifying algorithmic correctness", the answer is unequivocally FALSE. # 1. Strictly Permitted Static Targets (Must be tracked in Git) To be TRUE, the rule MUST uniquely target one of the following: - Raw Source Code syntax: Exact keywords, specific annotations/decorators (e.g., ‘@Component‘), explicit type declarations (e.g., ‘any‘). - File System Structure: Existence of specifically named source files (e.g., ‘tsconfig.json‘, ‘*route.dart‘). ...(Omitted content) # 2. STRICTLY BANNED (Will immediately result in FALSE) If the rule involves ANY of the following, you MUST mark can_detect: false: - [BANNED] Runtime/Execution: "Tests pass", "App loads fast", "Memory usage". ...(Omitted content) # Output Format Return a SINGLE JSON object: { "can_detect": true/false, "detection_targets": "...", "detection_logic": "..." } # Few-Shot Examples (Reference Cases) ...(Omitted content)

User Prompt # Input Rule { "change_type": "{change_type}", "first_level": "{first_level}", "second_level": "{second_level}", "rule_content": "{rule_content}" }

Fig. 7. The prompt structure used for LLM-based verifiability filtering.

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

19

commit, and all file Diffs from that commit. We defined specific evaluation logic and output constraints: the LLM was required to first perform an applicability check to bypass commits irrelevant to the rule’s target artifacts. During this step, if the LLM found no checkable compliance points in all preceding or all succeeding commits for a rule, we excluded that rule (removing 80 rules). Furthermore, the LLM produced structured JSON output that explicitly distinguishes between boolean rules, which verify a global state, and quantifiable rules, which evaluate each modified instance individually. We executed this evaluation pipeline on the 240 rules using Gemini 3 Flash Preview. Upon completion, we conducted a manual inspection of the 160 remaining generated results to ensure the validity of the automated evaluations. 4.3

Survey Design

The second phase of our study is a practitioner survey designed to investigate how developers configure and maintain rules in AI IDEs in practice. This survey aims to achieve methodological triangulation by complementing the repository mining findings from Section 4.2 with developers’ subjective perceptions. The complete workflow of the survey is illustrated in Figure 9. To reach a broad and diverse audience, we administered the survey online. The questionnaire comprises single-choice questions, multiple-choice questions, and Likert-scale rating questions. 4.3.1 Questionnaire Design. We designed the questionnaire to strictly align with our core objectives, ensuring that each Survey Question (SQ) maps directly to our Research Questions (RQs). The Welcome page of the survey outlines the background and purpose of the study. At the bottom of this page, we implemented a screening question (SQ0) to verify whether the respondent had practical experience using AI IDEs and their rule features. Only participants who answered affirmatively could proceed; otherwise, the survey terminated immediately. The main body of the questionnaire consists of four sections: (1) Demographics: Capturing basic background information, including location, education, professional experience, team role, and development domains. (2) AI IDE and Rule Usage: Identifying the specific AI IDEs used by participants and the duration of their experience with rule files. (3) Rule Creation and Categories (mapping to RQ1): Investigating strategies for rule creation and asking developers to rate the importance of various rule categories. (4) Rule Evolution (mapping to RQ2): Exploring the specific triggers for modifying rules (RQ2.2) and the practical impacts these changes have on their development processes (RQ2.3). The survey questions are outlined in Table 4 (the questionnaire is also available online5 ). Before finalization, the first and third authors collaboratively discussed the wording of each question to resolve any ambiguities. We then conducted two rounds of pilot surveys. In the first round, we sent the draft survey questionnaire to 20 contributors of GitHub projects containing rule files, received 2 valid responses, and 5 https://forms.gle/sTTcK2cvt4UzEdSL6

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

20

Cai et al.

Abridged Prompt for Rule Compliance Assessment

System Prompt [Role] You are a rigorous Empirical Software Engineering Auditor. Your task is to evaluate whether a specific Git Commit complies with an AI IDE Project Rule. You will receive the Rule information and ONE commit’s snapshot/diffs at a time.[Core Evaluation Guidelines] ...(Omitted content) STEP 2: Rule Classification Determine if the rule is QUANTIFIABLE or BOOLEAN: - QUANTIFIABLE (Incremental/Continuous): The rule applies to multiple instances (e.g., "all variables must be camelCase", "always use try-catch"). You must extract EVERY occurrence in the diffs and judge them individually. - BOOLEAN (State/Existence): The rule checks a global state or singular property (e.g., "directory ‘src/components‘ must exist", "use pnpm"). The commit as a whole is evaluated as True or False. **STEP 3: Applicability Check (Crucial)** Does this commit touch files, code, or metadata relevant to the target rule? If no artifacts corresponding to the rule are found in the commit (e.g., the rule dictates React UI conventions, but the commit only modifies backend Python files), you MUST mark the commit as "NOT_APPLICABLE" (‘is_applicable‘: false). Do not force an evaluation on irrelevant commits. ...(Omitted content) [Output Format Requirements] ...(Output JSON Schema Omitted)

User Prompt [Rule Information] ...(Omitted content) [Commit Information to Evaluate] ...(Omitted content) - Project Directory Tree (Snapshot after commit): project_tree_structure - File Diffs (Code Changes in this Commit): file_diff_contents *(Note: Strictly judge the developer’s behavior based on the ’+’ lines to avoid penalizing them for legacy code).* Please analyze the data above and output the evaluation in the required JSON format.

Fig. 8. The prompt structure used for rule compliance assessment.

refined the survey logic. In the second round, we sent the revised survey questionnaire to another 30 contributors, received 3 valid responses, and further optimized the formatting and phrasing to maximize readability. 4.3.2 Data Collection and Filtering. To invite developers with hands-on experience modifying rule files in AI IDEs, we queried the GitHub API to identify repositories ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

21

Welcome, introduction & consent

Part 1: Demographic information

• Study purpose • Confidentiality, estimated time (5–8 min), four sections • Screening: experience using AI IDEs and configuring/using rule features

• Country • Education background • Years of professional software experience • Roles in the team • Domains where AI IDE rules are used

Prerequisite question (before Part 1) Survey questions Q1–Q5

Part 2: Use of AI IDEs and rules

Part 3: Rule creation & categories

• Which AI IDEs are currently used ? • How long rule files have been part of the development workflow ?

• Primary strategy when creating rule files • Importance ratings (1–5) for rule sub-categories across Architecture, Implementation, Workflow, QA, and AI collaboration

Survey questions Q6–Q7

Survey questions Q8–Q9 for answering RQ1

Part 4: Rule evolution — impacts

Part 4: Rule evolution — reasons

• Observed practical impacts of maintaining/updating rules on code and process • End of questionnaire (optional email for study results)

• Primary reasons for modifying existing rule files (correction, refinement, synchronization, context enrichment, expansion, pruning, …)

Survey question Q11 for answering RQ2.3

Survey question Q10 for answering RQ2.2

Fig. 9. Overview of the survey questionnaire. Table 4. Overview of the Survey Questionnaire ID

Section

Question

Type

SQ0

Introduction

Do you have experience utilizing AI IDEs (e.g., Cursor, Windsurf) AND configuring their rule features (e.g., .cursorrules)?

Single Choice

SQ1 SQ2 SQ3

Demographic Demographic Demographic

Single Choice Single Choice Single Choice

SQ4 SQ5 SQ6 SQ7

Demographic Demographic Demographic Demographic

What is your primary location of work or study? What is your highest level of education? How many years of professional experience do you have in software development? What is your primary role in your team or organization? In which domains have you utilized AI IDEs and their rule features? Which AI IDEs do you currently use for development? How long have you been incorporating rule files into your development workflow?

SQ8 SQ9

RQ1 RQ1

When creating rule files, what is your primary strategy? Please rate the importance of 25 rule sub-categories across 5 primary categories. (1 = completely unimportant/rarely used; 5 = crucial/essential)

Single Choice Likert Scale

SQ10

RQ2.2

What are the primary triggers that prompt you to modify existing rule files?

Multiple Choice

SQ11

RQ2.3

What practical impacts have maintaining and updating these rules had on your project code or development process?

Multiple Choice

Multiple Choice Multiple Choice Multiple Choice Single Choice

containing commits that touched rule files or directories between June 2025 and January 2026. From these commits, we collected the publicly available email addresses ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

22

Cai et al.

configured by the authors in their Git metadata. We carefully removed duplicates across different repositories and applied regular expressions to filter out invalid or autogenerated addresses (e.g., users.noreply.github.com, localhost). Following this process across the 5 selected AI IDEs, we identified a total of 2,302 unique developer emails. To ensure ethical compliance and respect for developer privacy, all data collection adhered to GitHub’s Terms of Service regarding public information6 . The collected emails were used exclusively for sending survey invitations. Participation in the survey was entirely voluntary and anonymous, with no personally identifiable information (PII) linked to the survey responses. Furthermore, the invitation emails included a clear opt-out mechanism, and the mailing list was permanently destroyed upon the conclusion of the data collection phase. To maximize the response rate, we provided a localized version of the survey. Based on email suffixes, author names, and repository description languages, we identified 353 Chinese developers and sent them a translated Chinese version of the questionnaire. Out of the 2,302 emails dispatched, 162 bounced back, resulting in 2,140 effectively delivered invitations. Participation was entirely voluntary, and no financial compensation was provided. By the cutoff date of 20 April 2026, we received 117 initial responses (57 in English and 60 in Chinese). To ensure data quality, we conducted a rigorous filtering process on the collected responses: (1) Screening Failure: 12 respondents (2 in English, 10 in Chinese) who answered “No” to SQ0 (indicating a lack of experience with AI IDE rules) were immediately excluded. (2) Careless Responses: We identified and excluded 6 careless responses (2 in English, 4 in Chinese) that exhibited “straight-lining” behavior. These participants gave the exact same score for all 25 rule categories in SQ9 AND blindly selected every available option in both SQ10 and SQ11, rendering their feedback meaningless for discussion. After applying these exclusion criteria, we obtained a final dataset of 99 valid responses (53 in English and 46 in Chinese) for our subsequent analysis. 4.4

Mining Study Overview

To comprehensively understand the ecosystem of projects driven by AI IDEs, we conducted a statistical analysis of the fundamental characteristics of the 83 collected open-source repositories developed using AI IDEs. This overview encompasses the primary programming languages, application domains, project scales, and rule configuration densities. Languages and Domains. As shown in the left panel of Figure 10, because some projects use multiple programming languages, we classified each repository according to its predominant PL. TypeScript dominates the dataset, accounting for 50.6% (42 projects), followed by Python with 12 projects (14.4%) and JavaScript with 8 projects 6 https://docs.github.com/en/site-policy/github-terms/github-terms-of-service

ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study 50

42

40 30

Primary Language

3

2

er

4

Oth

7

nt ile A pp Web licatio n A (Bpaplicat cke ion nd) App Des lica ktop tion

8

Mob

PHP

0

8

Too ls

L

8

pme

2

er

2

Oth

2

Java

4

Go

t

on

Scri p

pt

4

17

10

7

Java

Pyth

Typ eSc ri

0

8

HTM

12

10

26

20

Vue

20

eve lo

30

Gam eD

40

Web Number of Projects A (Fuplplica l t i s o Web tac n k) A (Fproplicat nte ion nd Lea AI rnin / Ma ) g / A chin gen e t

Number of Projects

50

23

Project Domain

Fig. 10. Distribution of primary languages and project domains in the selected projects. 50 41

30

5

0

1

4

6-1

1

4

+

0

500

9 -49 200

9

99

-19 100

50-

9

Commit Count

2

20

10

10

3

13 5

0-4

30

20

2

13

10

40

26

1

20

Number of Projects

40

30

0

Number of Projects

50

40

Contributor Count

14 12 15 13

10 0

1

4 5 4 4

4

7

202 2024-09 2024-10 2024-11 2024-12 2025-01 2025-02 2025-03 2025-04 2025-05 2025-06 2025-07 2025-08 2025-09 2025-10 5-1 1

50

Number of Projects

50

Project Creation Month

Fig. 11. Distribution of commit count, number of contributors, and project creation month in the selected project.

(9.6%). The “Other” category comprises a diverse array of languages, including Elixir, Rust, Dart, C#, Kotlin, and Swift. Regarding application domains (in the right panel of Figure 10), Web Application development constitutes the overwhelming majority. Specifically, Full-Stack (26, 31.3%), Frontend (17, 20.4%), and Backend (4, 4.8%) projects cumulatively account for 56.6% of the collected projects. This suggests that Web engineering, with its strong emphasis on rapid iteration, is currently the most common adoption scenario for AI IDEs. Project Scale and Creation Time. Figure 11 presents the distribution of commit volumes, contributor counts, and project creation dates. Notably, projects developed entirely using AI IDEs are currently predominantly small-to-medium in scale. Over half of the projects (50, 60.2%) have fewer than 50 historical commits. In terms of team size, solo developers or micro-teams of 1 to 3 people account for the vast majority (1 person: 49.4%, 2 people: 31.3%, 3 people: 12.0%). This characteristic provides a profound reflection of the current landscape of projects developed by AI IDEs: they are primarily driven by independent developers or extremely small teams leveraging AI for agile development. Furthermore, based on the timestamp of the first commit, the rightmost chart shows that many of these projects were created between June 2025 and September 2025. This timeline closely aligns with the recent rapid updates and widespread adoption of mainstream AI IDE tools. Rule Distribution. Figure 12 reveals the distribution of rule files and individual segmented rules across the projects. The left chart indicates that most projects favor a ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

24

Cai et al.

30

16

14

10

3

9 -39

9

200

-19 100

99 50-

49 20-

21+

Rule File Count

0

6

9

3

1-1

2

20

0

6

11-

3

2

1

0

6

8

6-1

10

20

4-5

20

25 19

+

22

400

30

Number of Projects

40

36

Number of Projects

40

Rule Count

Fig. 12. Overview of the number of rule files and rule entries in each project.

centralized configuration pattern: projects with exactly 1 or 2 rule files account for 43.4% (36) and 26.5% (22), respectively. Repositories maintaining an extensive number of rule files (over 20) are extremely rare, comprising only 3.6% (3 projects). Following our semantic segmentation (see Section 4.2.2), the right chart displays the total number of individual rules per project. Projects containing 20~49 rules are the most common (25, 30.1%), followed by those with 1~19 rules (19, 22.9%), 50~99 rules (16, 19.3%), and 100~199 rules (14, 16.9%). Similarly, projects maintaining a massive context of over 400 rules are scarce (3, 3.6%). These distributions suggest that developers generally prefer to author mid-scale (dozens to a few hundred), high-density core rule sets for their AI assistants, rather than heavily fragmenting rule configurations across numerous files. 4.5

Survey Demographics

Our survey captured a diverse global sample of software practitioners. The 99 valid responses came from 30 different countries and regions. The top represented countries include China (43, 43.4%), India (5, 5.1%), and the United States (5, 5.1%), followed by Germany (4, 4.0%), Indonesia (4, 4.0%), and Japan (4, 4.0%). A complete geographical distribution of the participants is presented in Figure 13. Figure 14 illustrates the participants’ educational backgrounds, professional software development experience, and the duration of their experience in configuring AI IDE rules. Regarding education, the vast majority of respondents (59, 59.6%) hold a Bachelor’s degree, while only 2 respondents hold a PhD. In terms of professional experience, the distribution is relatively even and leans toward mid-to-senior levels: those with 1–2 years, 3–5 years, 6–10 years, and 11–20 years of experience account for 25.3% (25), 20.2% (20), 21.2% (21), and 21.2% (21) of the sample, respectively. Beginners with less than 1 year (7) and veterans with over 20 years of experience (5) constitute a minority. Concerning the duration of rule file usage, the largest segment of respondents has been incorporating rules into their workflows for 6 months to one year (32), followed by 3 to 6 months (26). Notably, 19 respondents are early adopters with over one year of experience, whereas only 5 respondents have used rule features for less than a month. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

25

1 1 1

50

1

1

43

1 1

1 4

5

1 1

3

43 1

1

1

5

40

3

1 1

2

3 1

30

4

3

1 4

2

2

20

10 5 0

5

4

4

4

3

3

3

3

2

2

2

1

1

1

1

1

U

ed nit

1

a ia ia ny an ece orea nka nam ralia razil ines gypt onia rgia ana gary land t a e Ind meric rma ones Jap t o p B E st Gh Hun Ire Gr th K ri L Es Ge ilip Vie Au A Ge Ind S u of Ph s So ate St a

in Ch

1

1

ly

Ita

Ma

s lay

ia

1

1

1

1

1

1

1

1

1

s d o d e in re es dom nd olan xic po Spa ailan krain irat g ga P Me herla U Th Em Kin t Sin Ne rab nited dA U e it Un

Fig. 13. Overview of countries of survey participants. 11 20 years (21) 6 10 years (21) More than 20 years (5) 3 5 years (20) 1 2 years (25) Less than 1 year (7)

Bachelor's Degree (59) High School or below (13) Master's Degree (25) Doctoral Degree (PhD) (2) 2.0%

7.1%

25.3%

21.2%

Less than 1 month (5) 1 3 months (16) 3 6 months (26) 6 months 1 year (32) More than 1 year (19) 19.4%

5.1% 16.3%

25.3%

13.1%

59.6%

21.2% 20.2%

Education Experience

32.7%

26.5%

5.1%

Professional Experience

Rule File Usage Duration

Fig. 14. Overview of education, professional experience and rule file usage duration of survey participants.

Figure 15 details the participants’ team roles, their development domains utilizing AI IDEs, and the specific tools they have adopted (note that these three questions allow multiple selections). For team roles, a dominant 72.7% (72) of the participants act as Software Developers/Engineers. Additionally, a substantial portion hold leadership or architectural positions, comprising 40 Tech Leads/Engineering Managers and 39 System/Software Architects. Students/Researchers account for 21 respondents. Conversely, only 3 respondents are QA/Test Engineers, which may imply that testing professionals ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

26

Cai et al.

Professional Role (SQ4)

Project Domain (SQ5)

25

er Oth

ilot

igra

Cop

e+

vity

16

Ant

23

Cod

6

34 24

dsu rf Qod er

15

Win

17

51 37

Kiro

27

57

Tra e

38

Number of Selections

41

sor

er

eer ngin

Oth

3

42

VS

Soft

8

47

80 70 60 50 40 30 20 10 0

Cur

12

st E

15

/ Te

21

79

ronWeb A ten pp d/B lica ack tion ll-setnd/Fu(F ack App Des ) k Fram licatiotonp ewo Lib rary r k / Mob ile A SDK / ppli Lea AI cation rnin / Ma g c Infr / Agehine astr Clo nt uct ud Dat DevOupre / / Dat a Sc s a Gam Anaielynce / tics eD eve lopm ent Oth er

39

QA

40

80 70 60 50 40 30 20 10 0

Number of Selections

72

war eD / Enevelo gin per EnTgech L eer in ea Sys Maneaerindg / tem ger /S Arcohftwa itecre t ResStude earc nt her / Dev O Prod ps / SR E uct Ma/ nProje agect r

Number of Selections

80 70 60 50 40 30 20 10 0

AI IDE Tool (SQ6)

Fig. 15. Overview of roles in the team, project domains developed using the AI IDE, and AI IDE usage of survey participants.

currently engage with generative AI coding tools less extensively than core developers. The “Other” category (8 respondents) further broadens our demographic, including roles such as Creative Director, VFX Supervisor, Technical Solo Developer, UX Designer, and Founder. In terms of project domains, Web Application development is overwhelmingly prevalent (79), followed by Desktop Applications (47), Libraries/Frameworks/SDKs (42), Mobile Applications (41), and AI/Machine Learning/Agent (38). Cloud/DevOps/Infrastructure (27), Data Science/Data Analytics (17), and Game Development (15) were less frequently selected. The “Other” field includes specialized areas such as drone communication and IoT, robotics simulation, and technical prototyping. Regarding tool preferences, Cursor emerges as the most widely adopted AI IDE among our respondents (57), closely followed by the traditional VS Code + Copilot setup (51). The other four selected AI IDEs in our study—Kiro, Trae, Windsurf, and Qoder—were utilized by 37, 24, 23, and 16 participants, respectively. Furthermore, 34 respondents reported using Antigravity, while tools like Claude Code (10), Codex (7), and OpenCode (4) were specified under the “Other” option (the detailed responses for all “Other” options across these figures are available in our replication package [21]). 5 5.1

Study Results RQ1: What are the categories of rules in OSS projects developed by AI IDEs?

To answer RQ1, we present our findings in three parts. First, we introduce the hierarchical rule taxonomy, comprising 5 primary and 25 secondary categories derived from the mining data with the open coding process (Section 5.1.1). Next, we characterize this taxonomy by comparing the prevalence of rules in the repositories with the importance ratings from developers (Section 5.1.2). Finally, we report on the strategies developers employ to create these rule files in practice (Section 5.1.3). 5.1.1 Overview of the Rule Taxonomy. Through an open coding process on the collected rules, we constructed a hierarchical taxonomy comprising 5 primary categories ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

27

and 25 secondary categories. In this section, we describe each category and illustrate its application using a representative example extracted from the repositories. 1. Architecture & Design. This primary category encompasses constraints regarding the high-level structure, technology selection, and macro-design philosophies of the project. • Technology Stack Selections. Defines the core languages, frameworks, libraries, databases, and infrastructure for the project. Example: “The project uses the following technologies: TypeScript, Next.js (App Router), Supabase for Database & Auth, Tailwind CSS with shadcn/ui components...”. • Design Principles & Patterns. Specifies the software engineering principles (e.g., SOLID) and common design patterns adopted in the system. Example: “Apply SOLID principles, Clean Architecture, and Domain-Driven Design (DDD)”. • System Architecture. Describes the high-level structure and organizational relationship of the system. Example: “System architecture is Gateway → Kiosk → ModbusController → Hardware”. • Design References & Constraints. Specifies external documentation, best practices, or specific constraints that the design must adhere to. Example: “Reference official AWS documentation for implementation patterns”. 2. Code Implementation. This category focuses on concrete coding practices, guiding the AI IDE on how to write code at the file and function levels. • Framework Usage. Best practices and key considerations when utilizing specific frameworks. Example: “Always use the latest Vue 3 Composition API syntax”. • Code Style Conventions. Conventions for naming identifiers (e.g., variables, functions, classes) and overall code formatting. Example: “Use UPPER_SNAKE_CASE for constants”. • Performance Optimization. Guidelines aimed at improving code execution efficiency. Example: “Use prepared statements and indexes for query optimization”. • Language Features. Best practices and restrictions regarding specific programming language syntax and semantics. Example: “Channel Closing: Only close channels from the sender”. • Error & Exception Handling. Unified mechanisms for error reporting, handling, and recovery. Example: “Implement a global error handling mechanism using an ErrorBoundary component that integrates with a reporting service like Sentry”. • Business Logic. Defines entity structures, business rules (e.g., validation logic), and specific functional processes within a specific domain to guide the AI IDE in accurately translating requirements into code. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

28

Cai et al.

Example: “Registration with Email Verification: Users must verify email before account activation”. 3. Development Workflow & Project Management. This category addresses rules governing software lifecycle management, collaboration processes, and the organization of project assets. • Workflow Conventions. Standardized development processes (e.g., issue management, PR workflows, CI/CD pipelines). Example: “MANDATORY WORKFLOW BEFORE COMMITTING: # 1. Run tests first make test # 2. If tests pass, stage and commit git add . git commit -m ‘type(scope): description’”. • Project Documentation. Standards for writing and managing project documentation (e.g., API docs, user manuals). Example: “API Documentation must detail each endpoint, including URL, method, purpose, request/response formats, error codes, security, and rate limits”. • Directory Structure. Rules dictating directory hierarchy, file placement, and naming conventions for files/directories. Example: “One package per directory, main package in cmd/ ”. • Environment Configuration. Configuration rules for development environments, dependencies, and toolchains. Example: “Use environment variables for all API keys and configuration”. • Version Control. Conventions for code history management, versioning, and commit practices. Example: “Write descriptive commit messages following this format: First line: Concise summary (50 chars or less); Body: Detailed explanation if needed (wrap at 72 chars)”. • Dependency Management. Standards for importing libraries, resolving version dependencies, and package management. Example: “For external imports, always define a version. For example, npm:@express should be written as npm:[email protected]”. 4. Quality Assurance. This category encompasses standards and practices designed to ensure software reliability, security, and maintainability. • Testing Strategy. Guidelines on how to design, write, and execute software tests. Example: “Write unit, integration, and e2e tests as appropriate”. • Security Practices. Defensive coding practices aimed at preventing security vulnerabilities. Example: “Use parameterized queries, never concatenate SQL”. • Code Quality Standards. Specific metrics and requirements for measuring and enforcing code quality. Example: “Code quality metric thresholds should be met: Code coverage >80%, Cyclomatic complexity <10, Code duplication rate <5%, Test execution time <5 minutes”. ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

29

• Logging Standards. Requirements regarding the formats, severity levels, and locations for logging. Example: “Purposeful Logging: Every log entry must provide actionable value for operations or debugging”. • Code Review. Processes and standards for evaluating code, particularly when utilizing AI as a reviewer. Example: “Code Review Requirements: All API endpoints must have error handling”. 5. AI Collaboration Specifications. This category acts as “meta-controls” to regulate the inherent behavior, interaction style, and tool usage of the Large Language Models in AI IDEs. • AI Behavior & Decision Strategies. Controls the AI IDE’s actions in specific situations, such as intent recognition, response logic, and mandatory safety checks (e.g., stopping upon encountering errors). Example: “Always verify information before presenting it. Do not make assumptions or speculate without clear evidence”. • AI Output Content Guidelines. Standardizes the AI IDE’s response format during interactions, including language requirements, tone, depth of explanation, and structural layout. Example: “Minimal Output: Answer directly, avoid unnecessary preambles/postambles”. • AI Context Management. Defines the AI’s persona and guides how it should maintain and utilize the project context. Example: “To maximize the effectiveness of this brief, reference specific Markdown files or sections based on the code you need the AI to generate”. • AI Tool Usage. Specifies the tools available to the AI IDE and imposes constraints on their invocation. Example: “You HAVE TO use generateTheme tool to generate the theme, do NOT just output XML type text for tool-call, that is not allowed”. 5.1.2 Distribution and Importance Analysis of Rule Categories. Table 5 presents the quantitative distribution of the collected rules alongside the importance ratings provided by survey participants. Concentration Effect in Mining Repositories. The distribution of rules across categories is uneven. Code Implementation and Development Workflow & Project Management contain the highest numbers of rules, with 1,921 (26.28%) and 1,898 (25.96%) rules, respectively. In contrast, AI Collaboration Specifications comprises the fewest rules (840, 11.49%). Among the 25 secondary categories, Workflow Conventions is the largest individual category with 679 rules (9.29%), followed by Testing Strategy with 640 rules (8.76%). The distribution is also concentrated within specific primary categories. For instance, within Architecture & Design, the rules are concentrated in the top two subcategories: Technology Stack Selections (515, 7.05%) and Design Principles & Patterns (493, 6.74%). A similar pattern exists in Quality Assurance, which is primarily represented by Testing Strategy (640, 9.76%). Conversely, the smallest subcategories ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

30

Cai et al.

Table 5. Taxonomy of AI IDE Rules and Elicited Importance from Survey

Primary Category

Secondary Category

Rule Count(#, %)

Importance (Mean ± SD)

Technology Stack Selections 515, 7.05% 1356, 18.55% Design Principles & Patterns 493, 6.74% 4.16 ±0.98 System Architecture 195, 2.67% Design References & Constraints 153, 2.09% Code Implementation Framework Usage 448, 6.13% 1921, 26.28% Code Style Conventions 407, 5.57% 3.94 ±1.14 Performance Optimization 330, 4.51% Language Features 273, 3.73% Error & Exception Handling 255, 3.49% Business Logic 208, 2.85% Development Workflow & Workflow Conventions 679, 9.29% Project Management Project Documentation 497, 6.80% 1898, 25.96% Directory Structure 288, 3.94% 3.87 ±1.14 Environment Configuration 174, 2.38% Version Control 157, 2.15% Dependency Management 103, 1.41% Quality Assurance Testing Strategy 640, 8.76% 1295, 17.72% Security Practices 290, 3.97% ±1.14 3.79 Code Quality Standards 254, 3.47% Logging Standards 58, 0.79% Code Review 53, 0.73% AI Collaboration Specifications AI Behavior & Decision Strategies 414, 5.66% 840, 11.49% AI Output Content Guidelines 218, 2.98% 3.95 ±1.17 AI Context Management 129, 1.76% AI Tool Usage 79, 1.08%

4.15 ±0.98 4.06 ±0.99 4.25 ±0.99 4.19 ±0.94 4.03 ±0.95 3.82 ±1.30 3.78 ±1.21 3.89 ±1.18 4.00 ±1.03 4.14 ±1.07 3.78 ±1.13 4.12 ±0.98 4.02 ±1.12 3.73 ±1.23 3.75 ±1.20 3.80 ±1.11 3.83 ±1.08 3.89 ±1.14 3.93 ±1.08 3.65 ±1.17 3.65 ±1.20 3.99 ±1.06 3.61 ±1.31 4.17 ±1.05 4.03 ±1.18

Architecture & Design

1 (Least Important)

2

3

4

5 (Most Important)

across the taxonomy include AI Tool Usage (1.08%), Logging Standards (0.79%), and Code Review (0.73%). Consensus and Divergence in Survey Perspectives. The survey results show that developers prioritize structural and high-level constraints. Architecture & Design receives the highest overall importance rating (4.16), whereas Quality Assurance— despite its high rule count—is rated the lowest (3.79). At the subcategory level, System Architecture (4.25), Design References & Constraints (4.19), and AI Context Management (4.17) receive the highest scores. Meanwhile, AI Output Content Guidelines (3.61) and Logging Standards (3.65) receive the lowest ratings. Furthermore, a Spearman’s rank correlation test [55] between the mean importance scores and their standard deviations across the 25 subcategories reveals a statistically significant negative correlation (𝜌 ≈ −0.82, 𝑝 < 0.001). This indicates a higher consensus (i.e., lower variance) among developers regarding the most important rules, though this result may be partially influenced by the ceiling effect of the 1~5 Likert scale, where average scores approaching the maximum limit inherently exhibit constrained variance. Triangulation Analysis via Scatter and Quartiles. To further explore the relationship between the prevalence of rules and developers’ perceptions, we mapped the 25 subcategories onto a scatter plot (Figure 16) with Rule Count on the X-axis and Importance Mean Score on the Y-axis. We performed an Ordinary Least Squares ACM Trans. Softw. Eng. Methodol., Vol. 1, No. 1, Article . Publication date: June 2026.

Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study

31

Related documents

• The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability#978496
• Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction#978497
• The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs#978498
• DeltaSelect: Affordable A/B Testing for Coding Agents#978499
• Code-as-Auditor: Executable Compliance Reasoning via Regulation-to-Code#978500