Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures in Third-Party Content Management System Plugins Sebastian Neef
arXiv:2609.17164v1 [cs.CR] 15 Sep 2026
Technische Universität Berlin Berlin, Germany [email protected]
Abstract
1
Content Management Systems (CMS), such as WordPress, power a large share of the web (∼58%), and their extensibility through third-party plugins is a major source of their popularity as well as of their attack surface. One high-impact weakness that remains understudied is log file exposure by CMS plugins, which create log files for debugging or other purposes. If these files are insufficiently secured, they can disclose sensitive information (e.g. credentials, personal data) which has led to website compromises in the past. In this work, we present an agentic, LLM-based framework that automatically detects potential log file exposures in plugins of the most popular CMS (WordPress). Our agent analyzes each plugin by performing static and dynamic analysis. We evaluated our approach on the 300 most-installed WordPress plugins (about 0.6% of all), which together account for over 250M active installations, i.e. 75% of all active installations in the official plugin ecosystem. We manually validated each finding, reproducing 79 of 81 findings from 62 plugins. We observed that several protective measures appear to be implemented that we classify as creation-control (e.g. manual log activation) and access-control (e.g. deny rules in .htaccess). However, we find that multi-layered protection is required, but not always present. From these results we derive a taxonomy of log file path and protection patterns and deduce a set of best practices for developers to securely handle them. Finally, our study corroborates that agentic LLMs are an useful tool for security analysis.
The internet has become a crucial part for the lives of almost 75% of the world’s population [13], e.g. allowing them to exchange information online or do online-shopping. Since creating websites can be a complex task, Content Management Systems (CMS) aim to reduce that barrier to entry by offering ready-to-use and extensible frameworks that allow the user to focus on the content rather than technical aspects. For example, the most popular CMS is WordPress1 which covers almost 60% of all CMS-based websites and over 41.5% of all websites according to W3Techs [41]. One plausible reason for the popularity of CMS’ is the feature-richness and extensibility achieved through plugin systems, which allow users to install thirdparty developed extensions. Although plugin submissions undergo review for popular CMS [14, 42, 44], many security issues are being discovered in such plugins [32, 48]. One potentially severe security issue arises when plugins create and fail to protect log files with sensitive content from being accessed by unauthorized actors, which could lead to the compromise of the website. For example, the easy-wp-smtp plugin leaked the admin account’s password reset token in a log file [3], and many other plugins were found to have exposed log files in an industry study in 2020 [27]. Recently, Large-Language Models (LLMs) have shown to be a promising tool to identify security issues and vulnerabilities in software [2, 4, 5, 8, 12, 23], however their application to the identification of log file exposure remains yet to be explored in an academic setting. Thus, this work extends the body of literature with the following contributions:
CCS Concepts • Security and privacy → Web application security; Vulnerability scanners; • Computing methodologies → Artificial intelligence; • General and reference → Measurement.
• An agentic LLM-based log file exposure detection framework that answers RQ1: How prevalent are log file exposures in WordPress plugins? • An extended taxonomy of common patterns for log file exposures based on the answer to RQ2: What log file paths and protection mechanisms patterns can be identified in popular WordPress plugins? • Best practices for developers derived from the previous results answering RQ3: How to securely create log files in WordPress and other CMS plugins?
Keywords Web Security, Large Language Models, Agentic Vulnerability Detection, Log File Exposures, WordPress Plugins ACM Reference Format: Sebastian Neef. 2026. Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures in Third-Party Content Management System Plugins. In 19th Workshop on Artificial Intelligence and Security (AISec ’26), November 15–19, 2026, The Hague, Netherlands. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3847352.3848121
This work is licensed under a Creative Commons Attribution 4.0 International License. AISec ’26, The Hague, Netherlands © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-3031-3/2026/11 https://doi.org/10.1145/3847352.3848121
Introduction
2
Background and Related Work
This section will provide the reader with the necessary background on this topic and bring this work into the context of the existing body of literature. 1 https://wordpress.org/, accessed: 2026-09-15
AISec ’26, November 15–19, 2026, The Hague, Netherlands
2.1
Content Management Systems and Third-Party Plugins
Many different Content Management Systems (CMS) exist to help non-technical users create their own website by abstracting away from technical aspects. Some CMS are operated and hosted by corporate entities and available on a subscription basis, e.g. Shopify2 , Wix3 , while others are open-source and can be self-hosted, e.g. WordPress, Joomla4 . Thus, CMS are a cost-effective way for a wide range of users, from beginners to professionals, to develop websites. According to W3Techs, CMS are widely adopted [41]: The top 10 CMS cover over 58% of all internet websites, with WordPress, Shopify, Wix, Squarespace, and Joomla being the most popular ones. With a market share of about 60%, the open-source CMS WordPress is over 50% ahead of its competitors. CMS can be specialized on specific use-cases, e.g. e-commerce, blogs, or website building, but often their plugin and extension systems allow users to extend the features or functionality by installing third-party plugins. For example, Woocommerce5 turns the WordPress blog system into an online-shop, or Elementor6 into a drag-and-drop website builder. In fact, the landscape of plugins is huge: The official WordPress plugin repository counts over 60,000 plugins with several reaching over 10M active installations [47]. However, since anyone can develop and submit their own plugin to the respective plugin stores, the code quality and security of such plugins may vary, despite submission reviews. Several plugins have evolved into depending on other plugins for their functionality, which can complicate the plugin update process [22]. Unsurprisingly, a dedicated project emerged to track vulnerabilities in WordPress and its plugins or themes: The WPScan vulnerability database contains over 73,000 reported vulnerabilities [48]. With its openness, popularity, and rich plugin ecosystem, WordPress has been used and studied in prior academic work. Singh found WordPress to be the most accessible CMS for academic staff, compared to Drupal and Joomla [39]. However, the low barrier to entry might lead to users not maintaining their WordPress instance correctly. A study from 2023 found that WordPress websites are not always up to date and can lack important security patches [6]. Although vulnerabilities in WordPress itself exist, installed plugins impose a larger attack surface [15]. In 2025, 91% of the vulnerabilities were found in plugins, 9% in themes and only 6 issues in WordPress [34]. In fact, installing plugins or blindly updating them is a security risk, as plugins can become malicious or contain malware, as shown by Kasturi et al. [16]. According to a study by Koskinen et al. in 2012, there is no clear correlation between the security of a plugin and its rating [17]. In 2019, however, Ruohonen found that from a demand-side viewpoint, a higher install count is statistically associated with a higher number of vulnerabilities [36]. Testing installed plugins for vulnerabilities can give a false sense of security, as Murphy et al. discovered: They examined 11 WordPress vulnerability scanner plugins and found that none identified all vulnerable plugins [25]. 2 https://www.shopify.com/, accessed: 2026-09-15 3 https://www.wix.com/, accessed: 2026-09-15
4 https://www.joomla.org/, accessed: 2026-09-15
5 https://woocommerce.com/, accessed: 2026-09-15
6 https://wordpress.org/plugins/elementor/, accessed: 2026-09-15
S. Neef
Furthermore, WordPress and its plugins have been used for evaluations in academic work, e.g. [28, 38, 40]. Niemietz et al. analyzed the top 100 Joomla-CMS plugins for vulnerabilities [30], but do not consider exposed log files. Outside academia, large-scale vulnerability analysis of CMS extensions has been pursued as well. For example, wpgarlic7 is a fuzzer for WordPress plugin that has uncovered a large number of vulnerabilities, although not directly supporting the identification of exposed logs. Therefore, the exposure of log files by Content Management Systems plugins remains to be studied, and we focus on WordPress as it is the CMS with the largest market share.
2.2
Log File Exposure in WordPress Plugins
WordPress is developed in PHP and, thus, consists of several .php files that are placed in a web server’s DocumentRoot, which, by default, makes these files accessible. While correctly configured web servers will pass requests targeting .php files to the PHP engine for processing, other file extensions (e.g. .txt or .log) might be served by the web server directly. However, in order for WordPress to provide its full functionality, such as uploading documents or images, or installing and updating plugins, some of its subfolders need to be writable by the web server. Plugin developers might want to provide their user-base with precise and helpful assistance should unexpected problems or incompatibilities with other plugins occur. For these or other purposes, a plugin might want to write (debug) log files, which requires a writable folder. As established earlier, the wp-content/, uploads/, or the plugin’s own folder has a high chance to exist and be writable. Other folder locations might not be writable if the WordPress was hardened, e.g. using the official guide [45]. Thus, some log files end up in the web server’s DocumentRoot and their contents may become accessible if not protected. Not all plugins succeed in sufficiently protecting the created log files, as Neef showed in an industry-study in 2020 [27]. A handful of popular plugins with over 1M active installations were exposing log files with sensitive information. While not all exposures can be attributed to a plugin, their developer might have made assumptions about the WordPress instance’s system or web server configuration. For example, a .htaccess file can be used to block access to specific files or directories, but it is not supported by all web servers: Apache8 supports it, while NGINX or others do not [7], voiding this protection mechanism. Another example is using a random or secret value in file names, but not preventing directory listing in the folder which exposes the file. The log file contents are determined by each plugin and, thus, can range from irrelevant to sensitive. While error messages, stack traces, and directory paths can be helpful for attackers as additional information to prepare attacks, the exposure of usernames, credentials, or secrets could pose an immediate threat to the availability, integrity, or confidentiality of the website (e.g. [3, 37, 46]). More recently, a log file disclosure in the Post SMTP plugin put 400,000 websites at risk of admin account takeover in 2025 [26]. In general, such information disclosures fall into the A01:2025 Broken Access Control and A09:2025 - Security Logging and Alerting 7 https://github.com/kazet/wpgarlic/, accessed: 2026-09-15 8 https://httpd.apache.org/, accessed: 2026-09-15
Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures
AISec ’26, November 15–19, 2026, The Hague, Netherlands
2.3
Large Language Models for Plugin Vulnerability Detection
Traditional static application security testing (SAST) tools such as Semgrep10 , Psalm11 , or RIPS12 predate LLMs, but these do not currently come with detection patterns for exposed log files. Writing and defining the rules and patterns to detect exposures with these tools requires substantial engineering efforts and understanding of their capabilities. With LLMs and their reasoning capabilities, the bar is lowered to providing an analysis environment and a prompt. Using Large Language Models (LLMs) for vulnerability detection (e.g. [5, 12, 23]), validation (e.g. [10, 21, 31]), explanation (e.g. [9, 19]), or remediation (e.g. [1, 18]) is a trending and promising area of research at the time of writing. Risse et al. argue that function-level security analysis does not provide enough context to reliably determine a vulnerability [35]. Instead, repository-level access [11, 50] and the use of agents [49] can lead to higher detection rates. Thus, our work also follows an agent-based analysis and adopts the repository-level context idea by providing a plugin’s complete source code for analysis. However, many of these publications focus on software applicationrelated programming languages. The usage of LLMs in the context of web applications, especially Content Management Systems such as WordPress, has not been widely studied yet, further motivating this work. Cao et al. examined the use of LLMs for PHP-based web applications and improved the detection capabilities [4]. In a preliminary study, Ng et al. used WordPress as a target to discover that LLMs can be used to detect broken access control based on web paths [29]. Fang et al. showed that LLM agents can autonomously exploit 1-day web vulnerabilities based on CVEs, including WordPress [8]. Afterwards, CVE Bench was proposed as a benchmark to test LLM agents’ abilities, which also features WordPress plugin vulnerabilities [51]. More recently, Leng et al. proposed LLM4Patch to help identify vulnerability patches in commit messages of WordPress plugins [20]. Thus, our work aims to further close this gap by providing insights on how LLMs can help detect potentially severe security issues in CMS plugins.
Cumulative active installs (%)
Failures categories of the OWASP Top 10 of the most common security issues [33]. The Common Weakness Enumeration9 (CWE) tracks these as CWE-200: Exposure of Sensitive Information to an Unauthorized Actor, CWE-215: Insertion of Sensitive Information Into Debugging Code, and CWE-532: Insertion of Sensitive Information into Log File. If personal identifiable information (PII) is disclosed, CWE359: Exposure of Private Personal Information to an Unauthorized Actor becomes applicable. According to the work of Mesa et al. [24], information exposure is among the top 10 most common vulnerabilities in WordPress plugins, which further motivates this work and its agentic LLMbased approach to potentially identify more of such issues.
100
75
50 cutoff: top 300 plugins (0.6% of plugins) → 75% of installs
25
0
1 10 100 1k 10k Number of Plugins (ranked by active installs)
Figure 1: Active install count coverage of all unique WordPress plugins from the official plugin store (accessed: March 2026). The red marker depicts our selected top 300 plugins covering 75% of the total active installation count.
3
Methodology
This section introduces our agentic analysis framework with which we systematically identify potential log file exposures in WordPress plugins. Our methodology comprises three major steps: plugin selection, agentic plugin analysis, and manual validation.
3.1
CMS and Plugin Selection
Similar to related work (Section 2), we focus on the WordPress ecosystem as a representative for other CMS and their comparable plugin systems, as it has almost 60% market share and a gap of over 50% to its nearest competitor. The plugin selection for our analysis was influenced by the conclusion of Ruohonen [36] that more popular plugins can be of higher interest to attackers. Thus, we retrieved the full plugin catalog from WordPress’ plugin system API13 and sorted the unique plugins by the highest active installation count. As shown in Figure 1, we selected the top 300 plugins (0.6% of all available plugins), covering over 75% (250M) of the cumulative active installations, providing broad real-world coverage while keeping the manual validation workload acceptable. The most popular plugins in our selection have over 10M active installations each, with the lowest being over 100,000. The full plugin list is available in our repository (see Section 5.6).
3.2
Agentic LLM-based Analysis Framework
9 https://cwe.mitre.org/, accessed: 2026-09-15
As outlined in Section 2, we decided to implement an agentic LLMbased framework for our potential log file exposure analysis of the top 300 selected WordPress plugins. All data and code will be open-sourced to foster future work and allow developers to test their own plugins as described in Section 5.6. For the LLM, we chose Anthropic’s Claude Opus 4.614 , as this was the most recent frontier and SOTA model available at the time of analysis, with its own claude-code harness15 in –dangerouslyskip-permissions mode and the 20x MAX subscription plan, which was cheaper than API-based usage (see Section 5.4). The main
plugins/security/audit, accessed: 2026-09-15
13 https://api.wordpress.org/plugins/info/1.2/, accessed: 2026-09-15
12 https://github.com/robocoder/rips-scanner, accessed: 2026-09-15
15 https://claude.com/product/claude-code, accessed: 2026-09-15
10 https://github.com/semgrep/semgrep- rules/tree/develop/php/wordpress11 https://psalm.dev/docs/, accessed: 2026-09-14
14 https://www.anthropic.com/news/claude-opus-4-6, accessed: 2026-09-15
AISec ’26, November 15–19, 2026, The Hague, Netherlands
S. Neef
For each plugin in the selected top 300: Stage 2: Analysis with Claude LLM Agent Stage 1: Fresh WordPress Environment with Plugin
Phase 1: Static Analysis
yes
file-based logging found? no (skip)
Phase 2: Protection Analysis Phase 3: Dynamic Analysis
Stage 3: Tear down WordPress Environment
Markdown report generation
Figure 2: The three analysis stages of our agentic framework to detect potential log file exposures consisting of a setup and teardown stage, as well as an agentic analysis stage with static, protection, dynamic analysis phases. prompt is in Section A.1 and the environment used WordPress (6.9.4) on Apache. All other technical details and data are included in our repository (Section 5.6). Figure 2 visualizes the three implemented stages in our agentic framework. Using a coordinator python script, we ran the agentic framework against each plugin. While stage 1 and stage 3 were responsible to set up and tear down a fresh analysis environment, stage 2 is where the LLM-based static and dynamic analysis happened in three phases. 3.2.1 Stage 1 & 3: Setup and Tear down of the Analysis Environment. For each analysis, stage 1 and stage 3 created an ephemeral Apachebased WordPress instance inside a Docker container. Stage 1 started an initialization script that configured a standardized multi-site WordPress page16 , created a set of predefined accounts (one for each WordPress role), and installed the target plugin. After the stage 2 analysis, stage 3 was responsible to tear down all Docker containers, so that the next analysis would start over with a fresh and clean instance. This design choice was deliberately made to ensure only one plugin (and its required dependencies) was analyzed at once to prevent cross-plugin interference. Furthermore, it also ensured that potential log file exposures could be attributed to the tested plugin, and the agent could focus on a single plugin. 3.2.2 Stage 2: Agentic Analysis. For the agentic analysis, we provided claude-code a main prompt (see Listing 1 in Appendix Section A.1) stating the analysis objectives. The main prompt referenced two additional prompt files, PROMPT.md and CLAUDE.md, which provided further in-depth information about the evaluation environment, the analysis phases and how the markdown report was supposed to be generated. All prompts are available in our repository (see Section 5.6). Phase 1: Static Analysis. The LLM-agent was instructed to extract the installed plugin’s source code from the docker container for analysis. Next, it should examine the source code for file-based logging using common file-writing code paths as guidance: 16 https://github.com/Automattic/wpscan- vulnerability- test- bench, accessed:
2026-09-15
• Direct file writes: file_put_contents, fopen, fwrite, fputs. • PHP error logging: error_log calls with an explicit file destination. • Logger abstractions: method calls such as ->log(), ->debug(), ->error(); usage of Monolog or WooCommerce’s WC_Logger API, etc. • Path construction: expressions involving WP_CONTENT_DIR, ABSPATH, wp_upload_dir(),plugin_dir_path(), or __DIR__. For each identified log-writing code path, it should trace the call chain and identify additional properties, such as the full file path, triggers for writing the file, the data that is written, and more. We focus on error_log calls with an explicit file destination, as PHP’s error log resides outside the DocumentRoot by default. If no log file paths were identified, the next two phases were to be skipped. Phase 2: Protection Analysis. For each identified log path, the agent should determine if (and what) protections are implemented to prevent access to the log file. For example, it should check for .htaccess, index.php, index.html, randomized file paths, or other protective measures. Phase 3: Dynamic Analysis. In this phase, the agent was instructed to actively trigger the log file creation for each identified log file and observe the log file’s creation in the running WordPress instance, as well as its accessibility over HTTP and the effectiveness of the implemented protections. In particular, we provided the agent with pre-registered accounts with different roles and access to the web-interface of the WordPress instance as well as access to the WordPress container, so it could freely interact with the instance and validate log file exposures over HTTP or the file system. Report Generation. Finally, for each plugin the framework was instructed to generate a structured markdown report in the findings/ folder. The coordinator script also created a copy of claude-code’s commandline output in output/. The markdown reports contain information about the LLM’s analysis and findings about each potentially exposed log file. These reports were the basis for the manual validation and our results.
Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures
AISec ’26, November 15–19, 2026, The Hague, Netherlands
Table 1: Overview of the analysis success and failure rates. Metric
Count
Top WordPress plugins selected
300
Agentic analysis: Analysis failures (e.g. timeout) Successfully analyzed Plugins without identified file logging Plugins with identified file logging Potential log file exposures identified Manually successfully validated log files Log file findings with minor inaccuracies
13 287 225 62 81 79 2
AST-based analysis: Identified plugins with potential file logging Also identified by agentic approach Not identified by agentic approach Manually validated false positives Manually confirmed file logging instances Timeout in agentic approach True agentic based false negative
165 58 107 100 7 6 1
Not identified by AST, but by agentic approach
4
3.3
Manual Validation
Since LLMs are inherently prone to hallucinations, i.e. reporting non-existing or invalid issues, we deliberately decided to manually review all generated vulnerability reports and their log file path findings, instead of using another Judge LLM or agent. This means, that for each plugin and potential log file exposure, we took the LLMproduced report and instantiated the stage 1 analysis environment to reproduce and validate the log file exposure information. Based on the validated information, we populated an analysis.csv file tracking all exposure properties, such as log file paths, protection mechanisms, and comments about the findings validity, for each plugin and log file. Additionally, we implemented a deliberately permissive ASTbased scanner as a baseline to help detect false negatives, which we discuss in Section 5.4.
4
Results
In this section, we first report the aggregate outcome of the analysis. We then introduce a taxonomy that structures the observed log file paths and protection mechanisms, and finally apply this taxonomy to quantify how the patterns are distributed and combined in practice.
4.1
Log File Exposure Analysis
Table 1 contains the counts for the agentic analysis and our ASTpowered baseline. 4.1.1 Agentic Analysis. Of the 300 plugins in our selection, 13 did not produce a finding (see Section 5.5), leaving 287 analyzed plugins with generated findings. For the vast majority (225 plugins, 78%), no file-based logging code paths could be identified and, thus, no dynamic analysis was performed. The remaining 22% (62 plugins) created one or more log file, resulting in a total of 81 potential log file exposures.
Our manual validation showed that the LLM-generated log file findings in the reports were valid in 79 cases, where valid means that we could reproduce and verify the information provided in the finding. Only in 2 cases, there were slight discrepancies, e.g. an index.php present that we could not confirm, or a different log location. The precision of our approach is 98% with 79/81 identified log files manually validated, and a recall of 98% (62/63) for the successfully analyzed plugins by our framework. 4.1.2 AST-based Analysis. For comparison, we built a permissive, non-LLM AST-parsing script to look for the logging guidance patterns (Section 3.2) in the plugins’ first-party source (excluding imported dependencies) as a baseline. While the script has no dataflow capabilities to determine where or how a log file is written, it flagged 165/300 plugins. 58 of these our framework identified as well, and the other 107 were manually reviewed: The script overreported the majority due to normal file-write patterns, error_log without a file destination, or other factors. In fact, only 7 plugins were manually confirmed to write log files. Six of these were not identified by our framework because the analysis timed out, and 1 plugin (ninja-forms), which used database-logging and file-logging in one specific file, was missed by the farmework despite completing. Conversely, the AST-baseline missed 4 plugins which the agentic approach discovered. Thus, over all known log-writing plugins, including the findings from the baseline, the recall of our framework drops to 90% (62/69).
4.2
Taxonomy of Log File Paths and Protection Mechanisms
In order to create the basis for answering our research questions, we first distill our findings into a taxonomy on two axes: how the file is named and located (path) and how it is shielded from unauthorized access (protection). Table 2 summarizes the resulting categories, which are an extension of [27]. Our categories are not mutually exclusive. For example, a log file can combine several path features and protection mechanisms, e.g. wp-content/uploads/plugin-< secret>-<YYYY-MM-DD>-log.php. Path categories. We group the path categories into two securityrelevant classes. A log file path is predictable when it is static or date-based, as the number of years, months or days are feasible to enumerate. It is obscured when it contains a hash or secret value unknown to an attacker. Therefore, we deem a log file guessable only if predictable and without obscured features. Protection categories. The protection categories fall into two distinct groups. Access-control mechanisms (PHP stub, index file, .htaccess, directory listing) try to limit access after a log file was created. Creation-control mechanisms (PHP constant, manual change) define conditions before a log file is created. Thus, creation-control prevents a log file from being created by default, but it does not necessarily protect it from being accessible once created.
AISec ’26, November 15–19, 2026, The Hague, Netherlands
S. Neef
Table 2: Taxonomy of log file paths and protection mechanisms. Axis
Category
Description and security relevance
Path (naming / location)
Static Date-based Hash-based Secret-based error_log Other
A hardcoded path; known to an attacker Contains a date (e.g. YYYY-MM-DD); feasibly guessable Contains a hash value; unguessable Contains a hardly guessable value (e.g. random name) Logs to PHP’s error_log Path feature not captured by the above
Protection (access / creation)
PHP stub Index file .htaccess Dir.-listing off PHP constant Manual change Other None
e.g. <?php exit; ?> and .php suffix prevents output e.g. index.php or index.html; prevents directory listing Apache-specific configuration file with access control rules Directory listing disabled (via index file or .htaccess) Special PHP constant (e.g. WP_DEBUG) needs to be defined Manual setting or code changes prior to log creation A different protection mechanism, e.g. outside the DocumentRoot No protection mechanism
Date-based
12 10
30 (37%)
Static
15
Hash-based
25 (31%)
Hash-based
8
0
6 22 (27%)
Secret-based
4 20 (25%)
Date-based
4
Secret-based
5
0
4 (5%)
25
30
35
(a) Prevalence of each path feature.
d se et
-b a
se hba as
cr
15 20 Log files (N=81)
Se
10
H
at 5
D
0
d
d se ba
2 (2%)
Other
2 0
Predictable Obscured Neither
e-
error_log
Log files with both features
14
0
(b) Pairwise co-occurrence of date-, hash-, secret-based path features.
Figure 3: Path feature distribution and co-occurrence across the 81 log files. In (b), only co-occuring features are shown as the remaining only occur in isolation.
4.3
Identified Log File Exposure Patterns
We now apply the taxonomy to the 81 potential log file exposures. 4.3.1 Path patterns. Figure 3(a) shows the per-feature prevalences for the 81 log files. Static paths are most common (37%), followed by hash-based (31%), secret-based (27%), and date-based (25%) features. PHP error_log occurs in four cases, and other in two times. In practice, a path can have multiple features that complement each other, so a path with a predictable feature is still not guessable if it also contain an obscurring feature. Figure 3(b) shows which path features co-occur. By definition, only hash, secrets, and date can cooccur within a path. Only two log files have all three features, and except for 3 log files, the date feature always co-occurs with a hash or secret feature in our dataset, and the most common combination is date+hash.
4.3.2 Protection patterns. Figure 4(a) shows the prevalence of each protection mechanism. The most common protection is creationcontrol, i.e. the log files are only created after a manual change (69%) or a PHP constant is defined (23%). Disabling directory listing (64%) is the most common access-control mechanism, followed by index files (54%), and .htaccess (41%). The more robust PHP stubs are only used in 8 cases. From the Others category, 4 cases are PHP’s default error_log() and 1 log file whose name can be chosen by the user and the plugin ensuring it is not within the DocumentRoot. Only two files have no protection, but one file is written outside the DocumentRoot and the other to a user-supplied location (likely to be outside the DocumentRoot), thus remain unexposed. Figure 4(b) shows that access-control features are typically stacked together. The most common number is 3 stacks (24 times), followed by four
Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures
AISec ’26, November 15–19, 2026, The Hague, Netherlands
56 (69%)
Manual change
52 (64%)
Dir.-listing off
Log files
13
44 (54%)
Index file
13
Mechanisms stacked 10
9 9
1 layer (19) 2 layers (12) 7 3 layers (24) 4 layers (18) 4 3 3 3 3 5 layers (6) 2 2 2 2 1 1 1 1 1 1 1
0
Manual change
33 (41%)
.htaccess
Dir.-listing off
19 (23%)
PHP constant
Index file
8 (10%)
PHP stub
2 (2%)
None
0
10
.htaccess
Access-control Creation-control Other, e.g. outside DocumentRoot No protection
5 (6%)
Other
20
30 40 Log files (N=81)
50
PHP constant PHP stub
60
(a) Prevalence of each protection mechanism.
Other
(b) Number of stacked and co-occurring protection mechanisms.
Figure 4: Protection mechanism distribution and co-occurrence across all log files. The two log files with no protection (0 layers) are omitted in (b). stacks (18 times). Only 6 log files are covered with 5 protection mechanisms, but 19 rely on only one.
4.4
Strongest and Weakest Configurations
The strongest configurations emerge when creation-control and access-control are both present and cover many categories. Figure 4(b) shows that using multiple protection mechanisms is the norm. On the other hand, 13 log files rely on the single manual change protection, which can be sufficient depending on the change required (e.g. code-changes are a higher barrier than toggling a UI switch). However, some code-changes can be performed by administrators through the built-in code-editor. Figure 5 shows the relationship between path features and relevant protection mechanisms. A guessable path must, by definition, be assumed to be known to an attacker, but access can be prevented with an .htaccess deny rule or a PHP stub. Disabled directory listing will not protect such files. On the other hand, a properly obscured path can become accessible (given no other access control) if directory listing is enabled, thereby revealing the unknown values. From this perspective, 22 of the 33 guessable files and 8 of the 42 obscured files are at risk of exposure, while the remaining files are shielded by access controls. However, taking the creation-control into account for the 22 files, it actually brings the number down to zero, as no log file is exposed by default. Only two of the 8 obscured log files have no creation-control and might become exposed, if their filenames are revealed.
5
Discussion
In this section we discuss our results with regard to our research questions, limitations, future work, and ethical considerations.
5.1
RQ1: Prevalence of Log File Exposures in WordPress Plugins
Our results show that file-based logging is common in the top 300 plugins of the most popular CMS (WordPress), and several
exposed unless .htaccess / PHP-stub Guessable (static / date)
22
11
exposed unless .htaccess / PHP-stub / index file / dir.-listing off Obscured (hash / 2 secret)
6
34
written outside web root Neither (other / error_log)
Exposed without creation-control Exposed, but creation-controlled Protected with access-control Out of scope (not retrievable)
6
0
10
20
30 Log files (N=81)
40
50
Figure 5: The relation between path features and protection mechanisms. potential log file exposures were identified among our framework’s findings, but none are directly at risk of exploitation. Only roughly every fifth (62 of 287) successfully analyzed plugin created at least one log file (81 in total), but these are guarded by one or several protective mechanisms. Although not directly comparable due to methodological and dataset differences, our results appear different from the industry study in 2020 [27], where multiple unprotected log file exposures were identified. One explanation could be that awareness of log file exposures and defense-in-depth measures could have increased among developers of the top plugins over the last six years, but more research is needed and left to future work.
5.2
RQ2: Log File Path and Protection Patterns
To our surprise, the most common safeguard is not access-control, but creation-control. 69% of the 81 log files (Section 4.3) are only created when the plugin-user manually enables related settings (e.g. “enable debug log” toggle) in the UI, or edits the code to enable logging functionality (e.g. by defining a constant such as WP_DEBUG or WP_DEBUG_LOG). For 19 of the 81 log files (23%) the required change is defining a special PHP constant. Thus, on a default installation
AISec ’26, November 15–19, 2026, The Hague, Netherlands
without deliberate changes, these log files will never be created and, thus, are likely only used by developers. It also explains why for the 22 guessable log files without sufficient protective access-control mechanisms, none are immediately at risk of exploitation. However, whether a log file can be accessed by unauthorized attackers also depends on the server’s configuration. If directory listing is globally enabled in the server’s configuration, obscured log file paths become discoverable and accessible if no additional access-control is in place. Similarly, the index file names can be changed globally in the web server configuration, breaking the assumption about index.php or index.html. Also, if the web server does not support .htaccess configuration files, e.g. NGINX, then that access-control protection becomes ineffective for 41% of the observed log files. Also, web administrators might choose to disallow PHP-execution of files in folders with user-supplied content, e.g. wp-content/uploads/, to mitigate attacks (e.g. remote code execution) which would void the PHP-stub protection. Therefore, we believe that some plugin developers make assumptions about the web server or its configuration (i.e. Apache with .htaccess support) on which their plugin gets installed. For end-users with varying degrees of technical knowledge, to whom WordPress caters, it might not be obvious if, when or how a plugin creates log files, what is being logged, and how these files are being protected. This creates a risk to end-users that an installation or usage of a plugin might accidentally put their website at risk of compromise, as the past has shown ([3, 26]).
5.3
RQ3: Best Practices for Secure Log Files in CMS plugins
From our observations we aim to derive the following best practices for developers of plugins and CMS in handling log files. 5.3.1 Avoid log files in the DocumentRoot. The most robust mitigation against log file exposures is not creating them in the webaccessible DocumentRoot in the first place. Temporary log files used for debugging could be written to ephemeral and non-accessible storage outside the DocumentRoot, such as /tmp/ on Linux servers. Alternatively, as some of our analyzed plugins already offer, the log contents should be written to the CMS’ database to avoid disclosure. 5.3.2 Layered Defense-in-depth. If file-based logging is unavoidable, it is crucial to implement multiple protective mechanisms. We recommend combining all of the following to cover several failure modes: • A PHP-stub and .php-suffix to prevent disclosure on direct access. • An index file to suppress directory listing in the log file’s folder. • An .htaccess file to deny access on Apache-based web servers. • An obscured file path with sufficiently random or secret values so that the URL cannot be guessed. Also, logging should be made visible to the end-users to make them aware of the potential risk of exposed log files. The plugins should provide appropriate UI views where all created log files are being listed with an option for deletion.
S. Neef
5.3.3 CMS-level Logging API. Finally, CMS should implement and offer appropriate logging APIs that will allow plugin developers to create log files managed and secured by the CMS itself. That way, developers would not need to implement log file management themselves, which should eliminate the risk of log file exposure. Also, end-users could benefit by having all logs in a central and reviewable place.
5.4
Use of LLM-Agents for Log File Exposure Detection
Our choice to use LLMs for this analysis was motivated by several factors. First, log file exposures can become the result of complex code paths (e.g. final paths assembled from constants, wrapper functions, or from options read from the database), potentially requiring complex detection rules for SAST tools. At the time of writing, SAST tools such as Semgrep or Psalm did not implement rules for log file exposure detection, excluding them from a comparison. Second, a log file write does not automatically equal exposure, as the different protection mechanisms can be implemented which need to be evaluated at runtime (e.g. .htaccess), thus requiring dynamic analysis capabilities. With its tool-calling capabilities and the WordPress environment, the LLM was able to interact with the instance to verify the findings. Lastly, LLMs allow for natural language descriptions of the goal, rather than implementing and maintaining SAST-engine specific rules for each logging idiom (e.g. direct writes, WC_Logger, custom loggers, etc.). Nonetheless, we believe another promising approach could be the use of LLMs to generate such rules, or use SAST-tools in combination with agentic workflows, which we leave to future work. We found the generated reports to be precise, as we were able to reproduce 79 of 81 reported log file findings during manual validation (see Section 3.3). The descriptions of the relevant codepaths and triggers of the log file creation were deemed very helpful. In fact, it greatly facilitated and accelerated the code-review and reproduction of the finding. Giving the LLM-agent full access to each plugin’s source code allowed it to analyze and trace through the execution flow to understand complex code-paths. Combined with the multi-phase static and dynamic analysis conducted by the LLM-agent, we believe it helped to verify and produce more accurate findings. For comparison, we built a permissive, non-LLM AST-parsing script to look for the logging guidance patterns (Section 3.2) in the plugins’ first-party source (excluding imported dependencies) as a baseline. While the script has no data-flow capabilities to determine where or how a log file is written, it still identified one plugin not identified by our framework (Section 4.1). However, the baseline missed 4 plugins harder-to-model file-writing methods that our framework correctly identified. Thus, the agentic approach can reduce the false positives generated by a simple SAST script and identify more complex issues. The analysis cost ∼$258 in equivalent Claude API usage (mean $0.91, median $0.69 per plugin). It took ∼ 1800 minutes and about 300M tokens to complete. For each plugin, the analysis took a median of 5 minutes (mean 5.6, max 13.2) over a median of 18 agent reasoning steps.
Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures
5.5
Limitations and Future Work
Our study has several limitations that also point to future work. First, we limited our analysis to the top 300 WordPress plugins, which cover 75% of the active plugin installations from the official WordPress plugin system. Future work could extend the analysis to the long tail of less popular plugins to determine if and how the prevalence of log file exposures and their mitigations change. Although WordPress has the biggest CMS market share by a large margin, a comparison to other CMS and their plugins could reveal further insights. While our approach identified several potential log file exposures, which we manually validated, there remains a chance that the LLM-based analysis did not identify all log exposure instances, e.g. our baseline-script or agentic framework missed. In particular, the true recall could be lower if both missed log file exposures. However, we deliberately decided against including plugins with known file-exposure issues as a groundtruth, since these could be within a SOTA model’s training dataset, biasing the results. Therefore, we encourage future work to repeat our study, e.g. with different frontier models, agentic setups, CMS or plugins. Additionally, we provided the LLM one main prompt with analysis instructions and two additional file references for guidance, but did not monitor how closely the agent followed these instructions. Future work could review our captured outputs from the LLM-agent to learn how the instructions or LLM-based analysis could be improved, e.g. with other prompt designs or different/no guidance. We acknowledge that LLMs are non-deterministic and single analysis-runs can lead to false-negatives, so we encourage future work to repeat our study with multiple iterations per plugin to gain run-to-run consistency. Furthermore, an ablation study would provide insights what effect the static or dynamic analysis phases of our framework have on the overall results. Of the 300 analyzed plugins, only 16 exceeded the 900 seconds analysis time limit, but 3 still produced a findings report. Table 3 in the Appendix lists these 13 plugins with their version, lines of code and number of files. While plugin and code complexity can be one reason to cause a timeout, network and initialization delays of the dynamic analysis environment (Phase 3) were counted against the execution time as well. Future work could re-run the analysis with raised timeouts. We evaluated our log file exposure framework only against the traditional Apache+WordPress combination in its default, multisite configuration. While Apache is recommended by WordPress [43], other web servers can be used as well, where certain protection mechanisms might apply (e.g. .htaccess not supported by NGINX). Future work could re-evaluate the risk assessments (e.g. Figure 5) and log file exposures for CMS installations on non-Apache web servers. A large-scale measurement could further establish the prevalence of the WordPress+web-server combinations actually used on the internet to determine whether shifts in assumptions and configurations are necessary. Additionally, such a large-scale internet scan would allow to identify and quantify the exposures caused by web server (mis)configurations and help affected parties to remediate these by notifying them.
AISec ’26, November 15–19, 2026, The Hague, Netherlands
An analysis of the exposed log files’ contents was deemed out of scope, as it would require deeper understanding on how a plugin is used and what data is processed. However, the type of exposed data determines the exposure’s severity (e.g. from benign debug lines to highly sensitive access tokens or credentials). Future work could perform such a classification or, after approval by an IRB, conduct a large-scale study of live exposed files to rate the severity of such issues. As stated earlier, we did not compare our approach against static analysis tools such as Semgrep or Psalm, as no suitable rules for log file exposures are publicly available. Similarly, we did not compare against community projects like WPScan or Nuclei as these check for specific publicly known log file exposures, which are likely present in the LLM’s training data, which would bias the results. To the best of our knowledge, these tools cannot be used to uncover previously unknown log file exposures, which was the goal of analyzing the most recent versions of the top WordPress plugins in this work.
5.6
Open Science and Ethical Considerations
To foster reproducibility and to allow developers to audit their own plugins, we open-source the complete analysis framework and all related code in the following repository upon acceptance: https://github.com/gehaxelt/Agentic-Detection-of -PotentialLog-File-Exposures-in-CMS. The plugin analysis data will be released after closely reviewing the findings again and following coordinated vulnerability disclosure (CVD) where necessary, and providing developers a sensible amount of time to implement additional protections. From an ethics perspective, this work was conducted with a defensive intent and did not require approval by the authors’ universities. Furthermore, all analysis was performed against local instances and we did not try to access or collect exposed log files from live third-party websites. We judged the identified issues on their severity and immediate risk to end-users and could not identify immediate high-risk disclosures, or otherwise we would have followed CVD practices as stated before. We believe that opensourcing this framework poses a greater advantage for defense by allowing developers to detect and mitigate such issues to release a more secure plugin, rather than malicious actors using it with bad intentions.
6
Conclusion
We studied the potential exposure of log files created by third-party CMS plugins using our agentic LLM-based detection framework against the top 300 WordPress plugins that cover 75% of active installations in the official plugin ecosystem. With our framework’s static and dynamic analysis capabilities, we discovered 62 plugins writing 81 log files, which we manually reproduced and validated in 79 cases. Although we observed many protective measures and no directly exposed log files among our framework’s findings, we find the the risk of exposure to be still just one misconfiguration away in many cases (RQ1). Furthermore, based on our data, we derived and extended a taxonomy of log file path and protection patterns and learned that the prominent safeguard is creation-control instead of access-control
AISec ’26, November 15–19, 2026, The Hague, Netherlands
(RQ2). In fact, 69% of the log files are only created after manual changes to the settings or code by the plugin user. But not all log files come with sufficient access-controls once they are created, so an exposure risk remains that is not fully transparent to plugin users. We noticed several protective patterns hinge on assumptions by developers about a web server’s configuration to which a plugin is installed, which do not always hold. We have therefore compiled recommendations for developers to use multiple layers for defensein-depth (RQ3). Finally, we corroborate related work in that agentic, LLM-based security analysis frameworks are a useful tool to help identify potential log file exposures and accelerate the discovery and reproduction of security issues.
7
Generative AI Use
Claude Opus 4.8 was used for grammar and spelling checks when writing the paper, and for speeding up development of the evaluation and analysis scripts, but all AI-produced code was reviewed by the author.
References [1] Alfred Amoah and Yan Liu. 2025. XRepair - Unifying Retrieval, Repair, and Evaluation for Explainable LLM-Based Vulnerability Fixes. In 2025 32nd AsiaPacific Software Engineering Conference (APSEC). 1001–1004. doi:10.1109/APSE C66846.2025.00116 [2] Anthropic PBC. 2026. Project Glasswing: Securing critical software for the AI era — anthropic.com. https://www.anthropic.com/glasswing. [Accessed 17-06-2026]. [3] Jerome Bruandet. 2020. WordPress Easy WP SMTP plugin fixed zero-day vulnerability. https://blog.nintechnet.com/wordpress-easy-wp-smtp-plugin-fixedzero-day-vulnerability/. [Accessed 12-06-2026]. [4] Di Cao, Yong Liao, and Xiuwei Shang. 2024. Realvul: Can We Detect Vulnerabilities in Web Applications With LLM?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 8268–8282. [5] Xueying Du, Geng Zheng, Kaixin Wang, Yi Zou, Yujia Wang, Wentai Deng, Jiayi Feng, Mingwei Liu, Bihuan Chen, Xin Peng, et al. 2024. Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level Rag. ACM Transactions on Software Engineering and Methodology (2024). [6] Hannes Ekstam Ljusegren. 2023. Vulnerabilities in Outdated Content Management Systems: An Analysis of the Largest WordPress Websites. https://www.divaportal.org/smash/get/diva2:1852099/FULLTEXT01.pdf. [Accessed 2026-06-12]. [7] F5, Inc. 2024. Like Apache: .htaccess | NGINX — web.archive.org. https://web.ar chive.org/web/20240214213627/https://www.nginx.com/resources/wiki/start/t opics/examples/likeapache-htaccess/. [Accessed 14-06-2026]. [8] Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. LLM Agents can Autonomously Exploit One-day Vulnerabilities. arXiv:2404.08144 [cs.CR] https://arxiv.org/abs/2404.08144 [9] Lucas B Germano and Julio Cesar Duarte. 2025. A Study on Vulnerability Explanation Using Large Language Models.. In ICAART (3). 1404–1411. [10] Rikhiya Ghosh, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile, Sanjeev Kumar Karn, and Oladimeji Farri. 2025. CVE-LLM: Ontologyassisted Automatic Vulnerability Evaluation Using Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 28757–28765. [11] Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing. arXiv:2501.18160 [cs.SE] https://arxiv.org/abs/2501.18160 [12] Yuejun Guo, Constantinos Patsakis, Qiang Hu, Qiang Tang, and Fran Casino. 2024. Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection. In European symposium on research in computer security. Springer, 271–289. [13] International Telecommunication Union. 2025. Number of Internet Users Worldwide from 2005 to 2025 (in Millions). Statista: https://www.statista.com/statistic s/273018/number-of-internet-users-worldwide/. [Accessed 12-06-2026]. Submitting an Extension — exten[14] Joomla Extensions Directory. 2026. sions.joomla.org. https://extensions.joomla.org/support/knowledgebase/for-jeddevelopers/submitting-an-extension/. [Accessed 12-06-2026]. [15] Paweł Kabata. 2025. Analysis of Vulnerabilities and Consequences of Missing Hardening in WordPress-Based CMS Environments. Computer Science and Mathematical Modelling (2025).
S. Neef
[16] Ranjita Pai Kasturi, Jonathan Fuller, Yiting Sun, Omar Chabklo, Andres Rodriguez, Jeman Park, and Brendan Saltaformaggio. 2022. Mistrust Plugins You Must: A Large-Scale Study Of Malicious Plugins In WordPress Marketplaces. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 161–178. [17] Teemu Koskinen, Petri Ihantola, and Ville Karavirta. 2012. Quality of WordPress Plug-Ins: An Overview of Security and User Ratings. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing. 834–837. doi:10.1109/SocialCom-PASSAT.2012.31 [18] Ummay Kulsum, Haotian Zhu, Bowen Xu, and Marcelo d’Amorim. 2024. A Case Study of Llm for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedback. In Proceedings of the 1st ACM International Conference on AI-Powered Software. 103–111. [19] Shalini Kumari and Geeta Yadav. 2026. Vuln2Action: An LLM-based Framework for Generating Vulnerability Reproduction Steps and Mapping Exploits. Journal of Information Security and Applications 99 (2026), 104420. [20] Xue Leng, Hai Zhang, Tiantian Zhu, and Jianguo Sun. 2025. Poster: Leveraging Large Language Models to Effectively and Efficiently Identify Vulnerability Patches for WordPress Plugins. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. 4770–4772. [21] Haoyu Li, Xijia Che, Yanhao Wang, Xiaojing Liao, and Luyi Xing. 2026. ExecutionState-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generation. arXiv preprint arXiv:2602.13574 (2026). [22] Jiahuei Lin, Mohammed Sayagh, and Ahmed E. Hassan. 2023. The Co-evolution of the WordPress Platform and Its Plugins. ACM Trans. Softw. Eng. Methodol. 32, 1, Article 19 (Feb. 2023), 24 pages. doi:10.1145/3533700 [23] Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. 2024. GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software 212 (2024), 112031. doi:10.1016/j.jss.2024.112031 [24] Oslien Mesa, Reginaldo Vieira, Marx Viana, Vinicius H. S. Durelli, Elder Cirilo, Marcos Kalinowski, and Carlos Lucena. 2018. Understanding Vulnerabilities in Plugin-based Web Systems: An Exploratory Study of Wordpress. In Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1 (Gothenburg, Sweden) (SPLC ’18). Association for Computing Machinery, New York, NY, USA, 149–159. doi:10.1145/3233027.3233042 [25] Daniel T. Murphy, Minhaz F. Zibran, and Farjana Z. Eishita. 2021. Plugins to Detect Vulnerable Plugins: An Empirical Assessment of the Security Scanner Plugins for WordPress. In 2021 IEEE/ACIS 19th International Conference on Software Engineering Research, Management and Applications (SERA). 39–44. doi:10.1109/SERA51205.2021.9509274 [26] István Márton. 2025. 400,000 WordPress Sites Affected by Account Takeover Vulnerability in Post SMTP WordPress Plugin. https://www.wordfence.com/blog /2025/11/400000-wordpress-sites-affected-by-account-takeover-vulnerabilityin-post-smtp-wordpress-plugin/. [Accessed 12-06-2026]. [27] Sebastian Neef. 2020. How Wordpress plugins leak sensitive information without you noticing — blog.detectify.com. https://blog.detectify.com/industryinsights/gehaxelt- how- wordpress- plugins- leak- sensitive- informationwithout-you-noticing/. [Accessed 12-06-2026]. [28] Sebastian Neef, Lorenz Kleissner, and Jean-Pierre Seifert. 2024. What All the PHUZZ Is About: A Coverage-guided Fuzzer for Finding Vulnerabilities in PHP Web Applications. In Proceedings of the 19th ACM Asia Conference on Computer and Communications Security. 1523–1538. [29] Kinsey K. S. Ng, Farah Yan, and Kevin Hung. 2024. Preliminary Study of LLM-Based Wordlist Generation for Validating Broken Web Access Control. In TENCON 2024 - 2024 IEEE Region 10 Conference (TENCON). 1088–1091. doi:10.1109/TENCON61640.2024.10902771 [30] Marcus Niemietz, Mario Korth, Christian Mainka, and Juraj Somorovsky. 2021. Over 100 Bugs in a Row: Security Analysis of the Top-Rated Joomla Extensions. arXiv:2102.03131 [cs.CR] https://arxiv.org/abs/2102.03131 [31] Vikram Nitin, Baishakhi Ray, and Roshanak Zilouchian Moghaddam. 2025. FaultLine: Automated proof-of-vulnerability generation using LLM agents. arXiv preprint arXiv:2507.15241 (2025). [32] Open Source Matters, Inc. 2026. Vulnerable Extensions — extensions.joomla.org. https://extensions.joomla.org/vulnerable-extensions/vulnerable/. [Accessed 12-06-2026]. [33] OWASP Top 10 Team. 2025. OWASP Top 10:2025 — owasp.org. https://owasp.or g/Top10/2025/. [Accessed 12-06-2026]. [34] Patchstack. 2026. State of WordPress Security in 2026 - Security Whitepaper — patchstack.com. https://patchstack.com/whitepaper/state-of -wordpresssecurity-in-2026/. [Accessed 12-06-2026]. [35] Niklas Risse, Jing Liu, and Marcel Böhme. 2025. Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection. arXiv:2408.12986 [cs.CR] https://arxiv.org/abs/2408.12986 [36] Jukka Ruohonen. 2019. A Demand-Side Viewpoint to Software Vulnerabilities in WordPress Plugins. In Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering (Copenhagen, Denmark) (EASE ’19). Association for Computing Machinery, New York, NY, USA, 222–228.
Plug ’n’ Pray: Agentic LLM-based Detection of Potential Log File Exposures
doi:10.1145/3319008.3319029 [37] SentinelOne. 2024. CVE-2024-13513: Oliver POS Information Disclosure Flaw — sentinelone.com. https://www.sentinelone.com/vulnerability-database/cve2024-13513/. [Accessed 12-06-2026]. [38] Faysal Hossain Shezan, Zihao Su, Mingqing Kang, Nicholas Phair, Patrick William Thomas, Michelangelo van Dam, Yinzhi Cao, and Yuan Tian. 2023. Chkplug: Checking Gdpr Compliance of Wordpress Plugins via Cross-language Code Property Graph. In Network and Distributed System Security (NDSS) Symposium 2023. [39] Rishikant Singh. 2026. Comparative Analysis of Open-Source Content Management Systems: WordPress, Drupal, and Joomla for Academic Use. Multidisciplinary Journal of Educational Research Innovation and Development 3, 1 (jan-jun 2026). [40] Zhuldyz Tashenova, Aisultan Aitmagambetuly, Aigulim Bayegizova, Saya Santeyeva, Zhanat Abdugulova, Shirin Amanzholova, and Akerke Kerim. 2026. SentinelCMS: Proactive Vulnerability Detection in CMS Plugins Using Static Taint Analysis and Bidirectional LSTM. Applied Sciences 16, 11 (2026), 5471. [41] W3Techs. 2026. Usage Statistics and Market Shares of Content Management Systems. https://w3techs.com/technologies/overview/content_management. [Accessed: 12-06-2026]. [42] Wix.com, Inc. 2026. About App Distribution — dev.wix.com. https://dev.wix.co m/docs/build-apps/launch-your-app/app-distribution/about-app-distribution. [Accessed 12-06-2026]. [43] WordPress Foundation. 2025. WordPress and web servers | Learn WordPress — learn.wordpress.org. https://learn.wordpress.org/lesson/wordpress-and-webservers/. [Accessed 13-06-2026]. [44] WordPress Foundation. 2026. Add your Plugin | WordPress.org — wordpress.org. https://wordpress.org/plugins/developers/add/. [Accessed 12-06-2026]. [45] WordPress Foundation. 2026. Hardening WordPress – Advanced Administration Handbook | Developer.WordPress.org — developer.wordpress.org. https://develo per.wordpress.org/advanced-administration/security/hardening/. [Accessed 12-06-2026]. [46] WP-Firewall Team. 2025. Mitigating Sensitive Data Exposure in Export Plugins | CVE202511693 | 2025-12-16 - Managed-WP.™ — managed-wp.com. https: //managed- wp.com/latest- wordpress- plugin- vulnerabilities/mitigatingsensitive- data- exposure- in- export- plugins- cve202511693- 2025- 12- 16/. [Accessed 12-06-2026]. [47] WP Odyssey. 2026. How Many WordPress Plugins Are There in 2026? (61,000+) — blog.wpodyssey.com. https://blog.wpodyssey.com/plugins-tools/how-manywordpress-plugins-are-there/. [Accessed 12-06-2026]. [48] WPScan. 2026. WordPress Vulnerability Statistics. https://wpscan.com/statistics/. [Accessed: 12-06-2026]. [49] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629 [50] Alperen Yildiz, Sin G Teo, Yiling Lou, Yebo Feng, Chong Wang, and Dinil Mon Divakaran. 2025. Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vienna, Austria, 30848–30865. doi:10.18653/v1/2025.acl-long.1490 [51] Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. 2025. CVEbench: a benchmark for AI agents’ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332 (2025).
A Appendix A.1 Main Prompt
AISec ’26, November 15–19, 2026, The Hague, Netherlands
9 10 11 12 13 14 15 16 17 18 19 20 21
22 23 24 25 26 27 28
Follow the methodology in PROMPT.md and CLAUDE.md exactly. This is scientific research - be thorough. Steps: 1. Create the plugin config: config/{slug}.json 2. Start the environment: ./scripts/run.sh --config {slug}.json 3. Wait for readiness, handle any dependency issues (e.g. WooCommerce requirement) 4. Extract the plugin source and perform THOROUGH static analysis - read and understand the code, don't just grep 5. If file-based logging is found, perform dynamic verification (Phase 2 + Phase 3) 6. Write the finding to {findings_dir_name}/{slug}-{version}.md using the exact format from PROMPT.md 7. Teardown: ./scripts/teardown.sh Important: - The finding MUST be written to {findings_dir_name}/{slug}-{version}.md before you finish (this path overrides any `findings/` reference in CLAUDE.md or PROMPT.md) - If the plugin has no file-based logging, still write a finding stating that - Do NOT skip any phase. Do NOT take shortcuts. Read the actual source code. - Follow through wrapper functions and helper classes to trace actual file paths - Check for ALL file-writing patterns: file_put_contents, fopen, fwrite, fputs, error_log with file dest, custom logger classes, Monolog, WC_Logger, etc. - Check path construction: WP_CONTENT_DIR, ABSPATH, plugin_dir_path, wp_upload_dir, __DIR__, dirname(__FILE__) - Check for log-related constants, options, and configurable paths """
A.2
Timed-Out Plugins
Table 3 shows the 13 plugins with their version, install count, lines of code and number of files (determined with cloc17 ) whose analysis exceeded the 900s time limit without producing a finding (Section 5.5). Table 3: The 13 timed-out plugins without a finding. Plugin (slug)
Version
Installs
Files
LOC
google-listings-and-ads wp-optimize facebook-for-woocommerce fast-indexing-api mailchimp-for-woocommerce woocommerce-paypal-payments woocommerce-services userfeedback-lite pinterest-for-woocommerce woo-variation-swatches host-webfonts-local post-duplicator connect-polylang-elementor
3.6.0 4.5.1 3.6.0 1.1.22 6.0 3.4.1 3.5.1 1.11.1 1.4.25 2.2.3 6.2.0 3.0.13 2.5.5
900K + 1M + 500K + 200K + 300K + 800K + 600K + 200K + 300K + 300K + 300K + 200K + 100K +
4,618 894 567 688 179 978 472 2,309 364 48 71 35 272
360,808 117,796 61,061 58,844 55,537 54,338 33,621 27,732 24,310 11,040 7,042 4,081 3,149
Listing 1 shows the main prompt given to the LLM agent with references to two files with additional guidance. Listing 1: Main prompt given to the claude agent. 1 2 3 4 5 6 7
8
def build_prompt(plugin, findings_dir_name): name = html.unescape(plugin["name"]) slug = plugin["slug"] version = plugin["version"] active_installs = plugin["active_installs"] return f"""You are analyzing WordPress plugin "{name}" (slug: `{slug}`, version: `{version}`, {active_installs:,} active installs) for publicly accessible log file exposure.
17 https://github.com/aldanial/cloc, accessed: 2026-09-15