arXiv CSopen access

Adversarial SQL Injection Generation with LLM-Based Architectures

Adversarial SQL Injection Generation with LLM-Based Architectures Ali Karakoc1 and H. Birkan Yilmaz1*

arXiv:2605.11188v1 [cs.CR] 11 May 2026

1

Department of Computer Engineering, NETLAB, Bogazici University, Istanbul, Turkey.

*Corresponding author(s). E-mail(s): [email protected]; Contributing authors: [email protected]; Abstract SQL injection (SQLi) attacks are still one of the serious attacks ranked in the Open Worldwide Application Security Project (OWASP) Top 10 threats. Today, with advances in Artificial Intelligence (AI), especially in Large Language Models (LLMs), an opportunity has been created for automating adversarial attack tests to measure the defense mechanisms. In this paper, we aim to create a comprehensive evaluation of use cases that utilize LLMs for adversarial SQL injection generation. We introduce two novel LLM-based systems, Retrieval Augmented Generation for Adversarial SQLi (RADAGAS) and Reflective Chain-of-Thought SQLi (RefleXQLi), and compare them with existing baselines against 10 Web Application Firewalls (WAFs) and one execution-based MySQL validator. To perform a comprehensive test, we used six rule-based open-source WAFs (ModSecurity PL1–3, Coraza PL1–3), 2 AI/ML-based WAFs (WAF Brain, CNN-WAF), and 2 commercial WAFs (AWS WAF and Cloudflare WAF). For the LLM models, we used GPT-4o, Claude 3.7 Sonnet, and DeepSeek R1. Our tests consist of 240 experiments that generate 240,000 payloads and perform 2.2 million tests against WAFs. Our comprehensive evaluation reveals that RADAGAS-GPT4o outperforms other baseline models with a 22.73% bypass rate. The proposed RADAGAS variants are highly successful on AI/ML-based WAFs (92.49% on WAF-Brain by RADAGAS-DeepSeek, 80.48% on CNN-WAF by RADAGASClaude), but struggle to bypass rule-based WAFs (0–5.70% on ModSecurity and Coraza). In addition to these findings, another observation is that creating less diverse payloads achieves more bypasses, however they show poor results if the initially chosen payload is not successful. We observe that our findings provide a comprehensive view on using LLM-based approaches in security testing. Keywords: SQL injection, Large language models, Web application firewalls, Adversarial testing, RAG-based generation, Security benchmarking

1

1 Introduction Web applications are the fundamental part of the modern digital world, and connect people and businesses in different critical domains like healthcare, finance, social networking, e-commerce and government services. Although it is one of the very known and initial web application attacks, SQL Injection vulnerabilities are still one of the major problems persisting on the Open Worldwide Application Security Project (OWASP)’s Top 10 most critical web application security risks [1]. The impact of SQL injection attacks can be leveraged with more sophisticated techniques to steal critical information, obtain remote services or denial of services. To protect the web application services against SQL Injection attacks, using advanced defense mechanisms is crucial. However, defense mechanisms do not work with simple installation; there are necessary configuration settings, fine tunings and manual or automatic rule creation that must be done to establish more robust and secure systems. This can be done by adversarial assessment on the systems to see the potential weaknesses. Traditional approaches to test web applications against SQL injection (SQLi) attacks rely on two main methodologies: manual penetration testing performed by real security experts and automated scanning via mainstream rule-based SQLi automation tools such as SQLmap [2]. While these methods have proven success revealing SQLi vulnerabilities, they face significant limitations against modern AI/ML based and commercial WAFs. Manual testing is time consuming, expensive and the success totally depends on the expertise of the security tester. Automated rule-based testing is not able to generate novel SQLi attacks to bypass next generation systems and they can be easily recognized by signature based defense systems. In addition to these, both approaches do not perform coverage tests for different variants of the SQLi attacks. The purpose of adversarial testing against defense systems is increasing the defense bar proactively even before an incident happens. The recent studies in Large Language Models (LLMs) have sparked significant interest in implementation of these approaches to security testing. Models such as GPT-4o [3], Claude Sonnet [4], and DeepSeek [5] demonstrated advanced capabilities of understanding and generating code-like structures including SQL syntax. Their ability of understanding syntax and making advanced reasoning positions them as a promising tool for generating adversarial SQL injection payloads. However, the existing work on adversarial testing with LLMs has several challenges: single-model focus, limited parameter exploration, insufficient diversity and feedback mechanisms. Single-Model Focus: Previous researches mainly evaluate the capability of a single LLM, that limits understanding the behavior of cross model setups and optimal parameter configurations. This prevents understanding the nature of LLM based adversarial SQLi generation comprehensively. Limited Parameter Exploration: Existing studies demonstrate the performance of adversarial generations under fixed or minimally varied hyperparameter settings. This prevents exploring model specific optimal parameter settings and understanding of their effectiveness. Insufficient Diversity and Feedback Mechanisms: Current LLM based approaches generally do not put diversity enforcements that result in repetitive SQLi

2

payloads and limit the attack coverage [6]. This makes the SQLi generator easily detectable by advanced defense systems [7]. To address these challenges and to analyze adversarial SQLi generation with LLMs, we introduce two novel architectures and a comprehensive benchmarking study against modern WAF defenses. Our contributions can be listed as follows: 1. Comprehensive Benchmarking Framework: We present a systematic evaluation framework to compare LLM SQLi generator models against ten comprehensive WAFs: AI/ML-based (WAF-Brain, CNN-WAF), rule based open source (ModSecurity PL1-3, Coraza PL1-3), commercial (Cloudflare, AWS WAF), and MySQL execution validation. We created 240 different experiment cases then generated 240,000 SQLi payloads and performed 2.2 million tests against ten WAFs and one execution test that provides an extensive benchmarking on LLM based adversarial SQLi generation. 2. Retrieval-Augmented Generation for Adversarial SQLi (RADAGAS): We introduce RADAGAS architecture that utilizes Retrieval Augmented Generation (RAG) and LLM generation with multi stage diversity and execution filtering. RADAGAS collects and retrieves proven successful attack patterns from curated knowledgebase (OWASP, Portswigger, Github) and generates adversarial SQLi payloads via RAG prompting. We experimented RADAGAS with different foundation models (GPT-4o, Claude 3.7 Sonnet and Deepseek-r1) to demonstrate its cross-model performance. 3. Reflective Chain-of-Thought SQLi (RefleXQLi): Our second proposed method is RefleXQLi that employs four step chain of thought reasoning (WAF Analysis, Strategy Formulation, Payload Design, Refinement) to generate high quality adversarial SQLi payloads followed by dual-LLM validation. The architecture consists of two modules: generator LLM creates SQLi payloads and discriminator LLM refines them iteratively that resulted in generation of payloads with 100% uniqueness. 4. The experiments demonstrated that RADAGAS variations (22.73%, 22.09%, 21.73%) and RefleXQLi (21.21%) outperformed existing traditional methods (15.01%), vanilla zero-shot LLM generator (12.90%) and GenSQLi (20.35%) [8] in terms of average WAF bypass rate, showing that both proposed methods are superior to the traditional approaches. 5. The experiments demonstrated that the temperature settings are model specific and counter intuitive [9]. While GPT-4o performed better in lower temperature (T=0.1, 22.73%), Deepseek-r1 performed better in high temperature (T=0.9, 22.09%) with more creativity and Claude 3.7 Sonnet has the optimal performance in medium temperature (T=0.6, 21.73%). This demonstrates LLMs have different characteristics under different temperature conditions in terms of exploitation versus exploration in adversarial generation. 6. MySQL Validation as Ground Truth: As an additional validation layer, we introduced a vulnerable web application backed by a MySQL server to validate the generated SQL injection payloads. The experiment results show that RADAGAS systems achieve 60-64% execution success, while the baseline systems stay at 2137% execution success against MySQL validation.

3

7. WAF Performance Characterization: We demonstrated system specific performance across all 10 WAFs and one MySQL validation layer. RADAGAS-DeepSeek performed high bypass rates on ML based WAFs with 92.49% on WAF-Brain, 78.39% on CNN-WAF, GenSQLi achieved the highest rule-based WAF bypass with 16.26% on ModSecurity PL1, and RADAGAS-GPT4o achieved the highest bypass rate on Cloudflare with 49.50%. This heterogeneous result set shows that there is no single system that outperforms universally, and the practical implementations should combine multiple approaches together. 8. RefleXQLi Chain of Thought Evaluation: We introduced RefleXQLi which is one of the first implementations of Chain of Thought reasoning combined with dual-LLM architecture for adversarial attack generation. RefleXQLi demonstrated 21.21% bypass rate with low variance σ = 0.37, that shows explicit reasoning provides more stable output for adversarial payload generation. However, 27.80% MySQL execution success is lower than RADAGAS’s 60-64% revealing a tradeoff between creative adversarial generation and semantic correctness of the generated outputs.

2 Related Work We organized related work section into three categories: Traditional SQLi generation, LLM based security testing, and diversity in adversarial generation.

2.1 Traditional SQL Injection Generation SQLi vulnerability is known for decades and SQLi testing tools and approaches are quite comprehensive and contain decades of expert knowledge embedded into rule based attack patterns. SQLMap [2] is one of the industry standard popular tools and extensively used in penetration testing to cover SQLi scenarios since 2006. The tool consists of thousands of hand crafted SQL injection payloads in its database, covering union based, error based, time based blind, boolean based blind and stacked queries across multiple database platforms like MySQL, PostgreSQL, Oracle and MSSQL. SQLMap utilizes automated parameter detection, DBMS fingerprinting and enumeration strategies to achieve comprehensive coverage. Surveys [10, 11] state the evolution of attack types and their countermeasures of traditional SQLi frameworks over the past two decades. Although traditional tools have extensive coverage in testing SQLi vulnerabilities, there are several limitations against modern defense systems. Since the tools generate deterministic payloads, those patterns can be captured easily by signature based WAFs [12]. ML based detection systems are quite effective on template based approaches since these approaches have weakness on contextual adaptation [13, 14] and ML based WAFs can learn the attack characteristics easily. In [12], Appelt et al. show that the commercial WAFs are significantly effective on detecting SQLMap generated attack payloads with 94% prevention rate. Even sophisticated manual tests are ineffective against multi layer defense in depth systems.

4

2.2 LLM-Based Security Testing SQLiGPT was one of the first uses of LLMs for adversarial security testing [15]. The tool examines the capability of GPT model when it generates and detects SQLi payloads with prompt engineering but in a limited scale. In [16], Yang et al. developed a system named LLMSqli to detect SQLi attacks using generalization capabilities of LLMs. Fang et al. introduced an agent based LLM architecture to learn exploiting web vulnerabilities via adaptive feedback loops [17]. In another approach, Deng et al. designed pentestGPT to perform automated penetration testing [18]. In [19], Liu et al. introduced a mechanism to detect adversarial prompt injection techniques against LLM based systems. Babaey and Ravindran introduced a comprehensive framework named GenSQLi to generate adversarial SQLi attacks and generate corresponding WAF rule to create defense for these payloads [8]. GenSQLi employs GPT-4o [3] with in context learning. They also evaluated Google Gemini Pro which performed lower bypass rate than GPT-4o, highlighting performance variance due to LLM dependency. In [20, 21], the authors worked on temperature scaling LLMs extensively. According to the experiments, the higher temperatures increase the diversity, but affect the coherence negatively. Top-k and top-p sampling provide alternative sampling methodology [20]. In [22, 23], the authors study on the effects of temperature on trade off in between functionality and creativity. The results show that the optimal parameter settings strongly depend on the model and task and need empirical evaluation for each domain.

2.3 Diversity in Adversarial Payload Generation Correlation of the diversity with successful adversarial attack generation is one of the assumptions of prior work. GenSQLi assumes payload variation increases the successful attack probability with genetic diversity through template mutation [8]. Effective defense rules are produced by grouping similar attack payloads via ML clustering approaches. They are aiming a comprehensive coverage by utilizing the diversity as a proxy. SQIRL [24] uses reinforcement learning [25] to optimize grey-box detection of SQLi vulnerabilities by exploring new payloads through diversity. If there are repeating payload patterns, the reward function penalizes them to achieve improvement in evasion by uniqueness. SQUIRREL [26] is an implementation of a fuzzing with mutation operators to achieve diverse query generation. The study utilizes a fuzzing principle [27, 28] to explore new and unknown payloads to search more edge cases. To achieve this, the study uses coverage guided fuzzing [29] and diversity metrics to optimize the exploration of new payloads. In information retrieval process, Maximum Marginal Relevance (MMR) [30] plays critical role to balance the diversity and relevance of information. While this mechanism selects items relevant with the query, it also considers how it is diverse from already selected items: MMR is used in retrieval and summarization extensively, and it has not been evaluated for adversarial attack testing extensively and systematically in the previous works.

5

Although there are studies that employ diversity to achieve more successful payload generation, diversity and WAF bypass correlation has still not been tested empirically and holistically. Our study provides extensive experiments and analysis on real world systems to show this correlation.

2.4 Positioning of This Work With the aim of establishing more robust and secure systems via adversarial assessment to see the potential weaknesses, our study is designed to cover holistic real world scenarios by employing seven SQLi generators (three base-line, and four our proposal) and test them against 10 WAFs (including commercial and AI/ML based WAFs) with validity check via real vulnerable web application backed by a MySQL server under identical conditions, providing: 1. Multi Metric Diversity Analysis: Seven complementary diversity metrics (uniqueness, semantic, lexical, contextual, n-gram, Abstract Syntax Tree (AST), and functional diversity) evaluated against generated SQL injection payloads across all generators. 2. Comprehensive System Comparison: Traditional SQL injection tools (SqlMap), recent studies GenSQLi, Vanilla GPT-4o (zero-shot), and our novel algorithms; three RADAGAS variants (GPT-4o, DeepSeek, Claude) and RefleXQLi (CoT+Adversarial). 3. Real World Scenarios: Two commercial WAFs, six Rule Based WAFs and two AI/ML based WAFs to test generated payloads against wide range of real world scenarios. 4. Model Specific Optimization Characteristics: Tuning LLM Parameters and diversity threshold to find optimal settings for generating successful SQL Injection payload.

3 Methodology This section provides detailed information on our experimental methodology, system designs, algorithms, diversity check pipeline, metrics, and WAF settings.

3.1 System Architecture The overall system is illustrated in Fig. 1. Our testbed consists of seven generators, ten WAFs, one SQL execution validator and six diversity filters. As a result of our tests, 240K payloads were generated and approximately 200K valid payloads were tested against 10 WAFs and one execution checker. In total we performed approximately 2.2M tests.

3.2 SQL Injection Generation Systems Our testbed consists of seven SQL injection generation systems including the existing studies and our novel systems to provide comprehensive comparison for LLM based SQL Injection generation techniques.

6

64/L*HQHUDWRUV

5$'$*$6

5HIOH;4/L

5$*005

&R7'XDO//0

*HQ64/L

7UDGLWLRQDO

9DQLOOD

7HPSODWH*HQHWLF

6WDWLF

=HURVKRW

6HHG$EODWLRQ5HIOH;4/L9_*HQ64/L9

a64/L3D\ORDGV

:$)0\64/([HFXWRU :$)%UDLQ_&11:$)

0RG6HF_&RUD]D

$,0/EDVHG

5XOHEDVHG3/

&ORXGIODUH_$:6

0\64/

&RPPHUFLDO

([HFXWLRQ

%<3$66%/2&.a0WHVWV

(YDOXDWLRQ $QDO\VLV %\SDVV5DWH

'LYHUVLW\

SHU:$)

PHWULFV

&RUUHODWLRQ

3DUDP2SWLPL]DWLRQ

Fig. 1: System architecture focusing on SQLi generation, defense, and evaluation layers.

3.2.1 Baselines Our experiment deploys three baseline systems to perform comprehensive analysis and comparison: traditional SQLi, vanilla GPT-4o, and GenSQLi. Traditional SQLi: We simulated a SQLMap variant to generate rule based deterministic payloads. This setup represents decades of hand crafted expert SQL injection payload database. This tool is still being used heavily in the industry to test web applications against SQL Injection attacks. We evaluated this system with 5 runs with 1000 generated payloads for each run for statistical consistency.

7

We implemented a very faithful implementation of SQLMap’s payload engine rather than using SQLMap directly, since the tool performs end to end SQL Injection exploitation including parameter detection, DBMS fingerprinting and code execution. Our simulation utilizes only SQLMap’s core payload database and generates payloads with random sampling. Vanilla GPT-4o: To test the natural capability of LLMs in generating adversarial payloads, we deployed a zero-shot GPT-4o without any fine tuning or special prompting. The system is evaluated in 5 runs each of which generated 1000 payloads. GenSQLi: A recent method depending on a template based framework [8] with in context learning for SQLi payload generation. We simulated the original methodology with 5 different runs each of which produces 1000 payloads. Since at the time of our experiments, the original GenSQLi implementation was not available publicly, we implemented the systems following the specifications in the GenSQLi paper[8]. Our GenSQLi implementation follows the principles of in-context learning pipeline with template based mutation faithfully. To test the system, we compared our simulation results with the paper and observed comparable performance.

3.2.2 RefleXQLi RefleXQLi is one of our proposal systems in this study which combines Chain-ofThought (CoT) reasoning with dual LLM architecture to generate SQLi payloads in multi step reasoning.

• Generator LLM decides the WAF bypass strategy, obfuscation, and SQL syntax mutation before generating the payloads. The CoT prompt is designed to plan sophisticated SQLi payload generation by considering the historic attack executions and their outcomes. • Discriminator LLM evaluates the generated payload to detect its malicious intent by acting like a filtering mechanism. Once discriminator LLM flags a payload, it sends feedback to the generator with reasons why it rejected the payload. • Adversarial loop occurs until a successful and creative payload bypasses the discriminator LLM. Repeated rejection from discriminator forces the generator LLM to generate more creative and successful attack payloads. This architecture is an adaptation of adversarial training with GANs [31] and CoT prompting in reasoning tasks [32] to produce sophisticated and successful SQLi payload generation. The complete picture of RefleXQLi pipeline is presented in Algorithm 1. The high level block diagram in Fig. 2 represents the logical relationships and interactions of RefleXQLi architecture and Dual LLM feedback loop. The generator and discriminator both use GPT-4o as a foundation model. We set maximum adversarial iterations to Imax = 3 and τthreshold = 7 (Maximum value is 10).

3.2.3 RADAGAS We designed and proposed another domain specific system RADAGAS to adapt RAG based SQLi payload generation. While the core functionalities (RAG, FAISS and

8

Algorithm 1 RefleXQLi: CoT + Adversarial Dual LLM Require: Target WAF characteristics W , payload count N , max iterations Imax Ensure: Accepted payloads P 1: P ← ∅ 2: for i = 1 to N do 3: 4:

Phase 1: Chain-of-Thought Reasoning

5: 6: 7: 8: 9:

analysis ← Generator-LLM: Analyze W strategy ← Generator-LLM: Formulate evasion strategy design ← Generator-LLM: Design payload structure refinement ← Generator-LLM: Refine for subtlety

10: 11:

Phase 2: Adversarial Dual LLM Generation

12:

iter ← 0; accepted ← false while iter < Imax and not accepted do 15: payload ← Generator-LLM(ref inement) 16: score ← Discriminator-LLM(payload, W ) 17: if score < τthreshold then 18: P ← P ∪ {payload} 19: accepted ← true 20: else 21: f eedback ← “REJECT!: Detected patterns: refine payload” 22: ref inement ← Generator-LLM(ref inement∥f eedback) 23: end if 24: iter ← iter + 1 25: end while 26: end for 27: return P

13: 14:

MMR) are already well established in retrieval process, their combination for adversarial generation with multi step diversity filtering and execution validation mechanisms represents a novel approach which has not been explored in prior works. RADAGAS uses RAG along with MMR to provide successful SQLi payloads while maintaining the diversity. The RADAGAS system is illustrated in Fig. 3 and consists of three phases:

• Offline phase is creating a knowledge base with embedding and indexing the known SQL Injection payloads from a catalog. Our catalog contains 82KB curated SQL injection context document consisting of SQL Injection methodologies from OWASP [1], PortSwigger Web Security Academy SQL injection and bypass techniques [33], popular GitHub repositories extensively used by security researchers (PayloadsAllTheThings) [34], specific syntax instructions for MySQL database, known obfuscation and mutation techniques and encoding strategies. As the chunking strategy the system uses RecursiveCharacterTextSplitter with chunk size=200 and chunk overlap=50 to keep the context while enabling accurate and well formed retrieval. For the semantic embedding the system uses sentencetransformers/paraphrase-MiniLM-L3-v2 [35, 36] that keeps balance between speed 9

:$)6SHFLILFDWLRQ 3KDVH&KDLQRI7KRXJKW

6WHS:$)$QDO\VLV 6WHS6WUDWHJ\6HOHFWLRQ 6WHS3D\ORDG'HVLJQ 6WHS5HILQHPHQW 3KDVH$GYHUVDULDO'XDO//0

*HQHUDWRU//0 'LVFULPLQDWRU//0 *372

5HMHFW

*372

$FFHSW" $FFHSW

)LQDO3D\ORDGV Fig. 2: RefleXQLi architecture consists of four step CoT reasoning: Analyzing WAF characteristics, developing a strategy, designing the payload and refinement that is followed by Dual LLM generation. Discriminator LLM provides feedback on the generated payload and iterates the process until generator creates high quality payload to bypass discriminator.

and accuracy. Our system uses FAISS (Facebook AI Similarity Search) [37] for similarity search. • Online Phase is where the system retrieves context from the query. To retrieve relevant but diverse context we configure MMR search with k = 3:

h

λ · Sim1 (di , Q) i −(1 − λ) · max Sim2 (di , dj )

MMR = arg max

di ∈R\S

(1)

dj ∈S

where candidate set as R, previously selected set as S , input query as Q, relevancediversity balance as λ, and similarity as Sim. • Generation Phase is where the LLM generates payloads with query and retrieved context until the generated payload passes the validity and diversity check. The generated query is executed in a vulnerable web application backed with MySQL

10

server. The queries that get execution errors are rejected. For the semantic diversity check, cosine similarity of payload is calculated against the accepted payloads. For checking distance to previously accepted queries, Levenshtein distance, i.e., lexical distance is utilized. For checking semantic similarity against previously accepted queries, BERTScore is evaluated and compared against a threshold. Once a query passes all validation and diversity checks, the system adds the query in the set of previously accepted queries. The detailed flow of the RADAGAS system is presented in Algorithm 2.

2IIOLQH3KDVH

3D\ORDG'DWDEDVH 6%(57(PEHGGLQJ )$,669HFWRU,QGH[ 2QOLQH3KDVH

4XHU\,QSXW 4XHU\9HFWRUL]DWLRQ 0055HWULHYDO *HQHUDWLRQ3KDVH

//0*HQHUDWRU

*372_&ODXGH_'HHS6HHN

'LYHUVLW\)LOWHULQJ

6HPDQWLF6LPLODULW\

5HMHFW

64/9DOLGDWLRQ

/HYHQVKWHLQ'LVWDQFH %(576FRUH 3DVV

$FFHSWHG3D\ORDGV Fig. 3: RADAGAS architecture: Three-phase pipeline with SBERT embedding, FAISS indexing, MMR retrieval, and LLM generation. The system balances relevance and diversity through Maximum Marginal Relevance, retrieving contextually appropriate payloads for generation.

11

Algorithm 2 RADAGAS: RAG-Based SQL Injection Generation Require: Query q, LLM model M , payload count N , diversity threshold θ, temperature T Ensure: Accepted payload set P 1: Offline Phase: 2: KB ← Load SQL injection knowledge base (OWASP, PortSwigger, PayloadsAllTheThings) 3: Chunks ← RecursiveCharacterTextSplitter(KB, size = 200, overlap = 50) 4: Embeddings ← SBERT(Chunks) 5: Index ← FAISS(Embeddings) 6: 7: Online Phase: 8: P ← ∅ 9: for i = 1 to N do 10: 11: 12:

qemb ← SBERT(q) Context ← MMR(qemb , Index, k = 3, λ) prompt ← ConstructPrompt(q, Context, P)

13: 14: 15:

Generation Phase: payload ← M (prompt, T )

16:

Multi-Stage Diversity Filtering: if SQLValid(payload) then 19: if Semantic(payload, P) > θ and Levenshtein(payload, P) BERTScore(payload, P) < θ then 20: P ← P ∪ {payload} 21: end if 22: end if 23: end for 24: return P 17: 18:

>

θ and

To evaluate model specific performance of RADAGAS, we created three variants that share the same MMR based RAG architecture with different foundation models: RADAGAS-GPT-4o using GPT-4o (OpenAI, 2024), a multimodal transformer with 128K context window; RADAGAS-DeepSeek using DeepSeek-R1 (DeepSeek, 2024), an open-weight reasoning model with 32K context window; RADAGASClaude using Claude 3.7 Sonnet (Anthropic, 2024), a Constitutional AI-trained model with 200K context window. In this setup, we wanted to evaluate the same generation foundational models to make a fair comparison. We used identical RAG pipeline, prompt templates, and diversity filtering for all RADAGAS variants. All variants were tested with different temperature and diversity threshold parameters.

3.3 Diversity Metrics In RADAGAS algorithm we used BERTScore and Levenshtein diversity checks as a feedback mechanism to increase the diversity and success rate of the generated payloads. We are using a multi faceted diversity framework to measure the diversity rate of all systems in the context of experiments. These calculations were not used 12

in any feedback mechanism, these are used for the analysis and the comparison of payload diversity rates across systems. To evaluate diversity performance of systems, we used six metrics:

3.3.1 Semantic Diversity This metric measures semantic differences of the payloads using sentence embeddings from MiniLM-L3-v2. The following formula represents the diversity calculation with cosine similarity for a new payload as pnew and payload set P : 
 Semantic(pnew , P ) = min 1 − cos emb(pnew ), emb(p) (2) p∈P

3.3.2 Character Level Lexical Diversity Character level normalized Levenshtein edit distance [38]:

Lex(pnew , P ) = min p∈P

Levenshtein(pnew , p) max(|pnew |, |p|)

(3)

3.3.3 Structural Level Lexical Diversity This metric measures lexical diversity with Jaccard distance over 2, 3, and 4-grams. As a complementary check over character level calculation, n-gram provides structural diversity validation:



|G(pnew ) ∩ G(p)| N gram(pnew , P ) = min 1 − p∈P |G(pnew ) ∪ G(p)|

 (4)

where G(p) is 2, 3, and 4-gram set of payload p.

3.3.4 Contextual Diversity Another semantic diversity metric to measure diversity in between a new payload and accepted payloads using BERTScore F1 [39].

3.3.5 Structural Level Syntactic Diversity This diversity looks at the tree edit distance of Abstract Syntax Trees (ASTs) of generated SQL payloads.

3.3.6 Functional Diversity This metric can be considered as a part of SQL execution validation and classifies execution results by their categories (data extraction, time delay, SQL error, no effect, incompatible). Filtering usage : Accept if functional behavior differs from existing payloads. Evaluation usage : Proportion of unique functional behaviors across the payload set.

13

3.3.7 Aggregated Diversity Score To show overall diversity posture of a generated payload, we aggregate all diversity results and take their average to obtain comprehensive scoring for comparison.

3.4 WAF Testing All generator systems are evaluated against ten diverse WAFs (traditional rule based, AI/ML based and commercial) and one MySQL execution test to understand real life behavior of the systems under variety of conditions. AI/ML based WAFs include WAF Brain and CNN WAF. WAF Brain is built on top of a Long Short Term Memory (LSTM) Deep Learning model and trained with SQL injection patterns. CNN WAF uses Convolutional Neural Network (CNN) to classify the SQL injection patterns. Rule based open source WAFs include ModSecurity PL1, PL2, PL3 [7] and Coraza PL1, PL2, PL3 [40]. ModSecurity is the OWASPs official WAF with three different paranoia levels. We experimented paranoia levels (PL1, PL2 and PL3) from the lowest to moderate to avoid false positive preventions. On the other hand, Coraza is the OWASP’s modern WAF written with GoLang. With similar approach we included lowest paranoia levels. Commercial cloud WAFs include Cloudflare WAF and AWS WAF. Cloudflare WAF is one of the best and popular WAF in the industry with enhanced CDN and ML enhanced detection capability. AWS WAF is the managed WAF service provided by AWS with custom rules. Other than WAFs, we created a vulnerable web application in a dockerized environment backed by a MySQL server (MySQL 8.0) to validate the execution and semantic correctness of generated SQLi payloads. We used this setup to test generated payloads in different environments and detection approaches. This allows us to avoid overfitting to a specific WAF and see real world behavior of SQLi generation success.

3.5 Prompt Engineering For each generator type we designed a structured prompt template to achieve a system specific task. For all RADAGAS variants we used an identical prompt template to achieve consistent comparison. By default foundation models have guardrails for restricting generation of potentially harmful content. We designed our prompts to enable authorized security research with specific instructions.

3.6 Parameter Grid Configuration This study explores optimal performance and behavior under different settings and conditions. We designed a comprehensive parameter grid to see the performance landscape of our models.

14

3.6.1 Temperature Controls creativity of models by random sampling. In RADAGAS model we tested GPT-4o with four different temperature values to measure the performance more granular. For other models we used two different temperatures to compare the performance of all three foundational models under optimal range.

• RADAGAS GPT-4o: 0.10, 0.30, 0.60, 0.90 (Granular search for optimal temperature) • RADAGAS Claude: 0.60, 0.90 (low and high temperature comparison across models) • RADAGAS DeepSeek: 0.60, 0.90 (low and high temperature comparison across models) For RefleXQLi and other baseline systems we used default temperature (T = 0.7). Although RefleXQLi is our novel CoT+Dual-LLM system, we fix the temperature to isolate the primary effect of structured reasoning to enable controlled comparison with RAG based and other baseline systems.

3.6.2 Diversity Threshold We employ diversity thresholds in the RADAGAS algorithm to promote the generation of diverse SQL injection payloads. Payloads that do not satisfy the specified diversity criteria are rejected based on a threshold value (θ). The threshold is applied uniformly across semantic diversity (cosine similarity), lexical diversity (Levenshtein distance), and contextual diversity (BERTScore). We evaluate threshold values of 0.70, 0.75, 0.80, 0.85, 0.90, and 0.95 for all foundational models used in RADAGAS variants. Total Configurations:

• RADAGAS GPT-4o: 4 temperatures × 6 thresholds = 24 configurations • RADAGAS Claude: 2 temperatures × 6 thresholds = 12 configurations • RADAGAS DeepSeek: 2 temperatures × 6 thresholds = 12 configurations • Total: 48 unique configurations Statistical Reliability: To avoid any potential runtime specific anomaly on the experiments we conducted 5 independent runs per configurations, each run generates 1,000 SQLi payloads, totaling: • RADAGAS GPT-4o: 24 configs × 5 runs × 1000 payloads = 120,000 payloads • RADAGAS Claude: 12 configs × 5 runs × 1000 payloads = 60,000 payloads • RADAGAS DeepSeek: 12 configs × 5 runs × 1000 payloads = 60,000 payloads • Grand Total: 240,000 payloads In RefleXQLi, we do not apply any diversity threshold during the SQLi generation. The system is designed to leverage pure CoT reasoning rather than parameter optimized diversity filtering.

15

3.7 Evaluation Protocol 3.7.1 Performance Measurement To evaluate system performances, each generated payload was tested against ten WAFs and execution testing. For each WAF, we measured the bypass rate and calculated overall mean bypass rate for all WAFs. To get a consistent and valid measurement, we repeated runs five times for each configuration setup and reported the mean bypass rate and standard deviation.

3.7.2 Diversity Measurement For all payload generation systems we compute execution validity, semantic diversity, character level lexical diversity, structural level lexical diversity, contextual diversity, structural level syntactic diversity and functional diversity of each generated payload. We also calculate the mean diversity score and correlate these with bypass rates to understand the relationship in between diversity and bypass rate.

3.7.3 Statistical Analysis To measure the correlation between diversity and bypass rate, we used the Pearson correlation coefficient [41]. A p-value of < 0.05 indicates a significant correlation, while p ≥ 0.05 does not suggest significant relationship between bypass rate and diversity. To verify the robustness of our findings, we also computed the Spearman rank correlation as a non-parametric alternative. Statistical significance tests [42, 43] were used to validate the results within appropriate confidence intervals.

3.8 Experimental Infrastructure and Simulation Parameters 3.8.1 Hardware Configuration • Compute Platform: Dedicated Bare-metal server, Cloud based virtual machines (AWS EC2), AWS Sagemaker, AWS Bedrock • CPU: AMD Ryzen AI 9 365 with Radeon 880M, 2000 Mhz, 10 Core(s), 20 Logical processors, for payload generation and data processing. • GPU: NVIDIA RTX A5000, NVIDIA RTX 5070 for local execution of LLMs and AI based WAF models • Memory: 32GB RAM per instance • Storage: 1TB SSD for payload storage and intermediate results • Network: 10 Gbps High bandwidth or API calls to LLM services

3.8.2 Software Environment • Operating System: Ubuntu 22.04 LTS • Python: 3.10.12 with venv isolation • LLM APIs: OpenAI GPT-4o (via API), Anthropic Claude 3.7 Sonnet (via Amazon Bedrock), DeepSeek-R1 (via API) • MySQL Database: 8.0 for execution validation

16

Table 1: Experiment configurations, usage of diversity and payload statistics across all SQL injection generators. To evaluate comprehensive diversity profile for all systems, six complementary Post-hoc metrics were implemented: (i) Semantic diversity (cosine distance), (ii) Contextual diversity (BERTScore), (iii) Character level lexical diversity (Levenshtein distance), (iv) Structural lexical diversity (n-gram), (v) Syntactic diversity (AST edit distance), and (vi) Functional diversity (execution result). Post-hoc diversity metrics are used for evaluation and correlation analysis, not included in generation process as a feedback mechanism. The diversity feedback RADAGAS uses is a completely different mechanism. Temperature = Sampling temperature. Diversity Feedback = Generation time diversity feedback. Diversity Threshold = Threshold applied in diversity feedback. Configs = Number of parameter configurations. Runs = Number of independent runs per configuration. Payloads = Number of generated SQLi payloads. System

Base Model

Temp. Used

Temp. Values

Div. Feedback

Div. Threshold

Post-hoc Div.

Configs

Runs

Payloads

RADAGAS-GPT4o RADAGAS-DeepSeek RADAGAS-Claude RefleXQLi GenSQLi Vanilla GPT-4o Traditional

GPT-4o DeepSeek-R1 Claude 3.7 GPT-4o GPT-4o GPT-4o Rule-based

Yes Yes Yes Yes Yes Yes No

0.10–0.90 0.60, 0.90 0.60, 0.90 0.70 0.70 0.70 N/A

Yes Yes Yes No No No No

0.70–0.95 0.70–0.95 0.70–0.95 N/A N/A N/A N/A

Yes Yes Yes Yes Yes Yes Yes

24 12 12 1 1 1 1

5 5 5 5 5 5 5

120K 60K 60K 5K 5K 5K 5K

• WAF Testing Framework: Custom Python wrapper integrating ModSecurity OWASP-CRS 4.8.0 Coraza OWASP-CRS 4.8.0, Cloudflare API, AWS WAF API

3.8.3 Simulation Parameters • Total Experiments: 48 parameter configurations × 5 runs = 240 experiments • Total Payloads Generated: 240,000 payloads • Total Payloads Tested against WAFs: Approximately 200,000 payloads (valid payloads) • WAF and Execution Test Calls: 200,000 payloads × 11 targets (10 WAFs + MySQL) = 2,200,000 tests

4 Results This section presents the experimental results of seven SQLi generator systems and their effectiveness against ten WAFs under different configuration sets given in Table 1. On top of that, we performed a seed effect ablation analysis to demonstrate how initial conditions and inputs affect the efficiency of the systems against WAFs. This section is organized in five subsections: overall performance of the systems (Section 4.1), diversity analysis (Section 4.2), parameter optimization for RADAGAS (Section 4.3), performance breakdown per WAF (Section 4.4), and seed effect analysis (Section 4.5).

4.1 Overall Performance of the Systems The overall performance of all seven SQLi generator systems against ten WAFs and an execution check are presented in Fig. 4. The result shows that RADAGAS occupies the top three ranks, and RefleXQLi follows RADAGAS systems and the remaining baseline systems performs significantly lower than the proposed methods. The overall ranking yields the following key observations: 17

6\VWHP3HUIRUPDQFH&RPSDULVRQ (UURUEDUV“VWGGHY



9DQLOOD*37R



7UDGLWLRQDO



*HQ64/L



5HIOH;4/L



5$'$*$6&ODXGH



5$'$*$6'HHS6HHN



5$'$*$6*372 











0HDQ%\SDVV5DWH Fig. 4: WAF bypass rate comparison across all seven systems and seed-driven ablation variant. RefleXQLi V2 (with seeds) achieves the highest bypass rate (35.78%), followed by RADAGAS variants (22.73%, 22.09%, 21.73%) and RefleXQLi (21.21%). Error bars represent standard deviation across independent runs. We observed that all RAG based RADAGAS systems are slightly outperforming the baseline systems with (22.73%, 22.09%, 21.73%). This demonstrates that for all foundational models, RAG architecture provides consistent performance. We analyzed the effect of RAG, comparing the performance of Vanilla LLM and RAG based generator with the same foundation model (GPT-4o) shows that RAG increases the success rate by 76.2%. This demonstrates that providing curated context has a key role in increasing the effectiveness of adversarial generation and raw LLM capability is insufficient to perform successful generation. Our another novel algorithm RefleXQLi achieves 21.21% success rate with 100% payload uniqueness and lowest variance with σ = 0.37%. RefleXQLi V2 is the seed driven version of RefleXQLi and achieves 35.78% bypass rate in our ablation tests (Section 4.5). Plain RefleXQLi does not use any prior context and the system is based on the foundation model only. This suggests, CoT combined with initial high quality seeds can amplify the performance further. When we focus on the variance, our results show that RefleXQLi shows lowest variance with σ = 0.37% and this is followed by RADAGAS-Claude with σ = 0.70%. Baseline model GenSQLi has the highest variance with σ = 2.04% indicating less stability across runs.

4.2 Diversity Analysis Our results show that there is a weak correlation with diversity and effectiveness of generated payloads. Even if the diversity rates are significantly different across the systems, the effect of diversity is minimal on success rate. The main reason of this

18

Table 2: Diversity Analysis: Bypass Rate vs. Six Post-Hoc Diversity Metrics System

Bypass (%)

Uniq. (%)

Semantic Div.

Lexical Div.

Context. Div.

N-gram Div.

AST Div.

Func. Div.

RADAGAS-GPT4o RADAGAS-DeepSeek RADAGAS-Claude RefleXQLi GenSQLi Traditional Vanilla GPT-4o

22.73 22.09 21.73 21.21 20.35 15.01 12.90

23.18 29.95 46.98 100.00 68.34 30.10 65.98

0.567 0.434 0.542 0.585 0.629 0.761 0.577

0.732 0.720 0.762 0.790 0.782 0.791 0.728

0.150 0.153 0.151 0.172 0.162 0.191 0.143

0.845 0.813 0.851 0.867 0.920 0.931 0.891

0.819 0.603 0.867 0.987 0.727 0.694 0.774

0.728 0.660 0.768 0.771 0.735 0.763 0.699

Pearson correlation (Uniqueness vs. Bypass): r = −0.093, p = 0.843 - not significant No metric shows significant correlation (p < 0.05)

Table 3: Cross-System Correlation Analysis (N = 7): Diversity Metrics vs. Bypass Rate Metric

Pearson r

p

Spearman ρ

p

Kendall τ

p

Uniqueness Semantic Lexical Contextual N-gram AST Functional Total

−0.093 −0.548 −0.070 −0.191 −0.654 +0.205 +0.029 −0.230

.843 .203 .881 .682 .111 .659 .950 .620

−0.571 −0.679 −0.357 −0.179 −0.857 +0.143 −0.107 −0.286

.180 .094 .432 .702 .014 .760 .819 .535

−0.429 −0.524 −0.333 −0.238 −0.714 +0.048 −0.048 −0.238

.239 .136 .381 .562 .030 1.00 1.00 .562

observation is the effect of high quality input context and foundation model itself that create more significant impact on the performance than diversity variation. Table 2 shows the success rate of the systems and six post-hoc diversity metric calculations of the generated payloads (Section 3.3). The data pattern shows us, there is no statistically significant correlation between successful generation rate and diversity metrics. On top of this, to measure the correlation and evaluate the robustness across parametric and nonparametric assumptions, we applied three methods: Pearson, Spearman rank and Kendall as shown in Table 3. None of three methods show statistically significant correlation between the six diversity metrics and successful bypass rate (p > 0.05). Only N-gram diversity demonstrates significance under Spearman (ρ = −0.857, p = 0.014) and Kendall (τ = −0.714, p = 0.030). Moreover, when we look at the total aggregated diversity values, there is no significant correlation for three methods (r = −0.230, ρ = −0.286, τ = −0.238 where p > 0.50). This shows the effect of diversity has very low significance compared to other factors on achieving high bypass rates.

4.2.1 Internal system validation To keep architectural system differences at minimum, and analyze the effect of diversity on bypass rate, we set a very comprehensive configuration with 48 configurations with RADAGAS. We set six diversity thresholds (θ ∈ {70, 75, 80, 85, 90, 95}) that control RADAGAS’s runtime feedback mechanism and with this, higher thresholds allow 19

Table 4: Optimal RADAGAS Configurations Identified Through Parameter Grid Search Model

Div. θ

Temp.

Mean (%)

σ (%)

GPT-4o DeepSeek Claude

90 90 75

0.1 0.9 0.6

22.73 22.09 21.73

1.55 1.26 0.70

greater similarity between payloads. The intra system analysis shows that there is a significant positive correlation between diversity threshold and bypass rate (Pearson r = +0.371, p = 0.009; Spearman ρ = +0.352, p = 0.014 and Kendall τ = +0.256, p = 0.016) under the same experimental setup condition (same architecture, same input context, same initial conditions). This result shows us that within a single architecture, having higher diversity does not always increase the bypass rate. Our analysis shows, the stricter diversity threshold θ = 70 sometimes performs poorly compared to higher θ values. Our statistical analysis on the experiment results reveals the correlation of diversity and bypass performance is weak due to three mechanisms: RADAGAS system uses RAG with MMR which allows the system generates high quality payload based on a curated knowledge base. Since the attack payload is successful initially, this mechanism does not force the algorithm to produce more diverse payloads and that causes the effect of diversity not visible. When we focus on the execution success, RADAGAS-GPT4o performs 64.20% MySQL execution success with poor uniqueness, while RefleXQLi has a perfect uniqueness but MySQL bypass score is low with 27.80%. This shows that, increasing the sophistication leads producing syntactically poor and invalid SQL injection payloads. Our experiment results show that optimal parameter settings are not common across all three RADAGAS systems as in the Table 4. The behavior of each RADAGAS variant over 48 configurations is illustrated in Fig. 5 Each RADAGAS variant shows different characteristics under parameter grid, which shows parameter tuning depends on the foundation model. RADAGAS-GPT4o (θ = 90, T = 0.1): While it is expected to get more successful results under high temperatures, GPT-4o shows better performance in low temperatures where the model is more deterministic. In addition to this, under high diversity thresholds where the system is less selective in terms of similar payloads, GPT-4o shows better performance with consistency. RADAGAS-DeepSeek (θ = 90, T = 0.9): In contrast to GPT-4o, DeepSeek shows better performance highest temperature(T = 0.9). Similar trend with GPT-4o is observed on diversity threshold variation as well. Similar to GPT-4o, high diversity thresholds provide better bypass rate in RADAGAS-DeepSeek. However the variation between runs is high in both diversity thresholds and temperature values, which indicates instability of the model. RADAGAS-Claude (θ = 75, T = 0.6): Under same temperature values, Claude 3.7 shows similar behavior with other RADAGAS variants. This shows the effect

20

'LYHUVLW\7KUHVKROG(IIHFWRQ3HUIRUPDQFH *372



&ODXGH

'HHS6HHN



%\SDVV5DWH

      









'LYHUVLW\7KUHVKROG













'LYHUVLW\7KUHVKROG













'LYHUVLW\7KUHVKROG





Fig. 5: Effect of diversity threshold feedback and temperature on bypass performance of RADAGAS variants.

Table 5: Per-WAF Bypass Rates Across Seven Systems and Ten WAF Categories and Execution Test Commercial

Valid.

Overall

System

WAF Brain

ML-Based CNN WAF

MSec PL1

MSec PL2

MSec PL3

Rule-Based Cora. PL1

Cora. PL2

Cora. PL3

Cloud flare

AWS WAF

MySQL Exec

Mean

RADAGAS-GPT4o RADAGAS-DeepSeek RADAGAS-Claude RefleXQLi GenSQLi Traditional Vanilla GPT-4o

74.56 92.49 78.20 85.77 56.98 38.18 60.28

61.72 78.39 80.48 70.01 57.40 50.72 51.16

0.00 0.28 0.06 5.70 16.26 12.94 0.54

0.00 0.00 0.00 0.00 0.28 0.00 0.00

0.00 0.00 0.00 0.00 0.26 0.00 0.00

0.00 0.16 0.04 0.67 14.74 11.54 0.32

0.00 0.06 0.00 0.00 2.10 0.00 0.00

0.00 0.06 0.00 0.00 2.08 0.00 0.00

49.50 10.18 14.70 10.39 20.58 13.70 7.28

0.00 0.42 4.98 33.00 16.18 14.96 0.92

64.20 60.96 60.52 27.80 36.96 23.12 21.44

22.73 22.09 21.73 21.21 20.35 15.01 12.90

of determinism is the same for all three models. Higher temperature lowers overall bypass performance of the systems. However, Claude 3.7 showed mixed performance under different diversity thresholds. There is no correlation captured between diversity threshold and bypass rate. These results show that there is no universal parameter configuration that makes the system runs its peak performance. Each system has its own optimal performance under different configurations. To get best performance from a system, model specific tuning must be done empirically. Table 5 shows comprehensive representation of bypass rates for all seven systems against ten bypass tests and one execution test. Fig. 6 presents system-WAF affinities and reveals the heterogeneous bypass success of overall systems.

4.3 Model Specific Parameter Optimization 4.4 Per-WAF Performance Breakdown The per-WAF success rate analysis shows several important patterns: ML-based WAF performance: ML based WAFs show weak prevention rate against RADAGAS variants with RADAGAS-DeepSeek 92.49% on WAF-Brain and 78.39% on CNN-WAF, followed by RADAGAS-Claude with 78.20% and 80.48% 21

5$'$*$6*372























5$'$*$6'HHS6HHN























5$'$*$6&ODXGH























5HIOH;4/L























*HQ64/L























7UDGLWLRQDO























9DQLOOD*37R























   

U )% :$

DLQ &1

$) 1:

&OR

XGI

ODUH

6 $:

) :$

F3

G6H 0R

/

F3

G6H 0R

/

F3

G6H 0R

/

:$)'HIHQVH6\VWHP

/

D3

UD] &R

/

D3

UD] &R

/

D3

UD] &R

%\SDVV5DWH

$WWDFN6\VWHP

3HU:$)%\SDVV3HUIRUPDQFH

 

HF

[ /(

64 0\

Fig. 6: Bypass performance heatmap shows higher bypass rates in darker shades.

respectively. We note that WAF-Brain [14] and CNN-WAF [13] are research prototypes and they were trained with publicly available datasets. Production grade ML based WAFs may show enhanced detection capabilities. Rule based WAF resilience: Interestingly, rule based WAFs show very strong prevention rate especially with higher paranoia levels with PL2 and PL3. Even in lowest paranoia level PL1, GenSQLi performed highest bypass level with 16.26%. However, qualitative analysis showed that GenSQLi’s 813 ModSecurity PL1 bypass payloads reveal that a significant portion are false positives with 11.2% that are malformed artifacts and 29.3% are tautological expressions which do not expose real threat. We manually classified all 813 bypass payloads into four categories:

• data exfiltration queries like (UNION SELECT) • tautological expressions like (OR 1=1, AND ”=”) • time-based blind (SLEEP, BENCHMARK) • malformed SQL Additionally, the experiment did not show a significant correlation (r = −0.093, p = 0.843) in between bypass rate and uniqueness as in the Fig. 7. If the bypass rate is higher in certain patterns, the systems may achieve high bypass rate with less unique payloads. However this creates a practical weakness where dynamic WAFs can prevent certain patterns and the generator needs to cover more diverse pattern to sustain bypass capability. MySQL execution: RADAGAS variants generated highest valid execution rate with 60–64%. This is followed by GenSQLi with 36.96% and RefleXQLi (27.80%). Traditional and Vanilla GPT-4o stayed at (23.12%) and (21.44%) respectively. Since RAG systems use curated knowledgebase, the execution rate is higher. Whereas GenSQLi generated tautologies which are similar simple payloads and passed the execution test. Traditional and zero-shot systems could not achieve that high rates. Commercial WAF performance: RADAGAS-GPT4o performed the best Cloudflare bypass rate with 49.50%. Our another novel algorithm RefleXQLi achieved 22

'LYHUVLW\YV8QLTXHQHVV 3HDUVRQU S   5$'$*$6*372 5$'$*$6&ODXGH

0HDQ%\SDVV5DWH



5HIOH;4/L

5$'$*$6'HHS6HHN



*HQ64/L

  7UDGLWLRQDO



5$'$*$6 5HIOH;4/L *HQ64/L 7UDGLWLRQDO 9DQLOOD*37R

9DQLOOD*37R

 





8QLTXHQHVV5DWH





Fig. 7: Uniqueness rate vs. WAF bypass rate across all seven systems (r = −0.093, p = 0.843, not significant). RADAGAS-GPT4o achieves the highest bypass (22.73%) with only 23.18% uniqueness, while RefleXQLi (100% unique) ranks fourth (21.21%), demonstrating that uniqueness alone does not determine bypass effectiveness.

33.00% bypass rate on AWS WAF. Whereas, GenSQLi performed 20.58% on Cloudflare. These results show that commercial WAF providers adopted heterogeneous detection strategies, since different generation systems bypass different rule gaps. Complementary strengths: Results revealed that the success rate is system specific and each system has different gaps and patterns. The highest bypass rates for WAF-Brain is 92.49% (RADAGAS-DeepSeek), CNN-WAF is 80.48% (RADAGASClaude), Cloudflare is 49.50% (GPT-4o) and rule based WAFs are dominated by GenSQLi for Coraza PL1 with 14.74% and ModSecurity PL1 with 16.26%. This shows that combination of different generation strategies can maximize the coverage.

4.5 Seed Effect Analysis This section presents the seed effect ablation experiment for RefleXQLi and GenSQLi. In the previous sections, we investigated the effect of configuration parameters. However there is still a big question on effectiveness of seed mechanism. To answer this question, we created a controlled seed effect experiment comparing two systems that use the same GenSQLi V2 paper seeds [8] and denoted these system variants as GenSQLi V2 and RefleXQLi V2.

23

6HHG(IIHFW$QDO\VLV6DPH96HHGV'LIIHUHQW3LSHOLQHV D%\SDVV5DWH6DPH96HHGV





SS 6HHGEDVHOLQH

  



SS

%\SDVV5DWH

0HDQ%\SDVV5DWH



E3HU:$)*HQ64/L9YV5HIOH;4/L9





*HQ64/L9 9 5HIOH;4/L



 



 

 6HHG2QO\ 96WULQJ

*HQ64/L9 7HPSODWH

5HIOH;4/L9 5HDVRQLQJ









 

 :$)%UDLQ &11:$) 0RG6HF 3/

&ORXGIODUH

$:6 :$)

0\64/ ([HF

Fig. 8: Seed effect analysis. Left: Overall bypass rate comparison showing raw V2 seeds (28.93%), GenSQLi V2 (+0.91pp), and RefleXQLi V2 (+6.85pp). CoT reasoning (RefleXQLi provides 7.5× more improvement than template-based mutation (GenSQLi). Right: Per-WAF comparison between GenSQLi V2 and RefleXQLi V2 using the same seeds

Table 6: Seed Effect: Seed-Only bypass rate represents the raw paper seeds tested directly against all 11 WAFs without any LLM processing. Configuration Seed-Only (V2 string) GenSQLi V2 RefleXQLi V2

Bypass (%)

σ (%)

MySQL (%)

∆ vs Seed

28.93 29.84 35.78

— 0.79 0.44

— 80.97 98.36

— +0.91 +6.85

4.5.1 System Comparison For the experiment, we prepared 11 GenSQLi seeds from the paper and converted them to string context with quote break prefix and comment suffix, since our backend services are vulnerable to the parameter in this format. Then we generated 5×1,000 payloads for each of systems. Fig. 8 visualizes the seed effect comparison, showing both bypass improvement of the seed and v2 models at left and breakdown of the performance per WAF at right. Table 6 shows a significant contrast between two V2 pipelines. While GenSQLi V2 adds +0.91pp performance with 29.84% bypass rate over original seed bypass rate (28.93%), RefleXQLi V2 achieves +6.85pp and reaches 35.78% bypass rate demonstrating that template based mutation techniques provide minimal improvement on raw seeds, while CoT based reasoning techniques achieve particularly significant improvement on the overall bypass rate. Valid MySQL execution rate further differentiates the systems where RefleXQLi V2 achieves 98.36% and GenSQLi V2 has 80.97% semantic correctness showing CoT produces both semantically and syntactically valid generation outputs rather than template based approaches.

24

Table 7: Per-WAF Bypass Rates with Same V2 Seeds WAF

GenSQLi V2 (%)

RefleXQLi V2 (%)

∆

WAF-Brain CNN-WAF ModSec PL1 ModSec PL2 ModSec PL3 Cloudflare AWS WAF MySQL Exec Coraza PL1 Coraza PL2 Coraza PL3

89.71 62.80 39.80 0.96 0.96 8.87 40.79 80.97 1.22 1.09 1.09

81.83 66.55 50.19 0.00 0.00 45.96 50.65 98.36 0.02 0.00 0.00

−7.88 +3.75 +10.39 −0.96 −0.96 +37.09 +9.86 +17.39 −1.20 −1.09 −1.09

Mean

29.84

35.78

+5.94

Table 8: Seed Effect Diversity: Same Seeds, Different Pipelines Metric

GenSQLi V2

RefleXQLi V2

Uniqueness (%) Semantic Lexical Contextual N-gram AST Functional

66.14 0.3787 0.6097 0.1265 0.7527 0.6814 0.3830

95.84 0.4385 0.6312 0.1257 0.8176 0.6941 0.3999

Total Diversity

0.4887

0.5178

4.5.2 Seed Effect Comparison Table 7 shows granular performance for two systems using V2 seeds per WAF which reveals the complementary strengths of both systems. The results reveal that RefleXQLi V2 shows higher performance than GenSQLi V2 on commercial WAFs with Cloudflare (+37.09pp) and AWS WAF (+9.86pp) suggesting that CoT reasoning discovers obfuscation techniques effectively on the commercial WAFs. This pattern is also similar in rule based WAF ModSecurity PL1 (+10.39pp) revealing the effectiveness of CoT over in-context learning generation. However, GenSQLi V2 shows better performance on WAF-Brain with +7.88pp which suggests that usage of both systems provides a complementary coverage on overall systems.

4.5.3 Diversity Comparison Under Same Seeds Table 8 shows the comparison of six post-hoc diversity metric performance of two V2 systems. The results reveal that RefleXQLi V2 has higher diversity levels for five out of six metrics with 0.5178 vs. 0.4887 overall diversity score and substantially higher

25

uniqueness rate with 95.84% vs. 66.14%. GenSQLi V2 outperforms RefleXQLi V2 for only contextual diversity (BERTScore) with 0.1265 vs. 0.1257 which is a negligible difference. This result demonstrates that CoT based reasoning provides higher diversity and produces more effective payloads than in-context learning generation.

4.5.4 Seed Quality and LLM Reasoning There are two significant findings of the seed effect analysis experiment. The first and most significant result suggests that the seed quality provides a strong baseline where the raw seeds achieved 28.93% success rate without any LLM intervention. The second finding is RefleXQLi V2 has a higher MySQL execution rate with 98.36% while GenSQLi has 80.97% levels suggesting that CoT reasoning is very effective in creating syntactically correct outputs. Overall results show that investment in both seed engineering and CoT reasoning yields successful returns for adversarial generation.

5 Discussion 5.1 Role of Diversity in Adversarial Generation The multi-method analysis across three correlation methods Pearson, Spearman, Kendall shows that out of six diversity metrics there is no significant correlation with bypass rate of the systems. The only significant finding is, negative Spearman correlation of N-gram diversity suggests that lower structural lexical diversity contributes higher bypass rates. Intra system analysis with 48 configurations for RADAGAS variants shows significant positive correlation between diversity threshold and bypass rate with r = +0.371, p = 0.009. That means, in a single setup and architecture, tuning diversity improves performance higher than architecturally different systems. That suggests that architectural design, retrieval strategy and reasoning mechanisms dominate over diversity metrics and that makes diversity effect negligible.

5.2 Model Specific Behavior and Training Methodology Out of 48 configurations, three foundation models of RADAGAS showed optimal performance under different T values with GPT-4o at T = 0.1, DeepSeek at T = 0.9, Claude at T = 0.6 suggesting that T values are model specific. These findings reveal direct implications for adversarial generation. There is no generic parameter set that can be transferred across models to obtain optimal efficiency for adversarial generation, since each foundation model requires independent empirical tuning.

5.3 Architectural Tradeoffs Our evaluation shows complementary strengths across different architectural designs: RAG based systems (RADAGAS) achieve highest performance across seven generation systems. The success lies in leveraging proven successful payloads from curated lists. RAG based systems had a very poor performance against rule based WAFs 26

where the established rules cover the known payloads very well. However, RAG based systems were successful against AI/ML based WAFs where DeepSeek showed 92.49% bypass rate on WAF-Brain suggesting ML based WAFs were not trained well to protect against known payloads. CoT+Adversarial (RefleXQLi) performed a competitive performance with 21.21%, σ = 0.37% in V1 mode. In V2 mode, seed effect ablation (Section 4.5) shows adding high quality seeds increases the bypass rate to 35.78% with 98.36% MySQL execution suggesting that seed quality and CoT reasoning have big impact on adversarial generation. Baseline systems perform lower. In-context learning algorithm GenSQLi achieves highest nominal bypass rates on rule based WAFs, with majority of payloads being tautological expressions. Vanilla GPT-4o and Traditional SQLi showed poor performance compared to other systems with 12.90%, 15.01% bypass rates respectively. Overall results show that, complementary WAF results suggests, combining CoT, RAG and In context learning generators together, maximizes the coverage across diverse WAFs.

5.4 Technical Limitations and Considerations There are five main aspects in terms of technical limitations. WAF coverage: While our experiment has a rich set of WAFs with commercial, rule based and AI/ML based, the results may not be generalized to all commercial WAFs or custom enterprise settings and deployments [44]. Our tests were performed against specific WAF versions in years 2025 and 2026. Attack type scope: We limited the attack scope with SQL Injection. However, generalization to other attack types (i.e XSS, Command Injection, SSRF) may require another empirical study with different approach. Database specificity: In our experiments, we used MySQL 8.0 to check SQLi validity. However, SQL dialects have differences across PostgreSQL, MSSQL and Oracle that may affect structural and semantic correctness results [11]. Sample size for cross system correlation: The cross system correlation analysis is done with N = 7 data points limiting statistical power and preventing us from definitive conclusions about the correlation. To mitigate this gap, we employed three complementary strategies. First, for small samples, we employed non parametric methods (Spearman, Kendall τ ). Second, to quantify uncertainty, we computed bootstrap 95% confidence intervals with 10,000 resamples and lastly, we validated the findings with intra-system analysis with 48 RADAGAS configurations (N = 48) which provided sufficient statistical power (β > 0.80). Temporal validity: Since the systems evolve continuously, our results reflect the state of the LLM and WAF detection systems as of early 2026.

5.5 Future Directions As a next step of this study, we have several research directions from our findings. Since our tests were only SQL injection specific, verifying the results with other attack types (XSS, SSRF, Command Injection etc.) will extend the results to provide more

27

generalized context. Another direction can be extending the coverage on other foundational models to analyze model specific optimal parameter settings and patterns more. Moreover, extending execution validation within multiple databases to provide more comprehensive landscape for all SQL injection types to generate DBMS specific attack generation strategies.

5.6 Ethical Considerations and Responsible Disclosure All experiments were conducted in isolated environments. The generated SQLi payloads were executed in self hosted environment with necessary authorizations. We tested Commercial WAFs with our own private testing accounts backed by specially crafted dummy web endpoints to measure the effectiveness of the rules. The generated payloads have never been used against unauthorized systems. The source code is kept as private to prevent any potential misuse of the system to generate malicious payloads.

6 Conclusion This paper presents an extensive evaluation of pure LLM based SQL Injection attack generation methods and their effectiveness through two novel adversarial generation systems RADAGAS (three variants, RAG based) and RefleXQLi (CoT based) benchmarked alongside three baseline systems. These seven systems were tested against ten WAFs and an execution test. The overall experiment generated 240,000 payloads and performed 2.2 million WAF tests constituting one of the largest benchmarking studies of LLM based adversarial SQL injection generation. Our experiments yielded findings that have theoretical and practical significance. While the expectation was that diversity drives the bypass effectiveness, the results show that the contribution of diversity is minimal or negligible considering the huge effect of seed effect combined with CoT reasoning. Practically, our results provide solid real life deployment guidelines where complementary usage of different systems has more coverage: RAG systems for AI/ML WAFs, CoT for stable adversarial generation, and seed driven CoT provides the highest bypass rate.

AI Transparency In this paper, generative AI methods are compared in a scientific framework. Apart from that, AI-based tools are utilized for grammar checks and corrections following the human-in-the-loop principle, i.e., all AI suggestions were carefully checked. The authors reviewed and edited all AI-assisted improvements and take full responsibility for the final manuscript.

Declarations • Competing interests: The authors have no relevant financial or non-financial interests to disclose. • Data availability: Data will be made available upon reasonable request. 28

• Funding: Not applicable • Conflict of interest: The authors declare that they have no conflict of interest. • Ethics approval and consent to participate: Not applicable • Consent for publication: Not applicable • Author contribution: A. Karakoc: Conceptualization, Methodology, Software, Investigation, Writing - Original Draft, Visualization. H. B. Yilmaz: Supervision, Conceptualization, Writing - Review & Editing.

References [1] OWASP Foundation: OWASP Top 10-2021: The Ten Most Critical Web Application Security Risks (2021). https://owasp.org/Top10/ [2] Damele, B., Stampar, M.: SQLMAP: Automatic SQL Injection and Database Takeover Tool. Open-source penetration testing tool (2006). https://sqlmap.org/ [3] OpenAI: GPT-4 Technical Report (2024). https://arxiv.org/abs/2303.08774 [4] Anthropic-AI: The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card 1(1), 4 (2024) [5] DeepSeek-AI: DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024). https://arxiv.org/abs/2406.11931 [6] Manes, V.J., Han, H., Han, C., Cha, S.K., Egele, M., Schwartz, E.J., Woo, M.: The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering 47(11), 2312–2331 (2019) [7] OWASP: ModSecurity: Open Source Web Application Firewall (2024). https: //modsecurity.org/ [8] Babaey, V., Ravindran, A.: GenSQLi: A generative artificial intelligence framework for automatically securing web application firewalls against structured query language injection attacks. Future Internet 17(1) (2025) https://doi.org/10.3390/ fi17010008 [9] Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., Sun, X., Li, L., Sui, Z.: A survey on in-context learning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1107– 1128. Association for Computational Linguistics, Miami, Florida, USA (2024). https://doi.org/10.18653/v1/2024.emnlp-main.64 [10] Kindy, D.A., Pathan, A.-S.K.: A survey on SQL injection: Vulnerabilities, attacks, and prevention techniques. In: IEEE International Symposium on Consumer Electronics (ISCE), pp. 468–471 (2011). https://doi.org/10.1109/ISCI.2011.5973873 [11] Clarke, J.: SQL Injection Attacks and Defense. Elsevier, Waltham, MA (2012)

29

[12] Appelt, D., Nguyen, C.D., Briand, L.: Behind an application firewall, are we safe from SQL injection attacks? In: IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10 (2015) [13] Pan, Y., Sun, F., Teng, Z., White, J., Schmidt, D.C., Staples, J., Krause, L.: Detecting web attacks with end-to-end deep learning. Journal of Internet Services and Applications 10(1), 1–22 (2019) [14] BBVA-Labs: WAF-Brain: Machine Learning Based Web Application Firewall (2018). https://github.com/BBVA/waf-brain [15] Gui, Z., Wang, E., Deng, B., Zhang, M., Chen, Y., Wei, S., Xie, W., Wang, B.: SqliGPT: Evaluating and utilizing large language models for automated SQL injection black-box detection. Applied Sciences 14(16) (2024) https://doi.org/10. 3390/app14166929 [16] Yang, T., Jiang, Z., Wang, Y.: LLMSQLi: A black-box web SQLi detection tool based on large language model. In: IEEE International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), pp. 629–633 (2024) [17] Sun, Y., Wu, D., Xue, Y., Liu, H., Ma, W., Zhang, L., Liu, Y., Li, Y.: LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning (2025). https://arxiv.org/abs/2401.16185 [18] Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In: USENIX Security Symposium (USENIX Security 24), pp. 847–864 (2024). https://www.usenix.org/conference/ usenixsecurity24/presentation/deng [19] Liu, Y., Deng, G., Li, Y., Wang, K., Wang, Z., Wang, X., Zhang, T., Liu, Y., Wang, H., Zheng, Y., Zhang, L.Y., Liu, Y.: Prompt Injection attack against LLMintegrated Applications (2025). https://arxiv.org/abs/2306.05499 [20] Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The Curious Case of Neural Text Degeneration (2020). https://arxiv.org/abs/1904.09751 [21] Meister, C., Pimentel, T., Wiher, G., Cotterell, R.: Locally typical sampling. Transactions of the Association for Computational Linguistics 11, 102–121 (2023) https://doi.org/10.1162/tacl a 00536 [22] Renze, M.: The effect of sampling temperature on problem solving in large language models. In: Findings of the Association for Computational Linguistics (EMNLP), pp. 7346–7356. Association for Computational Linguistics, Miami, Florida, USA (2024). https://doi.org/10.18653/v1/2024.findings-emnlp.432

30

[23] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language Models are Few-Shot Learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020). https://proceedings.neurips.cc/paper files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [24] Wahaibi, S.A., Foley, M., Maffeis, S.: SQIRL: Grey-Box detection of SQL injection vulnerabilities using reinforcement learning. In: USENIX Security Symposium (USENIX Security), pp. 6097–6114. USENIX Association, Anaheim, CA (2023). https://www.usenix.org/conference/usenixsecurity23/presentation/al-wahaibi [25] Hu, Z., Beuran, R., Tan, Y.: Automated penetration testing using deep reinforcement learning. In: IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 2–10 (2020) [26] Zhong, R., Chen, Y., Hu, H., Zhang, H., Lee, W., Wu, D.: SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback. In: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, New York, NY, USA, pp. 955–970 (2020). https://doi.org/10. 1145/3372297.3417260 [27] Zalewski, M.: American fuzzy lop (AFL) fuzzer (2017). https://lcamtuf. coredump.cx/afl [28] Lemieux, C., Sen, K.: Fairfuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage. In: Proceedings of the ACM/IEEE International Conference on Automated Software Engineering, New York, NY, USA, pp. 475–485 (2018). https://doi.org/10.1145/3238147.3238176 [29] Böhme, M., Pham, V.-T., Roychoudhury, A.: Coverage-based greybox fuzzing as markov chain. In: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, New York, NY, USA, pp. 1032–1043 (2016). https: //doi.org/10.1145/2976749.2978428 [30] Carbonell, J., Goldstein, J.: The use of MMR, diversity-based reranking for reordering documents and producing summaries. In: Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 335–336. Association for Computing Machinery, New York, NY, USA (1998). https://doi.org/10.1145/290941.291025 [31] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27 (2014). https://proceedings.neurips.cc/ paper files/paper/2014/file/f033ed80deb0234979a61f95710dbe25-Paper.pdf 31

[32] Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q.V., Zhou, D.: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837 (2022). https://proceedings.neurips.cc/paper files/paper/2022/ file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf [33] PortSwigger: Web Security Academy: SQL Injection (2024). https://portswigger. net/web-security/sql-injection [34] swisskyrepo, contributors: PayloadsAllTheThings: A List of Useful Payloads and Bypass for Web Application Security. Community-maintained security payload repository (2024). https://github.com/swisskyrepo/PayloadsAllTheThings [35] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982–3992 (2019). https://doi.org/10.18653/v1/D19-1410 [36] Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: MTEB: Massive Text Embedding Benchmark. In: Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, pp. 2014–2037 (2023). https://doi.org/10.18653/v1/2023.eacl-main.148 [37] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7(3), 535–547 (2019) [38] Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet Physics Doklady, vol. 10, pp. 707–710 (1966). Soviet Union [39] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evaluating Text Generation with BERT (2020). https://arxiv.org/abs/1904.09675 [40] OWASP: Coraza Web Application Firewall (2024). https://coraza.io/ [41] Pearson, K.: Notes on the History of Correlation. Biometrika 13(1), 25–45 (1920) [42] Fisher, R.A.: Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika 10(4), 507–521 (1915) [43] Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Routledge, New York (2013) [44] Razzaq, A., Latif, K., Ahmad, H.F., Hur, A., Anwar, Z., Bloodsworth, P.C.: Semantic security against web application attacks. Information Sciences 254, 19–38 (2014) https://doi.org/10.1016/j.ins.2013.08.007

32