RedShell: A Generative AI-Based Approach to Ethical Hacking Ricardo Bessa1,2 , João Trindade2 , Rui Claro2 , and João M. Lourenço1
arXiv:2604.11506v1 [cs.CR] 13 Apr 2026
1
NOVA University Lisbon — FCT & NOVA LINCS, Portugal [email protected] [email protected] 2 Layer8 - Shield Domain SA, Portugal [email protected] [email protected]
Abstract. The application of Machine Learning techniques in code generation is now a common practice for most developers. Tools such as ChatGPT from OpenAI leverage the natural language processing capabilities of Large Language Models to generate machine code from natural language descriptions. In the cybersecurity field, red teams can also take advantage of generative models to build malicious code generators, providing more automation to Pentest audits. However, the application of Large Language Models in malicious code generation remains challenging due to the lack of data to train and evaluate offensive code generators. In this work, we propose RedShell, a tool that allows ethical hackers to generate malicious PowerShell code. We also introduce a ground truth dataset, combining publicly available code samples to fine-tune models in malicious PowerShell generation. Our experiments demonstrate the strong capabilities of RedShell in generating syntactically valid PowerShell, with fewer than 10% of the generated samples resulting in parse errors. Furthermore, our specialized model was able to produce samples that were semantically consistent with reference snippets, achieving a competitive performance on standard output similarity metrics such as Edit Distance and METEOR, with their mean similarity scores exceeding 50% and 40%, respectively. This work sheds light on the state-of-the-art research in the field of Generative AI applied to Pentesting, and also serves as a steppingstone for future advancements, highlighting the potential benefits these models hold within such controlled environments. Keywords: Cybersecurity · Ethical Hacking · Large Language Models
1
Introduction
In recent years, Large Language Models (LLMs) sparked a revolution in natural language processing through advanced transformers [28]. Generative models such as ChatGPT [22] have demonstrated strong capabilities while performing generic tasks [30]. In addition, fine-tuning techniques have been employed to adapt LLMs to address more specific challenges, with Generative Artificial Intelligence (AI) becoming pervasive in the workflow of many software engineering tasks [9].
2
R. Bessa et al.
In the offensive cybersecurity field, ethical hackers can also take advantage of LLMs to develop malicious code generators, providing more automation to Pentest audits. Penetration Testing, often referred to as Pentesting, is a crucial activity for red teams, which are responsible for testing the cybersecurity effectiveness of a system through simulated cyber-attacks [29]. By detecting potential security lapses in a target system, Pentesters are able to prevent attacks that could cause harm to that system and its users. Ethical hackers typically use malicious code to exploit the previously identified vulnerabilities, assessing what real attackers could gain after a successful intrusion. However, regardless of the exploitation technique being employed, writing offensive code requires time and technical knowledge from the Pentesters. While the desired automation could be provided by LLMs, this strategy also remains challenging due to the lack of data to train offensive code generators. In this work, we propose RedShell, an AI-based malicious PowerShell generator designed to provide more automation to Pentesting activities targeting Microsoft Windows. We also introduce a malicious PowerShell ground truth dataset, combining code samples from various cybersecurity frameworks to train and evaluate models in offensive PowerShell generation. The evaluation focuses on gauging the quality of the generated snippets by examining their syntactic and semantic correctness, as well as performing a comparative analysis of the generation capabilities of different LLMs. Experiments show that our tool was able to generate syntactically valid PowerShell code, with fewer than 10% of the generated samples resulting in parse errors. Additionally, RedShell produced samples closely aligned with reference snippets, achieving high scores in common distance metrics such as Edit Distance and METEOR, with their mean similarity scores exceeding 50% and 40%, respectively. The remainder of this article is organized as follows: Section 2 introduces an extended version of a malicious PowerShell dataset from the literature; Section 3 details the design of RedShell, our specialized LLM fine-tuned on this dataset; Section 4 presents experiments validating RedShell’s generation capabilities; Section 5 reviews related work, and finally, Section 6 concludes the article with a discussion of RedShell’s limitations and directions for future work.
2
Ground Truth Dataset
Offensive code datasets are a crucial component for red teams to develop malicious code generators. However, there is a significant gap in the public availability of offensive code samples. Without collecting the required amount of snippets, the training and evaluation of malicious code generators is highly compromised. To develop RedShell, our approach was to build a ground truth dataset combining offensive PowerShell snippets and their corresponding descriptions in the English language. We built the ground truth dataset in two distinct phases, presenting the reference ground truth and the extended ground truth datasets. The reference ground truth dataset corresponds to the offensive PowerShell collection provided by Liguori et al. in [16]. The dataset is composed of 1,127
RedShell
3
samples of offensive PowerShell, capturing different real-world scenarios where Pentesters may take advantage of PowerShell to target Microsoft Windows. The snippets comprising the reference dataset were collected from communitydriven cybersecurity wikis and blogs about ethical hacking such as Red Team Recipe [23]. Additionally, snippets were also extracted from various cybersecurity frameworks, including Atomic Red Team [1]. The reference ground truth dataset covers 13 out of 14 offensive tactics described by MITRE ATT&CK [4], only leaving uncovered the Resource Development tactic. We built an extended version of the reference ground truth dataset by collecting new malicious samples, thus improving the overall quality and size of the dataset. We introduced 1,135 additional code samples of offensive PowerShell, effectively doubling the size of the original dataset. The collected samples were extracted from offensive PowerShell modules such as Mimikatz [6] and Nishang [19], and from tryhackme [25] walkthroughs on capture the flag challenges. 700
Number of Samples
600
681
Reference Samples [16] New Samples
581
500 400 300 200
272 159 118
100
82
74
73
72
65
31
25
18
12
D
D is ef en cov er se C y re de Ev nt asi on ia lA c La Ex ces te s ra ecu l M ti C ov on R om ec em o m an nna ent iss d an an d ce C o Pr nt P iv ro ile ers l ge ist Es enc e ca la C tion ol le ct io R Im n es ou Ex pa ct rc fil e D tra ev tio el op n In iti men al t A cc es s
0
Fig. 1: Ground truth dataset coverage of the MITRE ATT&CK tactics. While we manually classified the offensive tactic of each new collected sample according to MITRE ATT&CK documentation, for the reference dataset we relied on the classification provided by Liguori et al. in [16]. Figure 1 presents the number of collected PowerShell samples for each tactic, highlighting how both reference and extended datasets contribute to the final ground truth composition. The new collected samples improved significantly the dataset coverage of the offensive tactics typically employed by Pentesters, including Discovery, Defense Evasion, and Credential Access. The high availability of Discovery samples highlights the capabilities offered by PowerShell to perform various discovery activities on Microsoft Windows devices such as permission groups discovery. The ground truth dataset also provides a wide representation of Defense Evasion strategies, employed by ethical hackers to avoid detection and conceal traces of their malicious activities. Evasion techniques include disabling or mod-
4
R. Bessa et al.
ifying security software such as the Windows Defender and the Antimalware Scan Interface (AMSI), to allow the stealthy execution of malicious programs. Additionally, security professionals take advantage of Credential Access strategies to dump credentials and steal account names and passwords. Legitimate credentials can then be used to access sensitive information and services while making malicious activities harder to detect.
3
RedShell Design
To develop RedShell, we followed a methodology where the malicious PowerShell dataset described in Section 2 was employed as a knowledge base to fine-tune and compare three different LLMs. 3.1
Models
We selected LLMs based on the following criteria: i ) strong performance in coding tasks such as code generation, summarization, and reasoning; ii ) representation of the state-of-the-art model families and architectures; iii ) public availability of model weights to allow local fine-tuning; and iv ) relatively small model sizes to support efficient experimentation. The chosen models are: Qwen2.5-7B [24] Belongs to the latest series of Qwen LLMs, which offer significantly more knowledge than the previously released versions while providing improved capabilities in coding and mathematics. Qwen2.5-Coder-7B-Instruct [12] Belongs to the latest series of code-specific Qwen LLMs, offering strong capabilities in code generation, reasoning and fixing. Llama3.1-8B [10] Released on July of 2024 as part of the multilingual collection of LLMs from Meta, outperforming many of the available open-source and closed models on common industry coding benchmarks. The selected models were downloaded from HuggingFace [8] and fine-tuned locally, providing properties to our solution that overcome the limitations presented by closed models such as ChatGPT from OpenAI. When compared to proprietary LLMs, RedShell offers the following properties: Privacy. By manipulating our specialized models locally, we avoid to share sensitive data with private companies whose proprietary LLMs are only accessible through an external API. Specialization. Our specialized models were specifically trained to assist Pentesters in malicious PowerShell generation. The same training process could not be applied to closed LLMs since their weights are not publicly accessible.
RedShell
5
Ethical Boundaries. Closed LLMs are usually protected by ethical boundaries that restrict their generation capabilities. Although these protections can be often bypassed through prompt-engineering techniques, that manipulation requires time and effort from the Pentesters. In contrast, the generation capabilities of our specialized LLMs are not limited by ethical boundaries. While unrestricted LLMs may raise ethical concerns, attackers will inevitably exploit them, making it essential for security professionals to do the same to effectively prevent real threats. Additionally, to ensure the responsible use of RedShell, we excluded from its training data all the Impact samples that could potentially compromise the integrity of target systems, thus aligning our tool with the non-destructive nature of Pentesting. 3.2
Fine-Tuning
The training of the selected LLMs was conducted through Unsloth [26], a finetuning framework that manually patches complex mathematical steps and optimizes GPU kernels and VRAM (Video Random Access Memory) allocation to make training faster without any hardware changes. Unsloth also supports partial fine-tuning processes through the Low Rank Adaption Method (LoRA) [27], which allows the training to adjust only a small number of weights from the selected models, reducing the computational costs of the fine-tune. We adopted standard values for both LoRA and training parameters, following the recommendations provided by the Unsloth documentation. Additionally, to determine the most effective training configuration, we conducted a set of experiments evaluating the performance of models fine-tuned under different settings. We describe the experiments in more detail in Subsection 3.3. In particular, we parametrized LoRA with both rank and alpha equal to 64, and a 0 dropout. For the fine-tuning process, we employed a batch size equal to 8 and a learning rate equal to 2 × e−4 . We also provided the LLMs with a context that included a detailed description of their expected behavior. The context was defined as “Act as a malicious PowerShell generator. Generate commands in a single line, separated by semicolons and provide no further explanations”. The ground truth dataset was randomly split in two partitions, the training and the test datasets. Since the fine-tuning was conducted with a small and manually curated dataset, we adopted a 90/10 train-test split to maximize the amount of data available for training, optimizing the intended knowledge transfer. The test dataset, while comprising a significantly smaller part of ground truth, still provides a sufficient amount of unseen data to assess the generalization capabilities of the fine-tuned models and detect potential overfitting scenarios. Since the reference dataset provided only aggregate tactic counts rather than a tactical classification per snippet, the train-test split does not guarantee balanced representation across MITRE ATT&CK tactics. However, this does not represent a threat to the validity of RedShell since the most common Pentesting tactics such as Discovery, Defense Evasion and Credential Access dominate the dataset, meaning that random partitioning still ensures a high likelihood of their presence on both partitions.
6
R. Bessa et al.
The fine-tuning processes were executed on a local Linux machine, employing a single NVIDIA GeForce RTX 4090 GPU and 23.643 GB of VRAM. By taking advantage of the optimizations provided by Unsloth and LoRA, our strategy minimizes the time, energy and computational resources required to fine-tune the selected LLMs. Table 1 presents the training times of our specialized models (both in number of epochs and minutes), and the peak reserved VRAM for each model fine-tuned with the reference ground truth dataset. We can perceive that the observed VRAM peaks are significantly far below our max VRAM limit of 23.643 GB. Additionally, the number of epochs (i.e. complete model iterations through the training data) was defined based on experimentation (Subsection 3.3), reflecting the different learning abilities of each LLM. Table 1: Training report of LLMs fine-tuned with the reference dataset. Model
Epochs
Total Training Time (min)
Peak Reserved VRAM (GB)
Llama-3.1 (8B)
18
28
16.725
Qwen2.5-Coder (7B)
20
30
16.865
Qwen2.5 (7B)
28
47
17.600
3.3
Evaluation
An experimental evaluation was conducted to assess the effectiveness of employing the fine-tuned LLMs as malicious PowerShell generators. We inferred our specialized models using the previously unseen code descriptions from the test dataset. The evaluation aimed to validate the quality of the generated code blocks by examining their syntactic and semantic correctness. The syntactic correctness of the PowerShell snippets was evaluated based on the number and severity of the syntactic flaws identified by PSScriptAnalyzer [18], a static PowerShell code checker provided by Microsoft. The reported occurrences were the following: Parse Errors. High-severity errors that occur during the parsing of the PowerShell code, preventing the execution of the generated samples. Warnings. Flaws that may alert for the presence of bad coding practices or unexpected PowerShell patterns. Errors. High-severity flaws that alert for the violation of semantic and security rules from PowerShell. The presence of parse errors in the generated samples was a crucial metric to identify the snippets that could not be executed. In contrary, PowerShell warnings and errors typically do not prevent the code from executing. However, these occurrences allowed us to evaluate the quality of the generated samples in terms of the adherence to the best PowerShell practices.
RedShell
7
The results from the syntactic evaluation were then used to compute the parse error, warning, and error percentages in the samples generated by our specialized models. Since a single sample could potentially contain multiple parse errors, warnings and errors, simultaneously, our approach was to classify a sample as having a parse error if the code for that sample registered one or more parse errors, regardless of the additional presence of warnings or errors. Samples that did not register parse errors were then classified as containing warnings or errors if their code included at least one warning or one error, respectively. Samples (without parse errors) containing both warnings and errors were classified under both categories. Additionally, we defined the semantic correctness of the PowerShell code as a mean distance that statistically measures the similarity between the generated snippets and the corresponding expected snippets in the test dataset. To measure the semantic correctness of the PowerShell samples, we computed the standard output similarity metrics, described by Liguori et al. in [15]: ROUGE-L Measures the similarity between the reference and generated code samples based on the longest common subsequence metric, producing a score that ranges between 0 (perfect mismatch) and 1 (perfect matching). We employed the rouge [20] Python package to compute ROUGE-L. METEOR Measures the alignment between reference and generated samples by mapping unigrams, producing a score that ranges between 0 (perfect mismatch) and 1 (perfect matching). We computed METEOR by leveraging the evaluate [11] Python package from HuggingFace. BLEU Measures the n-gram intersection between the reference and generated snippets using a score that ranges between 0 (perfect mismatch) and 1 (perfect matching), penalizing the score of the generated samples that are longer than their corresponding references. We computed BLEU for n-grams with n = 4, taking advantage of the BLEU implementation provided by Microsoft in CodeXGLUE [17], a benchmark dataset on code intelligence. Edit Distance Measures the output distance by computing the minimum number of operations on single characters required to make each generated snippet equal to the reference sample. The ED score ranges between 0 (perfect matching) and a positive integer representing the number of character operations required to achieve a perfect matching. We computed ED through the pylcs [13] Python package, normalizing the produced score between 0 (perfect mismatch) and 1 (perfect matching). Exact-Match Measures the mean percentage of generated samples that perfectly match their corresponding reference samples in test dataset.
4
Syntactic and Semantic Assessment
The results of the syntactic evaluation of the models fine-tuned with the reference ground truth dataset can be observed in Figure 2. According to PSScriptAna-
8
R. Bessa et al.
Percentage (%)
lyzer, our specialized models were able to generate valid malicious PowerShell code. In fact, the generated samples registered low parse error and error percentages, highlighting the strong capabilities of the fine-tuned LLMs in generating syntactically correct PowerShell code. 30
Qwen2.5-Coder (7B) Qwen2.5 (7B)
20
Llama3.1 (8B)
10 0 Parse Errors
Warnings
Errors
Fig. 2: Syntactic evaluation of models fine-tuned with the reference dataset. From the three fine-tuned LLMs, Qwen2.5 and Qwen2.5-Coder were the models that produced samples with lower parse error percentages. Additionally, all the three specialized models produced snippets with significantly high warning percentages. However, since the LLMs were specifically trained to produce malicious code, it was expected that PSScriptAnalyzer would identify warnings in the generated samples regarding safety violations of PowerShell. For instance, Figure 3 presents all the syntax flaws identified by PSScriptAnalyzer in the samples generated by Qwen2.5-Coder fine-tuned with the reference ground truth dataset. One of the most registered warnings was “PSAvoidUsingInvokeExpression”, which alerts for the usage of a PowerShell command that enables the execution of potentially unsafe code via string representation. 2
PSAvoidUsingComputerNameHardcoded
Errors Warnings
5
PSAvoidUsingWMICmdlet
6
PSUseDeclaredVarsMoreThanAssignments
Parse Errors
PSAvoidUsingInvokeExpression
13
PSAvoidUsingCmdletAliases
13
UnexpectedCharactersAfterHereStringHeader
1
MissingEndParenthesisInExpression
1
MissingEndParenthesisInMethodCall
1 3
UnexpectedToken
0
5 10 Number of occurrences
15
Fig. 3: Syntax report of Qwen2.5-Coder fine-tuned with the reference dataset. The “UnexpectedToken” parse error was the most frequent occurrence to prevent the execution of the generated samples. Furthermore, “PSAvoidUsingComputerNameHardcoded” was the most registered PowerShell error. According
RedShell
9
to PSScriptAnalyzer, hardcoding the value of the ComputerName argument violates a security rule from PowerShell since it will potentially expose sensitive information regarding the target host. Figure 4 presents the results of the semantic evaluation performed for our specialized models. The fine-tuned LLMs achieved significantly high scores across different output similarity metrics, highlighting their strong capabilities to generate malicious PowerShell closely aligned with the expected code references. In particular, Qwen2.5-Coder outperformed Qwen2.5 and Llama3.1 in all output similarity metrics.
Score
0.6
Qwen2.5-Coder (7B) Qwen2.5 (7B)
0.4
Llama3.1 (8B)
0.2 0 ED
METEOR
ROUGE-L
BLEU-4
Exact Match
Fig. 4: Semantic evaluation of models fine-tuned with the reference dataset. To further validate the generation capabilities of our specialized models, we conducted a detailed analysis of the state-of-the-art solutions for automatic malicious PowerShell generation through LLMs. We used as reference the specialized versions of CodeGPT, CodeGen and CodeT5+ proposed in [16]. The reference models were fine-tuned by Liguori et al. using the reference ground truth dataset. Additionally, CodeT5+ and CodeGen were also pre-trained with a dataset composed of 89,814 generic PowerShell samples extracted from GitHub. Figure 5 presents the comparative analysis of the semantic evaluation results achieved by the reference models (reported in [16]) and our specialized version of Qwen2.5Coder, also fine-tuned with the reference ground truth dataset. 0.6
Qwen2.5-Coder
Score
CodeGPT [16]
0.4
CodeGen [16] CodeT5+ [16]
0.2 0 ED
METEOR
ROUGE-L
BLEU-4
Exact Match
Fig. 5: Semantic evaluation of Qwen2.5-Coder and reference models. Qwen2.5-Coder outperformed the reference models across all output similarity metrics. Notably, our specialized LLM was only partially fine-tuned through LoRA and Unsloth, using a single GPU under low VRAM consumption conditions, while the reference models were targeted by a complete fine-tune, with CodeT5+ and CodeGen also benefiting from a specialized pre-train in generic PowerShell.
10
R. Bessa et al.
We also compared the semantic performance of Qwen2.5-Coder with some of the most popular proprietary LLMs such as ChatGPT 3.5 [22] and DeepSeekChatV3 [5], as illustrated in Figure 6. While we inferred DeepSeekChat through its public API and evaluated the generated snippets, for the ChatGPT assessment we relied on the scores computed by Liguori et al. in [16] using the same code descriptions. Achieved results demonstrate that Qwen2.5-Coder is a strong alternative for malicious PowerShell generation when compared to the current state-of-the-art solutions.
Score
0.6
Qwen2.5-Coder ChatGPT 3.5 DeepSeekChat-V3
0.4 0.2 0 ED
METEOR
ROUGE-L
BLEU-4
Exact Match
Fig. 6: Semantic evaluation of Qwen2.5-Coder and closed models. Up to this point, conducted experiments focused on evaluating the selected models fine-tuned only with the reference ground truth dataset from [16]. To assess the training benefits of employing our extended dataset (previously introduced in Section 2), we compared the semantic evaluation performance of two different versions of Qwen2.5-Coder, one fine-tuned only with the original reference dataset, and the other fine-tuned with the extended ground truth dataset. As illustrated in Figure 7, the ED, METEOR and ROUGE-L scores exhibited negligible variation between the two versions of Qwen2.5-Coder. This happens since these metrics focus on overall structure and semantic similarity, areas where the model fine-tuned with the reference dataset already performed well. In contrast, the model fine-tuned with our extended dataset demonstrated more substantial improvements in BLEU-4 and Exact Match, stricter metrics that emphasize longer n-gram overlaps and exact matches. This highlights the positive impact of using a larger and more diverse training dataset, which primarily enhanced output precision rather than general structure or meaning.
Score
0.6
Reference ground truth [16] Extended ground truth
0.4 0.2 0 ED
METEOR
ROUGE-L
BLEU-4
Exact Match
Fig. 7: Semantic evaluation of Qwen2.5-Coder fine-tuned with different datasets. Based on the previous results, we defined the specialized version of Qwen2.5Coder as the best candidate to be incorporated in RedShell. Table 2 compares
RedShell
11
some illustrative examples of snippets generated by our tool and the corresponding ground truth references, visually emphasizing their strong similarity. Code differences are highlighted in red, where slashed text denotes unexpected elements. Table 2: Illustrative examples of snippets generated by RedShell. Label
Reference Snippet
Generated Snippet
List the members of Admins group.
Get-ADGroupMember -Identity "Admins"
Get-ADGroupMember -Identity "Admins"
Use Mimikatz to dump plain text passwords from memory.
Invoke-Mimikatz -Command ’"sekurlsa:: logonpasswords"’
Invoke-Mimikatz -Command ’"sekurlsa:: logonpasswords"’ | (((( Out-File -FilePath (((( ( C:\temp\creds.txt
Execute a malicious command encoded as a Base64 string.
powershell.exe -c "iex ([System.Text.Encoding] ::Unicode.GetString( [System.Convert]:: FromBase64String( ’Cmd’)))"
powershell.exe ((( -c" iex ( -Command ([System.Text.Encoding] :: UTF8.GetString( [System.Convert]:: FromBase64String ( (( base64EncodedCmd’)))" (’( (((
5
Related Work
LLMs have registered a high popularity in recent years, with the AI-community actively searching for new approaches to create small, and yet powerful, generative models. In addition, engineers are taking advantage of these novel technologies to provide more automation to complex and time-consuming tasks. In the offensive cybersecurity field, various AI-based solutions have been proposed to assist ethical hackers in Pentesting scenarios. Tools such as PentestGPT [7] and PENTEST-AI [2] aim to provide full automation to Pentest audits through complex LLM agents, providing mechanisms to detect, explore, and report different vulnerabilities with minimal human intervention. In contrast, RedShell provides a solution specifically designed to assist Pentesters in malicious PowerShell generation. Some literature also focused on the application of LLMs in offensive code generation. Liguori et al. in [14] proposed EVIL, an effective approach to generate assembly shellcodes and Python encoders through LLMs. The generation of shellcodes was also targeted by tools such as DualSC [31] and ExploitGen [32], using a shallow transformer and a template augmented approach, respectively. Chowdhary, Jha, and Zhao in [3] performed the Pentesting of web applications
12
R. Bessa et al.
through LLMs by employing generative adversarial networks while Natella et al. in [21] introduced a novel dataset to train models in malicious Python generation. RedShell, while being significantly simpler than a LLM agent such as PentestGPT, still provides an efficient code generator to assist ethical hackers performing Pentest audits with more automation. In addition, our solution minimizes the amount of computational resources such as GPUs and VRAM required to train and evaluate AI-based code generators, following a much lighter approach than the generative adversarial networks employed by Chowdhary, Jha, and Zhao in [3], and the pre-training strategies employed by Liguori et al. in [16]. Furthermore, our specialized version of Qwen2.5-Coder outperformed in the most common output similarity metrics the current state-of-the-art solutions for malicious PowerShell generation, including [16], as demonstrated in Section 4. Notably, RedShell was also able to outperform popular closed models such as ChatGPT 3.5 and DeepSeekChat-V3 in the semantic evaluation.
6
Conclusions
We demonstrated that RedShell is a competitive solution for assisting Pentesters in offensive PowerShell generation. Our tool generated malicious samples that were both syntactically valid and closely aligned with the reference snippets. We also built a ground truth dataset for malicious PowerShell, offering a strong knowledge base to train LLMs in offensive PowerShell generation. A potential threat to the validity of RedShell lies in the absence of a functional evaluation of the generated snippets. Future work will address that by testing those snippets in controlled environments that mimic real-world Pentesting scenarios.
References 1. Atomic Red Team, Atomic Red Team: Adversary Emulation for Cybersecurity, (2024). https://www.atomicredteam.io/ (visited on 01/31/2025) 2. Bianou, S.G., Batogna, R.G.: PENTEST-AI, an LLM-Powered Multi-Agents Framework for Penetration Testing Automation Leveraging Mitre Attack. In: 2024 IEEE International Conference on Cyber Security and Resilience (CSR), pp. 763–770. IEEE (2024). https://doi.org/10.1109/CSR61664.2024.10679480 3. Chowdhary, A., Jha, K., Zhao, M.: Generative Adversarial Network (GAN)-Based Autonomous Penetration Testing for Web Applications. Sensors 23(18), 1–18 (2023). https://doi.org/10.3390/s23188014 4. Corporation, M.: MITRE ATT&CK Framework, (2024). https://attack.mitre. org/ (visited on 01/31/2025) 5. DeepSeek-AI, DeepSeek Chat Platform, (2025). https : / / chat . deepseek . com/ (visited on 01/31/2025) 6. Delpy, B.: Mimikatz, (2011). https://github.com/gentilkiwi/mimikatz (visited on 01/31/2025)
RedShell
13
7. Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., Rass, S.: PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing. In: 33rd USENIX Security Symposium (USENIX Security 24), pp. 847–864. USENIX Association (2024). https://www. usenix.org/conference/usenixsecurity24/presentation/deng 8. Face, H.: Hugging Face, (2025). https://huggingface.co/ (visited on 01/31/2025) 9. Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., Zhang, J.M.: Large Language Models for Software Engineering: Survey and Open Problems. In: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), pp. 31–53. IEEE (2023). https://doi. org/10.1109/ICSE-FoSE59343.2023.00008 10. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A.: The Llama 3 Herd of Models, (2024). https : / / arxiv . org / abs / 2407 . 21783 (visited on 01/31/2025) 11. Hugging Face, evaluate: A Python library for model evaluation and comparison, (2025). https://pypi.org/project/evaluate/ (visited on 01/31/2025) 12. Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., Yang, A., Men, R., Huang, F., Ren, X., Ren, X., Zhou, J., Lin, J.: Qwen2.5 Coder Technical Report, (2024). https://arxiv.org/abs/2409.12186 (visited on 01/31/2025) 13. kuangzh, pylcs: A super fast C++ implementation of classic LCS problems using dynamic programming, (2023). https://pypi.org/project/pylcs/ (visited on 01/31/2025) 14. Liguori, P., Al-Hossami, E., Orbinato, V., Natella, R., Shaikh, S., Cotroneo, D., Cukic, B.: EVIL: Exploiting Software via Natural Language. In: 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), pp. 321– 332. IEEE (2021). https://doi.org/10.1109/ISSRE52982.2021.00042 15. Liguori, P., Improta, C., Natella, R., Cukic, B., Cotroneo, D.: Who evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators. Expert Systems with Applications 225 (2023). https://doi.org/10.1016/j. eswa.2023.120073 16. Liguori, P., Marescalco, C., Natella, R., Orbinato, V., Pianese, L.: The Power of Words: Generating PowerShell Attacks from Natural Language. In: 18th USENIX WOOT Conference on Offensive Technologies (WOOT 24), pp. 27–43. USENIX Association (2024). https://www.usenix.org/conference/woot24/presentation/ liguori 17. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C.B., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S.K., Fu, S., Liu, S.: CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. CoRR abs/2102.04664 (2021). https://www.microsoft.com/ en - us / research / publication / codexglue - a - machine - learning - benchmark dataset-for-code-understanding-and-generation/ 18. Microsoft Corporation, PSScriptAnalyzer, (2025). https://github.com/PowerShell/ PSScriptAnalyzer (visited on 01/31/2025) 19. Mittal, N.: Nishang - Offensive PowerShell for Red Teams, (2018). https : / / github.com/samratashok/nishang (visited on 01/31/2025)
14
R. Bessa et al.
20. Mora, S.: ROUGE: A pure Python implementation of the ROUGE metric, (2019). https://pypi.org/project/rouge/ (visited on 01/31/2025) 21. Natella, R., Liguori, P., Improta, C., Cukic, B., Cotroneo, D.: AI Code Generators for Security: Friend or Foe? IEEE Security and Privacy 22(5), 73–81 (2024). https: //doi.org/10.1109/MSEC.2024.3355713 22. OpenAI, ChatGPT: Overview and Features, (2025). https://openai.com/chatgpt/ overview/ (visited on 01/31/2025) 23. Recipe, R.T.: PowerShell tips & tricks, (2025). https : / / redteamrecipe . com / powershell-tips-tricks/ (visited on 01/31/2025) 24. Team, Q.: Qwen2.5: A Party of Foundation Models, (2024). https : / / qwenlm . github.io/blog/qwen2.5/ (visited on 01/31/2025) 25. TryHackMe, TryHackMe - Learn Cybersecurity, Penetration Testing, and Ethical Hacking, (2025). https://tryhackme.com/ (visited on 01/31/2025) 26. Unsloth AI, Unsloth: Open Source Fine-Tuning for LLMs, (2025). https : / / unsloth.ai/ (visited on 01/31/2025) 27. Unsloth Team, LoRA Hyperparameters Guide — Unsloth Docs, (2024). https: //docs.unsloth.ai/get-started/fine-tuning-guide/lora-hyperparametersguide (visited on 01/31/2025) 28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in Neural Information Processing Systems 30, 5999–6009 (2017). https://proceedings.neurips.cc/paper_ files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf 29. Vats, P., Mandot, M., Gosain, A.: A Comprehensive Literature Review of Penetration Testing and Its Applications. In: 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) ( ICRITO), pp. 674–680. IEEE (2020). https://doi.org/10.1109/ICRITO48877. 2020.9197961 30. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T., Gugger, S., Rush, A.: Transformers: State-ofthe-Art Natural Language Processing. In: EMNLP 2020 - Conference on Empirical Methods in Natural Language Processing, Proceedings of Systems Demonstrations, pp. 38–45 (2020). https://doi.org/10.18653/v1/2020.emnlp-demos.6 31. Yang, G., Chen, X., Zhou, Y., Yu, C.: DualSC: Automatic Generation and Summarization of Shellcode via Transformer and Dual Learning. In: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 361–372 (2022). https://doi.org/10.1109/SANER53432.2022.00052 32. Yang, G., Zhou, Y., Chen, X., Zhang, X., Han, T., Chen, T.: ExploitGen: Templateaugmented exploit code generation based on CodeBERT. Journal of Systems and Software 197 (2023). https://doi.org/10.1016/j.jss.2022.111577