F INGERPRINT-D RIVEN AUTOMATION : C OUPLING R ECONNAISSANCE WITH POC V ERIFICATION
Hongping Wang Hainan University Haikou, China [email protected]
Xiaoqi Li Hainan University Haikou, China [email protected]
arXiv:2607.26655v1 [cs.CR] 29 Jul 2026
A BSTRACT In the field of network security confrontation, reconnaissance is the first and most critical step. Accurate, efficient, and comprehensive reconnaissance can help network security workers more fully understand the target’s current state, identify potential weaknesses, and formulate a targeted attack strategy. However, there are some problems in the existing tools on the market, such as low accuracy of the collected information, lack of concealment, time-consuming, low or high integration of tools, which makes them difficult to start using. With the continuous development of technology, systems need to continuously upgrade their technologies and strategies, and expand new functions. The scalability of tool functions is also one of the factors that need to be taken into consideration. These problems in the practice of network security have a serious impact on user experience and work efficiency. In view of this, a more user-friendly, more automated reconnaissance and vulnerability verification tool is designed. This paper proposes to develop a highly automated reconnaissance and vulnerability verification tool. This paper presents the tool functions and conducts the basic design. Finally, we implement and test all the functions specifically to verify the feasibility of the tool functions and highlight the advantages of this tool in terms of automation, retrieval, and use in network security operations. Through the test of the online security simulation range, the results show that this tool can complete the automatic reconnaissance and in-depth data processing of specific targets, and conduct automatic vulnerability detection and verification. Keywords Cybersecurity · Reconnaissance · Vulnerability Detection
1
Introduction
In the field of cybersecurity, there are numerous tools available for security testing[1], which can be broadly categorized into three types: large scale automated vulnerability scanning systems (such as AWVS, which specializes in web application vulnerability scanning)[2][3][4], tools designed for specific functions (such as Dirsearch, which specially performs directory scanning), and vulnerability detection systems tailored to specific frameworks (such as WPscan, which is specifically designed to identify security risks in WordPress websites[5][6][7]). While the first category of large-scale systems offers a vast number of proof-of-concept (PoC) tests and comprehensive vulnerability detection, they require significant scanning time, often charge for certain functions, and present a steep learning curve for beginners[8][9]. The latter two categories, having been developed for specific functions, have a narrower scope of application[10]. Some tools suffer from design flaws or errors, for example, the directory scanning tool Dirsearch typically relies solely on HTTP response codes to determine page accessibility. If a site uses a custom error page or has incorrect server configuration, and the error page returns a 200 response code, the tool may misjudge the situation and return an invalid page. Additionally, many security tools are only semi-automated. When gathering basic information, users often have to manually inspect front-end code, JavaScript files, or HTML comments to extract sensitive data[11][12][13]. This process is time-consuming and error-prone, significantly reducing the efficiency of security work. Furthermore, the logic behind some service identification features is overly simplistic, relying solely on signature ports or response codes for inference. This lack of depth and comprehensiveness makes it difficult to adapt to the complex characteristics
of modern web applications. At the same time, many security systems have shortcomings in terms of functionality maintenance and expansion[14][15]. They were not designed to allow users to customize and extend tool functions, and when developers later abandon maintenance, the features fail to meet the demands of technological advancements, ultimately rendering the tool unusable. Therefore, to meet the practical needs of cybersecurity professionals, it is necessary to design and propose a security tool with good maintainability and scalability[16][17], ease of use, lightweight design, accurate results, and a high degree of automation[18][19]. This paper analyzes the shortcomings of traditional cybersecurity tools in practical applications. Based on the actual needs of security professionals and leveraging Python’s extensive third-party libraries[20][21][22][23][24], we design and develop a reconnaissance[25][26] and vulnerability verification tool for the cybersecurity reconnaissance phase that delivers accurate results, is highly automated, highly scalable, and lightweight yet comprehensive. This paper makes the following contributions: (1) We present a novel automated penetration testing framework that seamlessly auto-links the two separate and traditionally disjoint phases of reconnaissance and vulnerability verification into a security testing closed-loop pipeline. (2) We propose a fingerprint-based nexus mechanism as the main technical linkage of the two phases. During reconnaissance, we capture precise web fingerprints and dynamically map the fingerprints to flexibly invoke highly faithful PoC scripts, so as to make penetration testing targeted, avoid scanning blindness, and reduce network traffic overhead. (3) Instead of just using synthetic experiments, we do intensive evaluations in real scenarios on some authentic Capture the Flag (CTF) platform environments. These platform environments have complex and real vulnerable chains and authentic service responses. They allow the tool to be practically feasible, robust, and practical in real-world offensive security scenarios.
2
Related Work
The existing network security testing tools mainly cover three categories: large-scale automated vulnerability scanning systems, specific function-oriented special tools, and vulnerability detection systems developed for specific frameworks[27][28][29]. The first type of tool, represented by AWVS, although it provides a PoC test case library with a wide range of coverage and can detect various types of security vulnerabilities, its scanning process takes a long time, advanced functions usually need to be paid for use, and the interface and operation logic have high learning costs for beginners [8][9]. Comparatively speaking, tools such as Dirsearch, a directory scanning tool, or WPscan, a special scanner for WordPress, have advantages in terms of resource occupation and ease of operation, but at the cost of significantly narrowing the scope of application, they often only cover a specific link in the security testing workflow and are difficult to meet the comprehensive detection requirements in complex scenarios. In addition to the above classification characteristics, there are some common problems with various tools. In terms of detection accuracy, the design mechanism of some tools itself introduces bias. Dirsearch judges page accessibility only based on the HTTP response status code, which often leads to false judgments in actual tests. For example, when the target website is configured with a custom error page, and the page returns a 200 status code, Dirsearch will mark the path that does not actually exist as a valid resource. Testers will conduct subsequent analysis based on these results, which is inevitably misleading. In terms of the degree of automation, a considerable number of security tools only achieve semi-automatic workflow. The reconnaissance phase still relies on manual intervention. Users need to check the front-end code, JavaScript files, and HTML comments one by one to extract sensitive information that may be included [11][12], such as debugging interfaces, temporary credentials, or internal paths left by developers. This process is time-consuming and error-prone, especially in pages with large code volume, the risk of omission in manual review increases significantly. There is also a tendency to simplify the logic at the service identification level. Currently, many tools only rely on the default port number or the Server field in the response header to determine the type of back-end components. However, the technology stack of modern Web applications is becoming increasingly complex, and container deployment, content distribution network(CDN) acceleration, and the widespread application of reverse proxy make this single-dimension identification strategy difficult to work with and often misjudge or omit the actual running service types. In addition, the lack of tool scalability cannot be ignored. Most tools do not provide plug-in mechanisms or API interfaces, and users cannot add new detection modules or adjust existing logic according to their own needs. Once the original development team stops maintenance, these tools will quickly become disconnected from the new vulnerability types, and their practicability will decline [14][15][30]. 2
The more prominent problem is the fracture at the architecture level. The two stages of reconnaissance and vulnerability verification are usually designed as independent modules in the existing toolset, with independent input/output formats and execution logic. This means that security analysts must manually sort out all kinds of clues output in the early reconnaissance stage, screen out suspicious points, and then select appropriate verification scripts to perform the next step. This manual bridging process not only slows down the overall test rhythm, but also introduces the possibility of additional human errors, it is not uncommon for key clues to be missed or misread in the transmission process. In recent years, although some studies have tried to introduce large language models into the automated penetration test process [20][21][31], or use the agent framework to open the path between reconnaissance and verification, these schemes generally face the problem of high computing overhead and lack a lightweight middle layer mechanism to efficiently map the reconnaissance results to executable verification actions. In view of the above shortcomings, this paper proposes an automation framework based on fingerprint drive. The core design idea of this framework is to closely couple reconnaissance and verification to form a closed-loop security test process. Specifically, the contribution of this paper is reflected in three levels. First, we have built an end-to-end automated pipeline. The results from the reconnaissance phase are no longer handed over to manual processing as isolated data sets, but directly flow into the verification phase to trigger the corresponding detection tasks, thus eliminating the phase connection in the traditional workflow that relies on people. Secondly, we designed a fingerprintbased association mechanism. During the reconnaissance process, the tool extracts website fingerprints from multiple dimensions such as response content, response headers, and static resource characteristics, covering technology stack type, server version, and application framework information, and then dynamically matches and calls the corresponding PoC verification script based on this fingerprint information. This strategy makes the testing process more focused, avoids the network resource consumption caused by blindly traversing all known vulnerabilities, and also improves detection efficiency. Thirdly, we verified the tool in the real CTF competition environment. The test cases include a variety of target applications with real vulnerability chains and various abnormal service response scenarios. The experimental results show that the tool can stably complete the closed-loop process of detection verification when facing different target instances, and the rates of false positives and false negatives are controlled within an acceptable range, which indicates that the tool has good feasibility and robustness in actual combat attack and defense scenarios.
3
Design
3.1
Architecture Design
The overall functional architecture of this system aims to build an end-to-end automated security assessment pipeline. The architecture mainly consists of three parts: a multi-dimensional basic information reconnaissance module, a fingerprint-driven scanning and identification module, and a heterogeneous data aggregation and processing module[32]. These three functional modules maintain a high degree of independence in the implementation of the underlying code to facilitate subsequent independent iteration and plug-in expansion, however, at the logic and data flow level, synchronous collaboration is achieved through close series connection of standardized internal data interfaces. As the starting point of the assembly line, the reconnaissance module is responsible for non-intrusive asset mapping and basic information extraction of the target to build an initial target portrait, the scanning and identification module serves as the core hub connecting the previous and the following, relying on the portraits provided by pre-reconnaissance to accurately perform fingerprint matching and directional detection, effectively avoiding the high time cost and risk of false positives brought by traditional blind scanning, The data processing module is responsible for deep cleaning and structural processing of multi-source and heterogeneous detection echo data, providing effective data support for subsequent PoC triggering and vulnerability verification. The seamless connection of these three modules breaks the data island between traditional security tools and ensures the efficiency and robustness of the entire automated penetration process. The specific functional topology is shown in Figure 1. 3
Figure 1: Functional Diagram
3.2
Functional Design
The execution flow of the entire tool consists of the following steps: collection of basic information, vulnerability information matching, and generation of the scan report. The vulnerability information matching process requires the use of the necessary basic information parameters collected earlier, while the report includes two sections: basic information data and details regarding the presence of vulnerabilities. Figure 2 illustrates the execution flow of each function during system operation.
Figure 2: Function Flowchart
3.2.1
User Interface
The user interface is the initial presentation of the system to users. Whether it is a terminal command-line interface or a graphical web interface, it should be simple and clear, highlighting the tool’s core functions. The tool offers two interface options, a terminal command-line interface and a graphical web interface, to meet the needs of different users. 4
Deepscan’s terminal command-line interface (CLI) strictly follows the design principles of minimalism and operational efficiency, and is designed to provide a smooth and intuitive interactive experience for security assessors. In the system initialization phase, the tool first displays the system name by rendering the customized ASCII art font. This design not only gives the tool professional visual beauty and brand recognition, but also provides users with a clear visual anchor in the complex terminal data stream, thus helping the operator quickly establish the working context. Followed by a concise summary of the core parameters and basic usage of the system, this pre-interactive guidance effectively reduces the learning threshold of the tool and ensures that users can quickly master key instructions. More importantly, before any automatic scanning task is formally triggered, the interface will explicitly enumerate and echo the currently active global configuration parameters. This pre execution state confirmation mechanism not only ensures the transparency of the scanning process to the greatest extent, enabling users to visually verify the execution boundary and detection logic of the task, but also effectively avoids invalid network overhead or potential compliance risks caused by parameter mismatches, which further strengthens the precision and controllability of the security test process while improving the overall operational efficiency. To enhance user experience and ease of use, Deepscan has added a web-based graphical interface, allowing users to access and operate the tool directly via a browser. The web interface utilizes a modern design style and the Bootstrap framework to achieve a responsive layout, ensuring optimal display across various devices. 3.2.2
Webinfo Information
The functional design of the webinfo module aims to carry out a comprehensive network environment detection of the target web assets and provide more verification perspectives for subsequent accurate vulnerability verification. The module first obtains the real IP address of the target through deep domain name resolution, and constructs the network space geographic mapping combined with IP geographic location technology to establish the real network boundary of the target. The system further performs multi-dimensional feature extraction. It not only accurately captures the surface metadata such as the web page title, but also deeply analyzes the HTTP response message to obtain high-value web fingerprint data, middleware server types, and the details of the underlying operating system, so as to fully obtain the back-end technology ecology of the target. For the complex defense system in the modern web architecture, the webinfo module integrates the active network boundary detection mechanism, which can accurately determine whether the target front-end has deployed CDN or web application firewall (WAF), and further identify the specific WAF manufacturer characteristics and protection configuration. This series of in-depth detection logic from the outside to the inside, from the basic assets to the defense mechanism, not only effectively avoided the false positives and failures caused by CDN cache interference or WAF dynamic interception in the subsequent vulnerability verification stage, but also perfectly matched the core idea of ”fingerprint driven” in this system, laying a solid reconnaissance foundation for the smooth progress of automated penetration testing. 3.2.3
Port/Directory Scanning
Determine which ports are open and identify specific services based on characteristic ports, while utilizing multithreading for acceleration. Additionally, port spoofing must be considered, if more than 30 ports are open, port spoofing is highly probable. In the directory scanning module of automated penetration testing, to achieve efficient and accurate asset discovery, we need to comprehensively consider the dynamic generation of scanning dictionaries, the scheduling mechanism of high concurrent requests, and complex false positive suppression strategies. The core is to conduct a multi-dimensional in-depth analysis of the HTTP response content to confirm the true existence of the target file or directory. In order to effectively peel off the interference information returned by the server and greatly reduce the false positive rate, at the basic protocol response level, the system will perform strict status code verification to directly eliminate common server error responses, and synchronously review the HTTP response header and content type, so as to accurately intercept invalid probes that carry custom error IDs or MIME types that are seriously inconsistent with expectations. At the semantic analysis level of the response body, the module introduces a feature matching algorithm for the ”soft 404” phenomenon. By extracting and comparing the specific DOM structure or feature text in the general or custom 404 page, it thoroughly filters out those pseudo-live directories that return a 200 status code but have invalid actual content. In view of the complex routing behavior prevalent in modern web applications, the system will deeply track and verify the legitimacy and accessibility of the redirect target URL to prevent invalid jump interference caused by malicious configuration or interception by protective devices (such as WAF). In addition, the system also has a built-in dynamic special rule engine based on file attributes, which can perform customized fingerprint verification logic for different types of sensitive files (such as backup files, configuration files, or specific script suffixes). By implementing the above multi-layer comprehensive research and judgment strategy, not only the throughput efficiency of concurrent 5
scanning is maximized, but also the extremely high accuracy and practical reliability of directory enumeration results are fundamentally guaranteed.
3.2.4
Service Fingerprinting
Using the open-source Webeyes and Wappalyzer fingerprint libraries, crawl the target’s pages, response headers, the ‘src‘ attribute of ‘script‘ tags, and meta information, then perform regular expression matching against the fingerprint libraries to complete fingerprint identification.
3.2.5
Proof Of Concept
The tool employs fingerprinting technology to identify web fingerprints. During the reconnaissance phase, it collects specific field information from web applications, such as page keywords, special files and paths, based on URLs entered by the user. In the web fingerprinting phase, a fingerprint database is established to collect feature information and build a fingerprint feature repository. Simultaneously, the tool gathers feature information from the target web application and compares it with the data in the fingerprint feature repository to perform web fingerprinting. The fingerprint information in the fingerprint database is collected and synthesized from platforms such as Wappalyzer and FOFA, and is stored in JSON format. Fingerprint recognition technology is a critical step in the penetration testing process, helping us quickly formulate penetration strategies[33]. Based on the WebInfo and fingerprint information collected earlier, we determine whether the conditions for executing the PoC script are met. A dedicated verification function must be written to accomplish this task, thereby achieving the effectiveness of the PoC.
3.2.6
Database Design
In order to realize the efficient access management of data in the automatic penetration process, the system designs a structured core data model, which mainly covers four progressive key entities. First of all, webinfo (WEB basic information) entity as the cornerstone of building the target image is responsible for persisting the underlying technology stack and core fingerprint features of the target site, Secondly, ports entity accurately depicts the network exposure of the target, and records the open port status and its associated background service types in detail, On this basis, urls (resource path) entity is further extended to the application layer, which is specially used to map and store sensitive paths and key files found by deep directory enumeration, Finally, all security defects successfully verified by PoC scripts will be aggregated and written into the vulns (vulnerability information) entity in a standardized manner. These four entities are logically closely linked, completely mapping the data link from asset mapping, service identification to vulnerability verification, providing a solid and reliable database for the fingerprint-driven mechanism and automated decision-making of the system. The relationships between tables are presented as an entity-attribute diagram. A WebInfo entry represents a target website, and WebInfo maintains a 1:N relationship with all other entities, meaning each website may have N entries for port information, directory information, crawler results, and vulnerability information. 6
Domain
Page Name
Type
Service
Directory
Resp. Code
Resp. Length
Directory N
Time
MD5
N
Scan
1 OS
Firewall
Time
Detect Time
MD5
Time
Port
IP Address
Server
WebInfo
IP Location
MD5
Application
1
1
Mine Time
N
N
Crawler
Vulnerability
Domain
Port
Scan
1
Domain
MD5
Banner
Leaked Info
Vuln Info
MD5
Domain
Info Type
Figure 3: Entity-Relationship Diagram of the System
(1) Website Information Table (WebInfo): Primarily records the technical stack information of the target site, including domain name, server type, WAF, open services, and other details. The design of the Website Information Table is shown in Table 1.
Table 1: WebInfo Table Field Name
Data Type
Description
id time domain waf apps server ipaddr os md5
INTEGER TIMESTAMP VARCHAR(255) VARCHAR(100) TEXT VARCHAR(100) VARCHAR(45) VARCHAR(100) CHAR(32)
Auto-increment primary key Timestamp Domain name WAF type Application software used Server type IP address Operating system Unique identifier (MD5 hash)
(2) Port Information Table (Ports): Primarily records the open port information of the target site, including IP address, open ports, and services corresponding to specific ports. The design of the Port Information Table is shown in Table 2. 7
Table 2: Ports Table Field Name
Data Type
Description
id time ipaddr service port banner md5
INTEGER TIMESTAMP VARCHAR(45) VARCHAR(100) INT TEXT CHAR(32)
Auto-increment primary key Timestamp IP address Service type Port number Service Banner Unique identifier (MD5 hash)
(3) Directory Information Table (URLs): Primarily records valid directory and file information identified after scanning the target site, including domain name, page title, full web address, and response content. The design of the Directory Information Table is shown in Table 3. Table 3: Urls Table Field Name
Data Type
Description
id time domain title url contype rsp len rsp code md5
INTEGER TIMESTAMP VARCHAR(255) VARCHAR(255) VARCHAR(2048) VARCHAR(100) INT SMALLINT CHAR(32)
Auto-increment primary key Timestamp Domain name Page Title Complete URL Content Type Response Length Response Code Unique identifier (MD5 hash)
(4) Vulnerability Information Table (Vulns): Primarily records verified valid vulnerability information for the target site, including domain name and vulnerability details. The design of the Vulnerability Information Table is shown in Table 4. Table 4: Vulns Table Field Name
Data Type
Description
id time domain vuln md5
INTEGER TIMESTAMP VARCHAR(255) VARCHAR(512) CHAR(32)
Auto-increment primary key Timestamp Domain name Vulnerability Description Unique identifier (MD5 hash)
3.2.7
Report Generation
The report generation module of DeepScan aims to achieve efficient processing and multi-dimensional visualization of scanned data through the underlying database interaction mechanism. At the data persistence level, the system uses the lightweight SQLite3 as the storage engine and encapsulates the custom database operation class ’sqlDB’ to uniformly manage read and write requests. The system designs independent function methods for each business data table, so as to ensure that the massive heterogeneous data generated in the scanning process can be safely and structurally processed and stored. At the same time of data writing, in order to meet the real-time monitoring requirements of security testers for task progress, the system constructs a terminal dynamic feedback mechanism, which extracts the latest status data from the database in real time and calls a special ’console’ rendering function to convert the original data into a customized standardized terminal output format, which greatly improves the readability of complex logs and the tracking efficiency of vulnerability links under the command line interface. At the end of the life cycle of the scanning task, the system starts the final report delivery, and the core scheduler will fully evaluate the results from the database and serialize them into standard JSON data streams, Subsequently, the system introduces a predefined HTML template engine, reads the template file through a custom parsing function, uses data binding technology to accurately map JSON data and dynamically replace it into the corresponding placeholder of the HTML template, and finally 8
automatically renders a visual security assessment report with clear structure, good interaction and high readability, so as to completely realize the project from bottom-level data persistence, middle-level real-time monitoring to top-level visual delivery.
4
Implementation
The operation and empirical evaluation of this system rely on a highly standardized simulation security environment. The underlying operating system uses the Linux Kali 6.6.9-amd64 distribution, and the core operating environment is designated as Python 3.11.8. The target range used to verify the effectiveness of the system uses the BUUCTF online simulation platform and the vulhub open-source vulnerability Docker image collection to ensure the authenticity and vulnerability coverage of the test scenario. In the environment configuration and initialization phase, the system strictly requires that the Python version of the host computer should not be less than 3.10. Users only need to use the package manager to automatically and batch deploy all core third-party dependency libraries. In order to adapt to the interaction requirements in different security assessment scenarios, the system designed a flexible dual-mode startup mechanism: for the local test scenario that prefers the integration of efficient instruction operation and automated scripts, researchers can directly execute ’Python deepscan.py’ in the root directory of the system to invoke the lightweight CLI. For scenarios requiring visual data presentation or remote server deployment, the system provides a web-based graphical startup scheme by running the ’Python start web.py’ command in the root directory, the system will not only pull up the web service in the background and bind it to the whole network segment interface (0.0.0.0) by default to support cross-device remote access in the LAN, but also automatically invoke the local default browser and navigate to ’http://127.0.0.1:5000’. The core console provides a comprehensive test platform with both operational flexibility and architecture scalability for security evaluators. 4.1
User Interface
This tool offers two interaction methods: the terminal command line and a web-based graphical interface. When running in the system terminal, the pyfiglet library, which uses Python to generate ASCII art, is used to create a banner to enhance the user interface. Custom classes are employed to set the color and style of the text displayed on the interface. This code uses Python’s pyfiglet library to create a banner with stylized text. It also employs custom classes and ANSI escape codes to control text color and formatting in the terminal. The web graphical interface was developed to align with modern user habits and enhance usability. Based on the Flask framework[34][35], a system-level web graphical interface was created, allowing users to access the tool directly via a browser. The web interface utilizes the Bootstrap framework to achieve a responsive layout, ensuring optimal display across different devices[36]. 4.2
WebInfo Data
WebInfo data includes IP addresses, geographic locations, webpage title fingerprints, server types, operating system information, and CDN and WAF identification. The following describes the specific implementation approach for each function. (1) Retrieving IP Addresses and Geolocation. When resolving IP addresses, the ’dns.resolver.Resolver’ is used to perform DNS resolution, querying the A record of the domain name. An A record is a resource record type in the DNS protocol whose primary function is to map a domain name to an IPv4 address[37]. Resolution is then carried out using public DNS servers, such as 1.1.1.1 and 8.8.8.8. Invalid IP addresses, such as local loopback addresses, are filtered out. To obtain the geographic location, the ’geoip2’ library is used to read the ’GeoLite2-City.mmdb’ database file (which typically contains geographic location information associated with IP addresses). This allows us to query the city information for an IP address and retrieve the country, province, and city names[38][39]. (2) Operating System Information. In the automatic information collection and fingerprint identification module of the system, the underlying network discovery function integrates the ’Python nmap’ library. By calling the locally deployed open-source nmap engine, it realizes the Python automatic encapsulation of network topology detection and security audit capabilities, and provides accurate port and service information for subsequent vulnerability verification. In the stage of feature extraction at the web level, the system effectively captures the target server type (server), page title, and multidimensional web fingerprint information through the ’info’ method encapsulated in the ’webpage’ class, and introduces the set operation mechanism to strictly deduplicate the multi-source detection data, so as to ensure the uniqueness and accuracy of fingerprint features. Aiming at the problem of ”target IP distortion” caused by CDN widely existing in modern web architecture, the system designs a dual CDN recognition mechanism based on the combination of IP interval matching and ASN (autonomous system number) traceability[40][41][42]: on the one hand, the system has built-in a large IP address segment feature library covering the mainstream CDN service providers 9
at home and abroad, which can quickly determine whether the target IP belongs to the CDN edge node through the boundary matching algorithm, On the other hand, the system relies on the geoip2 database to query the ASN attribution of the target IP and cross compare it with the ASN list of known CDN providers[43][44], so as to accurately peel off the non real source IP and avoid invalid network overhead caused by subsequent vulnerability verification module on false targets. In addition, in the detection link of WAF, the system adopts a dynamic identification strategy combining active trigger and feature matching, that is, it injects malicious test payload into the target to induce the interception mechanism of WAF, and then captures and extracts the specific HTTP response features returned by WAF, and finally compares them with the locally constructed WAF fingerprint database in depth, so as to realize the accurate perception of the target defense environment and provide the basis for subsequent PoC scheduling. 4.3
Port Scanning
This program performs a port scan using fully established TCP connections and employs multithreading to increase the scanning speed[45]. It also includes a feature for identifying services based on port numbers. In service identification based on port, we first use the SIGNS list to match the response content with regular expressions to determine the service. If the match fails, we identify the service using the port: service dictionary defined in ’get-server’. In Python, the ‘concurrent.futures‘ module provides a high-level interface for asynchronously executing function calls. It includes two executors: ‘ThreadPoolExecutor‘ and ‘ProcessPoolExecutor‘, which are used to manage and schedule the execution of threads and processes, respectively. Here, multithreading is used to implement concurrent requests. 4.4
Directory Scanning
Directory scanning involves four key aspects: dictionary generation, 404 false positive detection, response content verification, and concurrent requests. The following provides a detailed explanation of each of these four components: (1) Dictionary Generation. The Cartesian class is used to compute the Cartesian product, generating a large number of possible directory and filename combinations from a small set of dictionary files[46][47][48]. The scan covers two categories: sensitive files and regular directory files. Sensitive files are identified by combining directory prefixes (e.g., /admin, /backup) with sensitive file extensions (e.g., .zip, .bak). Regular directory files are determined by using previously collected fingerprint information to identify file extensions, which are then combined with a standard directory dictionary to generate a more system-specific directory dictionary. (2) 404 False Positive Detection. The reason for performing 404 detection is that web administrators may customize 404 pages, which will return a 200 status code. Since we cannot rely on the status code to determine if a page is valid, the detection approach involves constructing invalid URLs to access 404 pages, recording their characteristics, and then comparing them with the 404 page characteristics during subsequent directory scans to improve detection accuracy. (3) Response Content Verification. Use the -verify method to parse each response to determine whether the directory or file actually exists. This involves filtering out common error codes and unexpected content types, detecting common 404 page text, analyzing whether redirect targets are valid, and applying specific validation logic for different file types. (4) Concurrent Requests. Select an appropriate acceleration method based on the runtime operating system: use multithreading on Windows and asynchronous I/O on Linux/Unix to implement concurrent requests and accelerate the scanning speed. This approach was chosen because port scanning is inherently a highly concurrent I/O operation, specifically, a large number of concurrent TCP connection attempts. Asynchronous I/O can manage tens of thousands of concurrent connection attempts with relative ease and without incurring significant thread overhead. The uvloop library[49] is a Python asynchronous I/O library based on Cython and libuv that enhances the performance of asyncio, however, it does not support the Windows platform and runs only on Unix/Linux systems[50]. 4.5
FingerPrint/Server Identification
This project employs a two-tier fingerprinting mechanism, utilizing the WebPage and Wappalyzer core classes to achieve service identification. At the basic identification layer, the WebPage class first uses the BeautifulSoup library to parse the HTML page of the target website[51], extracting the src attributes of all script tags, meta tag information, and HTTP response headers. It then performs rapid matching of the response headers and page content using regular expressions based on the feature rules defined in the ‘apps.txt‘ file, thereby achieving preliminary application identification. At the deep identification layer, the ‘Wappalyzer‘ class loads a more complex ‘apps.json‘ rule library capable of matching various service system fingerprints, such as CMS, forums, e-commerce systems, blogs, and more. Finally, the tool integrates the identification results from both layers to output complete web application fingerprint information. 10
4.6
Proof Of Concept
The key consideration for PoC verification is how to determine whether the target matches the PoC[52][53][54][55][56][57]. Implementation approach: Since WebInfo data has already been collected, information such as open ports and application fingerprints is passed to the ‘check‘ function in the PoC script. This function is used to verify whether the conditions for executing the PoC are met, as well as to evaluate and handle the execution results[58][59][60]. Additionally, multithreading is employed to run multiple POCs simultaneously for verification. The ‘check‘ function must be defined in accordance with the specific vulnerability mechanism, but its parameters are standardized.
4.7
Report Generation
The system’s report output primarily consists of three main components: real-time output on the terminal, generation of HTML-formatted reports for the corresponding targets, and writing report data to an SQLite database file. The following sections provide a detailed overview of the implementation details for each component[61]. (1) Real-time terminal output. Each scan result is displayed using the format ”Current Time — Function Module — Scan Result”, and custom classes are used to set the text color and style to achieve a highlighted effect. (2) The section for generating HTML-formatted reports. Use custom HTML templates based on Vue.js and Bootstrap. Each generated HTML file is named ”Deepscan-timestamp.html” and includes complete styling and interactive features. (3) Writing report data to a database file. This functionality is encapsulated in the sqldb.py file. The sqlite3 library is used to create an SQLite database, and the collected, scanned, and analyzed data is written to a DB file. The data can then be viewed using a database management tool such as Navicat. The web interface also provides a feature to view the DB file, enabling one-click access to scan results.
5
Evaluation
5.1
Effectiveness
The vulnerability environments used for functional testing are the online CTF range BUUCTF and the Vulhub range. BUUCTF is a CTF competition and training platform that provides online reproductions of real-world challenges and vulnerability environments. The Vulhub range is an open-source collection of pre-built Docker environments containing vulnerabilities, designed for security researchers and educators. All vulnerabilities are real-world, and each comes with detailed documentation explaining the vulnerability and exploitation steps. Users can quickly access these virtual environments via Docker. Functional testing consists of the following modules, and all test cases have passed. Specific test cases are shown in Tables 5 and 6.
Table 5: Table of Basic Execution Test Cases Test Case
Prerequisites
Parameter Processing
None
Start Execution
None
Function Call
None
Test Steps
Expected Result
Passed
Process user-input parameters, normalize parameters User launches the main program, scans specified targets After the main program runs, automatically calls various function modules
Processed parameters ensure normal function execution
✓
Scans normally, stable execution
✓
Standardized output of execution results for each function
✓
11
Table 6: Functional Execution Test Cases Table Test Case
Prerequisites
Test Steps
Expected Result
Passed
Liveness Check
None
Can correctly determine whether the target is alive
Can properly identify the target’s status
✓
DNS Resolution
None
IP Geolocation
Pre-configured mmdb database
WebInfo
None
WAF Detection
Uses signature database
CDN Detection OS Detection Side-Site Detection
Uses ASN database Uses nmap
PoC Detection
5.2
None
None
1. Determine whether user input Correctly returns IP address is domain name or IP regardless of input format 2. If domain name format, perform DNS resolution Uses geoip library to query IP location database Records website characteristics Send malicious payload to trigger WAF, record signature matches in database Determines if IP belongs to common IP ranges and ASNs Invokes nmap to identify OS Uses API interface for detection Based on collected fingerprints, pass PoC script to determine if execution conditions are met
✓
Returns geolocation for public IPs Correctly identifies website characteristics Triggers WAF and returns results
✓
Outputs CDN determination result Outputs identification result
✓
Returns results normally
✓
If PoC passes, display specific vulnerabilities in target
✓
✓ ✓
✓
Robustness
This tool utilizes a wide range of Python third-party library functions, however, some of these libraries do not support operation on Windows. For example, uvloop is a fast, scalable event loop for Python designed to replace the default event loop implementation in Python’s standard library asyncio and significantly improve the performance of asynchronous I/O operations. Therefore, for the directory scanning functionality, a multi-threaded Python thread pool is used when running on Windows, while an asynchronous I/O strategy is employed on Linux. This ensures that the tool runs normally on both Windows and Linux terminals. Empirical results confirm that the proposed fingerprint-driven mechanism successfully bridges the semantic gap between reconnaissance and utilization, and achieves efficiency and accuracy with extremely low network overhead.
6
Conclusion
This paper combines network scanning, reconnaissance, and penetration testing techniques to design and develop a lightweight yet powerful cybersecurity reconnaissance tool called Deepscan. The tool runs smoothly on various operating systems and has been optimized for performance across different operating system environments. It implements features such as directory scanning, port scanning, and fingerprinting. Deepscan involves a wide range of technical stacks, including network programming, multithreading, asynchronous programming, page parsing, data analysis, and database operations. Compared to existing open-source scanning systems, Deepscan’s advantages lie in its scanning depth, adaptability, and efficiency. It can automatically select the optimal technical solution based on the runtime environment, delivering reliable performance even with limited resources. It adopts a modular design with low coupling between functional components, facilitating easier expansion. DeepScan implements the entire workflow from reconnaissance to vulnerability verification, lowering the technical barrier for security assessments and making it easy for non-specialists to learn and use. DeepScan has been validated in various areas, including website security assessments and vulnerability discovery, confirming its practicality and effectiveness. 12
Acknowledgments We thank Meili Zhao for her valuable advice. AI-based tools are used for language polishing during manuscript preparation.
References [1] M. Alhamed and M. M. H. Rahman. A systematic literature review on penetration testing in networks: Future research directions. Applied Sciences, 13(12):6986, 2023. [2] N. Shiri Harzevili, A. Boaye Belle, J. Wang, S. Wang, Z. M. (Jack) Jiang, and N. Nagappan. A systematic literature review on automated software vulnerability detection using machine learning. ACM Computing Surveys, 57(3):1–36, 2025. [3] N. S. Harzevili, A. B. Belle, J. Wang, S. Wang, Z. Ming, Jiang, and N. Nagappan. A survey on automated software vulnerability detection using machine learning and deep learning. arXiv, 2023. Version 1. [4] B. Zhang, J. Li, J. Ren, and G. Huang. Efficiency and effectiveness of web application vulnerability detection approaches: A review. ACM Computing Surveys, 54(9):1–35, 2022. [5] Xiangfan Wu, Ju Xing, and Xiaoqi Li. Exploring vulnerabilities and concerns in Solana smart contracts. arXiv preprint arXiv:2504.07419, 2025. [6] Wenwen Zhou, Dongyang Lyu, and Xiaoqi Li. Blockchain security based on cryptography: A review. arXiv preprint arXiv:2508.01280, 2025. [7] Yaling Zhu, Jia Zeng, Fangchen Weng, Dan Han, Yiyu Yang, Xiaoqi Li, and Yuqing Zhang. Sybil attacks detection and traceability mechanism based on beacon packets in connected automobile vehicles. Sensors, 24(7):2153, 2024. [8] R. Amankwah, J. Chen, P. K. Kudjo, and D. Towey. An empirical comparison of commercial and open-source web vulnerability scanners. Software: Practice and Experience, 50(9):1842–1857, 2020. [9] J. Koman and M. Janiszewski. SCAnME - scanner comparative analysis and metrics for evaluation. International Journal of Information Security, 24(3):147, 2025. [10] W. Mazurczyk and L. Caviglione. Cyber reconnaissance techniques. Communications of the ACM, 64(3):86–95, 2021. [11] E. Ferrara, P. De Meo, G. Fiumara, and R. Baumgartner. Web data extraction, applications and techniques: A survey. Knowledge-Based Systems, 70:301–323, 2014. [12] Dr. S. Sarumathi, Ms. M. Sharmila, M. Saraswathy, and M. R. L. Priya. The benchmark analysis of different web scraping tools and techniques. IJARCCE, 13(5), 2024. [13] I. A. Essien, A. O. Adebayo, A. A. Afuwape, B. O. Eboseremen, F. Oladega, and O. M. Soneye. The ethics of web scraping in research: A review: Investigating the boundaries, legal implications, and societal acceptance of web scraping as a data collection method. Journal of Frontiers in Multidisciplinary Research, 4(1):529–538, 2023. [14] A. Rahman, R. Mahdavi-Hezaveh, and L. Williams. A systematic mapping study of infrastructure as code research. Information and Software Technology, 108:65–77, 2019. [15] L. Leite, C. Rocha, F. Kon, D. Milojicic, and P. Meirelles. A survey of devops concepts and challenges. ACM Computing Surveys, 52(6):1–35, 2020. [16] L. Ivančić, D. Suša Vugec, and V. Bosilj Vukšić. Robotic process automation: Systematic literature review. In C. Di Ciccio, R. Gabryelczyk, L. Garcı́a-Bañuelos, T. Hernaus, R. Hull, M. Indihar Štemberger, A. Kő, and M. Staples, editors, Business Process Management: Blockchain and Central and Eastern Europe Forum, volume 361, pages 280–295. Springer International Publishing, 2019. [17] K. Wiklund, S. Eldh, D. Sundmark, and K. Lundqvist. Impediments for software test automation: A systematic literature review. Software Testing, Verification and Reliability, 27(8):e1639, 2017. [18] S. G. Bianou and R. G. Batogna. PENTEST-AI, an LLM-powered multi-agents framework for penetration testing automation leveraging Mitre Attack. In 2024 IEEE International Conference on Cyber Security and Resilience (CSR), pages 763–770, 2024. [19] H. Xu, S. Wang, N. Li, K. Wang, Y. Zhao, K. Chen, T. Yu, Y. Liu, and H. Wang. Large language models for cyber security: A systematic literature review. ACM Transactions on Software Engineering and Methodology, page 3769676, 2025. 13
[20] R. Fang, R. Bindu, A. Gupta, Q. Zhan, and D. Kang. LLM agents can autonomously hack websites. arXiv, 2024. Version 3. [21] Z. Sheng, Z. Chen, S. Gu, H. Huang, G. Gu, and J. Huang. LLMs in software security: A survey of vulnerability detection techniques and insights. ACM Computing Surveys, 58(5):1–35, 2026. [22] Jiuyang Bu, Wenkai Li, Zongwei Li, Zeng Zhang, and Xiaoqi Li. SmartBugBert: BERT-enhanced vulnerability detection for smart contract bytecode. arXiv preprint arXiv:2504.05002, 2025. [23] H. Fang, J. Herbsleb, and B. Vasilescu. Novelty begets popularity, but curbs participation—A macroscopic view of the Python open-source ecosystem. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–11, 2024. [24] R. Paramitha and F. Massacci. Technical leverage analysis in the Python ecosystem. Empirical Software Engineering, 28(6):139, 2023. [25] A. Yadav, A. Kumar, and V. Singh. Open-source intelligence: A comprehensive review of the current state, applications and future perspectives in cyber security. Artificial Intelligence Review, 56(11):12407–12438, 2023. [26] J. R. G. Evangelista, R. J. Sassi, M. Romero, and D. Napolitano. Systematic literature review to investigate the application of open source intelligence (OSINT) with artificial intelligence. Journal of Applied Security Research, 16(3):345–369, 2021. [27] Wenkai Li, Xiaoqi Li, Yingjie Mao, and Yuqing Zhang. Interaction-aware vulnerability detection in smart contract bytecodes. IEEE Transactions on Dependable and Secure Computing, 23(1):298–315, 2026. [28] Yiming Shen, Kunhua Li, Lin Mao, Wenkai Li, and Xiaoqi Li. IntelliCon: Confidence-based approach for fine-grained vulnerability analysis in smart contracts. In International Conference on Blockchain and Trustworthy Systems, pages 45–59. Springer, 2023. [29] Xiaoqi Li, Wenkai Li, Zhiquan Liu, Yuqing Zhang, and Yingjie Mao. Penetrating the hostile: Detecting DeFi protocol exploits through cross-contract analysis. IEEE Transactions on Information Forensics and Security, 20:11759–11774, 2025. [30] N. K. Kahlon and W. Singh. A systematic review of web scraping: Techniques, LLM-enhanced approaches, performance metrics, and legal–ethical issues. Data & Knowledge Engineering, 164:102598, 2026. [31] Xiaoqi Li, Zongwei Li, Wenkai Li, Yuqing Zhang, and Xin Wang. No more hidden pitfalls? exposing smart contract bad practices with LLM-powered hybrid analysis. ACM Trans. Softw. Eng. Methodol., February 2026. [32] P. Di Francesco, P. Lago, and I. Malavolta. Architecting with microservices: A systematic mapping study. Journal of Systems and Software, 150:77–97, 2019. [33] M. Aydos, Ç. Aldan, E. Coşkun, and A. Soydan. Security testing of web applications: A systematic mapping of the literature. Journal of King Saud University - Computer and Information Sciences, 34(9):6775–6792, 2022. [34] B. Bednarz and M. Miłosz. Benchmarking the performance of Python web frameworks. Journal of Computer Sciences Institute, 36:336–341, 2025. [35] L. Albesher and R. Alfayez. An observational study on Flask web framework questions on stack overflow (SO). IET Software, 2024(1):1905538, 2024. [36] R. R, H. F. I, A. M, A. M. J, and D. S. Web application security testing framework using Flask. In 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), pages 1646–1652, 2023. [37] S. Liu, H. Duan, L. Heimes, M. Bearzi, J. Vieli, D. Basin, and A. Perrig. A formal framework for end-to-end DNS resolution. In Proceedings of the ACM SIGCOMM 2023 Conference, pages 932–949, 2023. [38] I. Livadariu, T. Dreibholz, A. S. Al-Selwi, H. Bryhni, O. Lysne, S. Bjørnstad, and A. Elmokashfi. On the accuracy of country-level IP geolocation. In Proceedings of the Applied Networking Research Workshop, pages 67–73, 2020. [39] Bin Zhu, Zhihong Tian, and Wenliang Duan. A survey on IP geolocation. In 2014 IEEE Workshop on Advanced Research and Technology in Industry Applications (WARTIA), pages 1039–1041, 2014. [40] B. Zolfaghari, G. Srivastava, S. Roy, H. R. Nemati, F. Afghah, T. Koshiba, A. Razi, K. Bibak, P. Mitra, and B. K. Rai. Content delivery networks: State of the art, trends, and future roadmap. ACM Computing Surveys, 53(2):1–34, 2021. [41] J. Wei, F. Zhu, and J. Jia. CDN-DF: A learning-based CDN domain name detection method with DNS deep feature extraction. In Proceedings of the 2023 6th International Conference on Information Science and Systems, pages 119–125, 2023. 14
[42] M. Ghaznavi, E. Jalalpour, M. A. Salahuddin, R. Boutaba, D. Migault, and S. Preda. Content delivery network security: A survey. IEEE Communications Surveys & Tutorials, 23(4):2166–2190, 2021. [43] K. Butler, T. R. Farley, P. McDaniel, and J. Rexford. A survey of BGP security issues and solutions. Proceedings of the IEEE, 98(1):100–122, 2010. [44] X. Dimitropoulos, D. Krioukov, M. Fomenkov, B. Huffaker, Y. Hyun, K. Claffy, and G. Riley. AS relationships: Inference and validation. ACM SIGCOMM Computer Communication Review, 37(1):29–40, 2007. [45] D. Everson and L. Cheng. A survey on network attack surface mapping. Digital Threats: Research and Practice, 5(2):1–25, 2024. [46] P. McMinn. Search-based software test data generation: A survey. Software Testing, Verification and Reliability, 14(2):105–156, 2004. [47] L. Han. Password cracking and countermeasures in computer security: A survey. arXiv, 2014. Version 3. [48] C. Nie and H. Leung. A survey of combinatorial testing. ACM Computing Surveys, 43(2):1–29, 2011. [49] Zejun Zhang, Zhenchang Xing, Xin Xia, Xiwei Xu, Liming Zhu, and Qinghua Lu. Faster or slower? performance mystery of Python idioms unveiled with empirical evidence. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023), pages 1495–1507. IEEE, 2023. [50] L. Sodian, J. P. Wen, L. Davidson, and P. Loskot. Concurrency and parallelism in speeding up I/O and CPU-bound tasks in Python 3.10. In 2022 2nd International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI), pages 560–564, 2022. [51] C. Olston and M. Najork. Web crawling. Foundations and Trends® in Information Retrieval, 4(3):175–246, 2010. [52] A. Sejfia, S. Das, S. Shafiq, and N. Medvidović. Toward improved deep learning-based vulnerability detection. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–12, 2024. [53] Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), volume 32, pages 10197–10207. Neural Information Processing Systems (NIPS), 2019. [54] J. Fan, Y. Li, S. Wang, and T. N. Nguyen. A C/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th International Conference on Mining Software Repositories, pages 508–512, 2020. [55] Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. SySeVR: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing, 19(4):2244–2258, 2022. [56] M. N. Uddin, Y. Zhang, and X. Hei. Deep learning aided software vulnerability detection: A survey. arXiv, 2025. Version 1. [57] X. Zhang, F. Zhang, B. Zhao, B. Zhou, and B. Xiao. VulD-Transformer: Source code vulnerability detection via transformer. In Proceedings of the 14th Asia-Pacific Symposium on Internetware, pages 185–193, 2023. [58] J. Li, B. Zhao, and C. Zhang. Fuzzing: A survey. Cybersecurity, 1(1):6, 2018. [59] P. Wang, X. Zhou, T. Yue, P. Lin, Y. Liu, and K. Lu. The progress, challenges, and perspectives of directed greybox fuzzing. Software Testing, Verification and Reliability, 34(2):e1869, 2024. [60] V. J. M. Manes, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering, 47(11):2312–2331, 2021. [61] S. Liu, W. Cui, Y. Wu, and M. Liu. A survey on information visualization: Recent advances and challenges. The Visual Computer, 30(12):1373–1393, 2014.
15