Finding Vulnerabilities via LLM-Augmented Semantics-Aware Type-Checking

arXiv:2608.14533v1 [cs.CR] 14 Aug 2026

Finding Vulnerabilities via LLM-Augmented Semantics-Aware Type-Checking Ruizhe Wang

Meng Xu

N. Asokan

University of Waterloo

University of Waterloo

University of Waterloo

a result, existing static-analysis tools often rely on manually crafted rules to approximate semantics and struggle to balance detection rates with false alarms. For example, CodeQL builds nearly 500 handwritten rules [2, 17] yet identifies less than 25% of bugs in practice [33]. On the other hand, improving bug detection rate often requires adopting more aggressive rules, which inevitably leads to high false positive rates [21, 29] that overwhelm developers, and erode their trust in the tool. These limitations largely stem from the difficulty of translating program semantics into manually crafted syntactic rules. Ideally, a static-analysis tool would operate directly on program semantics, precisely identifying bugs while minimizing false alarms. To address this gap, we propose a semantics-aware approach to statically detecting software vulnerabilities. We introduce S E T YPE, a type system that operates on program semantics rather than syntactic rules. In S E T YPE, every variable and expression is assigned a semantic type that captures its intended I. I NTRODUCTION meaning and behavior. Borrowing from the bidirectional type Static analysis is a widely adopted technique for identifying system [13], we design each operation to both check and infer potential software vulnerabilities. Instead of executing the the semantic types of its operands and results. If the entire program, static analysis tools examine the code through program type-checks, we consider it free of the vulnerabilities information such as syntax, structure (e.g., control flow), and detectable by S E T YPE. data flow, with the goal of finding insecure code patterns To illustrate, we present a path traversal vulnerability through a set of predefined rules [66]. (CWE-22) in Figure 1. S E T YPE checking starts from the To systematically identify vulnerabilities, static analysis review paper endpoint. Following the program execution flow, tools must reason about the security implications of operations it first infers pdf path as “user input” type (line 5) and and data flows in a program. For example, dereferencing a then abs pdf path as “file path from user” type (line 11). @Nullable pointer might lead to a null-pointer bug. This This “file path from user” type carries over to the pdf path resembles type systems in programming languages: static argument (line 20) through the function call (line 16) and analysis can be conceptualized as a form of type checking participate in the to markdown operation from the pymupdf4llm that aims to assign type labels to variables and expressions package (line 21), which is an agnostic file parser that reads and check whether certain type labels may participate in a the file at the given path, regardless of its source. As such, the given operation. In this context, the checking policy enforced review function expects a properly sanitized path argument, by the static analyzer would classify such a dereference as a and an argument of the “file path from user” type violates type-mismatch vulnerability. the typing rules and causes S E T YPE to raise a type error. However, while describing a vulnerability in natural language In the above analysis, the types inferred and checked by is often straightforward, precisely capturing its semantics S E T YPE (e.g., “file path from user”) are not conventional through a set of syntactic rules built upon abstract syntax programming types (e.g., int and string), whose meanings trees (ASTs), control-flow graphs (CFGs), or more generally are defined by the language. Instead, they are natural-language program-dependence graphs (PDGs) [17, 44, 49] is challenging. descriptions of the semantics of variables and expressions, Indeed, the problem of accurately capturing program semantics generated from the source code text rather than from ASTs, is known to be undecidable [45], meaning that no algorithm CFGs, or PDGs. Similarly, the S E T YPE checking described can determine the correctness of all possible programs. As above is not a mechanical algorithm. It relies on the natural-

Abstract—Vulnerability detection via static analysis traditionally relies on security experts encoding insecure coding patterns into algorithmic rules. However, this approach often focuses on syntactic patterns and overlooks deeper semantic information in the code, such as the meanings of variable and function names. As software systems grow more complex, modeling vulnerabilities using only syntactic rules becomes increasingly challenging. In this paper, we propose a semantics-aware approach to detecting software vulnerabilities. We present S E T YPE, a semanticsaware type system that can be derived directly from source code based solely on the meanings of symbols and expressions in natural language. In the S E T YPE type system, both type inference and checking are performed by Large Language Models (LLMs), and a failed type check indicates a potential vulnerability. We prototype P Y S E T YPE to demonstrate the feasibility of S E T YPE for detecting vulnerabilities in Python web applications. Our evaluation on real-world applications achieves 87% detection precision and 88% detection accuracy. Using P Y S E T YPE, we identified 15 potential zero-day vulnerabilities, nine of which were confirmed by developers.

1 2 3

@app.route("/api/review", methods=["POST"]) def review_paper(): """Review a paper using the Reviewer class"""

4 5 6 7 8 9 10 11 12 13

pdf_path = request.json.get("pdf_path") # Convert API path to absolute path if pdf_path.startswith("/api/files/"): # Remove /api/files/ prefix rel_path = pdf_path[len("/api/files/"):] gen_base = os.path.join(PROJECT_ROOT, "generated") abs_pdf_path = os.path.join(gen_base, rel_path) else: return "Invalid file path", 400

14 15 16 17

# Call reviewer.review() to get a single review review_result = Reviewer().review(abs_pdf_path) return review_result

18 19 20 21 22

def review(self, pdf_path: Optional[str] = None) -> str: text = pymupdf4llm.to_markdown(pdf_path=pdf_path) # ...

Fig. 1: Example path traversal vulnerability adopted from (CVE-202555149) [37]. The vulnerability arises from the unsanitized use of user input pdf path to access the file system in line 21.

Service (IRS). Nine of these vulnerabilities have been confirmed by respective development teams. P Y S E T YPE demonstrates practical scalability in this process—analysis on the largest codebase (195 API endpoints and 26k Python bytecodes) was completed in around fourteen hours with an estimated LLM inference cost of $3.19 using equivalent Qwen-Plus API pricing1 , making it a viable tool for real-world deployment. In summary, we make the following contributions: • A semantics-aware approach for vulnerability detection by introducing S E T YPE, a type system that enables vulnerability detection via automated code reasoning in natural language and harvests semantic information from source code that is not encoded in syntactic data structures (Section IV). • P Y S E T YPE , a prototype implementation that leverages LLMs to perform S E T YPE reasoning on Python programs to detect vulnerabilities common in web applications (Section V). • Extensive evaluation demonstrating P Y S E T YPE ’s effectiveness in detecting known vulnerabilities with high accuracy and precision, and its ability to discover new security issues in popular open-source projects while exhibiting practical scalability (Section VI).

language description of the path traversal vulnerability and is a fuzzy (non-deterministic) check at the natural-language level in practice. As all types in S E T YPE are in the natural-language domain, II. BACKGROUND we leverage Large Language Models (LLMs) as the backend to In this section, we present key background information that generate and reason. LLMs have demonstrated remarkable camotivates the design of P Y S E T YPE, including challenges in pabilities in understanding both code and natural language, and applying conventional static analysis and similarity between with their reasoning abilities, can analyze program semantics type checking and vulnerability modeling. effectively. We implement P Y S E T YPE, an extension of pytype [18] A. Static Analysis for Vulnerability Detection that incorporates S E T YPE reasoning for enhanced vulnerability Static analysis has always been a crucial tool for vulnerability detection in Python programs. Following the code-execution scanning in software development [3, 5, 12, 48, 65]. While flow inferred by pytype, P Y S E T YPE maintains a dynamic technically any tool that examines a program without executing representation of the program’s state and prompts an LLM to it with concrete inputs can be considered a static analyzer [4], in update the S E T YPE of variables and expressions following practice, most static analyzers (e.g., CodeQL [17], Bandit [44], the encountered operations (e.g., function calls or built-in and semgrep [49]) relies on predefined bug patterns expressed operations). We focus on detecting common vulnerabilities in over syntactic features such as AST, CFG, and PDG, which web applications, such as path traversal and SQL injection, with also include data flow information. security policies derived from Common Weakness Enumeration However, defining bug patterns is quite challenging. To (CWE) [39]. demonstrate, consider the path traversal vulnerability. In a We evaluate P Y S E T YPE through comprehensive experiments simplified scenario presented below, which both CodeQL on real-world Python codebases. As a static code analyzer, and Bandit can detect, an API endpoint get file takes P Y S E T YPE should follow generic criteria for static analysis— a file name parameter from user input. Without proper reporting as many genuine bugs as possible while minimizing validation, an attacker could inject directory traversal sequences false alarms. On a dataset of known vulnerabilities and their (e.g., ../../etc/passwd) into this parameter. When combined patches (SVEN [23]), P Y S E T YPE achieves 87% detection with a base directory path (e.g., FILE DIR), this malicious precision and 88% accuracy, demonstrating its effectiveness in input could allow access to files outside the intended directory vulnerability detection. To address memorization concerns, we scope, potentially exposing sensitive files like /etc/passwd. evaluate P Y S E T YPE on recent CVEs reported after the backend 1 @app.route("/get_file", methods=["POST"]) LLM’s training cutoff, and observe consistent performance 2 def get_file(): 3 file_name = request.form.get("file_name") across both datasets. 4 file_dir = os.path.abspath(FILE_DIR) We also deploy P Y S E T YPE to analyze 103 popular open5 path = os.path.join(file_dir, file_name) 6 source Python projects. P Y S E T YPE discovered 15 new vul7 with open(path, "rb") as f: nerabilities across high-profile codebases, including an official 8 data = f.read() 9 return data Microsoft library, Python Software Foundation (PSF) server 1 https://www.alibabacloud.com/help/en/model-studio/models infrastructure, and systems maintained by the Internal Revenue

catch this mismatch and executing this code would lead to a runtime TypeError exception.

In this scenario, the root cause is the direct use of usercontrolled input in file path construction without proper valida1 def retrieve_email(db: dict, name: str) -> str: 2 return db.get(name, "Email not found") tion or sanitization. Detecting such vulnerabilities followingly 3 requires identifying the data flow from user input to sensitive 4 # db = {"alice": "alice@example.com"} 5 retrieve_email(db, 23) operations (e.g., file access) and recognizing the absence of necessary validation steps. While this may seem straightforward, Similar to the syntactical TypeError, a vulnerability can be detecting such vulnerabilities effectively using static analysis viewed as a program construct of type X participating in an is challenging and requires addressing two key objectives: operation that expects type Y . However, unlike conventional 1) A comprehensive knowledge database. The tool must type systems that have formally defined type inference rules understand the purpose and behavior of each function used in (e.g., we can infer that v is of str type from the assignment the code (e.g., os.path.join, os.path.abspath, and open) and v = "abc"), it is hard to infer that a variable is a “user input” comprehend how they interact to form file paths. This requires type purely based on how the AST or CFG looks like, especially maintaining a comprehensive knowledge base of functions given the diversity of ways to obtain user inputs in realand their security implications, which becomes increasingly world codebases. This is partially the reason that many rules challenging as programming languages and libraries evolve for finding the path traversal vulnerabilities focus on better rapidly. For instance, consider the pymupdf4llm.to markdown definition of sources (APIs that return untrusted user inputs) function (shown in Figure 1). Despite this library being installed and sinks (APIs that consume paths and cause path traversal) millions of times per month2 , and its function signature clearly in static analysis [33, 63]. indicating that it reads a file, neither CodeQL nor Bandit The analogy between vulnerabilities and types extends to recognizes this behavior, resulting in missed vulnerabilities. type hierarchies and compositions. Just as complex types can 2) Detailed and versatile vulnerability patterns. The tool be built from simpler ones (e.g., List<String>), complex must recognize various sanitization and validation techniques vulnerabilities often arise from the interaction of multiple that prevent vulnerabilities. For instance, the patch for CVEsimpler conditions. For example, taking user input may not 2025-6776 (shown below) adds input validation allowed file always lead to a vulnerability, and a file operation is not in line 4, which CodeQL fail to recognize as a mitigation, inherently vulnerable by itself. It is when these two conditions although this behavior is semantically obvious: interact together in a specific way that can lead to a dangerous 1 @api.route(’/upload_to_local’) path traversal vulnerability. 2 3 4 5 6 7 8 9

def upload(): image = request.files.get(’image’, None) if image and allowed_file(image.filename): path = os.path.join(UPLOAD_DIR, image.filename) image.save(path) else: raise ParameterException() return Success()

Without understanding both vulnerable patterns and their mitigations, the tool risks missing actual vulnerabilities while raising false alarms on secure code. B. Type System and Vulnerability Modeling A type system is a set of rules that dictates how to assign and check types to various constructs in a program, such as variables, expressions, functions, and modules [41]. Type systems can also be bidirectional when rules are not fixed a priori [13]. The primary purpose of a type system is to ensure that operations in a program are performed on compatible types, thereby preventing type errors and enhancing program reliability. A static type system further requires type inference and checking at compile-time, and forcing a program that fails the static type check to execution (if possible) is almost doomed to trigger unexpected behaviors. For example, in the code snippet below, the function retrieve email (line 1) expects a string name as its second argument. When the function is called with an integer (line 5), a static type checker would flag this as a type error. Failing to 2 https://pypistats.org/packages/pymupdf4llm

III. T OWARDS LLM- ASSISTED S TATIC A NALYSIS While it is intuitive to see why LLMs are suitable candidates for addressing certain limitations of static analysis (e.g., finding sources and sinks autonomously, see Section II), how to systematically integrate LLMs into a vulnerability detection pipeline remains an open research challenge. Here, we present generic criteria for evaluating code analysis tools, followed by design options that we considered and justifications for our design choices. Design goals for an LLM-assisted static code analyzer: • Correctness: Avoid flagging safe code as vulnerable (i.e., high precision); • Completeness: Avoid flagging vulnerable code as safe (i.e., high recall); • Automation: Operate with minimal human input such as manual review or configuration; • Scalability: Analyze large codebases within a reasonable time frame and monetary cost. A. Design Candidates Option 1: Directly feed the entire source code and the description of a specific CWE to an LLM, and ask the LLM to conclude whether the code contains vulnerabilities. This option essentially offloads the entire task to the LLMs, which have complete control over how they explore the codebase and decide if the codebase contains vulnerabilities.

However, while modern LLMs have increased their input size LLMs focus on only one task. This also reduces uncertainty of capacity (up to 32k for Qwen3 [60]), they still face limitations LLM responses. At the same time, we do not restrict LLMs’ when processing large codebases or complex programs. Indeed, outputs to predefined templates, avoiding the brittleness of many models are initially trained on short text of code logical analysis rules by allowing LLMs to reason in natural snippets [20, 47] and struggle to maintain context when language directly. analyzing longer pieces of code. Furthermore, LLMs tend However, while this option preserves the strengths of static to prioritize some parts of the input over others, potentially analysis and LLM reasoning, we acknowledge that it also overlooking critical sections of code that are essential for inherits certain limitations from both sides (e.g., imprecise vulnerability detection. Prior works have mixed results when point-to results from static analysis and extra cost and nonapplying LLMs directly to large codebases, with some studies determinism from LLM queries). More detailed evaluation and reporting that less attention is paid to the middle of the discussion on the limitations can be found in Section VI-D, input [35], while others have found that LLMs indeed focus VI-E and Section VII. less on the end [50]. As a result, directly feeding the entire codebase to LLMs is not effective and may lead to missed IV. D ESIGN vulnerabilities or inaccurate assessments [26]. In this section, we describe the design details of P Y S E T YPE— Alternatively, one may consider breaking down the codebase an S E T YPE-aided static analyzer for Python programs. into smaller chunks and feeding them individually to the LLM. However, existing works have shown that the loss of the context Overview. P Y S E T YPE is a static-analysis framework that and interdependencies between different parts of the code can identifies potential vulnerabilities in Python programs using hinder the LLM’s ability to accurately interpret the code [27, S E T YPE. The analysis process traverses the program’s CFG 51]. An open question remains on how to effectively slice the while performing S E T YPE checking that combines syntactic codebase while preserving the necessary context for accurate type information with semantic reasoning about variable usage, derived from non-executable context such as variable names, vulnerability detection. Option 2: Task LLMs to create rules suitable for static comments, and docstrings. We illustrate this process with an analyzers to execute (e.g., CodeQL rules curated to the target example in Figure 2. 1 P Y S E T YPE begins by running vanilla pytype on the codebase). This option essentially delegates rule authoring to LLMs, leveraging their abilities to analyze source code and project to generate interface stubs for dependencies and build generate code-like rules while preserving the traditional static- the program’s CFG. This preprocessing step produces per-file object maps containing syntactic type information and control analysis workflow. However, enumerating comprehensive vulnerability rules is flow structures, which will be used in subsequent LLM-assisted challenging [28], and certain vulnerabilities cannot be easily analysis. Using the inferred syntactic types, P Y S E T YPE identifies captured by rigid logical rules [33]. Moreover, LLMs are inherently stochastic [53, 58], which can lead to hallucina- entry points through the main function and common server tions [25, 31, 64]. A recent study reports that the F1 score of endpoints (e.g., @app.route() in Flask and re path() in vulnerability detection using LLM-generated rules is 0.18 [33], Django), then simulates program execution from these points highlighting practical limitations. As a consequence, generated by traversing the CFG at the bytecode level via an extended rules necessitate additional verification prior to deployment pytype virtual machine. Entry points are analyzed once, while (e.g., via manually crafted examples [32]), which constrains other functions are revisited on each call to capture execution context precisely. scalability and automation. 2 During the simulation, P Y S E T YPE initializes the S E T YPE B. Seeking a Middle Ground of variables based on the non-executable context and instruOption 3 (our design): Grounding code exploration in a ments the bytecodes to maintain the S E T YPE of variables and deterministic and algorithmic traversal, while also allowing expressions encountered. For each operation that may change LLMs to comprehend source code semantics and improvise the S E T YPE (e.g., assignment, function call, branching, and vulnerability checking during code exploration. This option loop), P Y S E T YPE queries the LLM for the updated S E T YPE strikes a balance between Options 1 and 2: deterministic, after the operation with a prompt that includes the current algorithmic code exploration provides guaranteed coverage S E T YPE of the involved variables and the non-executable (reducing the chance of missed bugs), while LLMs remain context. The response from the LLM is then parsed and stored free to reason beyond rigid, predefined static-analysis rules and as the updated S E T YPE of the target variable or expression. Example prompts and the involved S E T YPE before and after adapt flexibly. Inspired by how vulnerability modeling is similar to type the operation are illustrated in Figure 3. systems (Section II-B), we adopt bidirectional typing [13] as 3 P Y S E T YPE uses S E T YPE to identify potential vulneraalgorithmic code exploration procedure (which touches every bilities. For each operation in the program, P Y S E T YPE follows reachable part of the program) and allow LLMs to reason a set of predefined vulnerability rules to determine whether about vulnerabilities in a localized type checking context. By it may introduce a security risk. If a mismatch is detected sending small queries (approximately 200 tokens), we keep between the S E T YPE of the involved variables and the expected

② SeType Reasoning

① Program Preparation pytype Target

Per-File Obj Map

@app.route('/api/count/{x}') def count_number(): n = count(x)

What is the semantic of calling `count` with variable `x`? The semantics of `count` is: … The semantic of calling … is …

pytype Dependencies

.pyi interfaces

14 PUSH_NULL 16 LOAD_NAME 2 (count) 18 LOAD_NAME 0 (x) 20 CALL_FUNCTION 2 22 STORE_NAME 1 (n) … …

③ Vulnerability Finding Do you think this call … True. CWE-22…

Fig. 2: Pipeline of P Y S E T YPE. P Y S E T YPE starts with preparing the program through pytype by generating interface stubs and program maps, and identifying the entry points ( 1 ). Next, it simulates the program execution by following the program’s IPCFG starting from the identified endpoints, during which it interacts with an LLM to infer the S E T YPE of variables ( 2 ) and perform type checking to identify potential vulnerabilities ( 3 ).

S E T YPE defined in the vulnerability rule, P Y S E T YPE flags it as a potential vulnerability.

as the container object (e.g., app or bp) is recognized as flask.app.Flask or flask.blueprints.Blueprint by pytype. A similar approach is applied to FastAPI. For Django, A. Program Preparation P Y S E T YPE extracts endpoints referenced in URL configuration P Y S E T YPE builds upon pytype, a static type analyzer for (e.g., path(...) and re path(...)) and treats the referenced Python that simulates bytecode execution to infer and check callables as endpoints. P Y S E T YPE additionally supports classsyntactic types [18]. Rather than reimplementing static analysis based views by recognizing view classes via pytype’s type inforfrom scratch, P Y S E T YPE extends pytype’s infrastructure by: mation: when pytype identifies a class inheriting from Django’s (1) leveraging its CFG, bytecode-level virtual machine, and per- view base classes (e.g., django.views.View), P Y S E T YPE treats file object maps for syntactic type information; (2) augmenting typical handler methods (such as get, post, and dispatch) the variable and frame abstractions to track S E T YPE and defined in the class as endpoints. All identified endpoints serve execution contexts; and (3) instrumenting the VM to query as starting points for the LLM-assisted S E T YPE inference ( 2 ) an LLM at semantically important instructions. This design and vulnerability finding process ( 3 ). enables P Y S E T YPE to combine pytype’s robust static analysis B. S E T YPE Inference with LLM-assisted semantic reasoning. Preprocessing. P Y S E T YPE first runs vanilla pytype on the We formally define S E T YPE in Section A. Here we describe project’s dependencies to generate interface stubs. These stubs how P Y S E T YPE infers S E T YPE during program analysis. capture the syntactic interfaces of external libraries and allow For program P , we extract its CFG G = (B, E), where B subsequent analysis to resolve types and call signatures for is the set of basic blocks and E is the set of edges connecting third-party and standard libraries. As LLMs are already trained the blocks. The analysis process traverses G and starts from on common public libraries, providing accurate stubs for the entry points identified in Section IV-A. S E T YPE of all dependencies is typically sufficient for LLMs to reason about variables are initialized to empty. the S E T YPE of variables involving these libraries. P Y S E T YPE analyzes each basic block in topological order After generating dependency stubs, P Y S E T YPE runs pytype within functions and depth-first order across functions. For on the target project to produce per-file object maps and to build each expression, P Y S E T YPE updates the variable’s S E T YPE by the program’s CFG. pytype records syntactic types of variables querying the LLM using templates from Table I. For example, at the basic-block level. These artifacts are then augmented in an assignment operation, the left-hand side variable’s with S E T YPE and execution-context during the LLM-assisted S E T YPE is updated based on the right-hand side expression’s analysis stage. S E T YPE and the name of the left-hand side variable. If Endpoint Identification. P Y S E T YPE first checks the main the right-hand side is a function call, P Y S E T YPE steps into function, which is the common entry point for Python scripts. the callee and infers the output’s S E T YPE before updating Additionally, P Y S E T YPE detects common endpoints to web the left-hand side variable’s S E T YPE. These S E T YPE are frameworks by matching framework-specific routing patterns. bounded to the corresponding basic block, and are inherited to This is straightforward as we can confirm which Python objects successor blocks following the CFG if it is the only predecessor. are framework-related by examining the syntactic types of these Alternatively, if a basic block has multiple predecessors (e.g., objects inferred by pytype. due to branching or loops), P Y S E T YPE merges the S E T YPE For Flask, P Y S E T YPE treats functions decorated with from all predecessors by querying the LLM with the merge app.route(...) or bp.route(...) as endpoints, as long prompt template in Table I.

TABLE I: Sample prompt templates for S E T YPE tracking in different operations. Comments and docstrings are included in the actual prompts but omitted here for brevity. Texts in monospace are dynamically filled based on the analysis context. Operation

Prompt Template

Assignment Call

I store the semantic rhs type to a variable named lhs name. Please describe the semantic of this new variable. I am calling a function named func name. This call takes the following parameters: param list. The semantic of the parameters are as following: param types. Please define the semantic of the output of this function call. After statically analyzing all possible executing branches, one variable is possible to have these possible semantic in different branches: semantic. Please consider all these possibilities and summarize its correct semantic. Variable var name has semantics old type. Now, I put this variable into a loop through an iterator iter. After the first iteration, the semantics of var name becomes into new type. Please define the final semantics of this variable after all iterations. I compare whether the lhs variable is operator the rhs variable. The lhs variable has the semantic lhs type. The rhs variable has the semantic rhs type. Suppose the comparison result is condition, summarize how this condition influences the execution of following code.

Merge Loop Branching

To handle variable updates within loops, P Y S E T YPE em- is queried about each item independently, and P Y S E T YPE ploys an induction process that considers three key aspects: aggregates the results to decide whether a potential semantic (1) the variable’s S E T YPE prior to entering the loop, (2) the violation exists. For example, the path traversal vulnerability variable’s S E T YPE after completing one iteration of the loop, is defined with the following checks: and (3) the S E T YPE of the loop iterator. By querying the LLM • is accessing (e.g., reading, writing, or deleting) a path; with these three S E T YPE, P Y S E T YPE infers the variable’s • has the accessed path constructed with a value provided S E T YPE upon exiting the loop. by an external client; Beyond variable S E T YPE, P Y S E T YPE also captures the • and, the called function is not a safe access function that execution context of each basic block. For example, as automatically sanitize the accessed path. illustrated below, a branching statement checks a variable’s Capturing Mismatches. P Y S E T YPE limits vulnerability type before an operation, the execution context within that checks to function calls whose callees are not defined within branch thus contains additional information about the vari- the analyzed project (i.e., external library functions). For able’s S E T YPE. This contextual information is essential each external function call, P Y S E T YPE constructs a prompt for accurate S E T YPE analysis and vulnerability detection. containing the current S E T YPE of involved variables and 1 if IS_ADMIN: queries the LLM to match against vulnerability rules. Upon 2 user = admin_user() For each basic block, we define its execution context as a set finding a match, P Y S E T YPE queries the LLM again with of conditions that must be satisfied for the block to be executed. the execution context to confirm the rule’s conditions are These conditions are derived from the branching statements satisfied. If confirmed, the operation is flagged as a potential leading to the block in the CFG. We use the “branching” vulnerability; otherwise, analysis continues without raising an templates presented in Table I to capture the execution context alert. To reduce LLM queries, P Y S E T YPE employs taint analysis during the analysis. to track untrusted data flows from external sources. If data C. Vulnerability Finding involved in an operation is identified as tainted, P Y S E T YPE With the inferred S E T YPE and execution contexts, skips the LLM query checking whether the data originates P Y S E T YPE performs type checking to identify potential from an external client. vulnerabilities in the program. Each time a function that matches a vulnerability rule is called (see details below on D. Illustrative Example how P Y S E T YPE checks for matches), P Y S E T YPE checks Putting all the components together, we present an illustrative whether the S E T YPE of the arguments passed into the operation example in Figure 3, which is a simplified version of a satisfy the expected S E T YPE of the operation defined in the vulnerability adopted from CVE-2025-55149 [61], same as vulnerability rule. If a type mismatch is detected, S E T YPE the one shown in Figure 1. In this example, we skip the flags it as a potential vulnerability. program preparation step ( 1 in Figure 2) and start with Vulnerability Rules. We focus on common, high-impact S E T YPE analysis ( 2 ) and vulnerability finding ( 3 ). P Y S E T YPE starts simulating the program execution from vulnerabilities, including SQL injection, command injection, cross-site scripting (XSS), and path traversal (see Section A the program entry point, which is the review paper function for the complete list of target CWEs). Each vulnerability rule in this example. The CFG of this function contains four basic is described following CWE definitions [39] and maintains a blocks: the entry block (BB1), the if block (BB2), the else high-level abstraction to facilitate generalization across different block (BB3), and the exit block (BB4). Analysis starts from the entry block (BB1). In line 5, P Y S E T YPE encounters a codebases. To mitigate limitations of LLM reasoning, P Y S E T YPE function call request.json.get("pdf path") that retrieves a employs a checklist-based verification strategy [9]. The LLM file path from the incoming request. P Y S E T YPE constructs a

1 @app.route("/api/review", methods=["POST"]) 2 def review_paper(): 3 """Review a paper using the Reviewer class""” 4 5 pdf_path = request.json.get("pdf_path") BB1 6 # Convert API path to absolute path 7 if pdf_path.startswith("/api/files/"): 8 # Remove /api/files/ prefix 9 rel_path = pdf_path[len("/api/files/"):] BB2 10 gen_base = os.path.join(ROOT, "generated") 11 abs_pdf_path = os.path.join(gen_base, rel_path) 12 else: 13 return "Invalid file path", 400 BB3 14 15 # Call reviewer.review() to get a single review 16 review_result = Reviewer().review(abs_pdf_path) BB4 17 return review_result 18 19 20 def review(self, pdf_path: Optional[str] = None) -> str: 21 formatter = InputFormatter() 22 text = pymupdf4llm.to_markdown(pdf_path=pdf_path) 23 # ...

Line 5: Call

What is the semantic of calling request.json.get with variable param? The semantics of param is: constant string pdf_path.

Line 5: Assign

What is the semantic of assigning A path obtained from client. to the variable pdf_path?

Line 7: Call

What is the semantic of calling start_with on the variable pdf_path with prefix? The semantics of prefix is constant string /api/files The semantics of pdf_path is … file.

Line 7: Branch T & F

I compare externally-provided pdf path starts with a prefix is equal to True. If the comparison result is {True/False}, summarize how this influence the following execution?

Line 14: Merge

No merge needed

Line 22: Check

I am calling function formatter.parse_paper_pdf_to_json, which takes a param pdf_path. We know the parameter:

BB1 tmp_var: … client pdf_path: … file.

BB2

BB3

pdf_path: … file.

CONTEXT: F

tmp_2: … prefix. rel_path: … CONTEXT: T

BB4 tmp_var: … client pdf_path: … file.

• pdf_path: A path obtained from client representing a pdf file. Do you think this call is accessing (reading or writing to) a file ?

CWE-22

rel_path: …

Fig. 3: Walkthrough with a simplified path traversal vulnerability adopted from CVE-2025-55149 [61]. The code snippet is the same as the one presented in Figure 1. We illustrate how P Y S E T YPE simulates the program execution following the CFG, infers the S E T YPE of variables, captures the execution context, and identifies the potential security vulnerability. We only illustrate the key communications with the LLM for brevity and clarity.

prompt using the call template in Table I and queries the LLM to infer the S E T YPE of the call output. Given that this output is not assigned to any variable, P Y S E T YPE temporarely stores it in a special variable tmp var with the inferred S E T YPE “a file path provided by an external client”. Still in this line, an assign operation assigns the value of tmp var to the variable pdf path. P Y S E T YPE constructs another prompt using the assignment template in Table I and queries the LLM to update the S E T YPE of pdf path to “a path provided by an external client pointing to a pdf file”. Both types are stored in the BB1 object in the pytype virtual machine. With some executions in between, the analysis reaches line 7, where P Y S E T YPE encounters a branching statement that checks whether the file at pdf path has a specific prefix. P Y S E T YPE constructs a prompt using the branching template in Table I and queries the LLM to capture the execution context for both branches. The resulting execution contexts are stored in BB2 and BB3 of the pytype virtual machine, respectively. Since these two basic blocks inherit from BB1, the S E T YPE of pdf path is also inherited. After analyzing both branches, the analysis reaches line 14, where the two branches merge. P Y S E T YPE similarly merges the execution contexts of BB2 and BB3. However, since no variable is assigned in either branch, the S E T YPE of all variables remains unchanged. Finally, when the analysis reaches line 21, P Y S E T YPE encounters a function call to pymupdf4llm.to markdown that processes the file at pdf path. P Y S E T YPE queries the LLM to check whether this operation is vulnerable. The LLM confirms that this operation involves path access, and the accessed path is constructed from untrusted input and is not sanitized, matching all conditions in the path traversal rule. Consequently, P Y S E T YPE flags this operation as a potential vulnerability. V. I MPLEMENTATION We implement P Y S E T YPE, an extension of pytype that incorporates LLM-based S E T YPE reasoning into syntactic

type analysis for Python projects. The implementation extends pytype release 2024.10.11 [18] with approximately 3,000 lines of code to support whole-project semantic analysis and Python 3.10. We use the Qwen3 32B model [60] as the LLM backend via the Ollama API [40], with top p=0.3 to reduce output randomness. All other parameters are at default values. P Y S E T YPE targets three popular Python web frameworks: Flask, FastAPI, and Django. We implement framework-specific rules to identify application endpoints, as detailed in Section IV-A. The system focuses on 12 common web vulnerability types, including SQL injection, XSS, and command injection. The complete list of target vulnerabilities is provided in Section A. For each vulnerability type, we design tailored prompts to assist the LLM in detecting S E T YPE inconsistencies associated with that vulnerability, adhering to the principles outlined in Section IV-C and employing the zero-shot prompting method. VI. E VALUATION In this section, we evaluate P Y S E T YPE against the design goals in Section III using real-world vulnerabilities. A. Evaluation Setup Environments. We run P Y S E T YPE on a server with AMD EPYC 7302 CPU and NVIDIA A100 40GB GPU. Comparison Baselines. We compare P Y S E T YPE with three baseline tools: • CodeQL [17] is a widely used static-analysis platform that detects security issues via built-in and custom queries. It supports multiple programming languages, including Python. We use CodeQL release 2.22.3 and the default Python security queries in our evaluation. • Bandit [44] is a Python-specific static-analysis tool developed by the Python Code Quality Authority (PyCQA). We run Bandit with its default rule set. • Vulnhuntr [43] is an LLM-based vulnerability detection tool developed by Protect AI (Palo Alto Networks). It combines

static analysis with LLM reasoning: the LLM first inspects the codebase and highlights snippets it considers potentially vulnerable, along with a list of functions or variables it wants to examine further. Vulnhuntr then uses static analysis to locate the definitions of these symbols and feeds the gathered snippets back to the LLM. This iterative process continues until the model requests no additional context. We use Vulnhuntr’s default configuration, which asks the LLM to assign a confidence score (0-10) and reports findings with score ≥7. We also patch Vulnhuntr to resolve certain parsing issues when it interacts with the LLM by extending its parser. Specifically, we extend Vulnhuntr’s parser to be more tolerant of common formatting deviations and to retry up to three times upon parsing failures. Datasets. We use three datasets to evaluate the effectiveness of P Y S E T YPE in detecting real-world vulnerabilities. Details of the construction and filteration of the three datasets is provided in Section A. • SVEN: Real-world Vulnerable Programs with Patches. We use the SVEN dataset [23], which contains real-world vulnerable Python programs with corresponding developer patches. After filtering out C projects, Python 2 projects, and those incompatible with current framework versions, we obtain 80 vulnerability instances from 27 repositories. The dataset spans four CWE classes: path traversal, OS command injection, XSS, and SQL injection, with vulnerabilities patched between 2016 and 2018. We evaluate on 160 total program versions (vulnerable and patched pairs), enabling comprehensive assessment of both detection recall and patch-aware accuracy across all instances. • Recent CVEs with (Partially Available) Patches. To address the potential risk of SVEN containing outdated examples present in the LLM training data, we constructed a more recent CVE-based dataset. We queried the CVE database [42] for vulnerabilities in projects utilizing Flask, Django, or FastAPI, focusing on CVEs published between 2024 and September 2025 (after the LLM’s knowledge cutoff of end-2023 3 ). After filtering out entries that are not server applications implemented with one of the three frameworks or whose vulnerabilities do not originate from the Python component, we identified 41 candidate CVE entries. We then excluded entries that (1) do not fall within any CWE category targeted by P Y S E T YPE or (2) are not open-sourced on GitHub. The final dataset comprises 14 program versions corresponding to 9 CVEs spanning five CWE classes, with 5 of them patched by the developers. • Popular Open-Source Projects. We evaluate P Y S E T YPE on a curated set of popular Python web projects that use Flask, Django, or FastAPI. We began with GitHub dependency data and used ghtopdep [1] to automate repository collection. From the resulting candidates, we inspected the top 100 starred repositories per framework and removed tutorials, plugins/metapackages, toy examples, and mismatched repositories (e.g., repositories that only mention a framework in lock files 3 https://github.com/QwenLM/Qwen3/issues/525#issuecomment-2159944 330

TABLE II: Performance evaluation on the SVEN dataset. Tool CodeQL Bandit Vulhuntr P Y S E T YPE

Precision

Recall

Accuracy

F1 Score

0.67 0.75 0.52 0.87

0.39 0.84 0.45 0.87

0.83 0.70 0.53 0.88

0.49 0.79 0.48 0.87

or tests). After manual validation, deduplication, and exclusion of misclassifications, the final evaluation set comprises 103 repositories. B. Vulnerability Detection Set up. We evaluate P Y S E T YPE on the SVEN dataset and compare it against the baselines described in Section VI-A. We run all tools on the vulnerable versions to assess detection recall and on the patched versions to measure the detection accuracy. Following our methodology, we analyze entire projects rather than individual files to obtain realistic vulnerability detection results. For each tool, especially Bandit and CodeQL that report numerous warnings (e.g., hardcoded password string), we manually inspect the findings reported by the tool to check if they are related to the original vulnerabilities. We report four standard metrics: precision, recall, accuracy, and F1 score in Table II. Compare to Baselines. We observe that P Y S E T YPE effectively detects real-world vulnerabilities, outperforming all baselines in all four metrics. While CodeQL achieves a similar high accuracy, its recall is significantly lower than P Y S E T YPE, indicating that many vulnerabilities are missed, aligning with the observations in prior work [33]. Bandit achieves an overall high performance but is still outperformed by P Y S E T YPE in all metrics. Most notably, Bandit achieves a relatively low accuracy, potentially due to its coarse-grained heuristics that lead to many false positives (e.g., flagging all use of Python string concatenation in SQL construction as vulnerable), which produces overly generalized findings. Vulnhuntr shows the lowest performance among all tools, which may be due to its design of depending on the LLM to analyze large code sections at once, making it more prone to hallucination and incomplete analysis. Validation through Recent CVEs. To validate that the effectiveness of P Y S E T YPE is not due to potential data leakage in the LLM training data, we further evaluate P Y S E T YPE on recent CVEs after the knowledge cutoff of the used LLM. Using the recent CVE dataset constructed in Section VI-A, we similarly run all the tools on the vulnerable and patched versions (if the vulnerable has been patched) and check the warnings. The results are shown in Table III. All tools show a decrease in performance. This is possibly because that the programs in this dataset are more complex and the vulnerabilities are more sophisticated. Specifically, we see that the precision of the two static analysis tools, CodeQL and Bandit, keep consistent compared to the results in Table II, while their accuracy and recall drop significantly. This indicates that while the static analysis rules are still effective

TABLE III: Performance evaluation with recent CVEs.

due to an insufficient patch by the developers. We discuss this case in Section A. 1) Cases Where P Y S E T YPE is More Effective: We present CodeQL 0.67 0.22 0.43 0.33 0.75 0.33 0.50 0.46 Bandit two concrete cases where P Y S E T YPE outperforms all baselines: Vulhuntr 0.67 0.44 0.50 0.53 the example illustrated in Figure 3 and CVE-2025-49126 [10], P Y S E T YPE 0.78 0.78 0.71 0.78 in which user input is directly returned by an endpoint and TABLE IV: Average execution time (t) and monetary cost (c) on used to generate HTML via get swagger ui html, a FastAPI the SVEN and new CVE dataset (in seconds and US cents). tp of function that produces an HTML page. This results in an XSS P Y S E T YPE refers to the time for static analysis and type inference vulnerability if the user input is maliciously crafted. ( 1 in Figure 2). Compare with Traditional Static Analysis. P Y S E T YPE leverages LLMs to reason about the security implications Tool (Metric) SVEN CVEs of functions without the need for predefined rules. In both CodeQL (t) 14.92 16.60 6.23 9.66 examples, the vulnerabilities are caused by the use of (relBandit (t) Vulhuntr (t) 230.90 413.64 atively) uncommon functions (i.e., get swagger ui html P Y S E T YPE (tp ) 48.59 200.61 and pymupdf4llm.to markdown) which are not marked as P Y S E T YPE (t) 1470.52 3558.59 dangerous sinks by static analysis tools. As a result, CodeQL Vulhuntr (c) 2.16 3.56 and Bandit fail to detect these vulnerabilities, while P Y S E T YPE P Y S E T YPE (c) 10.28 26.44 successfully detects them using the LLM’s knowledge and inference capabilities. in identifying certain vulnerabilities, they miss (potentially Compare with LLM-based Analysis. P Y S E T YPE adopts new) vulnerability patterns that are not covered by the rules. a checklist-based analysis to guide the LLM to perform The performance of Vulnhuntr and P Y S E T YPE drop slightly, systematic reasoning, to limit the analysis complexity, and indicating that the LLM unlikely has seen these vulnerabilities to follow execution paths to reduce hallucination. We observe in its training data, and the detection capability of LLMs that P Y S E T YPE can understand the context of code snippets generalizes to unseen vulnerabilities. and reason effectively about their security implications. In the example illustrated in Figure 3, we observe that C. Execution Cost P Y S E T YPE precisely identifies the root cause: “while there We present the average of execution time and monetary are checks for path prefixes and file extensions, and some cost of running all four tools on both the SVEN and recent validation of path components, there is no explicit check that CVE datasets in Table IV, following the scalability design normalizes the path or ensures that the final resolved path does goal in Section III. For monetary costs, as we deploy the LLM not escape the intended directory structure.” locally, we estimate the cost should a remote LLM service be In contrast, Vulnhuntr’s approach of asking the LLM to used based on tokens sent to and received from the LLM, using analyze large code sections in a single pass makes it more Qwen-Plus pricing (0.4 USD per 1M input tokens, 1.2 USD prone to hallucination and incomplete analysis. In its analysis of per 1M output tokens) as a reference since it offers similar CVE-2025-49126, Vulnhuntr initially identifies two potentially performance to our locally deployed Qwen3-32b model. vulnerable functions: one containing the vulnerability and the Two traditional static tools (CodeQL and Bandit) complete other benign. However, when the LLM is asked for deeper the analysis within seconds on average. Comparably, Vulnhuntr analysis, it shifts focus to the benign function only and later and P Y S E T YPE take significantly longer time due to the contradicts its initial assessment by concluding that both multiple rounds of LLM invocations. Notably, running the functions are unlikely to be vulnerable. In this case, the LLM’s perparation phase of P Y S E T YPE ( 1 in Figure 2) alone is inability to maintain focus on the vulnerable function leads to already slower than CodeQL and Bandit, as it involves static a missed detection by Vulnhuntr. type inference for all bytecode instructions in the project. In the other example, Vulnhuntr hallucinates a nonexistent Compared with Vulnhuntr, P Y S E T YPE takes a longer time due serve files function and reports a path traversal vulnerability to its more systematic and in-depth analysis of each endpoint, in it with high confidence (9/10). These false detections which involves multiple rounds of LLM queries per endpoint. highlight the challenges of applying LLMs to large codeBetween the two datasets, analyzing the CVE dataset takes bases without structured guidance, in contrast to P Y S E T YPE’s longer time for all tools, validating the observation that the CVE checklist-driven approach, which focuses the LLM’s reasoning dataset contains more complex programs and vulnerabilities. on specific execution paths and security properties. D. Case Studies 2) Cases Where P Y S E T YPE Misdetects: Although P Y S E T YPE combines static analysis with LLM-guided While in the previous sections we have shown the overall reasoning to balance their complementary strengths, limitations effectiveness of P Y S E T YPE in vulnerability detection, we in either component can cause misdetections. hereby present certain cases to help understand the strengths and weaknesses of P Y S E T YPE. Beyond the two discussions Statically Unreachable Code. P Y S E T YPE works by iterabelow, we also observe that P Y S E T YPE raises a false alarm tively analyzing the code snippets following the execution paths Tool

Precision

Recall

Accuracy

F1 Score

E. Finding Zero-day Vulnerabilities

Time (Second) Cost (USD)

50000

3.0 2.5

40000

2.0

Cost (USD)

Time (Second)

in the target project. It is unable to analyze code that is not reachable from the defined entry points. For example, in the SocialniSystem.cz4 project in the SVEN dataset, there is a vulnerability in the get context data method. This method overrides a parent class’s method and is called by the Django framework through polymorphism, rather than being directly invoked in the main execution paths. Another example is the use of dynamic features in Python, exemplified by the ScoringEngine5 project in the SVEN dataset. It dynamically decides which module to import based on specifications in a configuration file, which is beyond the capability of static analysis. These uncaught interactions make certain code snippets unreachable for P Y S E T YPE, and thus vulnerabilities in those snippets are missed by P Y S E T YPE. We present an additional example in Section A to illustrate the complexity of such unreachable code. LLM Hallucination. LLM reasoning can be inconsistent, producing both false negatives and false positives. For example, in the Python Wiki App6 there are five similar SQL-injection sites: P Y S E T YPE detects four but misses one because the LLM incorrectly judges db.query as non-sink in that case while correctly identifying it as an execution sink elsewhere. In another case (CLA Hub7 ), the model identifies the concrete template string (CE/home page.html) as user-controlled and produces a false finding.

30000

1.5 20000

1.0

10000

0.5 0.0

0 0

5000

10000

15000

Size (Instruction)

20000

25000

Fig. 4: Time (seconds), monetary costs (USD), and the sum of LoC of analyzed functions on popular GitHub projects.

Compare with Baselines. Out of the 15 vulnerabilities identified by P Y S E T YPE, CodeQL detects 9 of them, Bandit detects 2 of them, and Vulnhuntr detects 1 of them. Reasons of failures mainly align with those discussed in Section VI-D, such as the lack of rules in traditional static analysis tools and hallucination in LLM-based tools. False Alarms in Detection. P Y S E T YPE reports 33 findings across these projects, of which 15 we confirmed as true vulnerabilities, and 2 are less severe due to a limited attack surface, as discussed above. All other alarms are false positives. P Y S E T YPE raises more false positives than in the previous evaluations. We attribute this to these projects being more complex and generally larger than those in the previous datasets, which makes it more challenging for P Y S E T YPE to analyze them effectively. These false alarms are raised align with the reasons discussed in Section VI-D (i.e., hallucination and uncaught interactions). For example, the Flask flask.send from directory sends a file from a specified directory but is identified by P Y S E T YPE as a dangerous sink that may lead to an XSS. Another example is that the LLM incorrectly believe a username is a sensitive field that should not be directly stored in the database. It raises a false alarm for the cleartext-storage-of-sensitive-information vulnerability.

To further demonstrate the effectiveness of P Y S E T YPE, we run it on popular open-source projects to discover zero-day vulnerabilities. We explore the top-starred Python projects on GitHub that use one of the three frameworks, as described in Section VI-A. We run P Y S E T YPE on the latest versions of these projects and manually inspect the findings reported by P Y S E T YPE to identify potential zero-day vulnerabilities. We identify 15 previously unknown vulnerabilities that we consider security-critical, and 2 additional vulnerabilities that are less severe due to a limited attack surface as the server is configured for local, personal use. The identified vulnerabilities were responsibly disclosed to the developers, and 9 have been confirmed by the developers at the time of writing. Details of the disclosure process are provided in the ethics section. Of the 15 identified vulnerabilities, 9 are path traversal, 5 Costs and Program Characteristics. We analyze the time are open redirect, and 1 is SQL injection. The categories of the and monetary costs of running P Y S E T YPE on popular GitHub identified vulnerabilities are consistent with those in the SVEN projects. In Table V we present the average time, cost, and other and recent CVEs datasets, indicating that these types of vulner- key metrics of running P Y S E T YPE on these projects, grouped abilities remain prevalent in real-world applications. However, by framework. On average, a program has 15.66 endpoints we note that four out of five open-redirect vulnerabilities occur and takes 3,250.75 seconds (54 minutes) and costs $0.23 to in projects using the Django framework, and eight out of nine analyze. The preparation phase ( 1 in Figure 2) takes 419.35 path traversal vulnerabilities occur in projects using the Flask seconds (7 minutes) on average, accounting for 12.9% of the framework, suggesting that certain frameworks may be more total time. The remainder is spent on LLM-assisted analysis, prone to particular vulnerability types. We further discuss this which makes on average 1,168.93 calls to infer S E T YPE and 523.83 calls to check security properties. phenomenon in Section A. Per endpoint, analysis takes 223.34 seconds (3.7 minutes) 4 https://github.com/pirati-web/socialnisystem.cz and costs 1.64 US cents on average. Each endpoint involves 5 https://github.com/DSU-DefSec/ScoringEngine 6 https://github.com/Pumala/python wiki app redo analyzing 145.90 bytecode instructions on average, requiring 7 https://github.com/stevetasticsteve/CLA Hub 81.49 LLM queries to infer S E T YPE and 36.03 LLM queries

TABLE V: Average metrics of running P Y S E T YPE on the real-world popular projects. tp and t refer to the time in the preparation phase only ( 1 in Figure 4) and the total execution time respectively, and are in seconds. c is the LLM execution cost in US cents, # API is the number of analyzed endpoints, # Inst is the number of analyzed bytecodes, # Inf is the number of LLM queries for inferring S E T YPE, and # Sec is the number of LLM queries for checking security properties. Per-Project

Per-API

Metric

tp

t

c

# API

# Inf

# Sec

tp

t

c

# Inst

# Inf

# Sec

Flask Django FastAPI

168.08 768.11 297.78

1964.21 5221.27 2426.63

17.77 29.60 21.33

8.85 24.86 12.64

828.50 1453.39 1179.88

396.74 702.87 446.70

21.08 69.31 33.96

263.09 197.58 210.50

2.40 0.83 1.73

200.86 68.20 174.03

112.57 39.47 94.32

51.72 20.55 36.44

All

419.35

3250.75

23.03

15.66

1168.93

523.83

42.06

223.34

1.64

145.90

81.49

36.03

to check security properties. templates), and vulnerabilities may arise from these crossTo better understand the relationship between time, cost, and language interactions. For example, when a Python function program characteristics, we illustrate these metrics for each renders an HTML template that interpolates user input, XSS project in Figure 4. In the figure, we can observe that the cost may arise in the generated HTML even though the Pythonis generally proportional to the number of bytecodes analyzed, level value appears harmless, and vice versa. P Y S E T YPE as more code requires more LLM queries, leading to higher currently offers only heuristic handling of these cross-language costs. Alternatively, while execution time generally increases interactions (e.g., simple string-pattern checks and treating with the number of bytecodes analyzed, its correlation with template render calls as opaque), which is insufficient to model bytecode count is weaker than for monetary cost because language-specific escaping. A more robust solution would program structure and complexity also significantly affect require cross-language analysis that accurately models the runtime. In our experiments, the largest project analyzed has semantics and security properties of each involved language, 195 endpoints and passes roughly 26k bytecode instructions to which is an important direction for future work. the LLM. This takes 51,454 seconds (≈14 hours), and costs Implementation Limitations. P Y S E T YPE is intended as $3.19. a prototype to demonstrate the effectiveness of S E T YPE in Among frameworks, Django projects require more total detecting vulnerabilities. As such, it has several implementation analysis time but less time per endpoint than Flask and FastAPI. limitations that should be addressed in future work. First, This likely reflects that Django apps are larger and more P Y S E T YPE currently targets Python 3.10 only: bytecode and complex overall, whereas individual endpoints tend to be VM behavior differ across Python releases, so additional simpler. engineering is required to support other versions reliably. Second, the implementation covers a restricted set of language constructs and libraries; advanced or highly dynamic features VII. L IMITATIONS (e.g., metaclasses, complex decorators, and dynamic imports) While P Y S E T YPE presents a novel approach to achieve may not be modeled accurately. Third, our vulnerability checks S E T YPE inference and vulnerability detection, it has several focus on a limited set of semantic rules commonly observed limitations that need to be addressed in future work. in web applications; broader CWE coverage and finer-grained Statically Unknown Code. P Y S E T YPE is designed to analyze checks are needed for wider applicability. Python programs statically. However, certain dynamic features of Python may lead to uncaught interactions during the analysis. VIII. R ELATED W ORKS For example, if a program dynamically loads a module at A. Traditional Vulnerability Detection runtime, P Y S E T YPE cannot accurately infer the types of Static analysis tools for vulnerability detection have long variables associated with that module. Similarly, if the indirect call map constructed by pytype is incomplete, P Y S E T YPE relied on manually crafted rules and pattern matching to flag may miss certain function calls that could affect the type risky code constructs. Targeting on this, prior works can be inference process, and code paths involving these calls will grouped into two complementary directions: not be analyzed. Extending Search Space. One line of work targets on Furthermore, P Y S E T YPE only analyzes code belonging to expanding the expressiveness of vulnerability specifications the target project and limits itself to considering only the to capture richer program semantics through previously uninterfaces of code in dependent libraries. However, if dependent exploited relationships. Examples include inter-procedural libraries trigger certain behaviors of the target project (e.g., specifications that relate allocation/use/free lifecycles [62], through class inheritance or callback functions), P Y S E T YPE precise detection of multiple-read accesses [59], and leveraging may miss these interactions, leading to incomplete or incorrect class inheritance [34]. analysis results. The case in Section VI-D2, is an example. Automatic Pattern Inference. To reduce manual effort, a Interact with Other Languages. Many Python applications second line of work targets on deriving patterns automatically interact with other languages and runtimes (e.g., HTML from data. Approaches include mining patches and change

histories to infer repair-driven specifications [7], assuming normative behavior by majority voting over observed uses and flagging deviations [38], and detecting anomalies by crosschecking functions with similar intent across a codebase [36]. B. Learning-Based Vulnerability Detection Code Language Models (CodeLMs) are specialized language models trained on large corpora of source code, enabling them to understand programming languages and code structures. Examples of these models include CodeBERT [14], CodeLlama [47], and DeepSeek-Coder [19]. However, recent works have shown that these models struggle to reliably identify and reason about security vulnerabilities in large codebases and for uncommon vulnerability types [52]. Alternatively, researchers have trained models specifically for vulnerability detection. At different levels of detection granularity (e.g., repository-level [54], function-level [6, 22], slice-level [8], and line-level [15]), these methods train predictive models to identify whether a code snippet is vulnerable. These approaches can autonomously detect vulnerabilities without manually defined rules, but they suffer from limited interpretability and require large labeled datasets for effective training and generalization.

results. Conversely, static analysis offers complementary guarantees, such as soundness and provable absence of certain classes of bugs, but is constrained by a limited ability to understand program semantics at a higher level and to infer developer intent. Taken together, these complementary weaknesses suggest a hybrid direction. A middle ground that leverages the strengths of LLMs for reasoning and summarization, together with static analysis for analysis driven rigor, could yield more reliable and precise vulnerability detection. As a step toward this middle ground, we propose S E T YPE that associates both syntax and semantic types with program variables, enabling more precise type checking that can identify semantically inconsistent operations despite their syntactic correctness. An instantiation of S E T YPE, P Y S E T YPE tailored towards Python programs, orchestrates model queries and analysis passes, employs caching and summarization to mitigate context-length limitations, and uses iterative verification loops to reduce hallucinations and improve precision. Evaluation shows that it achieves notable detection rates, demonstrating the viability of this hybrid approach. We believe combining LLM-driven reasoning with rigorous static analysis offers a promising path toward more effective vulnerability detection in real-world software.

C. LLM-based Contextual Bug Analysis LLMs have been leveraged into the vulnerability analysis domain in two main ways: LLMs for Direct Vulnerability Detection. Recent works have explored using LLMs directly for vulnerability detection by leveraging their understanding of security concepts. For instance, researchers have used CWE descriptions to guide LLMs in identifying potential vulnerabilities [32] and locating vulnerable functions [57]. Others have focused on detecting inconsistencies between code and comments to reveal potential security issues [46]. There also exists frameworks that employ multiple LLM agents working collaboratively to reduce false positives [56]. LLMs for Assisting Traditional Analysis. LLMs have been particularly effective in identifying data flows and critical functions. Recent studies have shown considerable success in using LLMs to assist taint analysis by identifying source and sink functions [24, 33] and to map handlers to call sites in embedded systems [63]. LLMs have also been used to improve the precision of vulnerability detection. Several approaches have been proposed to reduce false positives through sophisticated analysis techniques. Some methods use LLMs to track execution constraints of variables [20, 30, 31], while others adopts LLMs to analyze broader code context [55, 58]. IX. C ONCLUDING D ISCUSSION While LLMs bring powerful pattern recognition and generative capabilities, they are not a silver bullet: they can produce hallucinations and are constrained by context length. Relying solely on LLMs for program analysis can lead to unreliable

R EFERENCES [1] andriyor. CLI tool for sorting dependents repo by stars. https://github.com/andriyor/ghtopdep, 2025. [2] Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. QL: Object-Oriented Queries on Relational Data. LIPIcs, Volume 56, ECOOP 2016, 56:2:1– 2:25, 2016. URL: https://drops.dagstuhl.de/entities/doc ument/10.4230/LIPIcs.ECOOP.2016.2. [3] Nathaniel Ayewah and William Pugh. The Google FindBugs fixit. In Proceedings of the 19th international symposium on Software testing and analysis, pages 241– 252, 2010. [4] Nathaniel Ayewah, William Pugh, David Hovemeyer, J David Morgenthaler, and John Penix. Using static analysis to find bugs. IEEE software, 25(5):22–29, 2008. [5] Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. Analyzing the state of static analysis: A large-scale evaluation in open source software. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), volume 1, pages 470–481. IEEE, 2016. [6] Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering, 48(9):3280–3296, 2021. [7] Wei Chen, Bowen Zhang, Chengpeng Wang, Wensheng Tang, and Charles Zhang. Seal: Towards diverse specification inference for linux interfaces from security patches. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1246–1262, 2025.

[8] Xiao Cheng, Haoyu Wang, Jiayi Hua, Guoai Xu, and Yulei Sui. Deepwukong: Statically detecting software vulnerabilities using deep graph neural network. ACM Transactions on Software Engineering and Methodology (TOSEM), 30(3):1–33, 2021. [9] Jonathan Cook, Tim Rocktaschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. Ticking all the boxes: Generated checklists improve llm evaluation and generation. arXiv:2410.03608, 2024. [10] CVE Program. CVE-2025-49126 — Visionatrix Reflected XSS. https://www.cve.org/CVERecord?id=CVE-2025-4 9126, 2025. [11] CVE Program. CVE-2025-6776. https://www.cve.org/ CVERecord?id=CVE-2025-6776, 2025. [12] Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W O’Hearn. Scaling static analyses at facebook. Communications of the ACM, 62(8):62–70, 2019. [13] Jana Dunfield and Neel Krishnaswami. Bidirectional typing. ACM Computing Surveys (CSUR), 54(5):1–38, 2021. [14] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv:2002.08155, 2020. [15] Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 608–620, 2022. [16] GeeksforGeeks. Django: Redirect to previous page after login. URL: https://www.geeksforgeeks.org/python/djang o-redirect-to-previous-page-after-login/. [17] GitHub. CodeQL, 2025. URL: https://codeql.github.com/. [18] Google. Pytype: A static type analyzer for Python code, 2024. URL: https://github.com/google/pytype/releases/ta g/2024.10.11. [19] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence, January 2024. URL: http://arxiv.org/abs/2401.14196. [20] Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. Repoaudit: An autonomous llm-agent for repository-level code auditing. In Proceedings of the 42nd International Conference on Machine Learning, 2025. [21] Zhaoqiang Guo, Tingting Tan, Shiran Liu, Xutong Liu, Wei Lai, Yibiao Yang, Yanhui Li, Lin Chen, Wei Dong, and Yuming Zhou. Mitigating false positive static analysis warnings: Progress, challenges, and opportunities. IEEE Transactions on Software Engineering, 49(12):5154–5188, 2023. [22] Sajal Halder, Muhammad Ejaz Ahmed, and Seyit Camtepe. Funcvul: An effective function level vulnerability detection model using llm and code chunk. In European Symposium on Research in Computer Security, pages

166–185. Springer, 2025. [23] Jingxuan He and Martin Vechev. Large language models for code: Security hardening and adversarial testing. In ACM CCS, 2023. URL: https://arxiv.org/abs/2302.05319. [24] Yuchen Ji, Ting Dai, Zhichao Zhou, Yutian Tang, and Jingzhu He. Artemis: Toward accurate detection of server-side request forgeries through llm-assisted interprocedural path-sensitive taint analysis. Proceedings of the ACM on Programming Languages, 9(OOPSLA1):1349– 1377, 2025. [25] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38, 2023. [26] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swebench: Can language models resolve real-world github issues? arXiv:2310.06770, 2023. [27] Avishree Khare, Saikat Dutta, Ziyang Li, Alaia SolkoBreslin, Rajeev Alur, and Mayur Naik. Understanding the effectiveness of large language models in detecting security vulnerabilities. In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 103–114. IEEE, 2025. [28] Davy Landman, Alexander Serebrenik, and Jurgen J. Vinju. Challenges for static analysis of java reflection literature review and empirical study. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 507–518, 2017. [29] Seongmin Lee, Shin Hong, Jungbae Yi, Taeksu Kim, ChulJoo Kim, and Shin Yoo. Classifying false positive static checker alarms in continuous integration using convolutional neural networks. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pages 391–401. IEEE, 2019. [30] Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, and Issa Khalil. LLMxCPG:Context-Aware vulnerability detection through code property Graph-Guided large language models. In 34th USENIX Security Symposium (USENIX Security 25), pages 489–507, 2025. [31] Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. Enhancing static analysis for practical bug detection: An llm-integrated approach. Proceedings of the ACM on Programming Languages, 8(OOPSLA1):474–499, 2024. [32] Penghui Li, Songchen Yao, Josef Sarfati Korich, Changhua Luo, Jianjia Yu, Yinzhi Cao, and Junfeng Yang. Automated static vulnerability detection via a holistic neuro-symbolic approach. arXiv:2504.16057, 2025. [33] Ziyang Li, Saikat Dutta, and Mayur Naik. IRIS: LLMAssisted Static Analysis for Detecting Security Vulnerabilities. The Thirteenth International Conference on Learning Representations (ICLR), 2025. [34] Miaoqian Lin, Kai Chen, Yi Yang, and Jinghua Liu. Uncovering the iceberg from the tip: Generating api specifications for bug detection via specification propagation analysis. In NDSS, 2025.

[35] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv:2307.03172, 2023. [36] Kangjie Lu, Aditya Pakki, and Qiushi Wu. Detecting missing-check bugs via semantic-and context-aware criticalness and constraints inferences. In 28th USENIX Security Symposium (USENIX Security 19), pages 1769– 1786, 2019. [37] Microsoft. RD-Agent, 2025. URL: https://github.com/m icrosoft/RD-Agent. [38] Changwoo Min, Sanidhya Kashyap, Byoungyoung Lee, Chengyu Song, and Taesoo Kim. Cross-checking semantic correctness: The case of finding file system bugs. In Proceedings of the 25th Symposium on Operating Systems Principles, pages 361–377, 2015. [39] Mitre. CWE - Common Weakness Enumeration, 2025. URL: https://cwe.mitre.org/index.html. [40] Ollama. Ollama: Run open large language models locally. https://github.com/ollama/ollama, 2025. [41] Benjamin C. Pierce. Types and Programming Languages. The MIT Press, 1st edition, 2002. [42] CVE Program. Common vulnerabilities and exposures (cve), 2025. URL: https://www.cve.org/. [43] ProtectAI. VulnHuntr: Zero shot vulnerability discovery using LLMs, 2024. URL: https://github.com/protectai/v ulnhuntr/tree/main. [44] PyCQA. Bandit. https://github.com/PyCQA/bandit, 2025. [45] H. G. Rice. Classes of Recursively Enumerable Sets and Their Decision Problems. Transactions of the American Mathematical Society, 74(2):358–366, 1953. URL: https: //www.jstor.org/stable/1990888. [46] Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. Code comment inconsistency detection and rectification using a large language model. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, pages 1832–1843, 2025. [47] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv:2308.12950, 2023. [48] Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. Lessons from building static analysis tools at google. Communications of the ACM, 61(4):58–66, 2018. [49] Semgrep. Semgrep: Lightweight static analysis for many languages. https://github.com/semgrep/semgrep, 2025. [50] Francesco Sovrano, Adam Bauer, and Alberto Bacchelli. Large language models for in-file vulnerability localization can be “lost in the end”. Proceedings of the ACM on Software Engineering, 2(FSE):891–913, 2025. [51] Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T Barr, and Wei Le. To err

is machine: Vulnerability detection challenges llm reasoning. arXiv:2403.17218, 2024. [52] Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In 2024 IEEE symposium on security and privacy (SP), pages 862–880. IEEE, 2024. [53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [54] Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. Reposvul: A repositorylevel high-quality vulnerability dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, pages 472–483, 2024. [55] Cheng Wen, Yuandao Cai, Bin Zhang, Jie Su, Zhiwu Xu, Dugang Liu, Shengchao Qin, Zhong Ming, and Tian Cong. Automatically inspecting thousands of static bug warnings with large language model: How far are we? ACM Transactions on Knowledge Discovery from Data, 18(7):1–34, 2024. [56] Xin-Cheng Wen. Collaboration to repository-level vulnerability detection. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1926–1928, 2024. [57] Yulun Wu, Ming Wen, Zeliang Yu, Xiaochen Guo, and Hai Jin. Effective vulnerable function identification based on cve description empowered by large language models. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 393–405, 2024. [58] Yifan Xia, Zichen Xie, Peiyu Liu, Kangjie Lu, Yan Liu, Wenhai Wang, and Shouling Ji. Beyond static pattern matching? rethinking automatic cryptographic api misuse detection in the era of llms. Proceedings of the ACM on Software Engineering, 2(ISSTA):113–136, 2025. [59] Meng Xu, Chenxiong Qian, Kangjie Lu, Michael Backes, and Taesoo Kim. Precise and scalable detection of doublefetch bugs in os kernels. In 2018 IEEE Symposium on Security and Privacy (SP), pages 661–678. IEEE, 2018. [60] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv:2505.09388, 2025. [61] Haofei Yu, Keyang Xuan, Fenghai Li, Kunlun Zhu, Zijie Lei, Jiaxun Zhang, Ziheng Qi, and Jiaxuan You. Tinyscientist: A lightweight framework for building research agents. https://github.com/ulab-uiuc/tiny-scientist, 2025. [62] Hang Zhang, Jangha Kim, Chuhong Yuan, Zhiyun Qian, and Taesoo Kim. Statically discover cross-entry use-afterfree vulnerabilities in the linux kernel. In Network and Distributed System Security (NDSS) Symposium, 2025. [63] Jiaxu Zhao, Yuekang Li, Yanyan Zou, Zhaohui Liang,

Yang Xiao, Yeting Li, Bingwei Peng, Nanyu Zhong, Xinyi Wang, Wei Wang, et al. Leveraging semantic relations in code and data to enhance taint analysis of embedded systems. In 33rd USENIX Security Symposium (USENIX Security 24), pages 7067–7084, 2024. [64] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv:2303.18223, 1(2), 2023. [65] Qingyang Zhou, Qiushi Wu, Dinghao Liu, Shouling Ji, and Kangjie Lu. Non-distinguishable inconsistencies as a deterministic oracle for detecting security bugs. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3253– 3267, 2022. [66] Xin Zhou, Duc-Manh Tran, Thanh Le-Cong, Ting Zhang, Ivana Clairine Irsan, Joshua Sumarlin, Bach Le, and David Lo. Comparison of static application security testing tools and large language models for repo-level vulnerability detection. arXiv:2407.16235, 2024. A PPENDIX In this section, we provide a formal description of the S E T YPE system. To start with, we define the S E T YPE of a variable or an expression e as τ in the typing environment Γ, denoted as Γ ⊢ e : τ . Here, we define the following rules: When a variable x is first encountered, we initialize its S E T YPE τx based on its non-executable context (e.g., variable name and comments) using a function inf (nx ), where nx is the non-executable context of x. x∈ / Dom(Γ) τx = inf (nx ) Γ ⊢ x : τx After initialization, the S E T YPE of x is retrieved from the typing environment Γ: x ∈ Dom(Γ) Γ ⊢ x : τx For an operation o with operands e1 , . . . , en , if each operand ei has the S E T YPE τei , then the operation has the S E T YPE τo : Γ ⊢ e1 : τe1 . . . Γ ⊢ en : τen τo = op(o, τe1 , . . . , τen ) Γ ⊢ o(e1 , . . . , en ) : τo When analyzing an assignment statement of the form x ← e, where x is a variable and e is an expression. We notate the variable x before the assignment as x̂. We first infer the S E T YPE τ of the expression e under the current typing environment Γ, i.e., Γ ⊢ e : τe . The assignment updates the S E T YPE of x in Γ by unifying its existing S E T YPE τx with the inferred S E T YPE τ : Γ ⊢ e : τe Γ(x̂) = τx Γ(x) = unify(τx , τe ) Γ⊢x←e Notably, in the above rules, the function inf , op, and unify are not formally defined. Indeed, these functions are implemented using LLMs in P Y S E T YPE, as described in Section IV. Type checking is performed within the op and unify functions to ensure that the operations are semantically valid. • CWE-22: Path Traversal

CWE-78: OS Command Injection CWE-79 (CWE-80): Cross-Site Scripting (XSS) • CWE-89: SQL Injection • CWE-200: Exposure of Sensitive Information to an Unauthorized Actor • CWE-312: Cleartext Storage of Sensitive Information • CWE-601: Open Redirect • CWE-617: Reachable Assertion • CWE-200: Information Exposure • CWE-807: Reliance on Untrusted Inputs in a Security Decision • CWE-918: Server-Side Request Forgery (SSRF) • •

Construction of Recent CVE Dataset. We following: 1 entry that has no CWE assigned, 4 are due to the usage of vulnerable third-party libraries, 5 entries that have the vulnerability level set to low, 5 entries that are not open-sourced in GitHub, 2 entries that focus on denial-of-service vulnerabilities, and three servers are designed for local usage only. Out of the remaining 23 entries, we take entries with their CWE number in the CWE types covered in the SVEN dataset or appears at least twice in the dataset. We eventually have 9 entries in the dataset, acrossing five CWE types, and 5 of them have been patched by the developers. Constrution of Popular Projects Dataset. We construct a dataset of popular open-source projects that use FastAPI, Django, or Flask frameworks. As of July 31th, 2025, we collected 37535 FastAPI projects, 30385 Django projects and 14141 Flask projects. Out of these repositories, 4734, 2440, and 2262 projects are public and has least one star. Among these projects, we select the top-100 starred projects for each framework. Among these projects, we manually filter out the following: • 144 projects that do not indeed use the framework in the code. This is because some projects only list the framework in the dependency file but do not actually use it in the code, or an older version of the project uses the framework but the latest version does not; • 19 projects that are framework-plugins or extensions; • 22 projects that are tutorials or the collection of code snippets; • 9 projects that uses a framework that is built on top of the target framework (e.g., 8 fastmcp), and • 3 projects that are duplicated repositories. In certain cases, the developer’s patch may not completely fix the vulnerability, leading to repeated detections by P Y S E T YPE. For example, in CVE-2025-6776 [11], multiple path traversal vulnerabilities are found where the user input is used to construct file paths without proper sanitization. The developer’s patch is to restrict the access to the endpoints to logged-in users. However, the underlying issue of unsanitized user input is not addressed in the patch. As a result, both CodeQL and P Y S E T YPE still report the vulnerabilities in both versions. 8 https://github.com/jlowin/fastmcp

1 2

class ResendEmailSerializer(serializers.Serializer): email = serializers.EmailField(label=("Email"))

3 4 5 6 7

def validate(self, attrs): user = get_object_or_404(User, email=attrs.get("email")) attrs["user"] = user return attrs

8 9 10

class ResendEmailView(GenericAPIView): serializer_class = ResendEmailSerializer

11 12 13 14 15 16 17

def post(self, request, *args, **kwargs): serializer = self.serializer_class(data=request.data) serializer.is_valid(raise_exception=True) user = serializer.validated_data["user"] if user.is_verified: return Response({"detail": "..."})

Here, we present a code example that illustrates the uncaught interactions during P Y S E T YPE’s analysis. The example is taken from the 9 MiniKala project in the explored popular projects dataset. In this example, the ResendEmailView class uses a serializer ResendEmailSerializer for validating the email address provided by the user. However, in the post method of the ResendEmailView class (line 13), the called is valid (line 15) and validated data (line 16) are not defined in the ResendEmailSerializer class. Instead, these methods are inherited from the parent class django.serializers.Serializer. Similarly, the validate method (line 4) in the ResendEmailSerializer class is also not called. However, this function will be invoked during the execution of the is valid method in the parent class to validate the email field. During P Y S E T YPE’s analysis, it does not explore this inheritance relationship and thus misses this interaction. As a result, P Y S E T YPE cannot infer the types of the variables email and validated data[’email’] accurately, leading to incomplete analysis results. While our work focuses on detection of vulnerabilities in the three frameworks, we observe that certain types of vulnerabilities are more prevalent in one framework compared to others. Open Redirect in Django. In Django, open redirect vulnerabilities are particularly common. This is primarily due to the framework’s use of the “next” parameter in its authentication views, which allows redirection to a specified URL after login. One potential reason for the prevalence of open redirect vulnerabilities in Django applications is the common mentioning of using the “next” parameter in various tutorials and documentation [16]. This widespread practice may lead developers to inadvertently introduce open redirect vulnerabilities if they do not properly validate the “next” parameter before using it for redirection. Alternatively, a similar design pattern is not commonly used in Flask or FastAPI, which may explain the lower incidence of open redirect vulnerabilities in applications built with these frameworks. File and Data Management. In Flask and FastAPI, we observe a higher occurrence of file and database management vulnerabilities, such as arbitrary file read and SQL injection. This could be attributed to the fact that both frameworks provide 9 https://github.com/MaryamKalantarii/MiniKala

more flexibility and commonly allows the user to choose their own libraries for handling file uploads and database interactions. This flexibility, while beneficial for customization, may lead to inconsistent security practices among developers, increasing the likelihood of vulnerabilities in these areas. Alternatively, Django’s built-in Object-Relational Mapper (ORM) avoids the need for raw SQL queries or filesystem interactions, which may contribute to a lower incidence of such vulnerabilities in Django applications.

Related documents

• Finding Vulnerabilities via LLM-Augmented Semantics-Aware Type-Checking#609689
• LLM-based Vulnerability Detection at Project Scale: An Empirical Study#1104949
• Towards an Automated Test of LLM Security Knowledge#386756
• HSMLog: Small Language Model-Assisted Hardware Security Module Log Anomaly Detection with Behavioral Analysis#798402
• HSMLog: Small Language Model-Assisted Hardware Security Module Log Anomaly Detection with Behavioral Analysis#796989