arXiv:2606.14881v1 [cs.DC] 12 Jun 2026
Evaluating Gemma4 Models as AI Teaching Assistants for Introductory Parallel Programming: A DataRaceBench Study Sabbir Hussain Meraj
Riham Chowdhury
The University of Texas at San Antonio San Antonio, USA [email protected]
The University of Texas at San Antonio San Antonio, USA [email protected]
Shimul Debnath
Wei Wang
The University of Texas at San Antonio San Antonio, USA [email protected]
The University of Texas at San Antonio San Antonio, USA [email protected]
Abstract Debugging data races is a major challenge for students learning parallel programming due to the non-deterministic nature of concurrent execution and the complexity of shared-memory semantics. Recent advances in Large Language Models (LLMs) suggest that they could serve as AI teaching assistants, but the capabilities of lower-cost open-weight models for parallel debugging remain unclear. In this paper, we evaluate two Gemma4 open-weight models, Gemma4-E4B and Gemma4-31B, on their ability to identify, explain, and repair data races in OpenMP programs from the DataRaceBench benchmark suite. We also investigate whether contextual hints, including ThreadSanitizer (TSan) reports and model-generated explanations, improve repair quality. Our results show that Gemma4-E4B correctly explained 82 of 104 race-condition programs and successfully repaired 73, while Gemma4-31B achieved 100 correct explanations and 98 successful repairs. Surprisingly, additional context did not consistently improve repair effectiveness and sometimes reduced performance. These findings suggest that open-weight LLMs can provide valuable support for student self-debugging, with larger models offering near-complete coverage of the benchmark suite.
CCS Concepts • Computing methodologies → Parallel computing methodologies; • Applied computing → Computer-assisted instruction.
Keywords Parallel Programming, Educational Tools, Large Language Model
1
Introduction
Undergraduate students learning parallel programming frequently struggle with the complexities of debugging multi-threaded code [9, 30]. The non-deterministic nature of data races, combined with the subtle semantics of shared memory models, creates a steep learning curve for novices. Supporting students during this process is equally challenging for instructors, who often have limited time to provide the personalized, line-by-line debugging assistance required in large introductory courses. Large Language Models (LLMs) have recently demonstrated significant capabilities in code analysis and debugging, suggesting they
could serve as effective AI teaching assistants [27–29]. By providing immediate feedback and explanations, LLMs could potentially empower students to debug their parallel programs independently. However, high-performance commercial LLM models are often expensive to students and universities. It is therefore worthwhile to explore whether open-weight LLM models, which can be deployed locally and at a lower cost, are sufficiently capable of supporting parallel debugging tasks. In this paper, we evaluate the performance of two Gemma4 openweight models [26] in identifying, explaining, and fixing data races. Our investigation is guided by three primary research questions: • RQ1: Identification and Explanation. To what extent can the Gemma4 models identify and explain the underlying causes of data races in OpenMP programs? • RQ2: Fix Generation Capability. Can the Gemma4 models generate correct and parallel-preserving fixes for data races across various OpenMP constructs? • RQ3: Impact of Contextual Hints. Does the inclusion of runtime tool output (ThreadSanitizer) and model-generated explanations improve the quality and correctness of the generated fixes? We conduct our evaluation using DataRaceBench, a benchmark suite of C/C++ programs with known data races [7, 19]. Our experimental setup involves a multi-step pipeline where models generate explanations and fixes based on various levels of context, ranging from raw code to code augmented with TSan reports [24]. We evaluate two scales of the Gemma model [26]: the small-scale Gemma4E4B and the larger Gemma4-31B, running on NVIDIA H200 GPUs. The correctness of the generated fixes is verified through a combination of runtime analysis with TSan and a multi-model audit involving Gemini 3.1 [13], Claude Opus 4.7 [1], and GPT 5.5 [22], as well as human manual evaluation. Our results show that Gemma4-E4B correctly explained 82 out of 104 race-condition programs and successfully repaired 73 programs. These findings suggest that the model has the potential to assist students in understanding and debugging race conditions. We also find that the larger Gemma4-31B model offers significantly better performance (with 100 explained and 98 fixed) and should be used when resources allow. Surprisingly, we observe that adding more context, such as TSan outputs, does not necessarily improve fix correctness and can sometimes lead to contextual distraction, diverging from prior research results [8].
Meraj et al.
Listing 1: Explanation prompt.
Listing 2: Basic fix prompt.
### Task : You are an expert in parallel programming and data race detection . Analyze the following code and explain ALL data races present . Instructions : 1. Start directly with " Data Race 1:" - no preamble 2. No introductory text like " Here is the analysis " or "I found " 3. No concluding remarks or summaries 4. Only the data race analysis
### Task : You are a code analysis assistant , expert in parallel programming . Correct the following code to remove the data race . Instructions : - Your response must be only contain the corrected code . - The corrected code must be in the same language as the original code . - The corrected code must be free of data races . - The corrected code must be free of undefined behavior . - Do NOT remove OpenMP pragmas or parallel regions . - Do NOT provide any explanations , reasoning , or extra text .
Format : Data Race 1: - Variable : [ name ] - Lines : [ numbers ] - Type : [ read - write / write - write ] - Reason : [ explanation ]
Code : { code }
Code : { code }
Listing 3: Fix with explanation prompt.
Analysis :
The primary contributions of this work are: (1) A thorough evaluation of open-weight models on their ability to explain complex OpenMP data race conditions. (2) A comprehensive analysis of race condition remediation using open-weight models across different levels of technical context and hints.
### Task : You are an expert in parallel programming . Fix the data races in the code based on the analysis provided . Instructions : 1. Output ONLY the corrected code - no explanations 2. Do NOT remove OpenMP pragmas or parallel regions 3. Add proper synchronization based on the data race analysis 4. Preserve all original functionality and parallelism 5. Start directly with # include or the first line of code 6. No markdown , no comments , no extra text Code : { code } Data Race Analysis : { explanation }
2
Evaluation Setup
Corrected Code :
In this section, we describe the environment, tools, and the multistep methodology used to evaluate Gemma’s capability in diagnosing and fixing data races. Listing 4: Fix with TSan prompt.
2.1
Environment
All experiments were conducted using llama.cpp with Gemma4 models. We evaluated two model scales: Gemma4-E4B (quantized to 8-bit) and Gemma4-31B (quantized to 16-bit). The inference was performed on a server equipped with NVIDIA H200 GPUs. We followed Google’s default parameters, i.e., temperature is 1.0, top_p is 0.95, and top_k is 64. Context size is set to 262144 for both models, although Gemma4-E4B can effectively use only 128K context.
2.2
Methodology
Our evaluation follows a five-step pipeline designed to mimic the workflow of a student using an AI assistant to debug parallel code. Step 1: Explanation Generation. We provided the source code of all 104 C/C++ programs with race conditions from DataRaceBench to the model (with all comments removed to prevent leaking the ground truth). The model is tasked with identifying and explaining all data races. The prompt used for this step is shown in Listing 1. Step 2: TSan Report Generation. To provide the model with technical runtime feedback, we execute the original DataRaceBench programs using ThreadSanitizer (TSan). We utilize libarcher [2] to improve TSan’s support for OpenMP and reduce false positives. The resulting TSan reports are captured for use in later steps. Step 3: Basic/Direct Fix Generation. As the basic code fix attempt, we provide the comment-free source code to the model and request a direct fix. The prompt (Listing 2) explicitly instructs the model to correct race condition while preserving parallelism.
### Task : You are an expert in parallel programming and thread safety . Fix the data races in the code using the ThreadSanitizer ( TSan ) report provided . Instructions : 1. Output ONLY the corrected code - no explanations 2. Do NOT remove or weaken any OpenMP pragmas or parallel regions 3. Fix ONLY what TSan reported - do not over - synchronize 4. Apply the minimal fix in this order of preference : - OpenMP : ` reduction `, `atomic `, `# pragma omp critical ` - C ++ std : `std :: atomic ` for simple variables , `std :: mutex ` for complex state 5. Preserve all original functionality and output behavior 6. Start directly with # include or the first line of code 7. No markdown , no comments , no extra text Code : { code } TSan Report : { tsan_output } Corrected Code :
Step 4: Context-Aware Fix Generation. To evaluate the impact of external hints, we provide the model with the same code along with its own explanation (from Step 1) and/or the TSan report (from Step 2). These "Context-Aware" prompts are shown in Listing 3, Listing 4, and Listing 5. Step 5: Verification. The correctness of the generated fixes is verified through a rigorous process. First, the fixed code is executed under TSan to check for any remaining or newly introduced data
Evaluating Gemma4 Models as AI Teaching Assistants for Introductory Parallel Programming: A DataRaceBench Study
Listing 5: Fix with race condition explanation and TSan prompt. ### Task : You are an expert in parallel programming and thread safety . Fix the data races in the code using the ThreadSanitizer ( TSan ) report provided . Instructions : 1. Output ONLY the corrected code - no explanations 2. Do NOT remove or weaken any OpenMP pragmas or parallel regions 3. Fix ONLY what TSan reported - do not over - synchronize 4. Apply the minimal fix in this order of preference : - OpenMP : ` reduction `, `atomic `, `# pragma omp critical ` - C ++ std : `std :: atomic ` for simple variables , `std :: mutex ` for complex state 5. Preserve all original functionality and output behavior 6. Ensure you do not force the program to be sequential if it was originally parallel , unless it is necessary to fix the race condition . 7. Start directly with # include or the first line of code 8. No markdown , no comments , no extra text , no code fence . Code : { code } Data Race Analysis : { explanation } TSan Report : { tsan_output } Corrected Code :
races. Second, the fixes are audited by three frontier models: Gemini 3.1, Claude Opus 4.7, and GPT 5.5. In cases where the models disagree, a manual review is conducted.
3
Evaluation and Results
This section presents the evaluation results of how open-weight Gemma4 models identify and repair data races in DataRaceBench. As stated previously, our evaluation is guided by the following research questions: • RQ1: Identification and Explanation. To what extent can the Gemma4 models identify and explain the underlying causes of data races in OpenMP programs? • RQ2: Fix Generation Capability. Can the Gemma4 models generate correct and parallel-preserving fixes for data races across various OpenMP constructs? • RQ3: Impact of Contextual Hints. Does the inclusion of runtime tool output (ThreadSanitizer) and model-generated explanations improve the quality and correctness of the generated fixes? Table 1: RQ1: Race condition explanation accuracy across two runs for Gemma4-E4B. Run
EXPLAINED
NOT_EXPLAINED
Success Rate
Run 1 Run 2
80 82
24 22
76.9% 78.8%
3.1
Gemma4-E4B Results
3.1.1 RQ1: Explanation Quality. The first phase of our evaluation focuses on the model’s ability to explain the data races present in
Table 2: RQ2 & RQ3: Audited fix outcomes across prompting strategies. Prompt Type
Run
Fixed
Not Fixed
Basic (Direct)
1 2
72 73
32 31
Explanation
1 2
57 66
47 38
TSan
1 2
56 54
48 50
Explanation + TSan
1 2
54 65
50 39
the DataRaceBench suite. An accurate explanation can significantly benefit students. Overall Performance and Consistency: As shown in Table 1, Gemma4-E4B demonstrated a robust ability to identify and explain data races across two independent runs. In Run 1, the model correctly explained 76.9% (80/104) of benchmarks, with 24 not correctly explained. Run 2 improved to 78.8% (82/104), leaving 22 benchmarks not explained. This 1.9% gap indicates moderate run-to-run stochasticity, but the overall baseline remains strong enough to suggest that the model captures many common race patterns. Note that although the second run generated more correct explanations overall, it failed on several benchmarks that were correctly explained in the first run, including DRB004. This suggests that Gemma4-E4B’s explanations can vary across runs; therefore, running the model multiple times may improve the chances of obtaining a correct explanation. Patterns of Successful Explanation: Gemma4-E4B consistently identified races involving fundamental OpenMP directives and data-sharing attributes. Across both runs, the model successfully explained many cases built around basic worksharing and synchronization constructs, including parallel, for, sections, critical, ordered, and common data-sharing clauses such as private, shared, default, and reduction. This suggests that the model can usually handle the foundational directives and clauses that dominate introductory parallel programming exercises. Failure Cases: Common failure patterns observed across both runs include: • Misidentifying the Actual Raced Object: The model often recognized that something was wrong near the true bug site, but still named the wrong variable or the wrong kind of defect. For example, in DRB014 (Listing 6), the nested loop causes an out-of-bounds array access that also induces a loop-carried cross-thread dependence. When j=0, b[i][j-1] evaluates to b[i][-1], which, due to linearized row-major storage in C, actually resolves to b[i-1][m-1] (the last element of the previous row). Since the outer loop i is parallelized, thread 𝑇1 processing row 𝑖 reads the memory location that thread 𝑇2 processing row 𝑖 − 1 is writing to, creating a true data race on b. The Gemma4E4B explanations did not capture this cleanly: Run 1 focuses on the out-of-bounds access and treats the case primarily
Meraj et al.
Listing 6: Out-of-bounds access causing a data race in DRB014.
Listing 9: Memory visibility issue due to incompatible synchronization constructs in DRB142.
# pragma omp parallel for private ( j ) for ( i =1; i < n ; i ++) for ( j =0; j < m ; j ++) b [ i ][ j ] = b [ i ][ j -1];
if ( thrd == 0) { # pragma omp critical { x = 10; } # pragma omp atomic write y = 1; } else { int tmp = 0; while ( tmp == 0) { # pragma omp atomic read acquire tmp = y ; } # pragma omp critical { if ( x !=10) printf ( " x ␣ = ␣ % d \ n " , x ); } }
Listing 7: Missing private clause for temporary variable in DRB020. int tmp ; // ... # pragma omp parallel for for ( i =0; i < len ; i ++) { tmp = a [ i ] + i ; a [ i ] = tmp ; }
Listing 8: True dependency incorrectly parallelized in DRB037. for ( i =0; i < n ; i ++) # pragma omp parallel for for ( j =1; j < m ; j ++) b [ i ][ j ] = b [ i ][ j -1];
as unsafe memory usage, while Run 2 instead hallucinates a race on loop index i. Across both runs, the consistent problem is not simply “seeing” the out-of-bounds symptom, but failing to name the documented raced object. • Index Confusion: A recurring failure mode involved the model incorrectly identifying the loop variable as the source of the race, while ignoring the actual shared variable that required privatization. For example, in DRB020 (Listing 7), variables declared outside the parallel region (like tmp) are shared by default. Because multiple threads execute the loop iterations concurrently, they simultaneously read from and write to the same shared memory location tmp, causing a data race. In both runs, Gemma4-E4B instead identifies the loop index i as the race source, despite i being implicitly privatized by the for directive. Similarly, for several benchmarks with complex race conditions, Gemma4-E4B incorrectly attributed the race condition to loop index variables. • Complex Dependencies: Higher-dimensional dependencies and subtle memory order remained challenging. The model often defaulted to a "no data race" conclusion by making incorrect assumptions about the parallelized region. For example, in DRB037 (Listing 8), the OpenMP directive parallelizes the inner j loop, which contains a loop-carried true dependency (b[i][j] depends on b[i][j-1]). In both runs, Gemma4-E4B incorrectly stated that there was no data race because it treated the outer loop i as the parallelized dimension. Note that the data race in this example is different than Listing 6, although both have complex dependencies.
Cases involving weak memory models also led to incorrect reasoning. For instance, in DRB142 (Listing 9), the model struggled to trace memory visibility requirements across thread boundaries. In this example, thread 0 updates x inside a critical section and then signals thread 1 by writing to y. Thread 1 spins on an atomic read acquire until y is 1, then enters its own critical section to read x. The documented race is therefore about the visibility of x, not about y. That is, in current x86 CPUs, updates to x may be visible after updates to y, causing a race condition. The Gemma4-E4B explanations again diverge from the true race condition: Run 1 incorrectly states that no race is present because all accesses appear synchronized, while Run 2 invents a race on y due to uninitialized-state reasoning. • Advanced OpenMP Directives: Gemma4-E4B struggled with the subtleties of OpenMP task synchronization rules. In DRB117 (Listing 10), the documented race here is on psum[1]: taskwait only guarantees the completion of direct child tasks, not descendant tasks, so the inner task computing psum[1] is not guaranteed to finish before sum is computed. However, the saved explanations from both runs do not identify that task-hierarchy issue. Instead, both runs claim there is a race between the loop that initializes a and the later task reads of a. This example therefore shows a broader failure to identify the correct synchronization boundary and raced variable, rather than a narrowly targeted misunderstanding of taskwait semantics. 3.1.2 RQ2: Correction Capability. The second phase of our evaluation focuses on whether Gemma4-E4B can move beyond diagnosis and produce a correct fix. For it to serve as a useful teaching assistant, it must demonstrate remediation strategies that remove races. Table 2 summarizes the fix outcomes across prompting strategies. Basic Fix Performance: When using the Basic (Direct) prompt, Gemma4-E4B achieved the strongest verified fix rate in both runs: 69.2% (72/104) in Run 1 and 70.2% (73/104) in Run 2. This high baseline indicates that the model already has a substantial working knowledge of common OpenMP repairs, such as adding private clauses, inserting critical sections, or restructuring simple loops.
Evaluating Gemma4 Models as AI Teaching Assistants for Introductory Parallel Programming: A DataRaceBench Study
Listing 10: Task synchronization failure in DRB117. # pragma omp single { # pragma omp task { # pragma omp task { psum [1] = a [2] + a [3]; } psum [0] = a [0] + a [1]; } # pragma omp taskwait sum = psum [1] + psum [0]; }
Listing 12: Compilation-failing fix in DRB009. # pragma omp parallel for private ( i ) for ( i =0; i < len ; i ++) { # pragma omp atomic x=i; }
Listing 13: Fix with race condition in DRB025. # pragma omp simd for ( i =0; i < len -1; i ++) a [ i +1] = a [ i ] * b [ i ];
Listing 11: Serialized fix in DRB010. # pragma omp parallel for private ( i ) for ( i =0; i < len ; i ++) { # pragma omp critical x = i; }
Again, although the two runs of Gemma4-E4B fixed a similar number of race conditions, the sets of benchmarks that were successfully fixed differed between the runs. Therefore, users of Gemma4-E4B may benefit from executing the model multiple times to increase the likelihood of obtaining a correct fix. Serialized Repairs in Basic Fixes: Some fixes remove unsafe access pattern but leaves the program sequential. Representative cases include edits that appear safe locally yet fail to preserve the intended OpenMP semantics. For example, in DRB010 (Listing 11), the generated repair replaces the original shared assignment with a critical section. This transformation serializes writes to x, but it does not implement the benchmark’s intended lastprivate-style behavior of preserving the final loop iteration’s value. Note that some benchmarks in DataRaceBench are synthetic, for which serialization may be the only viable solution. Therefore, we consider serialized fixes as successful fixes in our analysis. Failed Cases in Basic Fix: Gemma4-E4B’s unsuccessful fixes fall into the following cases: • Compilation Errors: Some generated repairs are invalid at the language or OpenMP syntax level. Most of the compilations errors were due to misuse of OpenMP directives or clauses, such as malformed or ineffective atomic/ordered usage, and structurally invalid directive placement. These failures never produce a usable fix, even before semantic correctness is considered. For example, in DRB009 (Listing 12), the model attempted to protect a variable assignment with omp atomic. This edit looks superficially plausible, but the atomic clause is not valid for pure assignments, hence the generated fix does not compile. • Fixes still have race condition: A second class of errors consists of outputs that compile but still leave the documented race in place. Common examples include critical sections that fail to break loop-carried dependences, incomplete taskwait/depend repairs, or edits that protect the
Listing 14: Semantically changed fix for DRB025. # pragma omp simd for ( i =0; i < len -1; i ++) a [ i +1] = a [ i +1] * b [ i ];
wrong operation while the documented race remains. A representative case is DRB025 (Listing 13), where the generated fix keeps the SIMD loop intact. That is, because the loop still carries a true dependence from a[i] to a[i+1], the transformed program compiles but does not eliminate the underlying race. • Semantically Changed Solutions. Some fixes would change the intended behavior of program. Again, take DRB025 as an example, in a run with Gemma4-E4B, the code is fixed as shown in (Listing 14). The fixed code changed the right hand of the assignment from a[i]*b[i] to a[i+1]*b[i]. Although this fix is race condition free, it is still incorrect as it changed the program’s behavior. 3.1.3 RQ3: Impact of Contextual Hints. Our results do not show a consistent benefit from adding explanations or ThreadSanitizer output. As shown in Table 2, in Run 1, the direct prompt produced the highest fix count (72/104), while the explanation, TSan, and explanation+TSan conditions reached 57/104, 56/104, and 54/104, respectively. In Run 2, the same ordering largely held: the direct prompt again led with 73/104 fixes, explanation reached 66/104, explanation+TSan reached 65/104, and TSan alone dropped to 54/104. Moreover, we observed that providing additional contextual information often increased the number of compilation failures. Adding contextual hints hurt fix quality is different from the observation of prior work [8]. One possible explanation is that the extra context introduced additional noise into the prompt, making it more difficult for the model to generate syntactically correct code modifications. Since our study employs smaller open-weight models instead of the larger commercial models used in prior work, these models may experience a greater degradation in performance as the amount of contextual information increases. Overall, Gemma4-E4B demonstrates promising capability for automated race-condition repair. Nevertheless, its outputs remain imperfect, exhibiting both incorrect fixes and compilation failures.
Meraj et al.
Table 3: Gemma4-31B explanation results. Total
EXPLAINED
NOT_EXPLAINED
104
100
4
Table 4: Gemma4-31B fix results across prompting strategies.
As a result, the generated repairs should be viewed as useful starting points for student learning and code review rather than as guaranteed correct solutions.
3.2
Gemma4-31B Results
To assess whether a larger open-weight model improves performance on the same benchmark suite, we also evaluated Gemma431B on the 104 positive DataRaceBench cases. Tables 3 and 4 summarize the explanation and fix results. 3.2.1 RQ1: Explanation Quality. Gemma4-31B achieved very strong explanation performance, correctly explaining 100 of the 104 positive benchmarks. This leaves only four cases not correctly explained, and those failures are concentrated in a small number of subtle synchronization patterns rather than in basic OpenMP constructs. The four incorrectly explained cases are DRB129, DRB142, DRB173, and DRB175. In DRB129, the model stated that no race was present, missing the benchmark’s mergeable-task race on x. In DRB142, the model again concluded that the code was race-free because it fails to recognize that the weak memory-ordering bug is a race on x, not on the signaling variable y. The remaining two failures, DRB173 and DRB175, are non-sibling task-dependence cases. In both, the model mentioned the correct variable a, but it explained the wrong mechanism: it described a read by printf racing with task writes, whereas the documented bug is a write/write race between nonsibling tasks. Thus, the 31B model’s remaining explanation errors stem mainly from subtle tasking semantics and memory-ordering logic, not from a broad inability to identify shared-state races. 3.2.2 RQ2: Fix Capabilities. Throughout this section, unresolved cases are conservatively treated as not fixed. The direct prompt is the clearest summary of Gemma4-31B’s repair capability. Under this setting, the model correctly fixed 98 of the 104 benchmarks, leaving only 6 not fixed. This result indicates that the larger model can usually synthesize an effective repair without relying on extra explanation or runtime context. The six direct-prompt failures are DRB025, DRB027, DRB037, DRB123, DRB138, and DRB175. Two of these are dependence cases where the generated code does not preserve the benchmark’s intended computation even if it reduces or hides the observed race pattern. In DRB025 (Listing 14), the model changes the recurrence from a[i+1] = a[i] * b[i] to a[i+1] = a[i+1] * b[i], which breaks the original semantics. In DRB138, the unsafe SIMD loop is effectively left unchanged, so the original loop-carried dependence on b[i] and b[i-m] remains. The other four failures are tasking or synchronization cases that the model does not repair correctly. DRB027 failed due to the misuse of atomic clause. DRB037 failed because the transformed code leaves the shared loop variable j racing. In DRB123, the model adds an atomic operation, but the loop variable remains shared and the task-undeferred pattern is not repaired correctly. Finally,
Prompt Type
Total
FIXED
NOT_FIXED
Basic (Direct) Explanation TSan Explanation + TSan
104 104 104 104
98 98 75 71
6 6 29 33
DRB175 remains not fixed because the non-sibling task race is not eliminated: the generated change does not correctly synchronize the competing task updates. Surprisingly, although the race condition DRB142 was not correctly explained, it was correctly fixed by Gemma4-31B. 3.2.3 RQ3: Impact of Additional Context. For Gemma4-31B, adding the model’s own explanation as extra context does not improve over the direct prompt: both settings fix 98 of the 104 benchmarks, although with different sets of fixed benchmarks. By contrast, adding ThreadSanitizer context substantially reduces performance. The fix count drops from 98 to 75 with TSan alone, and further to 71 when explanation and TSan are provided together. In other words, the extra runtime trace does not help the larger model refine its repairs; instead, it appears to push the model toward less reliable transformations, especially on benchmarks with subtle dependence or tasking semantics. Overall, for Gemma4-31B, direct prompting remains the strongest strategy, explanation is effectively neutral, and TSan context is detrimental.
4
Discussion: Implications for Parallel Programming Education
The results of our evaluation of Gemma4-E4B and Gemma4-31B provide several insights into the viability of open-weight models as AI teaching assistants for introductory parallel programming.
4.1
Pedagogical Usage
Gemma4-E4B’s strong performance on foundational OpenMP constructs, such as private clauses and standard parallel for loops, suggests that it can effectively resolve the majority of common errors encountered by undergraduate students. However, the explanations attributing race conditions to wrong variables like Listing 6 (where memory safety errors mask data races) present a pedagogical risk. If a student relies solely on the AI’s explanation, they may fix the surface-level memory violation while remaining unaware of the underlying synchronization logic failure. Instructors must therefore emphasize that AI-generated feedback is a starting point for investigation rather than a definitive proof of correctness. Our analysis of failure modes in complex tasking (e.g., Listing 10) and memory consistency (e.g., Listing 9) also helps define the boundaries of AI assistance. While Gemma4-E4B can filter out approximately 70% of foundational bugs, it consistently struggles with hierarchical synchronization and relaxed memory models. These failure points represent a natural "hand-off" point where the student should be encouraged to seek assistance from an instructor.
Evaluating Gemma4 Models as AI Teaching Assistants for Introductory Parallel Programming: A DataRaceBench Study
Nonetheless, by automating the resolution of common errors, openweight models allow instructors to focus their limited time on these more cognitively demanding parallel programming concepts.
4.2
The Model Contextual Overload
A surprising finding in our study was that adding technical context, such as ThreadSanitizer reports, often degraded the fix quality of the Gemma4 models. We hypothesize that for smaller-scale models, a detailed TSan report acts as "technical noise" that consumes the model’s limited attention and interferes with its instructionfollowing capabilities, leading to brittle fixes rather than holistic reasoning about the program’s concurrency model. This suggests that for small-scale deployment in education, "less is more" regarding the technical detail provided in the initial prompt.
4.3
Data Contamination and Model Viability
A potential threat to the validity of LLM-based evaluations is data contamination, as benchmark suites like DataRaceBench are publicly available and likely have been included in the models’ training corpora. However, we argue that the strong performance of the smaller Gemma4-E4B model remains a significant result. Even if the model’s success is partially attributed to pattern memorization from training, the fact that a 4B parameter model can reliably retrieve and apply these complex parallel synchronization patterns indicates that open-weight models have reached a level of foundational competence suitable for real-world educational assistance.
4.4
Deployment Cost
The ability to run these models on local hardware using openweight frameworks provides significant advantages for education. 4B models can run on consumer GPU cards or laptops with CPU/GPU unified memory. Hence, they can be deployed directly to students’ computers for local inference, without incurring the high costs or privacy concerns associated with commercial, proprietary APIs. Universities can also deploy private "debugging servers" that offer students immediate assistance and further reduce students’ burden. The significant performance leap seen in the 31B model also suggests that institutional investment in mid-to-large scale local hardware can provide students with a nearly frontier-level debugging assistant at a fraction of the long-term operational cost.
5 Related Work 5.1 Data Race Detection Data race detection has been studied extensively by prior studies. Dynamic detectors such as Eraser, FastTrack, and ThreadSanitizer monitor executions to identify conflicting accesses that occur under insufficient synchronization [12, 23, 24]. Other dynamic and hybrid systems target particular concurrency models or deployment settings, including OpenMP-focused detection in ARCHER [2]. Static approaches such as RacerX and RacerD reason about possible races without requiring a specific failing execution [4, 11]. Chen et al. evaluated data race detection on DataRaceBench using commercial and open weight models [6]. HPC-GPT is a fine-tuned LLM model for race condition detection [10]. These detection systems provide
the foundation for race debugging, but they do not explain the bug to a learner or synthesize a correct repair. Benchmarking work complements these detectors by providing standardized programs for evaluation. DataRaceBench [19] and its extensions [7] provide OpenMP programs with documented racy and non-racy variants. Our study is built on top of DataRaceBench.
5.2
Program Repair for Concurrency
Concurrency repair systems attempt to move beyond detection by modifying programs to eliminate concurrency bugs. Early systems focused on specific bug classes, such as automated atomicityviolation repair in AFix that fixed single-variable atomicity violations [15]. CFix [16] generalized this direction to a broader class of concurrency bugs by decomposing each failure-inducing interleaving into mutual-exclusion and ordering constraints, then synthesizing synchronization operations that simultaneously eliminate the bug and avoid introducing deadlocks. Other systems, including ARC, Grail, and PFix, explored genetic-algorithms-based, contextaware, or memory-pattern-based repair strategies for concurrency bugs [18, 20, 21]. For structured parallel programs, Surendran et al. studied test-driven repair of data races by inserting synchronization into parallel programs [25]. These systems demonstrate that concurrency repair requires more than removing a single conflicting access: a repair must preserve ordering, avoid new deadlocks, and maintain program semantics. DR.FIX extends this line of work to industry-scale Go programs by combining program analysis, retrieval, validation, and LLM-generated patches [3]. Our work studies a different setting: OpenMP benchmarks and an educational use case, where the model must provide both a natural-language explanation and a repair suitable for helping students understand parallel programming errors with low deployment cost.
5.3
GenAI-Based Bug Fixing
Recent automated program repair work increasingly uses large language models to synthesize patches. RepairAgent frames repair as an autonomous agent workflow, InferFix combines LLM repair with static-analysis diagnostics, and ChatRepair uses conversational feedback to iteratively refine patches [5, 17, 29]. Other studies evaluate LLMs for general program repair or combine LLMs with code-completion engines [27, 28]. Closer to concurrency, Jin et al. explored GenAI-based data-race fixes for real-world Go programs [14], and DR.FIX showed that retrieval and validation can make LLM-based race fixing effective in an industrial workflow [3]. Our work is complementary to these systems. Rather than building a production repair pipeline around a commercial model, we evaluate open-weight Gemma models. This lets us isolate how well such models explain OpenMP race mechanisms, how often they produce correct fixes under different prompting contexts, and whether larger open-weight models improve reliability enough to support an introductory parallel-programming assistant.
6
Conclusion
Our evaluation demonstrates that the Gemma4-E4B model is capable of accurately explaining a significant number of OpenMP data race conditions and providing correct parallel-preserving fixes
Meraj et al.
for a substantial portion of the DataRaceBench suite. Given its low execution cost and robust performance on foundational OpenMP constructs, Gemma4-E4B is well-suited to serve as an AI teaching assistant, helping students in introductory parallel programming courses debug their code independently. While instructor intervention remains necessary for complex synchronization scenarios or subtle memory consistency issues that the model fails to diagnose correctly, the total volume of such cases is significantly reduced when using Gemma4-E4B as a first-line debugger. Furthermore, our results show that the larger Gemma4-31B model provides significantly better diagnostic and remediation performance and should be prioritized whenever computational resources permit. Finally, we observe that contrary to expectations from prior work, the addition of contextual hints such as model-generated explanations or technical ThreadSanitizer reports does not necessarily improve the correctness of the generated fixes. We suspect this is due to a contextual distraction or pollution issue, where the additional technical noise and technical reasoning overhead overwhelm the model’s instruction-following capabilities, particularly in smaller models like Gemma4-E4B.
References [1] Anthropic. 2026. Claude Models Overview. https://platform.claude.com/docs/ en/about-claude/models/overview. Accessed: 2026-06-08. [2] Simone Atzeni, Ganesh Gopalakrishnan, Zvonimir Rakamaric, Dong H. Ahn, Ignacio Laguna, Martin Schulz, Gregory L. Lee, Joachim Protze, and Matthias S. Müller. 2016. ARCHER: Effectively Spotting Data Races in Large OpenMP Applications. In Proceedings of the IEEE International Parallel and Distributed Processing Symposium. IEEE, Los Alamitos, CA, USA, 53–62. [3] Farnaz Behrang, Zhizhou Zhang, Georgian-Vlad Saioc, Peng Liu, and Milind Chabbi. 2025. DR.FIX: Automatically Fixing Data Races at Industry Scale. Proceedings of the ACM on Programming Languages 9, PLDI, Article 166 (2025), 28 pages. doi:10.1145/3729265 [4] Sam Blackshear, Nikos Gorogiannis, Peter W. O’Hearn, and Ilya Sergey. 2018. RacerD: Compositional Static Race Detection. Proceedings of the ACM on Programming Languages 2, OOPSLA, Article 144 (2018), 28 pages. doi:10.1145/3276514 [5] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2024. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. arXiv:2403.17134 [cs.SE] [6] Le Chen, Xianzhong Ding, Murali Emani, Tristan Vanderbruggen, Pei-Hung Lin, and Chunhua Liao. 2023. Data Race Detection Using Large Language Models. In Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis. [7] Le Chen, Wenhao Wu, Stephen F. Siegel, Pei-Hung Lin, and Chunhua Liao. 2023. DataRaceBench V1.4.1 and DataRaceBench-ML V0.1: Benchmark Suites for Data Race Detection. In Proceedings of the Seventh International Workshop on Software Correctness for HPC Applications (Correctness ’23). ACM, New York, NY, USA, 1–4. doi:10.1145/3624062.3624231 [8] Xinyun Chen, Maxwell Lin, Nathanael Schaerli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024. 8746–8825. [9] Andrew Danner, Tia Newhall, and Kevin C. Webb. 2019. ParaVis: A Library for Visualizing and Debugging Parallel Applications. In 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). [10] Xianzhong Ding, Le Chen, Murali Emani, Chunhua Liao, Pei-Hung Lin, Tristan Vanderbruggen, Zhen Xie, Alberto Cerpa, and Wan Du. 2023. HPC-GPT: Integrating Large Language Model for High-Performance Computing. In Proceedings of the SC ’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis. [11] Dawson Engler and Ken Ashcraft. 2003. RacerX: Effective, Static Detection of Race Conditions and Deadlocks. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles (SOSP ’03). ACM, New York, NY, USA, 237–252. doi:10.1145/945445.945468 [12] Cormac Flanagan and Stephen N. Freund. 2009. FastTrack: Efficient and Precise Dynamic Race Detection. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, New York, NY, USA, 121–133. [13] Google. 2026. Gemini API Models. https://ai.google.dev/gemini-api/docs/models. Accessed: 2026-06-08.
[14] Feiyang Jin, Zhizhou Zhang, Rajkishore Barik, Gautam Korlam, and Milind Chabbi. 2023. Early Notice: GenAI-Based Datarace Fix for Real-World Golang Programs. In Workshop on ML for Systems at NeurIPS. [15] Guoliang Jin, Linhai Song, Wei Zhang, Shan Lu, and Ben Liblit. 2011. Automated Atomicity-Violation Fixing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, New York, NY, USA, 389–400. [16] Guoliang Jin, Wei Zhang, Dongdong Deng, Ben Liblit, and Shan Lu. 2012. Automated Concurrency-Bug Fixing. In 10th USENIX Symposium on Operating Systems Design and Implementation. USENIX Association, Berkeley, CA, USA, 221–236. [17] Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. InferFix: End-to-End Program Repair with LLMs. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, USA, 1646–1656. [18] David Kelk, Kevin Jalbert, and Jeremy S. Bradbury. 2013. Automatically Repairing Concurrency Bugs with ARC. In Multicore Software Engineering, Performance, and Tools. Springer, Berlin, Germany, 73–84. [19] Chunhua Liao, Pei-Hung Lin, Joshua Asplund, Markus Schordan, and Ian Karlin. 2017. DataRaceBench: A Benchmark Suite for Systematic Evaluation of Data Race Detection Tools. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’17). ACM, New York, NY, USA, 11:1–11:14. doi:10.1145/3126908.3126944 [20] Huarui Lin, Zan Wang, Shuang Liu, Jun Sun, Dongdi Zhang, and Guangning Wei. 2018. PFix: Fixing Concurrency Bugs Based on Memory Access Patterns. In Proceedings of the ACM/IEEE International Conference on Automated Software Engineering. ACM, New York, NY, USA, 589–600. [21] Peng Liu, Omer Tripp, and Charles Zhang. 2014. Grail: Context-Aware Fixing of Concurrency Bugs. In Proceedings of the ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, New York, NY, USA, 318–329. [22] OpenAI. 2026. OpenAI Models. https://developers.openai.com/api/docs/models. Accessed: 2026-06-08. [23] Stefan Savage, Michael Burrows, Greg Nelson, Patrick Sobalvarro, and Thomas Anderson. 1997. Eraser: A Dynamic Data Race Detector for Multithreaded Programs. ACM Transactions on Computer Systems 15, 4 (1997), 391–411. [24] Konstantin Serebryany and Timur Iskhodzhanov. 2009. ThreadSanitizer: Data Race Detection in Practice. In Proceedings of the Workshop on Binary Instrumentation and Applications. ACM, New York, NY, USA, 62–71. [25] Rishi Surendran, Raghavan Raman, Swarat Chaudhuri, John Mellor-Crummey, and Vivek Sarkar. 2014. Test-Driven Repair of Data Races in Structured Parallel Programs. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, New York, NY, USA, 15–25. [26] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv:2403.08295 [cs.CL] https://arxiv.org/abs/2403.08295 [27] Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, USA, 172–184. [28] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated Program Repair in the Era of Large Pre-trained Language Models. In Proceedings of the IEEE/ACM International Conference on Software Engineering. IEEE, Los Alamitos, CA, USA, 1482–1494. [29] Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each Using ChatGPT. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing
Evaluating Gemma4 Models as AI Teaching Assistants for Introductory Parallel Programming: A DataRaceBench Study
and Analysis. ACM, New York, NY, USA, 819–831. [30] Yuxiao Zhang, Jiang Li, Di Wu, and Yunfei Du. 2018. Improving Student Skills on Parallel Programming via Code Evaluation and Feedback Debugging. In 2018 IEEE
International Conference on Teaching, Assessment, and Learning for Engineering (TALE).