arXiv CSopen access

Geographic Variation in Stack Overflow Code Quality: Evidence from a Cross-Regional Study of Coding Practices

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

software-architecture software-engineering testing

software engineering, software architecture, testing

Geographic Variation in Stack Overflow Code Quality: Evidence from a Cross-Regional Study of Coding Practices Elijah Zolduoarrati*, Sherlock A. Licorish, Nigel Stanger Department of Information Science University of Otago Dunedin, New Zealand

[elijah.zolduoarrati, sherlock.licorish, nigel.stanger]@otago.ac.nz

ABSTRACT Developers routinely integrate Stack Overflow code snippets into their codebases, making the platform ubiquitous. However, the quality of snippets embedded in users’ answers remain elusive. Existing evaluations of code quality tend to be language-specific, necessitating holistic investigations that span multiple languages. Moreover, literature has found that contribution patterns vary depending on geographical locales, creating an unexplained rift between code quality, user location, and latent contextual regional factors. This study evaluates the quality of SQL, JavaScript, Python, Ruby, and Java snippets across reliability, readability, performance, and security dimensions. We benchmark findings across states in the US, assessing how different diversity indicators correlate against code quality violations. We conducted a series of inductive content analyses that qualitatively supplement prior quantitative findings. Outcomes show that readability violations are the most prevalent across all languages, followed by reliability, performance, and security. Readability violations largely comprise improper whitespace, while reliability issues mainly involve program flow alterations. Performance violations encompass resource allocation while security vulnerabilities include unvalidated inputs that risk injection attacks. Tech hubs like California show moderate reliability and readability violations, while tech-sparse areas like Maine exhibit fewer reliability and performance violations. Regions with equitable wealth distribution and higher per capita income tend to have better code quality, possibly due to increased investment in education and access to technology. Disparities in coding practices were also surmised to stem from a combination of gender-inherent traits, diversified tech workforce, emerging non-tech industries, and the prevalence of startup culture within these regions. Keywords: Code quality, Stack Overflow, readability, reliability, performance, security, programming languages, regional disparities, diversity indicators, content analysis.

INTRODUCTION

In the digital age, the rapid evolution of technology has ushered in an era of unprecedented collaboration and information sharing [1]. Community Question Answering (CQA) platforms, exemplified by popular websites like Stack Overflow1 and Quora2, represent a remarkable outcome of this digital revolution [2]. These platforms have revolutionised information dissemination by enabling users to pose questions in natural language and receive tailored responses [2]. Stack Overflow, in particular, stands at the digital epicentre, where developers gather to share insights, seek guidance, and collaborate with peers regarding the technical challenges they face [3, 4]. The hallmark of this platform is its emphasis on high-quality content, including accurate, well-structured posts that provide robust solutions. Posts on Stack Overflow provide developers with more than just answers; they offer well-crafted solutions that address the root causes of issues, provide context, and illuminate underlying computer science-related concepts. Therefore, it is paramount that the information presented on Stack Overflow is of high quality, so that developers facing similar challenges are empowered to make informed decisions and build more and more robust solutions in the future. Owing to its importance, quality evaluations on Stack Overflow have garnered major popularity in recent literature. However, scholarly works have primarily focussed on semantic (e.g., readability or clarity) and community-related factors (e.g., popularity or content curation activity of users) [5, 6, 7]. Studies focussed on semantics have oftentimes side-lined the facet of code quality, which is no less important given that developers frequently use Stack Overflow code snippets in their projects [8]. * Corresponding author 1 https://stackoverflow.com 2 https://www.quora.com

Existing evaluations of code quality tend to be language-specific instead of encompassing the broader picture. For instance, JavaScript, Java, and Python code snippets were investigated separately in three different research endeavours [9, 10, 11]. Moreover, previous work [12] has hinted that diversity complications in the tech field also propagate to Stack Overflow. For instance, it was shown that diversity-related problems range across human-inherent features (e.g., age and gender), across socioeconomic constructs, technological accessibility, and workforce concentration [12]. While subsequent literature has tried to bridge the gap between contribution and diversity [13], the unexplained rift between code quality and the construct of diversity remains elusive. Thus, the need for a research agenda arises aimed at integrating these different facets under a unified framework. This work aims to comprehensively investigate code quality in Stack Overflow by assessing multiple quality dimensions, namely reliability, readability, performance, and security. These dimensions were drawn from recent literature, acknowledging the absence of a singular, universally applicable definition for the construct of quality [11]. In addition, we progress the agenda of exploring diversity and how this affects SE-related practices by investigating regional variations in coding practices, aiming to shed light on how different regions vary with regard to producing reliable, readable, performant, and secure code. Our study builds upon the work of Meldrum et al. [11], whose approach we largely adopt in this investigation, particularly in terms of how code quality is defined. We leverage the Stack Overflow Database obtained from the Stack Exchange Data Dump, which has been used previously [7, 14]. Our study focusses exclusively on answers, aligning with user behaviour, which predominantly involves seeking answers and copying code solutions [15, 16]. Further, we narrow our investigation to examine code snippets originating from the United States of America (US) for several reasons3. Firstly, Stack Overflow’s origin in 2008 was in the US before its global expansion [17]. Secondly, as the largest English-speaking nation [18], the US serves as a suitable setting for studying software development, as English is the lingua franca within this field [19]. Third, the US has been consistently positioned at the forefront of technological innovation [20, 21], providing a diverse pool of developers and access to cutting-edge resources. Finally, the prevalence of US users on Stack Overflow is among the highest in comparison to other countries, rendering it a representative platform for exploring the dynamics of contributors in CQA settings [12]. While Stack Overflow data is primarily user-generated (e.g., posts, comments, profile bios), prior research has demonstrated its usefulness as a proxy for exploring SErelated practices [22, 23]. Moreover, previous work had subjected the data to rigorous checks [12] and thus the data quality employed in this work is assured. SQL, JavaScript, Python, Ruby, and Java were selected for coverage in our study due to their widespread adoption among all US users (refer to Section 3.3). All five languages have also been studied extensively in the code quality niche [24, 25], yet findings from prior works remain compartmentalised. To focus our study and achieve the planned objective, we have formulated the following research questions: RQ1: What is the quality of code snippets across different programming languages on Stack Overflow? RQ2: How do coding practices among US contributors differ across states and cities in the United States? RQ3: How do diversity indicators of US regions affect code quality? RQ4: What themes are prevalent in code quality violations, and how do these themes differ across US regions? RQ1 provides insights into the quality of Stack Overflow code snippets, frequently used by developers, with respect to four quality dimensions: reliability, readability, performance, and security. The motivation behind RQ1 is to assess the platform’s effectiveness in providing trustworthy solutions, as well as to ascertain whether the given solutions are comprehensible, performant, and are free from security vulnerabilities. RQ2 presents a granular analysis across all states and cities throughout the US to explore disparities in reliable, readable, performant, and secure coding practices. This will shed light on the causative factors behind such observed shifts, in turn identifying why these variations occur due 3

Our data covers 50 states and the District of Columbia. Our geocoding API (refer to Section 3.2) treated the US territories, namely American Samoa, Guam, Northern Mariana Islands, Puerto Rico, US Minor Outlying Islands, and US Virgin Islands, as distinct entities, but the US Census Bureau excludes these territories from the main 50 states.

to inherent contextual and cultural factors within each region. RQ3 focusses on identifying regional factors that may influence code quality within Stack Overflow snippets. We aim to understand how latent characteristics of a region, such as socioeconomic factors, access to infrastructure, or tech and R&D density may be able to impact the reliability, readability, performance, and security of the code snippets contributed by developers living in those regions. RQ4 delves into the nature of code quality violations, enabling the qualitative identification of any recurring themes across the four key dimensions of code quality. For example, within the dimension of readability, issues related to improper whitespace formatting may be more prevalent, as opposed to less widespread problems like function annotation or poorly named variables. We surmise that a deeper qualitative exploration will enrich the community’s understanding into which specific types of code quality violations are prevalent. Addressing these research questions offers contributions for both theory and practice. Our study contributes to existing literature by bridging a theoretical gap wherein multiple programming languages are assessed under the same quality dimensions. We expand this line of inquiry by taking into account census diversity indicators, thereby bridging the gap between technology and socioeconomics. From a practitioner standpoint, our insights promote reliable, readable, performant, and secure coding practices, empowering developers to adopt these enhancements into their daily development processes. As developers adopt higher-quality coding practices, the overall codebase of projects can improve, leading to better-performing, more maintainable, and more secure repositories. A qualitative examination of common themes within each code quality dimension is also presented, which may empower practitioners to proactively avoid common pitfalls in their own software development practices. Finally, while generative language models (e.g., ChatGPT or Gemini) have been linked to a decrease in overall Stack Overflow participation, their solutions are mostly error-prone [26]. Conversely, Stack Overflow offers solutions that are vetted by the community and specifically tailored to address niche problems, and thus its unique value stems from the quality of the shared code snippets. Our study thus remains valuable for developers, enabling them to write and identify high-quality code even in the age of generative language models. The remainder of this manuscript is organised as follows. Section 2 provides the theoretical underpinnings of our study. Section 3 elucidates the methodologies employed for data collection, processing, and analysis. Section 4 presents the findings of the study with respect to each research question, before we discuss our results in Section 5. In Section 6, we discuss the limitations of the study, and we conclude with final remarks and suggestions for future research in Section 7. Additionally, a replication package is provided for those interested in further examining our research methodology and performing follow up analyses [27]4.

2 BACKGROUND 2.1 Diversity in Software Engineering The inherent socio-technical nature of software engineering (SE) makes it prone to the accumulation of social debt, particularly related to diversity within the field [28]. However, the relationship between developer background diversity and project outcomes remains elusive. On the one hand, diversity in factors like age, gender, culture, and education can foster synergy, creativity, and a broader range of perspectives when tackling problems [29, 30]. On the other, they can also lead to increased potential for conflict [31]. Language barriers are also a threat, where the dominance of English as the lingua franca in software development creates a barrier for non-native speakers, hindering their ability to keep pace with advances in the field [32]. Unequal access to infrastructure and communication services, also known as the digital divide, further exacerbates the existing challenges of diversity within SE-related practices. This disparity permeates to all layers of society across individuals, households, businesses, and entire geographical regions, and has become even more pronounced since the emergence of COVID19 [33]. These factors result in a complex interplay that impedes diversity in SE [12]. External challenges, such as limited access to infrastructure and economic disparities, act as barriers to participation. Internally, unconscious biases and exclusionary practices can stem from inherent human characteristics like educational background, age, gender, and cultural constructs [12]. In fact, seemingly technical challenges in SE-related processes such as insufficient documentation and software defects are often intertwined with latent, underlying human aspects [34]. To this end, a comprehensive tertiary 4

https://zenodo.org/records/13622420

review regarding what human factors influence SE processes has emphasised the need for tailored approaches to bridge diversity gaps, rather than resorting to one-size-fits-all solutions [34]. While research has extensively explored individual human aspects like gender, race, and culture [35, 36, 37], external factors such as demographics and socioeconomic background remain understudied [34]. Prior research has demonstrated a potential association between regional development and participation in the technology sector [12, 13]. Regions with greater access to education, strong economies, healthcare, infrastructure (including internet), and consumer technology tend to have a larger presence in the tech industry. This translates to a higher user base and increased activity levels on platforms like Stack Overflow, which holds true not only on a global scale but also within more granular levels, such as across states and cities [12]. An interesting dynamic exists in how users from different geographic locations interact within the community, where urban users tend to adopt a more assertive and solutionoriented approach, actively contributing information. In contrast, rural users primarily engage through inquiries and discussions, often incorporating personal experiences, expressions of gratitude, and conciliatory language [12]. Another separate study attempts to unveil more of this mosaic by going beyond simple quantitative enumeration [13], developing metrics to assess user participation, behaviour, and the value they contribute to the community, enabling operationalisations of constructs that are not readily available (e.g., user popularity, sentiment of posts, and readability), across various US states and cities [13]. Interestingly, users from rural states tend to post more frequently, curate content more actively, and produce content that is more readable, positive-toned, and with fewer errors. Those residing in cities without a strong tech presence show signs of lower participation, increased lurking behaviour, and a tendency to write longer code snippets. In fact, users from tech hubs were more inclined to present technical jargon and collaborative knowledge-sharing, often incorporating code examples, debugging tips, and relevant anecdotes. Conversely, rural users express a wider range of emotions, including hope, frustration, and satisfaction, alongside their manifold questions [13]. While previous studies have explored various aspects of user interaction on Stack Overflow, the impact of diversity and sociotechnical factors on code snippet quality remains unexamined. This current research aims to bridge this knowledge gap.

2.2

Code Quality

The software development landscape is marked by rapid evolution, characterised by a diverse range of programming languages, frameworks, and paradigms [38, 39]. However, these progressions do not always adhere to practices that ensure software quality [40], thereby calling for quality control mechanisms. Software quality encompasses diverse characteristics like functionality, reliability, usability, efficiency, maintainability and portability, traditionally defined by standards like ISO/IEC 9126 and ISO/IEC 25010 [41, 42]. However, translating these abstractions into measurable metrics presents a challenge [43]. In response to this challenge, more pragmatic quality models have emerged, such as SQALE and Quamoco, bridging the abstraction-implementation gap by providing concrete frameworks for quality assessment [43]. Furthermore, the emergence of specialised maintainability models like SIG and Delta address particular concerns like technical debt [44, 45]. Readability models tackle the subjective nature of code comprehension and program understanding, as done by Scalabrino et al. [46] as well as Buse and Weimer [47]. Moreover, the concept of code quality may also be classified into either code smell, internal, or external quality attributes [48, 49]. There are also external quality attributes including compatibility, configurability, extensibility, and functionality [49]. In spite of these developments, their applicability to smaller, modular code snippets like those found on Stack Overflow remains a challenge. A multitude of studies have thus assessed the quality of code snippets hosted on the platform across various dimensions. Security has been a primary focus, with investigations into vulnerabilities in C/C++ and C# [50, 51, 52]. Code readability has also been studied, highlighting the influence of less nuanced factors like line length, whitespace, and formatting on both code quality and comprehension [53]. Studies have even explored compliance with style guides. For instance, Bafatakis et al. [9] examined whether Python snippets comply to Python Enhancement Proposal (PEP) 8 [54], whereas Campos et al. [10] investigated rule violations in JavaScript snippets and found that 82.9% of the violations pertain to stylistic issues. Ahmad and Ó Cinnéide [55] conducted an inquiry into the code cohesion of Java code snippets that were copied from Stack Overflow into GitHub projects. Results indicate that copied code snippets actually lowers cohesion, thereby making the codebase less robust [56]. Further research by Subramanian and Holmes on Java [8] illuminate the issue of code snippet incompleteness stemming from specific aspects (e.g., variable declarations) being frequently omitted.

While studies like Zerouali et al. [57] demonstrate the value of analysing multiple languages for specific quality dimensions, they have not yet been applied to Stack Overflow itself. Their work explored security vulnerabilities in JavaScript, Python, and Ruby packages within Docker Hub images, showcasing the feasibility of such multi-language analysis. On the other side of the coin, several works have holistically examined multiple dimensions under a unified umbrella. Meldrum et al. [11] investigated Java snippets across Stack Overflow answers through the lens of reliability and conformance to programming rules, readability, performance, as well as security. These findings reveal that the most occurring violation is related to readability, averaging approximately 10.5 readability violations per snippet, followed by 4.8 of reliability and conformance to programming rules, 0.5 of performance, and 0.01 of security. Geremia et al. [58] have done conducted a similar study, operationalising code quality to encompass snippets’ total lines of code, cyclomatic complexity [59], and readability. Results underscore that these quality dimensions influence the likelihood of solutions being adopted by fellow developers. Pantiuchina et al. [60] delineate code quality across four dimensions: cohesion, coupling, readability, and complexity, while Gonzalez et al. [61] consider total refactors, code documentation, complexity and introduced issues (e.g., code smells, bugs, and vulnerabilities). The aforementioned studies highlight the multifaceted nature of code quality, yet there remains a dearth of research that comprehensively assesses code quality on Stack Overflow across multiple dimensions and programming languages. Hence, our study endeavours to address this gap in the literature by offering a comprehensive snapshot of the platform’s code quality landscape. Moreover, we investigate how code quality relates to census diversity indicators, enabling the identification of inherent regional factors that may drive (or deter) the quality of code snippets. We acknowledge the existence of other CQA platforms that allow code snippets in their posts, such as Cross Validated5, Ask Ubuntu6, or Code Review7, yet our work focusses solely on Stack Overflow due to its wealth of information and peer-topeer interactions that allow for subtle behavioural tendencies to be studied [62]. The following section delineates the methods that were employed in this study.

METHODS

In this section, we first reflect on established literature to define our study scope regarding code quality dimensions. Subsequently, we elucidate our strategy for data collection and processing, followed by outlining our methodology for selecting the languages to examine. We conclude this section by delineating our approach to address each research question.

3.1

Code Quality Definition

Considering the difficulty of defining code quality [63], literature has identified various dimensions that define such a construct. Specifically, we identified ten pertinent quality dimensions from prior studies (see Section 2), namely security, readability, performance, complexity, completeness, reusability, conformance to programming rules, reliability, coupling, and code cohesion. Several dimensions exhibit overlap upon closer examination. Meldrum et al. [11], examined “reliability” alongside “conformance to programming rules,” but we observed that the former encompasses the latter since style guides prescribe consistent code presentation [9]. Boogerd and Moonen [64] identified 10 convention violations as significant predictors of faults in C code, implying a causal relationship between rule adherence and reliability. Due to this interrelationship, we merged “conformance to programming rules” and “reliability” under the latter. Similarly, we observed the phenomenon of “code smells,” which describes potential design problems [65]. Recent works associate code smells with reliability due to their link to error-prone systems [66], as the intuition is that error-prone code is inherently less reliable. In light of this, we also consider our “reliability” dimension to include code smells. Another overlapping dimension is “performance” and “complexity,” which exhibit an inverse relationship (i.e., as one increases, the other decreases) [67]. For example, larger codebases with more features have increased complexity, leading to performance degradation [67]. Similarly, finite state machines, widely used in game development, experience performance drops as complexity increases with more game behaviours [68].

https://stats.stackexchange.com https://askubuntu.com 7 https://codereview.stackexchange.com 6

As such, the decision was made to include “performance,” subsequently representing “complexity” inside the former with the intuition that high-performing code snippets are less complex. Certain dimensions are not applicable to individual code snippets and are designed for analysing large codebases. For instance, “code cohesion” measures the relatedness of elements within a class, while “coupling” assesses the strength of inter-file or inter-class dependencies [55, 60]. This inherent reliance on class-based object-oriented structure renders them inapplicable to individual code snippets, particularly in interpreted languages like Python and Ruby where classes and complex dependencies might be less prevalent. “Completeness” assesses whether snippets lack proper class declarations, which is unsuitable as SO posts are often missing crucial parts [8]. Finally, “reusability” requires a separate dataset to compare snippet reuse across platforms like F-Droid, GitHub, and BitBucket [69, 70]. Consequently, we focussed on four dimensions: reliability, readability, performance, and security, aligning with Meldrum et al. [11], being the most relevant study to our own pertaining to code quality on Stack Overflow across multiple quality dimensions. Notably, their study addresses the largest number of quality dimensions, thus ensuring the robustness of our subsequent analyses. As these dimensions were initially defined for Java, we sought to generalise them to other programming languages. Finally, these four dimensions closely align with the ones identified by Ndukwe et al. [71] following a set of semi-structured interviews of 50 practitioners: “Functionality”, “Readability”, “Efficiency”, “Security” and “Reliability”. Firstly, the “Functionality” dimension is encompassed within “Reliability”, as the rationale is that reliable code inherently fulfils its intended function. Similarly, we consider “Efficiency” to be synonymous with “Performance,” defining performant code as code that executes the desired task with minimal processing steps. Table 1 documents our chosen quality dimensions, the reasoning behind their selection, and their relevance to our case. Table 1. Quality dimensions and their definitions Dimension

Definition

Reliability

Code snippets ought not to be broken, introduce bugs, or prone to errors [11].

Readability

Code snippets should adhere to the readability conventions specific to each programming language [9, 11].

Performance

Code snippets should be efficient and capable of performing the intended task with minimal processing steps [11].

Security

3.2

Code snippets ought not to compromise security or present potential vulnerabilities that could be exploited [11, 50].

Motivation It is assumed that users who contribute code snippets are responsible for providing accurate solutions, and thus their solutions should be free of errors in order to be deemed reliable [11]. Adhering to established conventions ensure that code can be readily maintained in the future [11]. In contrast, if code is unreadable, it may introduce maintainability challenges or inhibit program comprehension [72]. Software ecosystems that exhibit greater efficiency and less complexity would render them more scalable, meaning that they would perform well even when presented with larger amounts of input [73], as opposed to those that are less efficient. In order to protect oneself from malicious attacks and hackers’ intrusions, it is imperative to prioritise the avoidance of vulnerabilities and weaknesses when writing software systems [74].

Implications The reliability of code may enable us to discern whether certain users have a propensity for writing more dependable code than others. The readability of code may allow us to infer whether certain users tend to neglect programming conventions, thus rendering their code snippets difficult to comprehend. Code snippet performance may enable us to determine whether certain users are more efficient in their implementation of solutions. The security of code snippets may uncover certain users’ inclinations to produce less secure code, jeopardising the security of others who incorporate these insecure snippets into their codebases.

Data Collection and Processing

Our study leverages the Stack Overflow (SO) Database, sourced from the Stack Exchange Data Dump8 superset seen in previous literature [22]. The relational database was stored on our local instance of 8

https://archive.org/details/stackexchange

Microsoft SQL Server, containing a myriad of activities and raw measures across different tables, namely users’ posts (questions and answers), comments, badges, and edits. Being the latest at the time of our investigation, we used the June 2022 release9, which included 56,264,788 posts, 85,467,182 comments, and 22,796,157 edits from 17,922,426 unique users. Our replication package [27] presents the interactive database schema10 and the total rows for each table11. To explore how diversity indicators influence code quality within the US, our study leverages census data from a prior study [12]. This data encompasses ten sets of census indicators for each state within the US, carefully refined from an initial pool of 465, thereby ensuring the scientific rigour and analytical merit of the chosen diversity indicators. These sets cover a multitude of diversity aspects, such as economy, infrastructure, and education. Table 2 below lists these census diversity indicators, and what diversity aspect each indicator represents. Table 2. Selected state-level census indicators [12] Diversity Aspect Economy Education Ethnicity Gender Health Poverty Science & Technology Social Protection & Labour

Indicator Name Per capita income in the past 12 months (in 2020 inflation-adjusted dollars) School Enrolment by Level of School for the Population 3 Years and Over: Enrolled up to Grade 12 Race: White alone (% of all population) Total Males Total Females Females with health insurance coverage Gini index Households with one or more types of computing devices Internet Subscriptions in Household: With an Internet subscription Total Unemployment

Several data processing measures were then undertaken. To focus our scope on code quality within the confines of the US, we needed to identify which users hail from this country. Therefore, the first objective was to determine users’ country of origin, to which we excluded 14,174,843 users with null location fields, retaining data for 3,814,901 unique users. Second, inconsistencies in naming conventions were found, such as variations like “SF Bay Area” and “San Fran” both denoting San Francisco but treated as distinct locations. This inconsistency prevented automatic aggregation of user data with corresponding census diversity indicators at regional levels. Consequently, the inability to link individual regions with their specific diversity indicator hindered our investigation into how these indicators might influence code quality. To address such mismatches and enable aggregation, we explored 15 proprietary geocoding APIs from numerous GIS providers, namely ArcGIS, Bing Maps, GeocodeEarth, Geodict, Geograpy, Geonames, Geopy, Geotext, Google Maps, MapQuest, Nominatim, OpenCage, OpenStreetMaps, Photon, and TomTom. The selection considered factors such as suitability for the study, commercial availability, and insights from recent literature. Following a rigorous evaluation process across city, state, and country levels (see our replication package [27]12 for more details), MapQuest was ultimately selected to standardise all user “Location” entries and detect invalid entities. Several entities that were deemed as invalid include “Tatooine” (Star Wars) for Users.Id13 11896718 and “Valhalla” (Norse mythology) for Users.Id 10938505. Moreover, some users provided non-geographical locations like “Mostly at home” for Users.Id 2736499. As 67,318 invalid location entries were then discarded, we ensure that city, state, and country entries are entirely accurate. As a demonstration, the string “Manhattan, NYC” would return “New York City” as the city, “New York” as the state, and “United States” as the country. Our scope is limited to only users from the US, given the origin of Stack Overflow and the largest English-speaking country with English as the lingua franca of SE-related rigours [19]. It is also a leader in tech innovation [20, 21] and hosts the highest number of Stack Overflow users [12], making it a suitable choice for this study. Therefore, we lastly filtered out

https://archive.org/details/stackexchange_20220606 Replication package » Figures » SO Database Schema.html 11 Replication package » Figures » SO Database Rows.png 12 Replication package » Demonstrations » GeoAPI Evaluation Process 13 Prefixes denote the table that a particular column is in. For instance, “Users.Id” refers to the column Id within the Users table of the Stack Overflow Database. 10

2,986,774 users whose locations point to other countries, yielding a final sample of 760,809 users with 8,951,562 total posts (6,596,105 from answers) that only encompasses US users. To extract code snippets from their posts, we followed recent literature where we only examined code blocks in posts rather than in-line code snippets [11]. The rationale behind this is that in-line code snippets were typically added as part of text answers when users needed to provide an explanation. Code blocks, on the other hand, provided actual implementations to solve the problem at hand [11]. For instance, Figure 1 depicts the answer Posts.Id 72504547 about using C integer types in a Fortran program. Technical keywords such as import in the answer were inserted as in-line snippets (the red square), whereas example implementations were inserted as code blocks (blue square). Prior literature deemed it unfair to assess the quality of such in-line snippets [11], therefore our analysis looks only at code blocks.

Figure 1. Example answer (Posts.Id 72504547) with in-line code snippet (red) and code block (blue) Our analysis of the Stack Overflow database revealed that code snippets are stored within the Body column wrapped in HTML tags. In-line code snippets are enclosed by <code> tags, while full code blocks are surrounded by <pre><code> tags. We employed the string-matching algorithm of the Pandas14 library to extract code blocks from answer posts, excluding in-line code snippets. We thus eliminated 2,884,589 answers that did not contain a code block, whilst keeping 3,711,516 answers having 6,213,554 distinct code blocks. As an example of the former, Figure 2 below depicts an answer with Posts.Id 72505278 remarking how transfer learning requires developers to exclude the last layer of a pretrained neural network. The given answer only provided a conceptualised theory without a code snippet, and hence was excluded from subsequent analysis. As one answer may contain several code blocks, we stored the extracted outcomes in a separate table called USCodes, where individual rows correspond to distinct code snippets represented by SnippetId. Each row also includes information about the parent answer, as well as authorship details such as OwnerUserId, city, and state (refer to our replication package for the database schema [27]15).

Figure 2. Example post that does not contain a code block (Posts.Id 72505278)

3.3

Language Inference and Selection

Following the extraction of code snippets from each answer, our subsequent task entails identifying the programming language(s) for further examination. This step was essential to facilitate pertinent preprocessing procedures such as insertion into classes [11] and exporting to appropriate file formats (e.g., .cpp for C++ or .js for JavaScript). Moreover, this knowledge enabled us to select specific languages representative of all users from the US. We initially considered harnessing the Posts.Tags column that 14 15

https://pandas.pydata.org/docs/reference/api/pandas.Series.str.contains.html Replication package » Figures » Snippets Database Schema.png

often indicated the programming language in question (e.g., ‘Java’) and the technology being inquired (e.g., ‘Maven’) to determine which answers correspond to which questions. However, we encountered threads with multiple programming languages, rendering this approach unreliable. For example, Figure 3 depicts an answer with Posts.Id 38995602 outlining a workaround to display a dropdown menu. While the main question was tagged and posted in HTML, this particular answer was written in JavaScript.

Figure 3. Example answer (Posts.Id 38995602) in JavaScript In light of this constraint, we pivoted towards Guesslang16 to detect programming languages from plaintext. Guesslang has been proven to yield good effect in identifying code snippets on Stack Overflow [50, 51], and is the backbone for various developer tools such as Visual Studio Code17 and Chameledit18. Prior to running the tool on our extracted code blocks, we did an empirical evaluation of 385 answers to ascertain its utility. Our evaluation revealed an accuracy of 84.67% (refer to our replication package [27] for full evaluation results19). Prior studies have established that accuracy exceeding 80% is considered acceptable [75]. Thus, our achievement of 84.67% ensured the reliability of our Guesslang application, enabling its application to our entire code snippet dataset. Results were then saved onto another column USCodes.GuesslangTag. While Table 3 below documents the identification of 54 distinct technologies after the tool has finished running, we also uncovered 4,551 snippets where Guesslang returned null values instead of the predicted technology. These were excluded, as they represented a small percentage (0.073%) of the entire pool of 6,213,554 snippets. Table 3. 54 technologies identifiable by Guesslang Assembly Batchfile C C# C++ Clojure CMake COBOL CoffeeScript CSS

CSV Dart DM Dockerfile Elixir Erlang Fortran Go Groovy Haskell

HTML INI Java JavaScript JSON Julia Kotlin Lisp Lua Makefile

Markdown Matlab Objective-C OCaml Pascal Perl PHP PowerShell Prolog Python

R Verilog Ruby Visual Basic Rust XML Scala YAML Shell SQL Swift TeX TOML TypeScript

https://github.com/yoeo/guesslang https://code.visualstudio.com/updates/v1_60#_automatic-language-detection 18 https://gitlab.com/yoeo/chameledit 19 Replication package » Assessments » Guesslang Sample Answers.xlsx 17

Within our new pool of 6,209,003 snippets, we discerned that several of these technologies are not programming languages, but rather data storage types, configuration files, as well as markup languages and UNIX-style commands such as sed (stream editor) or Shell scripts. For instance, JSON is a technology used to store data in nested key-value pairs, commonly returned as a GET or POST response from an API request. Another example pertains to INI, a file format mainly used for storing configuration settings in an application. Similar to JSON, the INI file format stores information in keyvalue pairs and thus has its own syntax so that the corresponding application can correctly read the written data. Focussing primarily on programming languages, most prior research (see Section 2) omits analysis for configuration files, data storage formats, markup technologies, and UNIX-like commands. We posit that these technologies hold minimal relevance for code quality analysis, as applying our predetermined dimensions (see Table 1) to snippets that are not programming languages would be irrelevant. To first determine which of these predicted technologies belong to which category, we conducted a brief review of existing literature, because definitions of programming languages, data storage types, configuration files, markup languages, and UNIX-style commands can be subjective and thus vary from person to person. For instance, some might argue that HTML is a programming language, while others may see it as a markup language [76]. Similarly, some developers may not consider SQL to be a programming language since it cannot be used to create standalone applications. Yet the literature revealed that SQL is, in fact, a set-based, domain-specific, declarative programming language and thus should be categorised as such [77, 78]. Our replication package [27] documents our categorisation results, as well as their tallied occurrences20. We thus removed 891,764 configuration snippets, 87,149 data storage formats, 578,592 markup technologies, and 970,851 UNIX-like commands. This left us with a total of 3,680,647 snippets, encompassing 38 distinct programming languages. Despite having removed irrelevant snippets, the volume of data remains substantial, thereby bolstering the reliability of our findings. Previous research has demonstrated that even smaller code snippet pools can unearth novel insights [9, 11], and thus our study’s downstream analyses based on this pool remain relevant. Next, we observed substantial variations in terms of their total number of snippets, hinting that languages may not be comparable between each other. For instance, SQL had 776,836 snippets, while Fortran only had 3,252. To compound this issue, certain languages were completely absent in particular states (e.g., no COBOL answers in Alabama). Another potential confounder is that we observed significant disparities in snippet counts across programming languages within individual states. For instance, Kansas had only 42 Scala snippets compared to 1,421 SQL and 951 JavaScript snippets. This phenomenon was prevalent among less popular languages like COBOL, Verilog, and DM, while widely used languages like SQL, JavaScript, and Python consistently exhibited higher snippet counts regardless of state. To determine the languages for inclusion, we computed the mean number of snippets across all languages as a threshold, aligning with prior literature [79, 80]. Yielding an average of 96,859, 27 languages were excluded where their total snippets fell below this number, in turn retaining the top 11 languages. Sorted from most to least snippets, the top 11 languages are SQL (776,836 snippets), JavaScript (486,175), Python (329,396), Ruby (206,090), Java (184,903), C# (179,723), TypeScript (135,223), PHP (132,191), R (139,448), Groovy (121,668), and finally CoffeeScript (99,379). To conclude our funnelling process, our next objective was to ascertain which of these 11 languages were among the top 10 languages for each state. The rationale was that we would pick languages that offer comprehensive representation across all regions of the US. This was achieved by a simple quantitative analysis that tallied the number of code snippets in each language within each state. Our findings are documented in Table 4, where checkmarks denote whether the language is ranked within the top 10 languages in the corresponding state. Table 4 reveals that SQL, JavaScript, Python, Ruby, and Java are the only languages that are present in the top 10 languages of all 50 states, as well as the District of Columbia (refer to our replication package [27] for the tallied numbers21).

20 21

Replication package » Results » Predicted Technologies.docx Replication package » Language Counts » Programming Languages - State.xlsx

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓

✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓

✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓

✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

CoffeeScript

Java

Groovy

Ruby

Python

PHP

JavaScript

Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming

TypeScript

State

SQL

Table 4. Prevalence of programming language for each state

✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓

✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓

In contrast, other languages do not demonstrate similar levels of ubiquity. For example, C# does not feature among the top 10 languages of Rhode Island, Utah, or Virginia. Similarly, TypeScript is absent from the top 10 languages of Alabama, Alaska, Connecticut, and New Mexico. Thus, given the pervasiveness of SQL, JavaScript, Python, Ruby, and Java, we decided to investigate their code snippets across the established quality dimensions. Selection of all five languages ensures a fair and balanced sample, guaranteeing comparable results across regions. However, these five programming languages might not reflect global popularity as revealed by the Stack Overflow Developer Survey conducted in the same year [81]. This limitation is discussed in Section 6. As a final measure to ensure the accuracy of Guesslang’s language identification, we randomly sampled 385 code snippets, one from each question post tagged with a specific language (i.e., resulting in five sets of 385 snippets, one for each language). This sample was then manually verified to confirm the correctness of Guesslang’s predictions, and the results are documented in Table 5 below. All five evaluated sets exhibited relatively high accuracy, around 87-94%, which ensures the validity of such predictions. Our replication package [27] contains the full results of the manual language checks22. Table 5. Guesslang evaluation results for each language Language SQL JavaScript Python Ruby Java

3.4

Question Tag Identifier <sql> <javascript> <python> <ruby> <java>

Accuracy 94.29% 91.42% 89.61% 87.79% 90.91%

Code Snippet Extraction

Following language selection, we seek to extract the code snippets. We applied filtering to our USCodes table, selecting rows where the GuesslangTag column matched the target language. For example, to filter SQL snippets, we isolated rows where the GuesslangTag equalled “SQL” before writing them as individual files. Each snippet’s filename comprised a language prefix and its corresponding SnippetId, so that results could be easily linked back to their corresponding posts and users. For example, the filename SQL_613.sql refers to an SQL snippet having a SnippetId of 613.For dynamically-typed languages SQL, JavaScript, Python and Ruby [82], snippets were extracted without any additional preprocessing measures or snippet repairs (e.g., importing classes or deleting spaces), as doing so would confound our results [11]. Moreover, dynamically-typed languages afford greater flexibility in variable types and do not enforce strict type checking at compile time, therefore bypassing the need for enclosing code within public classes (such as in C or C++) [83]. When extracting Python snippets, Windows Defender identified 3 as Keylogger scripts [84], which are SnippetId 4214594 from Posts.Id 57747365, 6165888 from 65077377, and 6165943 from 65258604. We manually inspected these snippets to ensure safety, subsequently adding these snippets to our local machine’s whitelist. Java, on the other hand, is classified as a statically-typed language wherein variable types must be explicitly declared before usage [82, 83]. Following recent literature [11, 85], we firstly checked whether the code snippets contained import, package or class keywords. Should these keywords be present, we saved the file as it is. Otherwise, we encapsulated the snippets in a public class structure, with the class name containing the corresponding SnippetId. For instance, Figure 4Error! Reference source not found. illustrates SnippetId 561 enclosed in a public class structure before being saved as Java_561.java. Having prepared the data for analysis, Table 6 presents the methods employed to answer each RQ, which are then explored in detailed within the following subsections. Table 6. Methods used to answer each RQ RQ#

Research Question

RQ1

What is the quality of code snippets across different programming languages on Stack Overflow?

Method Quantitative analysis was conducted via linting tools, followed by a detailed exploration how the identified violations for each language might shed light on users’ habits on SE-related practices.

Replication package » Assessments » Guesslang Per Language

RQ#

Research Question

RQ2

How do coding practices among US contributors differ across states and cities in the United States?

RQ3

How do diversity indicators of US regions affect code quality?

RQ4

What themes are prevalent in code quality violations, and how do these themes differ across US regions?

Method Violation densities for each region are calculated, defining total violations per logical line of code. A comparative analysis of these violation densities is then performed across different regions to identify variations in coding practices. Pearson’s correlation analysis was employed to investigate the relationships between state-level diversity indicators and the violation densities of each state. At a 5% significance level, the null hypothesis was that there is no correlation, while the alternate hypothesis was that a correlation exists. Inductive content analysis was applied to code snippets from three representative states. This allowed us to delve into the latent characteristics associated with each state and code quality dimension.

Figure 4. Example Java snippet (SnippetId 561) being enclosed in a public class structure

3.5

Code Snippet Quality (RQ1)

To assess code snippet quality, we initially considered using off-the-shelf automated static analysis tools such as SonarQube and Codacy, which offer quantifiable quality metrics and support for various programming languages [86]. However, we found that these tools are primarily designed for analysing software repositories with interdependent files and established continuous integration and continuous delivery (CI/CD) workflows [87]. These inherent design misalignments, coupled with the fact that these automated code review tools fundamentally employ linting tools as their underlying mechanisms [88], pivoted us to harness linting tools instead. For instance, Codacy was shown to employ ESLint and PMD as built-in code pattern checkers, all of which are linting tools23. Therefore, we employed numerous linting tools to operationalise code quality across the four dimensions of reliability, readability, performance, and security. The majority of linting tools examined in our study are predominantly designed as command-line interface (CLI) utilities. Consequently, we redirected their outputs to text files in order to systematically store the results. Following this, we employed text mining techniques utilising regular expressions to extract both the SnippetIds and violation names. A number of tools exhibited the capability to detect a significant number of violations. To ensure clarity and focus we report only the ten most frequently detected violations identified by the tools for each dimension. Presenting an exhaustive list would be impractical, as some tools generate hundreds of violations for a single dimension. The definitions and examples for these violations are sourced from the respective tool documentation, unless otherwise specified. Additional results, such as definitions and examples24, as well as tallied occurrences for each state and city25, can be found in our replication package [27]. Our selection of linting tools was tailored to each research question and programming language, ensuring that the chosen tools could effectively identify issues related to the specified quality dimensions, and reflecting similar prior research. Some 23

https://docs.codacy.com/repositories-configure/configuring-code-patterns Replication package » Violations » Reliability to Security 25 Replication package » Results » Tallied Occurrences » Reliability to Security 24

tools may be able to address multiple research questions (i.e., inspect more than one quality dimension), and be supplemented with plugins to further expand their scope. Our tools of choice are outlined in Table 7. Table 7. Tools choice by dimension and language Programming Language SQL

Tool Name

Dimension(s)

Violations to Check

SQLLint [25] SQLFluff [89]

Reliability Readability Performance Security Reliability Readability Performance Security Reliability Readability Performance Security Reliability Readability Security

9 59 21 7 23 31 9 14 33 52 8 62 90 – Lint category 305 – Layout, Naming, and Style categories 5 – Security category 24

SQLCheck [90]

JavaScript

ESLint [10] ESLint Security Plugin* [91]

Python

flake8 [92] flake8-bandit* [93] Rubocop [94, 95]

Ruby Rubocop Performance* [94, 95] PMD [11, 94] Checkstyle [11, 96]

Performance

Reliability Readability Java Performance SpotBugs [11] Security *) Plugins that work in conjunction with the original tool.

220 56 – Google checks 37 – Performance category 41 – Security and Malicious Code categories

In total, there are 1,106 violations to check across all dimensions. Several implementation details are to be considered, given that each tool works differently. Firstly, flake8 and SQLFluff report violations using codes, requiring users to consult each tool’s documentation for interpretation. Next, Meldrum et al. [11] used FindBugs for Performance and Security analysis, but due to its discontinuation, we opted for SpotBugs, its actively maintained successor. Like FindBugs, SpotBugs requires compiled code (.class) to identify violations. Moreover, Rubocop and SpotBugs automatically categorise violations, enabling us to quantify each dimension’s violations by consulting the documentation. For instance, Rubocop’s ‘Lint’ violation category is explicitly defined to scrutinise potential ambiguities and detect error-prone behaviour26. As this specific use-case aligns with our definition of reliability, this category would quantify the reliability dimension for Ruby. Each language’s quality assessment is first conducted by analysing reliability, as our chosen tools in Table 7 can determine the proportion of parsable and unparsable snippets. Afterwards, the readability, performance and security dimensions are assessed. Determining the degree of unparsable snippets is crucial to account for potential inaccuracies in language prediction by Guesslang. Figure 5 exemplifies such a case, displaying SnippetId 104673. SQL_104673.sql RoomId = db.Database.ExecuteSqlCommand("SELECT RoomId FROM dbo.Rooms WHERE RoomCategory = '" + roomCategory + "'"); return RoomId;

Figure 5. Example snippet SQL_104673.sql. Red text denotes SQL syntax This snippet written in C# executes a database query using driver code, and contains embedded SQL syntax (highlighted in red). Consequently, Guesslang misidentified its language and returned SQL instead of C#. Similar limitations are further addressed in Section 6. Our study primarily reports and discusses the top 10 violations for each language, tool, and quality dimension because these represent over 75% of each language’s violations within each dimension, except for the Ruby readability dimension, which covers only 55.99% (addressed further in Section 6). Refer to our replication package 26

https://docs.rubocop.org/rubocop/cops.html#lint

[27] for the complete list of violations27. Aside from brevity, we posit that reporting the top 10 violations effectively presents the overall code quality scene for each region across all four dimensions. Following tool execution, we rigorously examined all 1,106 violations for suitability in snippet-level analysis, employing the method conducted by Meldrum et al. [11]. This ensured each violation was indeed applicable for snippet-level analysis within Stack Overflow, as opposed to the file-level analysis typically used in traditional codebases. We subsequently found 21 violations that were only suitable for file-level code assessments, and thus not applicable to the snippet-level nature of Stack Overflow data. For instance, the violation no-undef by ESLint (JavaScript) flags undefined variables. However, variable definitions within the snippet itself may not be necessary if the question thread presents sufficient context. Figure 6 depicts such a case where the definitions of array and value are not assigned prior to being used, yet contributors can still grasp the main question being asked. Moreover, there are certain violations that were raised by our pre-processing steps when extracting and exporting snippets to their respective file extensions. For example the violation ClassWithOnlyPrivateConstructorsShouldBeFinal by PMD (Java) dictates that classes with private constructors should be declared as final, since such classes may not be subclassed either way. Consequently, private constructors encased in a public class structure—which is our pre-processing step as seen in Error! Reference source not found.—raises this v iolation. Details for these violations are documented in Table 8, and we excluded these violations from the subsequent analysis and findings [11] as they do not reflect a genuine issue within the actual snippets.

Figure 6. Example unassigned variable usage in JavaScript Table 8. Omitted violations for each language and tool Language

Tool

Violation Name

JavaScript

ESLint

no-undef

Python

flake8

F821 (undefined name)

PMD

Java

Checkstyle

SpotBugs 27

Reason for Omission Variable definitions may be omitted when the question thread provides sufficient context.

UnusedPrivateField UnusedPrivateMethod UncommentedEmptyConstructor UncommentedEmptyMethodBody UnusedImports UseUtilityClass ClassWithOnlyPrivateConstructorsShouldBeFinal CloneMethodMustImplementCloneable CloneMethodReturnTypeMustMatchClassName ProperLogger OuterTypeFilename Indentation WhitespaceAround CommentsIndentation OneTopLevelClass JavadocMethod UPM_UNCALLED_PRIVATE_METHOD

Replication package » Results » Code Quality Violations (RQ1)

Irrelevant to analyse at snippet-level [11].

Violations caused by pre-processing steps [11]. Violations caused by pre-processing steps [11]. Irrelevant to analyse at snippet-level [11].

Violating Snippets 280,894 106,163 4,240 4,301 1,139 1,849 0 25,233 2,285 33 49 18 94,395 92,392 29,631 4,762 3,373 99 284

URF_UNREAD_FIELD UUF_UNUSED_FIELD

3.6

Irrelevant to analyse at snippet-level [11]

234 126

Code Snippet Violation Densities (RQ2)

The fundamental essence of our work lies in examining regional variations, to reveal insights on how different states and cities vary in reliable, readable, performant, and secure coding practices. However, relying solely on the number of violations would be unfair as it might disproportionately penalise regions with lower activity, and therefore would yield fewer parsable snippets and fewer violations. Our preliminary investigation revealed that regions with greater technological engagement, like California and New York, yielded more parsable snippets and consequently, a higher number of violations across all dimensions and languages. This has the potential to skew insights in more rural states such as North Dakota or West Virginia. To avoid obscuring such findings, we adopted a “violation density” measure from prior literature, enabling us to uniformly compare the concentration of violations across regions with respect to all four dimensions and five languages [97, 98, 99]. Such a measure is usually computed as the ratio of the number of static analysis violations, divided by the product size (i.e., size of the code) [97]. Typically, lines-of-code (LOC) serves as the product size [98], leading to the intuitive interpretation that a density exceeding 1 indicates more than one violation per line on average. LOC takes into account all code elements, such as logical statements, comments, and blank lines. Yet this approach can be inaccurate for snippets containing numerous blank lines or comments, as counting these non-contributing lines as LOC would skew the density metric and eventually create an unfair assessment. Literature has thus suggested to use logical lines-of-code (LLOC) which only analyses actual lines of code, thereby excluding empty lines and comments [100]. However, Error! Reference s ource not found. readability violations often pertain to whitespace and comments, so therefore utilising LLOC for the readability dimension would render these violations invisible and obfuscate the nuances we were trying to uncover in the first place. In light of this dilemma, we will analyse violation densities with respect to LOC exclusively for the readability dimension, and use LLOC for the other three dimensions. We harnessed a simple Python script to first tally the total LOC for each snippet, followed by a comprehensive parsing process with regular expressions to differentiate between single-line comments, multi-line comments, and whitespace within the code structure. Through this procedure, we effectively strip away extraneous elements, ultimately obtaining the desired LLOC of each snippet. Live demonstrations28 and comment identifiers29 for our regular expressions can be seen in our replication package [27]. Afterwards, we averaged the violation densities of individual code snippets authored by users from each city and state of the US. This approach provided us with a representative average violation density for each region, enabling us to conduct a comparative analysis to identify regional disparities in violation densities.

3.7

Code Snippet Relations (RQ3)

To investigate the relationships between various diversity factors and code quality across different dimensions and programming languages, we conducted Pearson’s correlation analysis at the state level, employing a 5% significance level. This decision was guided by prior research [12] that indicated that only 179 of all US cities satisfy the necessary data quality assumptions. Conducting such an analysis at the city level would likely yield unreliable results due to an incomplete sample. Pearson’s correlation analysis was employed due to its effectiveness in prior research [12, 34] and its robustness to normality deviations. Given our dataset encompasses 51 data points (50 states and the District of Columbia), it is suitable for our scope [101]. Following the calculation of average violation densities per state (used to answer RQ2), these values were merged with state-level diversity indicators obtained from a prior study [12] (Table 2). This process resulted in a comprehensive dataset, where each row represents a distinct state and each column houses state-level information: violation densities across different programming languages and code quality dimensions, alongside the corresponding census diversity indicators. Only correlation pairs with p ≤ 28 29

Replication package » Regex Demonstrations » Regex Demonstrations.txt Replication package » Regex Demonstrations » Comment Identifiers.png

0.05 were kept to ensure meaningful results. All correlation pairs and their corresponding p-values were computed using the Python package SciPy30.

3.8

Code Snippet Themes (RQ4)

Quantitative exploration on all 50 US states and the District of Columbia was conducted for RQ2 and RQ3 to establish an understanding of code quality trends across the US. However, quantitative analyses alone may not provide sufficient depth to fully grasp the nuances of user-provided code snippets. Content analysis (CA) was thus selected as a confirmatory step, offering a qualitative perspective to complement prior quantitative findings. CA is a well-established method widely used in computer science and related fields (e.g., software engineering and information systems), serving as a valuable tool to unveil latent textual patterns that are otherwise often missed by simple quantitative measurements [12, 102]. Studies with similar design often harness CA using either inductive or deductive approaches [103]. Inductive analysis follows a bottom-up strategy, where researchers begin with the data and allow themes and categories to emerge organically, leading to theory development at the conclusion [104]. Conversely, deductive content analysis takes a top-down approach, where pre-existing theoretical frameworks guide the identification of codes and categories. Given the exploratory nature of our research, we opted for inductive CA. Our approach is centred specifically on code snippets within post bodies, excluding other content types like post paragraphs, user profiles, and temporal data. This focus on code snippets ensures consistency across the various code quality dimensions we examine, maintaining alignment with the findings from the three prior RQs. Furthermore, focussing on only code snippets mitigates biases that might rise from user attributes (e.g., age, interests, or profession), ensuring the impartiality of subsequent findings. Lastly, this approach draws inspiration from prior research that employed inductive CA on users’ posts’ paragraphs to uncover novel insights [13]. We thus apply the same technique to users’ code snippets, offering a fresh perspective on user-generated content through direct analysis of the code instead. Following established practices [105], this study defines three key units for content analysis: sampling unit, recording unit, and context unit. The sampling unit refers to each individual data instance that is chosen for examination. The recording unit is the text component that is subjected to categorisation during the coding phase, whereas the context unit denotes the text necessary to establish additional context for the recording unit [104]. Our study selected individual code snippets from answer posts as the sampling unit, complementing previous RQs and allowing thematic investigation embedded within user-provided code solutions. Additionally, answers are the central content of any CQA platform [106]. Naturally, the recording unit is the code snippets themselves, because they constitute the core of user solutions and are commonly found in both open-source and commercial software [11]. Finally, question threads are designated as the context unit. The rationale is that analysing code snippets within the context of a question thread may provide additional insights into how they contribute to (or deviate from) the central line of inquiry. A manual qualitative analysis of all code snippets across all states would be impractical, as all five languages tally to 1,983,400 snippets. Given our study’s focus on regional variations in coding practices, we therefore employed a targeted sampling approach to select which states would be subjected to CA. This selection process considered factors relevant to both the technological landscape and workforce diversity within each state. Factors to consider are the tech workforce concentration, LOC and LLOC volume, the volume of tech job postings, tech job added, the economic impact of the tech sector, and Simpson’s diversity index for that state – which served as a measure of racial and ethnic diversity among tech workers in a state [107]. Higher values indicate more even representation of otherwise underrepresented minorities (e.g., African American or Asian). Based on previous work [13] we also consider user base, urbanisation level, technology workforce concentration, R&D activity, and educational attainment. This ensures that our selection is not arbitrary, enabling a heterogeneous representation across manifold diversity facets. Subsequently, California, Utah, and North Dakota were selected as study sites due to their diverse profiles aligning with these key considerations. For instance, the state California boasted 1,487,864 tech jobs in 2022, whilst Utah had 123,450 and North Dakota only 13,272 within the same year (see Table 9 for details). This selection of states aligns with prior research [13], where these same geographical locations were chosen as study sites.

https://scipy.org

Table 9. States to be subjected to content analysis State California Utah

Tech Employment

Avg. LOC per Snippet

Avg. LLOC per Snippet

Tech Business Concerns

Economic Impact to Tech Sector

1,487,864

14.575

11.801

55,868

16.7%

123,450

15.379

12.532

9,286

10.0%

North 13,272 16.247 13.212 1,219 3.1% Dakota * Calculated as 1 − 𝐷 [108]; not to be confused with Simpson’s Reciprocal Index 1/𝐷.

Growth of Tech Job Openings +38,186 (+2.6%) +5,130 (+4.3%) +324 (+2.5%)

Simpson’s Diversity Index* 0.649 0.336 0.216

An interesting observation emerged when comparing LOC and LLOC within Table 9. California exhibited the lowest LOC and LLOC values, while North Dakota had the highest. This finding appears to align with established practices within the Stack Overflow community, which generally favours more concise and succinct code implementations [109]. Reflecting on the community’s preference, California’s code might be considered “better” despite having lower overall LOC and LLOC volumes. Following established practices [110], we adopted a probabilistic random sampling and selected 385 snippets for each state and each dimension. It is worth noting that our CA does not differentiate languages within each dimension, as findings in RQ1 revealed that violations largely correspond to the same thematic categories across all five programming languages. For instance, readability violations consistently involve aspects like code clutter and improper whitespace, while security vulnerabilities mainly revolve around unvalidated user inputs that could lead to injection attacks. This means that there would be 12 sets (4 dimensions × 3 states) where each contains 385 sample snippets to be subjected to CA. Specifically, we sampled only those where Guesslang’s language identification predicted one of either SQL, JavaScript, Python, Ruby, or Java. Thus, for each dimension, there would be a total of 1,155 code snippets (385 for each state). Within each dimension, the first two authors conducted a pilot reliability test to calibrate inter-coder consistency and subsequently draft the coding themes [111]. Development of the initial set of coding themes was supplemented by the use of Google’s Gemini 1.5 Pro generative language model, chosen for its superior reasoning capabilities compared to similar models like ChatGPT [112]. It is important to emphasise that Gemini only served as a complementary aid, and that the final coding decisions remained the responsibility of the researchers. A 10% subsample was drawn for this purpose [111, 113], resulting in a total of 116 code snippets. The researchers then analysed the content of this subsample across two iterations of the pilot test [114], identifying emergent codes which would then form the first set of coding themes. The second iteration then refined this obtained set, merging similar themes, renaming themes for clarity, and making necessary adjustments to ensure a balance between generalisability and specificity. For instance, under the Security dimension, the themes Exec Usage and Eval Usage were merged into the theme of Code Evaluation, as both constitute parsing arbitrary strings as code. Similarly, the Performance dimension saw the introduction of Usage of Inefficient Methods to identify situations where faster methods can substitute certain implementations. Each emergent coding theme was assigned the value 1 if said theme was apparent within the code snippet, and 0 otherwise. A final coding theme Not Coded was added to facilitate code snippets that either could not be assigned to any of the emergent categories [22], or that did not raise any violations. Following each iteration, coders discussed differences to build consensus [113]. Lastly, Cohen’s Kappa (κ) coefficient was used to calculate inter-coder reliability [115]. Table 10 documents how the coding themes evolved across both iterations as well as the resulting κ values. Our replication package [27] contains all reliability test results31. We employed pairwise chi-squared tests to ascertain thematic variations across user-generated content from the three states across all quality dimensions [13, 22]. The null hypothesis assumed uniformity in thematic prevalence across the states at p < 0.05. Finally, theme-specific frequency and percentage analyses quantitatively explicate the extent and direction of state-level divergences in code snippet violations. 31

Replication package » Content Analysis (RQ4) » Reliability to Security

Table 10. Evolution of coding themes 1st Iteration Dimension

No. of Themes

Pilot κ

19 14 7 15

0.762 0.772 0.784 0.800

Reliability Readability Performance Security

2nd Iteration No. of Pilot Themes κ 17 0.823 13 0.796 8 0.845 10 0.855

Final κ 0.839 0.825 0.859 0.869

RESULTS

We present a holistic overview in Table 11 by showcasing the total number of parsable and unparsable code snippets for each language. Furthermore, for Java, we record the proportion of compilable snippets within its parsable pool. Table 11. Code snippet breakdown Language SQL JavaScript Python Ruby Java

Initial Pool 776,836 486,175 329,396 206,090 184,903

Unparsable

Parsable

Uncompilable

Compilable

418,850 145,750 101,347 91,646 86,421

357,986 340,425 228,049 114,444 98,482

– – – – 90,032

– – – – 8,450

Despite our analysis being limited to 8,450 compilable Java snippets from the US, this sample size is likely sufficient to draw meaningful conclusions. Prior research by Meldrum et al. [11] achieved good results using 8,010 snippets which constitutes snippets from all countries (i.e., not only US). This suggests that our sample size that only encompasses those from the US can still yield robust findings. Nonetheless, we still acknowledge this small subset as a limitation (see Section 6). Table 12 provides a comprehensive overview of the total number of occurrences for each of the top 10 violations across all languages, tools, and dimensions. Certain entries may not reach 10, however, due to limitations of the specific tools. For example, SQLLint, used for evaluating SQL reliability, only identifies 9 violations, and thus only 9 entries are included in the table for that tool. For violations beyond the top 10, results are documented in our accompanying replication package [27]32.

4.1

Code Snippet Quality (RQ1)

This section presents results for RQ1: What is the quality of code snippets across different programming languages on Stack Overflow? We first observe patterns in the reliability dimension. Given that code snippets on Stack Overflow are generally presumed to offer correct solutions and not intentionally misleading information [11], our definition for reliability is the code should not be broken, introduce bugs, or be prone to errors. Within SQL, only a small portion (10,090; 2.81%) were found to raise at least one reliability violation. From Table 12, the most common violation is unmatched-parentheses, which occurred 31.33% times, indicating that users tend to overlook whether parentheses are opened and closed properly when constructing queries. This was followed by my-sql-invalid-create-option (23.73%) and hungariannotation (17.58%). The former hints that users tend to erroneously use the CREATE statement when constructing tables, whereas the latter indicates continued reliance on Hungarian notation despite its declining popularity in favour of standardised SQL conventions [116].

Replication package » Code Quality Violations (RQ1)

Table 12. Violations’ occurrences and definitions for each language, tool, and dimension. Lang.

Tool

Violation Name

Definition

Occurrences

3,612 2,735 2,027 1,487 972 383 228 78 6 989 522 215

31.33 23.73 17.58 12.9 8.43 3.32 1.98 0.68 0.05 45.31 23.91 9.85

153

7.01

94 45 33 35 25 26

4.31 2.06 1.51 1.60 1.15 1.19

20,182

60.61

6,323 2,230 2,111 1,098 287 225 125 184 114 19,352

18.99 6.70 6.34 3.30 0.86 0.68 0.38 0.55 0.34 49.92

5,955

15.36

2,724 1,981 1,214 558

7.03 5.11 3.13 1.44

SQLLint

unmatched-parentheses my-sql-invalid-create-option hungarian-notation missing-where my-sql-invalid-alter-option my-sql-invalid-drop-option invalid-limit-quantifier my-sql-invalid-truncate-option odd-code-point no-prototype-builtins no-cond-assign no-global-assign no-unreachable

ESLint

JavaScript

SQL

Reliability Dimension

no-inner-declarations no-dupe-keys valid-typeof no-control-regex no-const-assign no-async-promise-executor

flake8 Rubocop

Ruby

Python

F405 (name may be undefined) W605 (invalid escape) E722 (bare except) F706 (return outside function) F722 (forward annotation error) F704 (yield outside function) F632 (is used for literals) F823 (undefined local) F633 (print redirection) F701 (break outside loop) UselessAssignment Void NumberConversion UnusedBlockArgument UnusedMethodArgument MultipleComparison

Number of parentheses is unbalanced. A CREATE statement is being given an invalid option. Queries contain the identifier sp_ or tbl_ A WHERE clause is absent from the DELETE statement. An ALTER statement is being given an invalid option. DROP statements are given an invalid option. Non-numerical inputs are present inside LIMIT statements. TRUNCATE statements are given an invalid option. Queries contain unsupported code points. Usage of certain Object.prototype methods that can produce unexpected results. Assignment operators occur in conditional expressions. Assignment to built-in global variables (e.g., undefined) Presence of statements located after the return, throw, break, and continue keyword. Declaration of functions and variables in nested blocks. Duplicate keys within object literals. Typos in string literals within typeof function (e.g., “strnig” instead of “string”) Use of escape characters (e.g., \x00 or \x1F) in regular expressions. Reassigning a const variable. Use of async function with promise executors. Seemingly-undefined variable usage ambiguously imported with the wildcard import statement. Usage of a backslash-character pair that does not form a valid escape sequence. Usage of a bare except clause. Usage of return statement outside functions. Syntax error in type annotation (type hinting). Usage of yield statement outside functions. Comparison of string, byte, or integers using is or is not keywords. References of local variables prior to their definition. Usage of >> operator is used with the print() function. Usage of break keyword outside loops. Unused variables within local scopes. Usage of operators, variables, literals, lambdas, and nonmutating methods in void context. Number conversions with the to_i, to_f, to_r, or to_c methods. Block arguments are not utilised anywhere in the block. Method arguments are not utilised within that method. Comparisons of three or more values within a single line.

Lang.

Tool

Violation Name AmbiguousBlockAssociation AssignmentInCondition TopLevelReturnWithArgument AmbiguousOperator LocalVariableCouldBeFinal MethodArgumentCouldBeFinal SystemPrintln

PMD

Java

LawOfDemeter ImmutableField ConstructorCallsOverridableMethod AvoidInstantiatingObjectsInLoops DoNotUseThreads UnusedLocalVariable TestClassWithoutTestCases

Definition A parameter is passed in a method without parentheses. If/while/until blocks contain an assignment operator (=) rather than the equality operator (==). A non-empty return statement is used in the top-level object. Ambiguous operators are found in the first argument of a method invocation without parentheses. A local variable assigned only once is not declared as final. A method argument is not declared as final. Usage of System.out.print or System.err.print instead of logging statements. A class is interacting with the internals of related objects or their collaborators (tight coupling). Private non-final fields whose value never changes are not marked as final. A non-private constructor calls an overridable method from within a class. New objects are instantiated within loops. Threads are directly used within J2EE components (Java 2 Platform, Enterprise Edition) A local variable is declared and/or assigned, but not used. Test classes do not contain any test cases nor test methods.

Occurrences 504

% 1.30

462

1.19

429

1.11

390

1.01

169,031 139,483 52,108

31.88 26.31 9.83

24,025

4.53

17,839 11,048 6,455

3.36 2.08 1.22

6,179

1.17

6,017 5,569

1.14 1.05

Incorrect indentation in the query. Inappropriate spacing, such as excessive or trailing whitespace. Multiple SELECT targets are placed on the same line, except when there’s only one target. Capitalisation inconsistencies of unquoted identifiers, such as column names. Tables are aliased without using the AS keyword. Capitalisation inconsistencies of keywords, such as SELECT or FROM. A query generates an unspecified number of result columns. Excessively long queries without a line break, leading to complex and hard-to-read queries. Column expressions are implicitly aliased without using the AS keyword. Inconsistent references as prefixes. Unused variable, contributing to code bloat. Semicolons that are unnecessarily written. Redeclarations uses var keyword. Escape character usage (backslash) when there is nothing to escape. Declared but unused labels. Constant expressions in conditional statements. Unicode whitespace characters are used yet not visually distinguishable from regular spaces. Occurrences of empty block statements. Usage of certain operators (such as && and :) consecutively within a single line. JavaScript global objects and restricted identifiers are renamed.

176,834 150,405

15.27 12.99

130,087

11.23

105,311 99,896 57,373 55,382

9.09 8.63 4.95 4.78

45,886

3.96

32,573 28,894 94,604 2,857 2,559 2,514 845 379

2.81 2.50 90.12 2.72 2.44 2.40 0.81 0.36

353

0.34

262 175 141

0.25 0.17 0.13

Readability Dimension LT02 (layout.indent) LT01 (layout.spacing)

SQLFluff

SQL

LT09 (layout.select_targets) CP02 (capitalisation.identifiers) AL01 (aliasing.table) CP01 (capitalisation.keywords) AM04 (ambiguous.column_count)

ESLint

JavaScript

LT05 (layout.long_lines) AL03 (aliasing.expression) RF03 (references.consistent) no-unused-vars no-extra-semi no-redeclare no-useless-escape no-unused-labels no-constant-condition no-irregular-whitespace no-empty no-unexpected-multiline no-shadow-restricted-names

Lang.

Tool

Violation Name E231 (no space around delimiters) E501 (line too long) E111 (indentation invalid multiple)

flake-8

Python

E302 (no blank line after functions) W291 (trailing whitespace) E225 (no space around operator) E251 (space around default keywords) E261 (two spaces before inline comment) E265 (no space after pound) E128 (continuation under-indented) Style/StringLiterals Style/MethodCallWithArgsParentheses Style/DocumentationMethod

Rubocop

Ruby

Style/HashSyntax Layout/SpaceAroundOperators Layout/TrailingWhitespace Style/Documentation Style/TopLevelMethodDefinition Layout/IndentationWidth Layout/SpaceAfterComma MissingJavadocMethod LeftCurly LineLength

Checkstyle

Java

ParenPad EmptyLineSeparator CustomImportOrder NeedBraces MemberName AbbreviationAsWordInName AvoidStarImport

Definition Absence of whitespace after the comma, colons, or semicolons. Lines that span longer than 79 characters. Indentation used is not a multiple of four spaces. Two blank lines are not found between functions and classes declared in the global scope. Presence of trailing whitespace after the final character in a line. Absence of a single whitespace around comparison operators. Presence of spaces before or after the “=” sign in a function definition. Presence of insufficient spacing before inline comments. Absence of whitespace before the pound sign and the comment itself within block comments. Insufficient indentation for continuation lines in a supposedly-single line statement. Double quotation marks (") are used to construct a string literal. Absence of parentheses in method calls containing parameters. Absence of documentation comments for public methods. Inconsistent use of the Ruby 1.9 syntax (e.g., {a: 1}) when the hash’s keys are of Symbol type. Absence of whitespaces around arithmetic, assignment, comparison, and logical operators. Presence of trailing whitespace. Absence of top-level documentation of classes and modules, except for empty classes and modules. Top-level method declarations are not organised within their appropriate classes. Indentation does not comply to a multiple of 2 spaces. Comma not followed by any space. Absence of Javadoc comments for a method or constructor.

Occurrences 267,156 101,985 74,068

% 29.46 11.25 8.17

65,804

7.26

61,904 53,920 47,977 33,228

6.83 5.95 5.29 3.66

28,457

3.14

16,756

1.85

74,910 45,401 34,826

13.56 8.22 6.30

33,779

6.11

23,010

4.16

22,406

4.05

21,509

3.89

19,622 17,131 16,854 55,304

3.55 3.10 3.05 17.45

Placement of left curly braces (“{“) for code blocks is not at the end of line. Lines that span longer than 100 characters. Inconsistent padding around parentheses in method or constructor declarations, method calls, and conditionals. Absence of empty line separators before package, imports, fields, methods, classes, and static/instance initialisers. Import statements in a source file do not follow any specified order.

46,318 27,312

14.61 8.62

26,556

8.38

20,869

6.59

19,011

6.00

Absence of braces inside single-line statements. Instance variable names do not match to the regular expression ^[a-z][a-zA-Z09]*$. Variable, method, or class names contain abbreviations that are treated as separate words. import statements uses the * notation.

16,581

5.23

14,525

4.58

12,037

3.80

10,081

3.18

Lang.

Tool

Violation Name

Definition

Occurrences

48,257 22,592 19,063 8,564 5,124 4,151 1,870 1,214 760 563 2,559

42.85 20.06 16.93 7.60 4.55 3.69 1.66 1.08 0.68 0.50 93.57

SQLCheck

SQL

Performance Dimension SELECT * Spaghetti Query Alert UNION Usage Implicit Column Usage Generic Primary Key Metadata Tribbles Index Attribute Order Multi-Valued Attribute ORDER BY RAND Usage DISTINCT & JOIN Usage no-redeclare

ESLint

JavaScript

no-debugger no-fallthrough no-misleading-character-class for-direction require-yield no-useless-backreference

flake8

Python

no-invalid-regexp F401 (unused imports) F841 (unused variable) E402 (import placement) F403 (wildcard imports) E712 (comparison to bool) F811 (redundant imports) F842 (unused annotation) F622 (multiple starred expressions)

Rubocop Performance

Ruby

RegexpMatch StringReplacement RedundantBlockCall RedundantMatch TimesMap CompareWithBlock RedundantMerge RangeInclude BindCall

Query does not return an explicit number of columns. Presence of large spaghetti queries. Usage of UNION instead of UNION ALL. Wildcard usage and unnamed columns in querying. A generic name of primary key column (e.g., Id) for tables exist. Mixing of metadata with data. Query attributes are not in the same order as the index attributes. An attribute has multiple values for a single instance of an entity. Query selects a randomised set of rows from the result. Query contains the DISTINCT and JOIN clauses, instead of EXISTS. Redeclaration of variables using the var keyword. Usage of debugger statements, which stops execution at the current point in the code. Absence of break keyword within switch-case statements. Usage of certain character classes (such as Emojis) in regular expressions. Presence of infinite for-loops due to a wrong loop direction. Usage of yield keyword outside functions. Unnecessary backreferences in regular expressions. Usage of invalid regular expression patterns (e.g., “[“ or “.”) in RegExp constructors. Presence of imported but unused modules. Defined but unused local variables. Module-level imports are not located at the top of the file. Usage of the from module import * statement (known as wildcard import). Comparison of a variable to the Boolean value True. Modules are imported multiple times in the same namespace. Defined but unused local, annotated variables. Use of multiple starred expressions in assignment statements, such as a, *b = *c, d. Usage of conventional match methods within regular expressions. Usage of gsub() method with string literals, which should be replaced by tr or delete. Usage of &block and block.call in generators, which should be replaced by yield. Usage of Regexp#match or String#match instead of using =~. Usage of times.map, where an explicit array creation is a recommended substitute. Usage of sort { |a, b| a.foo <⇒ b.foo } which should be replaced by sort_by(&:foo). Usage of Hash#merge! – which should be replaced by Hash#[]=. Usage of Range#include? and Range#member?, which should be replaced with Range#cover? Usage of bind(obj).call(args, …) instead of bind_call(obj, args, …).

94 51 14 9 5 2

3.44 1.87 0.51 0.33 0.18 0.07 0.04

1 18,933 6,555 4,923 4,095 1,317 774 3

51.73 17.91 13.45 11.19 3.60 2.12 0.01

0.00

466

28.68

238 168 131 124 64 62 59 51

14.65 10.34 8.06 7.63 3.94 3.82 3.63 3.14

Lang.

Tool

Violation Name Casecmp

Definition Case-insensitive string lexicographical comparisons are not done with casecmp.

SBSC_USE_STRINGBUFFER_CONCATENATION SIC_INNER_SHOULD_BE_STATIC_ANON

Presence of string concatenation using the ‘+’ operator in a loop. An anonymous inner class is not declared as static. A class contains an instance final field that is initialised to a compile-time static value. An inner class (not necessarily anonymous) is not declared as static. The nextInt() method is not used in random number generation. A boxed primitive is created from a String, just to extract the unboxed primitive value. The constructor of the java.lang.String class is used to create a new string object. Explicit calls to garbage collection. Creation of an empty strings using new String() constructor. Usage of String.toString() operation.

SpotBugs

Java

SS_SHOULD_BE_STATIC SIC_INNER_SHOULD_BE_STATIC DM_NEXTINT_VIA_NEXTDOUBLE DM_BOXED_PRIMITIVE_FOR_PARSING DM_STRING_CTOR DM_GC DM_STRING_VOID_CTOR DM_STRING_TOSTRING

Occurrences

% 3.02

49 58 46

25.89 20.54

11.16

21 19

9.38 8.48

8.04

18 4 3 3

8.04 1.79 1.34 1.34

48,257 47,652 22,592 3,695

38.22 37.74 17.89 2.93

3,565 499

2.82 0.40

SQLCheck ESLint Security Plugin flake8bandit

Python

JavaScript

SQL

Security Dimension SELECT * NULL Usage Spaghetti Query Alert String Concatenation Imprecise Data Type Readable Passwords object-injection non-literal-regexp non-literal-fs-filename unsafe-regex eval-with-expression non-literal-require new-buffer possible-timing-attacks child-process bidi-characters S311 (pseudorandom) S101 (assert usage) S113 (request without timeout)

Query does not return an explicit number of columns. NULL is used to conduct comparison operations. Presence of large spaghetti queries. Usage dynamic of string concatenation on querying. Usage of FLOAT instead of DOUBLE and REAL when inserting values for the database. Password are stored in plain-text, or is decipherable by a cryptographic function. Presence of square bracket notation in objects as a left- or right-hand assignment operand. Dynamic variable usage in RegExp() functions, rendering expressions non-literal. Usage of module33 that is made with a dynamic filename argument. Usage of unsafe regular expressions, which may block the event loop. Usage of eval(), which executes a string of characters as code instead. Non-literals (e.g., variables or user input) being passed into the require() mechanism. A new Buffer object is created with a non-literal value for its length. Usage of insecure comparisons (==, !=, !== and ===) to check input sequentially. A call to the child_process module is made with a non-literal argument. Usage of bidirectional characters, such as RLO34 and PDI35. Usage of standard pseudo-random generators, specifically those from the random standard library. Usage of assert keyword as a safeguard mechanism. A request with the requests library is made without specifying a timeout.

https://nodejs.org/api/fs.html https://www.compart.com/en/unicode/U+202E 35 https://www.compart.com/en/unicode/U+2069 34

80,582 1,363 916 800 798 793 241 99 56 2

94.08 1.59 1.07 0.93 0.93 0.93 0.28 0.12 0.07 0.00

6,708

23.40

4,194 3,143

14.63 10.96

Lang.

Tool

Violation Name S603 (subprocess without shell)

Rubocop

Ruby

S607 (process spawned with partial path) S404 (subprocess imports) S310 (urlopen with file scheme) S105 (hardcoded passwords) S108 (hardcoded temp) S605 (process spawned with shell) Open Eval YAMLLoad MarshalLoad JSONLoad MS_SHOULD_BE_FINAL MC_OVERRIDABLE_METHOD_CALL_IN_CONSTRUCTOR MS_FINAL_PKGPROTECT

SpotBugs

Java

MS_EXPOSE_REP EI_EXPOSE_REP EI_EXPOSE_REP2 MS_PKGPROTECT DP_DO_INSIDE_DO_PRIVILEGED REFLC_REFLECTION_MAY_INCREASE_ACCESSIBILITY_OF_CLASS DMI_CONSTANT_DB_PASSWORD

Definition Usage of subprocess module without explicitly setting shell=True for command execution. Filesystem paths do not start at the root (i.e., lack of a leading forward slash character). Importation of the subprocess standard library to spawn child processes. Usage of the urlopen() function from the urllib standard library. String literals resemble hardcoded passwords. Usage of hardcoded temporary file or directory paths. Calls that start a process with a shell in an unsafe manner. Usage of Kernel#open and URI.open with dynamic data (i.e., unsanitised input). Usage of Kernel#eval and Binding#eval methods, which evaluates strings as code. Usage of YAML class methods to load YAML objects. Usage of Marshal class methods, especially the methods load and restore. Usage of JSON class methods, especially the methods load and restore. Public static fields are not declared as final. An overridable (non-static, non-final, public) method is called within a constructor. A mutable public static field is neither declared as protected nor final. A public static method returns a reference to an array that is part of the static state of the class. A mutable internal object is exposed by pointing a reference to it – from an object or a publicly-accessible field. Code stores a reference to an externally mutable object into the internal representation of the object. Presence of a mutable, non-protected static field. Method invocations require a security permission check. The invocation needs to occur inside a doPrivileged block. Usage of reflection to increase accessibility of classes, methods or fields. A database password is hardcoded into the code as a string.

Occurrences

1,785

6.23

1,470

5.13

1,449 1,427 739 685 589 561 272 161 24 22 65 38 12

5.05 4.98 2.58 2.39 2.05 53.94 26.15 15.48 2.31 2.12 40.12 23.46 7.41

6.17

5.56

1.85

2 2

1.24 1.24

JavaScript snippets showed much fewer reliability violations compared to SQL where only 1,768 (0.52%) managed to flag at least one. The most common problem was no-prototype-builtins, accounting for 45.30% of all violations, possibly due to habits formed during the migration to ECMAScript 5.136. This was followed by no-cond-assign (23.91%) and no-global-assign (9.85%). The former signals a proclivity for assignments within conditional blocks. The latter, while less frequent, is somewhat more severe: inadvertently modifying built-in global variables (e.g., undefined) which may have detrimental consequences and lead to unanticipated behaviour. Python showed a moderate amount of reliability violations with 10,234 (4.49%) across all 228,049 parsable snippets. F405 (name may be undefined) occurred the most, accounting for 60.61% of all violations, suggesting the usage of variables prior to its definition – likely due to the usage of wildcard, implicit imports. W605 (invalid escape) followed closely at 18.99% as well as E722 (bare except) at 6.7%. These figures hint at the widespread misuse of escape characters and omission of proper exception handling in try-except blocks. The latter may obscure the true cause of errors and thus make the code behave unexpectedly [11]. Ruby code exhibited more reliability violations at 24,302 (21.23%) across all 114,444 snippets. UselessAssignment occurred the most (49.92%) followed by Void (15.36%). The former suggests a widespread adoption of declaring variables that ultimately remain unused, whereas the latter is more severe; such practices would mask underlying program logic due to syntax errors within the enclosing code block. NumberConversion accounted for 7.03% of all violations, suggesting users tend to use less resilient number conversion techniques, leading to unexpected numerical outcomes37. Finally, 84.36% of Java snippets (83,080 of 98,482 parsable) raised at least one violation. LocalVariableCouldBeFinal is the most prevalent – covering 31.88% of all violations – hinting a widespread practice to avoid declaring variables as final despite being assigned only once, posing risks to immutability (or lack thereof) [117]. MethodArgumentCouldBeFinal (26.31%) and SystemPrintln (9.83%) were found next. The former is similar to LocalVariableCouldBeFinal, whereas the latter highlights users’ penchant to forgo Java’s logging functionality and instead redirect outputs to their standard output streams. Regarding readability, code should adhere to the readability conventions specific to each programming language, and also be readable to minimise confusion encountered by fellow programmers in an effort to minimise technical debt [11]. In SQL, 262,442 of 357,986 parsable snippets violated readability (73.31%). LT02 (layout.indent) was the most widespread, covering 15.27% of all violations, followed by LT01 (layout.spacing) at 12.99%. Both violations suggest a widespread lack of attention to consistent spacing with respect to the code’s visual structure. Additionally, LT09 (layout.select_targets) covered 11.23%, suggesting a tendency among users to cram multiple columns into a single line in SELECT statements. This practice may quickly hinder comprehension if the query is complex. 72,858 (21.40%) JavaScript snippets violated readability conventions. no-unused-vars is the most-occurring, apparent in 90.12% of all violations, followed by no-extra-semi in 2.72%. Both contribute to code bloat, especially the latter given that JavaScript syntax does not necessitate semicolons at the end of statements. noredeclare covered 2.44% violations, indicating that users are inclined to redeclare variables. For Python, 157,993 snippets (69.28%) returned at least one readability violation. E231 (no space around delimiters) was the most widespread, at 29.46%, implying that users tend to omit spacing between punctuation. E501 (line too long) followed suit at 11.25% and E111 (indentation invalid multiple) emerged at 8.17%. The former may be attributed to users’ tendency to write long Python oneliners, inhibiting the specific elements’ identification. The latter indicate inconsistencies in indentation depth, obscuring visual separation. In Ruby, 97,725 snippets (85.39%) raised at least one violation. Style/StringLiterals is the most common, accounting for 13.56% of violations. This observation suggests users tend to use double quotation marks for constructing string literals instead of singles. While both styles are syntactically valid, adhering to a 36 37

https://262.ecma-international.org/5.1 https://0.30000000000000004.com/#ruby

consistent style (i.e., single quotes) enhances comprehensibility. Style/MethodCallWithArgsParentheses appeared in 8.22% and Style/DocumentationMethod in 6.30%. These findings indicate that users frequently included unnecessary parentheses in method calls and omitted documentation comments for public methods, making complex code harder to understand. Lastly, 72,169 (73.28%) of Java snippets violated readability. The most recurring violation is MissingJavadocMethod (17.45%), indicative of users’ lack of attention to include Javadoc comments where necessary. This was then succeeded by LeftCurly (14.61%) and LineLength (8.62%). These hint that users frequently started code blocks on new lines, yet also tend to compose lines exceeding 100 characters. Extended lines may hinder effective scanning, making the code challenging to understand. In terms of the performance dimension, snippets should be efficient in performing the intended task with minimal processing steps, in a way that would exhibit efficacy. For SQL, 89,121 (24.90%) returned at least one performance violation. SELECT * was the most common one (42.85%) which corroborates prior findings that database programmers tend to favour wildcard selects [118], ostensibly driven by a reluctance to manually type out column names. Following closely behind were Spaghetti Query Alert (20.06%) and UNION Usage (16.93%). These findings hint at users’ preference to write complex, tightly-coupled queries, as well as their lack of awareness regarding the more efficient UNION ALL statement, which achieves the same functional outcome as UNION without performing duplicate elimination [118]. Interestingly, only a small portion (1,776; 0.52%) of JavaScript snippets raised at least one performance violation. The most notable issue is no-redeclare (93.56%), identifying users’ tendency to redeclare variables using the var keyword. no-debugger was observed in 3.44%, indicative of debugger statements within snippets, while no-fallthrough appeared in 1.86%. This is indicative that users tend to forgo break keywords in switch-case statements. Similarly for Python, a small portion (22,505; 9.87%) of all snippets violated performance rules. F401 (unused imports) is the most occurring violation (51.73%) which hints at the prevalence of unused package imports, creating performance overhead. In a similar fashion, F841 (unused variable) followed at 17.91%, suggesting that defined variables are mostly left unused. E402 (import placement) came next at 13.45%, indicating that users do not place import statements at the top of their snippets. Under Ruby, only 1,322 snippets (1.16%) violated performance rules. The most common problems pertain to the widespread usage of slower methods despite faster alternatives being available. For one, RegexpMatch was observed the most (28.68%), suggesting the widespread usage of conventional match instead of the faster match?. Another example is StringReplacement at 14.65%, hinting a widespread usage of gsub methods in string literals despite alternatives like tr or delete. RedundantBlockCall followed at 10.34% which implies users’ tendency to use &block and block.call in generators instead of the widely-accepted yield keyword. Regarding Java, out of all 8,450 compilable snippets, 198 (2.34%) returned at least one performance violation. The most occurring violation is SBSC_USE_STRINGBUFFER_CONCATENATION (25.89%), implying inefficient string concatenation using loops and the plus operator. Other prevalent violations largely pertain to the lack of static modifiers, such as SIC_INNER_SHOULD_BE_STATIC_ANON (20.54%), SS_SHOULD_BE_STATIC (11.16%), and SIC_INNER_SHOULD_BE_STATIC (9.38%). Finally, security vulnerabilities are defined as codes that can be exploited by malicious third-party attackers. We inspected the degree of security hotspots raised for each language. First, 103,892 (29.02%) of parsable SQL snippets raised at least one violation. Similar to the performance dimension, SELECT * was the most frequent (38.22%), followed by NULL Usage (37.74%), which is raised when users use NULL in comparison expressions. Spaghetti Query Alert was also prevalent (17.89%) hinting at the continued usage of complex queries. For JavaScript, 39,215 snippets (11.52%) had security vulnerabilities. The most frequent issue was object-injection (94.08%), hinting at the almost-universal practice of using square bracket notation in objects as operands. The violations non-literal-regexp (1.59%) and non-literal-fs-filename (1.07%)

pertain to dynamic string literals. The former relates to dynamic inputs in regular expressions whereas the latter relates to file paths. Within Python, 18,025 snippets (7.90%) raised security violations. S311 (pseudorandom) stood out as the most prevalent (23.4%), presenting as a stark contrast to JavaScript, given that none of its violations pertain to pseudo-random generators. This finding suggests that Python developers have a higher tendency to utilise dedicated pseudo-random number generators. S101 (assert usage) followed closely (14.63%), points to the frequent usage of assert statements. S113 (request without timeout) appeared in 10.96% of all violations, hinting at the usage of requests without timeout mechanisms. Ruby violations are much less widespread, with only 944 (0.82%) security hotspots. Open was the most frequent hotspot (53.94%) highlighting the frequent practice of opening URLs directly with unsanitised inputs. Related to code injection, Eval was also frequent (26.15%) which is raised whenever string literals are evaluated as code. YAMLLoad followed next (15.48%), flagging the unsafe parsing and loading of YAML objects. In Java, 107 (1.27% out of all compilable) snippets exhibited security violations. Most violations relate to access modifiers. MS_SHOULD_BE_FINAL was the most prevalent (40.12%), and signifies users’ penchant to omit final non-access modifiers in public static fields. MC_OVERRIDABLE_METHOD_CALL_IN_CONSTRUCTOR at 23.46% implies overridable methods are often called within class constructors. MS_FINAL_PKGPROTECT was flagged in 7.41% of all instances, again hinting at the omission of final and protected within mutable public static fields.

4.2

Code Snippet Violation Densities (RQ2)

This section presents results for RQ2: How do coding practices among US contributors differ across states and cities in the United States? To delve beyond initial insights, we shift towards a fine-grained examination of code quality variations within US states and cities. While comprehensive city-level results are available in our replication package [27]38, Table 13 highlights that states with established tech hubs (California, New York, Washington, Massachusetts, Texas) consistently produce more total and parsable code snippets in all five studied languages, which may be explained by their proportionately larger pools of users. To enable meaningful comparisons, we divided the total number of snippets for each state by the number of users in that state to get the average number of snippets per user in that particular state, with higher values indicating higher snippet output. Major tech hubs like California and New York consistently produce a high volume of code snippets, yet they do not necessarily exhibit the highest snippet output per user. Instead, these states typically fall within the upper-middle range for each language. For instance, an average user in California would output 0.903 Python snippets (0.576 parsable), both figures being the 7th highest nationwide. Interestingly, rural states often exhibit either the highest or lowest snippet output nationwide, occupying both extremes. Continuing our Python example, an average user in Alaska would have 1.466 snippets (1.169 parsable), which are the highest figures nationwide. Yet at the same time, users of North Dakota on average contribute 0.234 Python snippets (0.142 parsable), both being the lowest figures across all states. This trend is consistent for other languages as well (refer to our replication package [27] for the full snippet-per-user results39). These findings suggest that snippet output is not solely determined by the presence of technology hubs or whether a state is considered more rural or more urban. To account for potential biases and ensure a fair comparison of code quality across regions, we employed violation density (see Section 3.6). This metric controls for the influence of parsable code volume, preventing regions with more snippets (e.g., California) from appearing to have higher violation rates and thus erroneously suggesting worse code quality.

38 39

Replication package » Results » City-Level Parsables.xlsx Replication package » Results » Snippets per User.xlsx

Table 13. Parsable snippets for each language and state State Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming

SQL Snippets (Parsable) 3,013 (1,045) 1,592 (608) 18,426 (10,722) 3,709 (1,751) 165,304 (57,102) 35,320 (15,601) 7,571 (3,987) 1,371 (327) 29,976 (12,602) 29,509 (13,004) 22,285 (8,319) 1,836 (931) 4,417 (2,019) 32,378 (15,275) 7,766 (3,115) 5,554 (2,578) 2,384 (888) 4,368 (2,117) 9,905 (3,078) 2,293 (851) 13,909 (5,593) 49,563 (16,654) 15,608 (7,096) 15,024 (7,141) 985 (444) 20,063 (10,869) 927 (297) 2,830 (1,005) 4,445 (1,265) 2,893 (1,202) 17,897 (7,188) 5,123 (801) 172,753 (117,263) 33,665 (12,753) 363 (145) 20,162 (8,378) 4,023 (1,496) 32,290 (15,350) 22,459 (8,260) 4,711 (3,663) 6,210 (2,306) 1,012 (500) 8,346 (3,337) 76,958 (27,485) 10,224 (3,848) 1,841 (754) 14,351 (5,834) 53,827 (18,460) 5,573 (2,286) 10,014 (3,381) 672 (228)

JavaScript Snippets (Parsable) 3,149 (1,871) 1,195 (791) 9,699 (6,497) 2,279 (1,541) 145,492 (98,173) 23,230 (15,404) 5,403 (3,844) 903 (619) 20,933 (14,554) 26,820 (18,526) 15,388 (9,920) 1,300 (820) 2,150 (1,351) 23,870 (15,049) 6,575 (4,405) 4,066 (2,677) 1,944 (1,340) 3,229 (2,299) 6,921 (4,618) 2,007 (1,432) 11,909 (8,059) 36,129 (24,464) 11,899 (8,219) 11,466 (7,744) 940 (591) 8,066 (5,329) 1,009 (623) 3,787 (2,757) 3,497 (2,232) 2,197 (1,545) 12,286 (8,227) 2,274 (1,099) 69,159 (47,345) 33,345 (25,216) 421 (279) 18,295 (13,195) 3,826 (2,687) 21,918 (14,992) 20,557 (13,720) 1,376 (958) 5,932 (4,135) 870 (671) 9,755 (6,858) 63,533 (43,091) 10,960 (7,468) 1,308 (845) 11,186 (7,484) 41,497 (26,536) 3,638 (2,266) 12,196 (8,262) 512 (310)

Python Snippets (Parsable) 1,503 (1,063) 1,769 (1,411) 4,373 (2,765) 1,557 (1,056) 124,984 (79,722) 13,126 (8,603) 4,113 (2,893) 660 (457) 17,502 (11,597) 10,058 (6,752) 9,277 (6,260) 2,348 (1,892) 1,637 (1,064) 15,148 (10,360) 3,631 (2,464) 1,929 (1,328) 1,176 (847) 1,131 (714) 4,399 (2,913) 755 (503) 10,038 (7,456) 40,617 (29,631) 5,409 (3,466) 4,863 (2,994) 524 (342) 4,997 (3,321) 506 (319) 681 (454) 1,937 (1,371) 794 (492) 7,862 (5,282) 1,575 (983) 51,391 (33,739) 12,972 (8,784) 181 (110) 7,175 (4,936) 5,171 (4,208) 12,855 (8,740) 13,438 (8,649) 674 (410) 2,706 (1,716) 233 (161) 3,277 (2,144) 35,244 (23,995) 6,716 (4,614) 1,014 (695) 6,633 (4,143) 40,039 (25,313) 2,349 (1,638) 3,238 (2,231) 347 (224)

Ruby Snippets (Parsable) 1,063 (580) 440 (221) 9,886 (7,276) 943 (445) 76,587 (43,072) 11,406 (6,473) 2,790 (1,566) 411 (197) 11,763 (6,305) 9,285 (5,162) 7,375 (4,159) 776 (473) 915 (428) 10,250 (5,530) 3,116 (1,858) 2,114 (1,505) 659 (340) 1,102 (636) 3,004 (1,435) 422 (182) 3,832 (1,957) 18,258 (9,804) 4,434 (2,405) 4,667 (2,549) 193 (61) 4,066 (2,092) 327 (130) 1,095 (704) 2,160 (1,323) 895 (439) 3,201 (1,372) 1,380 (676) 36,746 (20,724) 10,368 (5,407) 216 (122) 5,409 (2,965) 1,297 (628) 11,719 (6,418) 8,684 (4,943) 502 (270) 1,685 (914) 154 (51) 3,056 (1,775) 24,589 (13,226) 3,816 (2,148) 1,237 (800) 4,782 (2,580) 19,019 (10,299) 1,946 (904) 3,109 (1,659) 368 (233)

Java Snippets (Parsable) 2,324 (1,450) 285 (127) 4,190 (2,233) 1,318 (714) 63,237 (34,870) 10,029 (5,411) 1,526 (787) 414 (214) 9,278 (4,882) 8,643 (4,582) 12,473 (6,652) 402 (212) 1,433 (844) 9,020 (4,802) 3,304 (1,648) 1,409 (717) 738 (360) 1,018 (530) 5,180 (3,505) 473 (223) 5,698 (3,173) 17,308 (9,343) 5,005 (2,785) 4,780 (2,828) 375 (214) 4,269 (2,324) 350 (207) 941 (470) 1,161 (528) 941 (503) 5,654 (3,002) 1,017 (440) 25,700 (13,670) 11,595 (6,470) 163 (84) 6,182 (3,426) 1,158 (635) 7,914 (4,152) 11,892 (6,852) 384 (215) 6,569 (4,900) 365 (221) 2,688 (1,459) 25,840 (14,090) 3,497 (1,859) 366 (162) 5,858 (3,179) 21,822 (11,628) 2,046 (1,115) 3,763 (1,967) 198 (109)

For the reliability dimension, New Mexico held the highest density for SQL (0.101) and JavaScript (0.576), suggesting concerns regarding error-proneness for its users. Similarly, Mississippi had the highest violation density for Ruby (0.374) and Java (0.325), while Missouri led in Python violations (0.482). Counter to what might be intuitively expected, major tech hubs like California and New York were concentrated in the middle of the violation density spectrum, while smaller tech states had lower densities. These findings suggest that factors beyond tech industry presence influence code reliability, which is explored in Section 5.2. The pattern of moderate tech ecosystems having higher violation densities extends to the city level. Champaign, IL exemplifies this, with the highest observed JavaScript reliability violation density (0.592). Interestingly, even major tech hubs like San Francisco, CA and Seattle, WA do not necessarily have the highest or lowest violations. For one, San Francisco’s reliability violation density for Python (0.328) is clustered on the median (43rd highest), and for Ruby (0.112) is lower than most cities (75th). Looking at the readability dimension reveals a surprising trend. Unlike reliability, states with less developed tech sectors (Alabama, Maine, North Dakota, Mississippi) exhibited high violation densities (e.g., Alabama’s highest SQL density at 0.772). These findings suggest a relationship between tech maturity of a state and developer code clarity. However, increasing technological influence may not offer a panacea, as prominent tech hubs (California, New York) also had significant readability violations (e.g., Massachusetts’s Python density of 0.386, 15th highest). Conversely, states with moderate tech ecosystems had the lowest readability violations (South Carolina’s lowest Java density at 0.081). This suggests a potential “sweet spot” where established practices promote clarity without the complexities of advanced ecosystems. Similar patterns emerge at the city level, with Davis, CA (large suburban area) having a low Java violation density (0.056). This reinforces the notion that a balanced level of technological integration might be optimal for code clarity, avoiding both the pitfalls of nascent and highly advanced ecosystems. However, deviations exist, like Lansing, MI (state capital) having the lowest Ruby density (0.545). These findings highlight the multifaceted nature of factors influencing code readability, beyond just technological maturity. In states with advanced tech sectors, for example, a diverse range of developers with varying levels of experience may be attracted. This diversity could lead to a balancing effect on code readability, preventing extreme deviations from the average level and resulting in a more moderate level of readability violations (explored further in Section 5.2). Analysis of performance violation density reveals that states with established tech sectors (California, New York, Massachusetts) have lower densities across languages (e.g., New York’s SQL density of 0.017). These findings suggest the demand for performant code within these competitive, large, and vibrant tech landscapes (see Section 5.2). Conversely, states with less developed tech sectors (West Virginia, Vermont, and North Dakota) have higher densities (West Virginia’s highest SQL at 0.178). However, this trend does not hold for all languages, exemplified by Maryland with lowest JavaScript density of 0.003. This suggests other factors beyond tech maturity influence performance violations, like regional practices (e.g., Washington’s low Ruby density of 0.006). Interestingly, city-level data shows an inverse relationship — larger cities that are not necessarily tech hubs have higher densities. For instance, Irving, TX has 0.271 for SQL. These discoveries accentuate the presence of less nuanced, multifaceted factors which determines performant coding practices. Like performance, security violation density analysis suggests a link to urbanisation. States with larger urban populations (California, New Jersey, and New York) have lower densities. Examples include New York’s density of 0.0015 and New Jersey’s 0.0017 for Python. However, the trends were less stark than the performance dimension, in which several deviations still occurred. Examples include Nevada’s high Java violation density (0.233) and Florida’s high Ruby density (0.0086), highlighting the non-monotonic relationship between the urbanisation of a state and their secure coding practices. An inverse trend emerged on rural states, exhibiting higher violation densities. North Dakota and Maine, for instance, boasted the highest violation densities for JavaScript at 0.037 and 0.036, respectively. These findings can also be seen to ripple towards city-level despite some fluctuations. Bustling metropolises generally exhibited lower violation densities across various programming languages. Boca Raton, FL, has one of the lowest densities for Java (0.055) and Houston, TX exhibiting the same notion for Python (0.004). In general, our analysis reveals a potential causal relationship between violation densities across all dimensions and regional urbanisation.

4.3

Code Snippet Relations (RQ3)

This section presents results for RQ3: How do diversity indicators of US regions affect code quality? Our preliminary findings indicate that certain census diversity indicators exhibit moderate to strong negative correlations with the states’ violation densities. Negative correlation infers that an increase of that indicator would lead to a decrease in violation density (i.e., better quality). Results showed all pairs’ p-values to be less than 0.05, confirming that all correlation pairs are unlikely due to chance. In subsequent reporting of the results, only medium (r = 0.3–0.5) to strong correlations (r ≥ 0.5) (as per Cohen [119]) were highlighted, allowing examination and comparison of cross-region diversity and code quality densities. Firstly, the indicator Gini index was shown to moderately correlate with JavaScript Performance (𝑟 = -0.392), as well as Ruby Readability (-0.3) and Reliability (-0.323). This implies that states with higher Gini index, which signals better wealth equality, tend to also yield less violations. We also found the indicator households with one or more types of computing devices exhibited strong negative correlations with SQL Performance (-0.597) and Ruby Readability (-0.527), and also moderate negative correlations with multiple violation densities including Python Reliability (0.457) and Performance (-0.363), as well as SQL Security (-0.455) and Readability (-0.383). These results hint that the prevalence of domestic information and communications technology (ICT) access at the state-level tends to positively impact code quality. Another ICT-related indicator that is worth noting is Internet Subscriptions in Household: With an Internet subscription, exhibiting strong negative correlations with Java (-0.659) and Python Security (-0.652). It also presented moderate negative correlations against Ruby Security (-0.447), Python Readability (-0.443) and Reliability (-0.348), Java Performance (-0.406) and Readability (-0.395), as well as JavaScript Performance (-0.315) and Readability (-0.306). This supports the idea that domestic ICT access can bolster code quality regardless of dimensions and languages. Apart from ICT-related indicators, per capita income in the past 12 months negatively correlates with Java (-0.578) and Python Security (-0.408), i.e., higher regional income is correlated with more secure coding practices. The racial diversity indicator white alone (% of all population) positively correlates with Java Readability (0.314), i.e., lower racial diversity in a state correlates with lower code readability. Other sets of positive correlations were found between total unemployment and Python (0.358) and SQL Readability (0.417), i.e., higher unemployment correlates with less readable code. In terms of education, school enrolment by level of school for the population 3 years and over: enrolled up to grade 12 negatively correlates with three performance violation densities: Java (-0.45), JavaScript (-0.404), and Python (-0.318). That is, broader coverage of compulsory basic education correlates with more performant code. Finally, Total Females was found to correlate with Ruby Readability (-0.653) and Security (-0.564), JavaScript Security (-0.585) and Readability (-0.524), Python Performance (-0.49) and Readability (-0.43), and Java Security (-0.409) and Performance (-0.321); while Total Males was found to correlate with: Ruby Security (-0.616), Python Reliability (-0.536) and Security (-0.415), Java Reliability (-0.469) and Security (-0.48), and SQL Reliability (-0.385). These findings may hint at a relationship between gender composition and code quality metrics, but further research is clearly needed to understand these dynamics. Table 14 below documents the strong and moderate correlations for all census indicators and code quality violation densities, while our replication package hosts the full correlation matrix40 [27]. Table 14. State-level indicators’ correlations Diversity Indicator Gini index

Households with one or more types of computing devices

Code Quality Violation Density Ruby Reliability Ruby Readability JavaScript Performance SQL Security SQL Readability SQL Performance Ruby Readability Ruby Performance Python Reliability Python Performance JavaScript Reliability

Pearson’s r -0.323 -0.300 -0.392 -0.455 -0.383 -0.597 -0.527 -0.492 -0.457 -0.363 -0.302

Replication package » Results » Census Indicators Correlations (RQ3) » Census Indicators Correlations.xlsx

Diversity Indicator

Internet Subscriptions in Household: With an Internet subscription

Per capita income in the past 12 months (in 2020 inflation-adjusted dollars) Race: White alone (% of all population) School Enrolment by Level of School for the Population 3 Years and Over: Enrolled up to Grade 12

Total Females

Total Males

Total unemployment

4.4

Code Quality Violation Density Java Performance Ruby Security Python Security Python Reliability Python Readability JavaScript Readability JavaScript Performance Java Security Java Readability Java Performance Python Security Java Security Java Readability Python Performance JavaScript Performance Java Performance Ruby Security Ruby Readability Python Readability Python Performance JavaScript Security JavaScript Readability Java Security Java Performance SQL Reliability Ruby Security Python Security Python Reliability Java Security Java Reliability SQL Readability Python Readability

Pearson’s r -0.327 -0.447 -0.652 -0.348 -0.443 -0.306 -0.315 -0.659 -0.395 -0.406 -0.408 -0.578 0.314 -0.318 -0.404 -0.450 -0.564 -0.653 -0.430 -0.490 -0.585 -0.524 -0.409 -0.321 -0.385 -0.616 -0.415 -0.536 -0.480 -0.469 0.417 0.358

Code Snippet Themes (RQ4)

This section presents results for RQ4: What themes are prevalent in code quality violations, and how do these themes differ across US regions? Following our inductive content analysis, results were aggregated using frequency counts and percentages. Snippets with more than one violation were assigned multiple coding themes. Table 15 summarises the emergent coding themes discovered across all quality dimensions, their descriptions, and chi-squared test results. Significance levels are appended at the end of each test statistic following the ‘three-star system’ (i.e., *p ≤ 0.05, **p ≤ 0.01, ***p ≤ 0.001) [120]. To further illustrate discovered trends, Figure 7 provides a visual representation of the quantitative analysis results for all three states across each quality dimension. Detailed examples and analytical merit of each coding theme may otherwise be seen in our replication package [27]41. Firstly, California exhibited the highest prevalence of complex non-basic reliability errors. These errors, exemplified by Unclosed Resource (34 occurrences, 6.79%) are notably more frequent in California compared to Utah (25 occurrences, 3.86%) and North Dakota (14 occurrences, 2.08%). Several other coding themes that adhere to this trend are Race Conditions (45; 8.98%), and Field Mutability (40; 7.98%) in California, which is the highest of all three states. In contrast, somewhat basic reliability violations such as Incorrect Branching (10 times; 7.30%) and Incorrect Loops (8 times; 5.84%) were the least frequent in California. These findings suggest that a state’s tech workforce and R&D presence may impact the complexity of their developers’ reliability violations. However, certain deviations still exist, such as the prevalence of Usage Mismatch violations (68 occurrences, 13.57%) in California compared to Utah (68 occurrences, 10.49%), and North Dakota (29, 4.31%).

Replication package » Coding Themes » Reliability to Security

Table 15. Discovered content analysis coding themes

Readability

Reliability

Dimension

Scale

Coding Theme

Description

χ²

Field Mutability

Fields that are only used once are not given immutable access modifiers.

2.268 (ns)

Built-In Mutability

Immutable built-in data types are modified.

10.634**

Excessive Imports

Numerous imported packages that remain unutilised, making the code brittle.

2.032 (ns)

Usage Mismatch

Non-built-in methods are used in a way that does not comply with its intended use-case.

21.509***

Redundant Statements

Unnecessary statements that risk accidental field misuse or modification.

11.594**

Empty Exceptions

Exceptions are caught yet are ignored and unhandled.

27.908***

Overly-Broad Exceptions

General, non-explanatory raw exception types.

14.851***

Lack of Interfaces

Insufficient abstractions and interfaces, hindering versatility and extensibility.

8.109*

God Class

Classes that contain too many responsibilities and functionalities.

6.591*

Syntax Errors

General syntax errors that will prevent the code from running.

19.289***

Threading Errors

Errors that are related to multithreading and will result in deadlocks.

18.697***

Data Structures

Improper utilisation for data structures’ functionalities

0.327 (ns)

Built-in Methods

Built-in methods are used in a way that deviates from its original design.

44.189***

Incorrect Loops

Loops that end prematurely or goes on infinitely.

24.56***

Incorrect Branching

Program flow may deviate from the expected behaviour.

18.221***

Race Conditions

Multiple threads process a shared item where the final result depends on the order of executions.

17.345***

Unclosed Resource

Connections and resources that are not closed properly following its utilisation.

8.803*

Not Coded

—

0.779 (ns)

Implicit Syntax

Non-self-explanatory syntaxes and expressions.

7.022*

Advanced Language Features

Language-specific features that are unintuitive.

15.145***

Complicated Methods

Complex methods despite easier alternatives can be implemented.

20.086***

Long one-liners

Lines that excessively chain instructions, hindering interpretability.

6.976*

Separable Statements

Lines that contain several statements that would be more interpretable if separated.

2.439 (ns)

Redundant Stylistic Choice

Stylistic choices that are unnecessary and does not affect program behaviour.

9.909**

Irregular Spacing

Inconsistent whitespaces and indentations between array elements, code blocks, and statements.

3.006 (ns)

Irregular Comments

Comments that are inconsistent or misleading.

9.052*

Irregular Empty Lines

Too much or too little empty lines between statements.

12.566**

Security

Performance

Dimension

Scale

Coding Theme

Description

χ²

Lack of Documentation

Methods, functions, or classes that are undocumented.

7.965*

Incomplete Documentation

Documentations that are vague, unclear, and/or hardly understandable.

0.058 (ns)

Single-Letter Names

Object or variable names that only contain a single letter.

5.115 (ns)

Ambiguous Names

Non-self-explanatory object or variable names.

15.542***

Not Coded

—

4.511 (ns)

Small Memory Allocation

Minor performance overheads that only slightly affects performance.

22.509***

Invalid Regular Expressions

Regular expressions that does not return any matches.

3.813 (ns)

Invalid Character Classes

Multiple code point characters in character class syntax.

15.71***

Unnecessary Computation

Usage of complex algorithmic steps where a faster alternative may be implemented.

73.349***

No Lazy Computation

Lack of lazy implementations where computations are not done ad-hoc.

14.962***

Comparisons to Sort

Usage of comparison operators to sort values.

27.862***

Usage of Inefficient Methods

Usage of built-in language features and methods that are slower than others.

10.553**

Fall-through

Switch-case statements without breaking each case.

3.863 (ns)

Not Coded

—

0.839 (ns)

Dynamic Paths

Accession to non-hard-coded file paths.

18.269***

File Parsing

Attempts at parsing unsanitised files such as XML.

8.673*

Dynamic Input

Parsing non-literal and non-deterministic inputs.

25.181***

Unsafe Shell Injection

Usage of unsafe shell methods.

1.598 (ns)

Code Evaluation

Parsing arbitrary strings and evaluating them as code statements.

2.763 (ns)

Unsafe Deserialisation

Deserialising external arbitrary files without prior validation nor sanitisation.

32.229***

Readable Passwords

Passwords stored as plaintext or is decipherable through reverse encoding.

11.879**

Silent Errors

Program may not behave unexpectedly in a way that security vulnerabilities are opened.

0.214 (ns)

Pseudorandom Numbers

Usage of pseudo-random number generators instead of true random sequences.

5.826 (ns)

Arbitrary File Opening

Exposure to insecure APIs or external functionalities that permits arbitrary local file opening.

5.096 (ns)

Not Coded

—

4.904 (ns)

*p ≤ 0.05; **p ≤ 0.01; ***p ≤ 0.001; (ns) = not significant.

8.00%

6.00%

61 4.00%

37 34

0 11 12 2.00%

0.00%

(2A)

18 13

39 48 46

29 29 25

43 59 61

29 32

61 44 43

37 25

17 17

4.00%

120

100

60 101

0 24

2.00%

0.00%

18.00%

16.00%

14.00%

12.00%

10.00%

8.00%

6.00%

4.00%

2.00%

0.00%

(2B)

(1B)

42 46 49

39 42 40 56

62 62

38 38

21 18

9 14

31 24

Not Coded

107

6.00%

Single-Letter Names

10.00%

36 36 44 68

Irregular Empty Lines

12.00%

Separable Statements

80 8.00%

Redundant Stylistic Choice

Long one-liners

14.00%

Irregular Spacing

100 16.00%

10.00%

Lack of Documentation

Utah

Irregular Comments

18.00%

Implicit Syntax

120

(1A)

12.00%

Incomplete Documentation

Built-In Mutability Built-in Methods Data Structures Empty Exceptions Excessive Imports Field Mutability God Class Incorrect Branching Incorrect Loops Lack of Interfaces Overly-Broad Exceptions Race conditions Redundant Statements Syntax Errors Threading Errors Unclosed Resource Usage Mismatch Not Coded

0.00%

Complicated Methods

37 20

14.00%

Ambiguous Names

0 2.00%

100

Advanced Language Features

23 30

Not Coded

71 33 34 4.00%

Single-Letter Names

6.00%

Irregular Empty Lines

Separable Statements

Long one-liners

8.00%

Redundant Stylistic Choice

Irregular Spacing

Lack of Documentation

10.00%

Irregular Comments

Incomplete Documentation

12.00%

Implicit Syntax

Complicated Methods

14.00%

Ambiguous Names

100

Advanced Language Features

Not Coded

60 29

Single-Letter Names

53 28 42

Irregular Empty Lines

Separable Statements

10 8 22

Long one-liners

Redundant Stylistic Choice

82 86

Irregular Spacing

30 25

Lack of Documentation

40 15 12

Irregular Comments

Implicit Syntax

Incomplete Documentation

20 18

Ambiguous Names

Complicated Methods

Reliability 10

Advanced Language Features

Readability

California North Dakota

100 14.00%

12.00%

70 10.00%

8.00%

87 6.00%

49 4.00%

29 25

120

100 80

60 72 54

0 16 16

2.00%

0.00%

(1C)

18.00%

16.00%

14.00%

12.00%

10.00%

8.00%

6.00%

4.00%

2.00%

0.00%

(2C)

Sum %

(4A) Sum

36 %

4.00% 20

2.00% 10

0.00% 0

(4B)

Figure 7. Content analysis results per dimension and state 80 16.00%

14.00% 70 14.00%

60 12.00% 60 12.00% 60 12.00%

50 10.00% 50 10.00% 50 10.00%

40 8.00% 40 8.00% 40

30 6.00% 30 6.00% 30 66

Sum

46 40

59 31

Not Coded

Fall-through

Invalid Character Classes

Invalid Regular Expressions

Unnecessary Computation 43

Not Coded

16.00%

Small Memory Allocation

Unsafe Deserialisation

14.00%

Usage of Inefficient Methods

(3B)

Unsafe Shell Injection

Silent Errors

No Lazy Computation

0.00%

Readable Passwords

16.00%

Comparisons to Sort

5.00% 61

Pseudorandom Numbers

Not Coded

Fall-through

Invalid Character Classes

Utah

File Parsing

38 76

Code Evaluation

42 30 40

Dynamic Paths

Invalid Regular Expressions

10.00%

Dynamic Input

Unnecessary Computation

Arbitrary File Opening

0 31

Not Coded

0.00%

Unsafe Deserialisation

Unsafe Shell Injection

0 20

2.00%

Silent Errors

4 4.00%

Readable Passwords

30 48

Pseudorandom Numbers

(3A) Usage of Inefficient Methods

0.00%

File Parsing

20 62

Small Memory Allocation

0 5.00% 56

Code Evaluation

No Lazy Computation

10.00%

Dynamic Paths

Comparisons to Sort

Dynamic Input

Not Coded

120 30.00% 120 30.00% 120 30.00%

100 25.00% 100 25.00% 100 25.00%

80 20.00% 80 20.00% 80 20.00%

15.00% 60 15.00% 60

108

Arbitrary File Opening

Fall-through

Invalid Character Classes

Invalid Regular Expressions

Not Coded

36 34

Unsafe Deserialisation

Unsafe Shell Injection

Silent Errors

Readable Passwords

Unnecessary Computation

Pseudorandom Numbers

Usage of Inefficient Methods

File Parsing

Small Memory Allocation

Code Evaluation

No Lazy Computation

Comparisons to Sort

Dynamic Paths

Dynamic Input

Performance 20

Arbitrary File Opening

Security

California North Dakota

105 15.00%

65 10.00%

17 33

5.00%

0.00%

(3C)

8.00%

6.00%

4.00%

2.00%

0.00%

(4C)

On the other hand, North Dakota exhibited a tendency towards simpler reliability violations. This is evident in the high frequency of Syntax Errors (87 times; 12.93%) compared to Utah (61; 9.41%) and California (42; 8.38%), which prevent a program from running and thus are relatively basic mistakes. Similarly, North Dakota had a higher proportion of violations related to Overly-Broad Exceptions (56 times; 8.32%) and Built-In Methods (61 times; 9.06%) compared to the other states. These findings further suggest that a region’s technological advancement can influence the reliability of code contributions. North Dakota, with a more nascent tech industry, appears to have a higher prevalence of these simpler reliability violations. When examining readability, California users exhibited a tendency to employ more advanced language features, doing so 53 times (7.57%), compared to Utah (39 times; 5.94%) and North Dakota (21; 3.63%). This preference for less intuitive code in California is further supported by the Implicit Syntax coding theme, being raised most frequently in California (86 times; 12.29%) compared to the other states. In general, these findings suggest that California users tend to contribute code that is more complex and less explicit, making it less intuitive for beginners. Regarding stylistic choices, both California (107; 15.29%) and Utah (101; 15.37%) exhibited a high prevalence of the Redundant Stylistic Choice violation. This finding suggests that developers in these states may deviate more frequently from established coding conventions, employing a wider variety of stylistic approaches. Certain trends were consistent across all three states, however, suggesting that user behaviour may transcend geographical boundaries. For one, all states exhibited a high frequency of the Irregular Spacing violation. Utah and North Dakota were found to have the most instances, with 90 (13.70%) and 85 (14.68%), respectively. These results imply that consistent spacing practices are generally overlooked regardless of where users reside. An analysis of performance-related coding violations revealed that Utah’s violations consistently fell below 60 across all coding themes, which contrasts with California and North Dakota where violations for one theme could surpass 100 occurrences. In fact, the most frequent violation in Utah, No Lazy Computation, only occurred 62 times (15.05%). Initial findings suggest a generally higher adherence to performant coding practices in Utah. On the other hand, the most frequent violations in California were Small Memory Allocation (108 times; 26.02%) and Usage of Inefficient Methods (85 times; 20.48%). This suggests that users in California might be more prone to contribute snippets with slight performance overheads that would be detrimental if they accumulate, or that use less efficient methods when faster alternatives exist. Similarly in North Dakota, Unnecessary Computation occurred the most often at 105 times (21.52%) which hints that their users tend to contribute unnecessarily complex implementations despite the possibility of a faster alternative. Overall, though, Invalid Regular Expressions and Fall-through were consistently low across all three states. California had the highest frequency of the former at 34 occurrences (8.19%), while North Dakota had the most Fall-through themes at 17 instances (3.48%). These low figures imply that developers in all three states generally handle regular expressions and switch-case constructs effectively. Finally, in terms of security, California exhibited a relatively low number of security violations overall. However, two themes, Arbitrary File Opening (52 times; 13.47%) and Silent Errors (44 times; 11.40%), were more frequent in California compared to others. Interestingly, Arbitrary File Opening was also prevalent in North Dakota (57 times; 11.63%) but less frequent in Utah (37 times; 9.66%). These findings suggest that users in California and North Dakota might be more likely to provide answers with connection-related mistakes that could accidentally open files. The prevalence of the Readable Passwords coding theme served as another indicator of shared practices between California and North Dakota users. Such behaviours were exceptionally rare in both states, occurring only 4 times (1.04%) in the former and 10 times (2.04%) in the latter, suggesting that users in these regions are likely aware of the security pitfalls associated with storing passwords in plain text or unsalted hashes. Finally, snippets from Utah signalled concerning trends wherein Dynamic Input (48 occurrences; 12.53%) and Code Evaluation (55; 14.36%) were among the most frequent coding themes in that state. These findings hint that users of Utah might be less likely to sanitise user input and have a tendency to evaluate strings as code, which can introduce vulnerabilities to injection attacks.

5 DISCUSSION AND IMPLICATIONS 5.1 Code Snippet Quality (RQ1) In this section, we discuss snippet quality with respect to RQ1: What is the quality of code snippets across different programming languages on Stack Overflow?

Our analysis revealed a relatively low prevalence of reliability violations. In SQL (2.81%), JavaScript (0.52%), and Python (4.49%) we saw only a small fraction of parsable snippets are prone to errors and bugs. Reliability violations were more frequent in Ruby (21.23%), but Java exhibited the highest rate (84.36%). These findings suggest that users may have an easier time writing more reliable snippets in SQL, JavaScript, and Python compared to Ruby and Java. This difference might be due to Java’s complexity and its strict type checking, which can make debugging trickier [24, 83]. As a result, ensuring code reliability in Java might be more challenging. Looking at the violations, unmatched-parentheses are the most common error in SQL code snippets shared online. This prevalence likely arises from the context-specific nature of SQL queries. Unlike code intended for independent execution, SQL queries necessitate database access to test its correctness, making pre-testing impractical. We posit this reliance on external execution might lead to a higher frequency of basic syntax errors like unpaired parentheses, compared to more complex errors. Holzmann [121] found that large software repositories were found to harbour more minor errors than major ones, and based on our results, this pattern can also be seen in SQL snippets. In JavaScript, misuse of prototypes and omitting variable definitions are frequent, suggesting a lack of understanding of the language’s core prototype concepts. Python shares the issue of missing variable declarations, corroborating Reid et al. [122] where snippets were often found to omit variable declarations, as users are expected to manually check such omissions before using them. Ruby suffers from unutilised code remnants, hinting at residuals of incomplete refactoring [123]. Java developers tend to forgo declaring constants, suggesting a tendency to waive immutability and hence increase the risk of introducing bugs [124]. We also reveal a spectrum of error severity across languages. Some violations, like missing conditions in SQL or type conversions in Ruby, might not cause immediate errors but introduce potential for unexpected behaviour. Others, like undeclared variables in Python or invalid assignments in JavaScript, will halt program execution. Finally, specific violations like improper error handling practices (e.g., using bare except in Python) highlight the need for robust code to ensure resilience against failures. Under the readability dimension, Ruby suffers the most readability violations (85.39%), followed by SQL (73.31%), Java (73.28%), and Python (69.28%). This might be due to less strict adherence to community conventions in these languages, especially for Python and SQL, which are known for readability [125, 126]. The inherent readability of these languages might lead users to subconsciously overlook established conventions, assuming the code’s syntactic clarity is sufficient. Moreover, high readability violations can also be attributed to different coding conventions within each language. For example, SQL has various dialects beyond the ISO standard, such as Oracle Native or BigQuery. Compared to reliability, readability violations were noticeably higher, suggesting a prioritisation of code that works over code that is understandable. Our results corroborate Meldrum et al. [11] in highlighting a potential disregard for readability conventions within the online coding community. The high readability violations might also be partially due to the nature of answer snippets themselves, which tend to be short and focussed on solving a specific problem [11, 85]. Unlike code written for independent use, readability might be less of a concern on Stack Overflow where users prioritise finding solutions quickly [8], leading to less emphasis on readability conventions. The analysis of language-specific violations reveals that improper indentation is the most common issue for SQL, which likely stems from a focus on quick solutions for one-time use cases (i.e., ad-hoc queries) where readability might be less important [127, 128]. JavaScript exhibits frequent violations regarding unused variables. While this could be a stylistic choice, it can disrupt code flow and reduce readability [72, 129], especially when variables are declared in one snippet and used in another within the same post. Readers may experience loss of continuity due to context switching. Figure 8 depicts such an example where this.var is first declared in the first (upper) snippet but not used until the second. In Python, frequent omission of spacing around delimiters in array-like data structures might be due to a perceived lack of impact on functionality, despite going against readability best practices as specified in the PEP 8 style guide [54, 130]. Similarly, Ruby’s Style/StringLiterals violation, while seemingly minor, reflects a preference for single quotes to avoid unintended variable interpolation within strings using double quotes [131]. For example, “Hello, #{name}!” would insert the value of the name variable into the string. In contrast, single quotation marks do not present as much versatility, preventing accidental variable substitution and imparting a clearer semantic context to readers. Java violations mainly involve lack of Javadoc comments, which may be due to code snippets being miniscule (as noted

above) and therefore contextual information surrounding the snippet should adequately elucidate its functionality [11].

Figure 8. Example post (Posts.Id 8566320) where variable is declared and used in different snippets Overall, violations are mostly focussed in practices that would create technical debt by making code harder to understand in the long run. This includes enforcing consistent capitalisation (capitalisation.keywords in SQL), avoiding abbreviations (AbbreviationAsWordInName in Java), using escape characters properly (no-useless-escape in JavaScript), and maintaining clear visual separation between code elements (LT05 in Python) [132]. For performance, SQL (24.90%) has the most violations, followed by Python (9.87%), Java (2.34%), Ruby (1.16%), and JavaScript (0.52%). These findings suggest users struggle more with optimising SQL queries compared to other languages, potentially owing to the inherent complexity of database operations [133] and of the SQL language itself. This aligns with Lyu et al. [134], where they identified the persistence of SQL performance anti-patterns even in performance-critical applications, significantly impacting both runtime and energy consumption. For instance, using SELECT * was shown to consume ten times more energy on average and hinder scalability because it retrieves excessive data [134]. Notably, wildcard queries were violated most often. Behaviour-wise, wildcard querying may be attributed to an unwillingness to manually type column names [118]. JavaScript exhibits frequent variable redeclarations (no-redeclare), a practice that can degrade performance over time, even though it might seem like a shortcut initially [135]. In Python, unnecessary imports are somewhat common, especially for standard libraries like sys or os. This is likely because developers are more familiar with these libraries and might import them “just in case,” whereas third-party libraries with specific purposes are less likely to be imported and then not used [156]. Ruby developers frequently use basic RegexpMatch statements for regular expressions despite faster alternatives existing [136]. This suggests a lack of awareness, or a preference for more familiar methods instead of novel ones. Similarly, Java developers often resort to the plus operator (+) for string concatenation (SBSC_USE_STRINGBUFFER_CONCATENATION) instead of more efficient built-in functions [137]. This again might be due to unfamiliarity, or to a misunderstanding of Java’s immutable strings, where concatenation creates new objects and potentially consumes more memory [11]. From a contextual standpoint, performance violations vary in severity. Less critical examples include variable redeclaration (no-redeclare) in JavaScript and import placement (E402) in Python. Conversely, more serious violations involve inefficient regular expressions (no-invalid-regexp) in JavaScript and manual memory management (DM_GC) in Java [11, 138]. These findings highlight the importance of code optimisation even in short snippets, as seemingly minor design choices can accumulate and impact performance. Under the security dimension, SQL queries exhibit the most violations (29.02%) compared to JavaScript (11.52%) and Python (7.90%), while Ruby (0.82%) and Java (1.27%) exhibited lower percentages. The higher frequency of security violations in SQL snippets may partially explain the prevalence of SQL injection attacks observed in the broader SE industry for over two decades, as documented by the Open Web Application Security Project (OWASP)42 [139]. Even with various defence mechanisms, new 42

https://owasp.org

injection threats continue to emerge [139]. Conversely, security violations are rare in Java code snippets, aligning with the study by Meldrum et al. [11] where Java security vulnerabilities are somewhat minimal. In terms of total violation counts, SELECT * was flagged most frequently in SQL, which risks accidental exposure of more data than intended [140]. This suggests a trade-off between efficiency and security, where users might prioritise brevity over explicit column names [118]. Similarly, JavaScript exhibits frequent object injection hotspots due to a lack of input validation, opening doors for cross-site scripting (XSS) attacks [141]. Python shows misuse of the random43 standard library (S311), with developers potentially unaware that it generates pseudo-random numbers instead of truly random ones [142]. This could have security implications if the generator’s state is compromised (known as state compromise extension attacks) [143]. Unsanitised URL inputs are common in Ruby, raising the risk of injection attacks, unauthorised access (e.g., root access) and telnet daemons [144]. Finally, Java developers tend to omit the final modifier for public static fields (MS_SHOULD_BE_FINAL). While the contents of the object can change, the reference itself cannot be reassigned, limiting unintended access reference [11]. This aligns with Meldrum et al. [11], who found mutable static fields being a prevalent security concern in Java. Overall, the majority of security violations stemmed from unsanitised inputs, which can manifest in various ways depending on the programming language. For instance, SQL code might be vulnerable to string concatenation attacks, while JavaScript could be compromised through techniques like non-literal requires. Similarly, Ruby code might be susceptible to security risks if it employs the Eval function. Another category of security violations involved hardcoded passwords within the code itself. Examples include readable passwords found in SQL snippets, or the presence of hardcoded credentials (e.g., S105 in Python and DMI_CONSTANT_DB_PASSWORD in Java). We also found certain threats that are more severe albeit less frequent: violations related to bidirectional characters in JavaScript code, which can evade human detection during code review [145].

5.2

Code Snippet Violation Densities (RQ2)

In this section, we discuss snippet violation densities as outlined in RQ2: How do coding practices among US contributors differ across states and cities in the United States? Looking at our results, the size of the tech industry and economy seems to have an effect towards the quantity of parsable code snippets. States like New York, California, and Texas, renowned for their sizable tech workforces (e.g., Texas with a 5.5% net growth in tech jobs in 2022) [146], yielded a higher number of parsable snippets, which can likely be attributed to the volume of code production in these regions. Interestingly, these same states (New York, California, and Texas) exhibited lower violation rates across all code quality dimensions and programming languages. Particularly for reliability and readability, their violation densities tended to cluster around the median, suggesting a balance between non-compliance (i.e., high violation density) and adherence to coding best practices (low violation density). One possible explanation for this observation lies in the age and experience diversity of the tech hubs’ coding communities, which ranges from seasoned professionals to apprentices [147]. While senior developers are more likely to adhere to established practices, resulting in reliable and readable code, contributions from less experienced individuals might introduce deviations from these standards [148, 149]. Therefore, we posit that this age diversity within the workforce might influence the overall conformance (or departure) from best practices. California and New York also exhibit lower violation rates in performance and security dimensions, which may be attributed to the fiercely competitive nature of these tech ecosystems. Being home to giants like Google, IBM, and Bloomberg might incentivise a stronger emphasis on performance and security best practices [150, 151], which could be seen as a way to gain a competitive advantage in the software market and attract consumers who prioritise a smooth and secure user experience [177]. Moreover, as tech hubs are often at the forefront of cutting-edge technology [152, 153], projects in such regions might prioritise performant and secure code to remain competitive. For instance, enhanced code security was found to mitigate data leaks, contributing to user retention [150]. Conversely, states with less established tech sectors, like Vermont or North Dakota, showed higher readability violation densities. This might be because smaller companies lack the formalised coding conventions found in larger firms (e.g., Google Java Style44), leading to more stylistic inconsistencies. Interestingly, states without a tech presence exhibited lower reliability violation rates compared to 43 44

https://docs.python.org/3/library/random.html https://google.github.io/styleguide/javaguide.html

their tech-heavy counterparts. One potential explanation is their prioritisation of code functionality rather than business-related factors (e.g., time-to-market) that are otherwise prevalent in startupintensive environments [154]. Finally, our findings hinted at a trend beyond just the presence of tech hubs. States like Maryland and Washington, with varying tech industry sizes, displayed the lowest performance violation rates for JavaScript and Ruby. A closer investigation revealed that these states offer some of the highest salaries for web developers [155]. For instance, Washington boasts the highest median web developer salary nationwide ($138,780), followed closely by Virginia ($101,060) [155]. Considering the prevalence of JavaScript and Ruby in web development [156], this observation hints at a potential link between developer compensation and adherence to best practices. In particular, developers may be incentivised to enhance their JavaScript and Ruby performant coding practices to remain competitive in a high-paying job market. Our city-level analysis revealed similar, though less pronounced, patterns. Tech hubs with large R&D clusters (e.g., San Francisco, CA and Seattle, WA) displayed moderate levels of reliability and readability violations. This aligns with our prior findings where diverse workforces in these regions, with both experienced and junior developers [147], influence how violations are flagged [148, 149]. Interestingly, large cities (not necessarily dominated by tech), exhibited higher performance violation rates than smaller ones. This contrasts with findings at the state-level, which could be due to the specific industries prevalent within these cities. Unlike Silicon Valley with its deep tech roots, cities like Irving, TX, whose largest employers reside in sectors like investment banking (Citigroup; 6,162 total employees) and insurance (Allstate Insurance; 3,068 total employees) [157], may not possess a strong tech identity. These non-tech industries may prioritise functionality over speed, which may translate to higher performance violations for users in these locations. Security violations, however, mirrored readability and reliability findings. Larger cities with a strong R&D presence, like Houston, TX exhibited lower security violations. This emphasis on security likely stems from the inherent risks associated with data during the rapid experimentation that characterises R&D activities [150, 158].

5.3

Code Snippet Relations (RQ3)

In this section, we discuss how diversity indicators affect violation densities as outlined in RQ3: How do diversity indicators of US regions affect code quality? Our findings suggest that certain diversity aspects indeed affect code quality. One key finding emerged from examining socioeconomic aspects, as measured by the indicator Gini Index which reflects regional wealth distribution [159]. We found a negative correlation where regions with more equitable wealth distribution had lower violation densities. This correlation could be partially explained by increased investment in education within these regions, which aligns with Wilterdink [160] where it is argued that decreased expenditure on public education may inherently speed up wealth inequality. Inversely, regions with more equitable wealth distribution might allocate more resources towards education, fostering workforces with the skills necessary to write overall better code with fewer errors. For instance, Florida’s substantial investment in higher education (approximately 5.691 billion USD in 2021) – one of the highest figures nationwide [161] – also exhibits a comparatively high Gini index of 0.43 (3rd highest across all states) and small violation densities for Ruby Readability at 1.118. Another interesting finding regarding socioeconomic diversity is the possible association between higher regional income (measured by per capita income in the past 12 months) and more secure coding practices, particularly evident in Java and Python snippets. Wealthier regions might have an advantage in attracting developers with strong cybersecurity expertise, or given the high cost of implementing proper cybersecurity practices [162], they may be able to more effectively invest in training programs focussed on secure coding practices. Thus, we postulate that firms residing in these high-income regions are better-equipped to commit these investments. Moreover, the paramount importance of cybersecurity in enterprise applications (which are often developed using these popular languages [163, 164]) corroborates our findings. Next, we found a positive correlation between ICT domestic access and code quality, evident within the indicators households with one or more types of computing devices and Internet subscriptions in household: with an Internet subscription. Looking closer, it appears that this correlation holds regardless of language and dimension, which suggests that any potential causal influence is broad. In fact, our findings somewhat align with Silva et al. [165] who suggest that early domestic access to ICT, such as computers in childhood, fosters the development of computational thinking skills in later life. In the

context of our study, we postulate that problem-solving and algorithmic pedagogic approaches that give way to programming and robotics [165] would eventually lead to the production of higher quality code, as initial findings suggest. We also found a negative association between the percentage of white population in a state and the readability of Java snippets. However, this observation must be considered with extreme caution, as it is highly unlikely that this correlation is solely attributable to racial distribution. Exploring potential mediating factors instead sheds light into more subtle socioeconomic facets. For instance, higher percentages of white population are mostly found in more rural states (e.g., Vermont at 94.89%, West Virginia at 93.01%) [12], and these rural states suffer from lower investment in infrastructure and education, as well as high income disparity [166]. In the context of our results, limited educational coverage may explain the observed prevalence of readability violations in Java code, which are otherwise found in more urban states such as Nevada (81.50% white population) and California (72.46%). Similarly, our findings regarding how educational attainment affect code quality extends to the indicator school enrolment by level of school for the population 3 years and over: enrolled up to grade 12. Substantiating the prior observation regarding limited educational resources in rural areas, this indicator also revealed that higher completion rates of compulsory basic education is associated with more performant snippets. We can therefore infer that a well-educated population may contribute to a stronger foundation in coding practices. In fact, several developed nations including Singapore and South Korea, have already begun integrating coding into their K-12 curriculum [167]. This decision is rooted in the rationale where introducing such concepts from an early age will benefit even those whose aim is not to be professional programmers [167], as these skillsets foster creativity and advance their spatial and reasoning skills [168]. In terms of gender, we found states with a higher male proportion tend to also have fewer reliability violations regardless of language. Due to the perceived reliability of code snippets, this could potentially explain the observation that males tend to accrue more reputation points, as outlined by Vasilescu et al. [169]. In contrast, we also found states with higher female proportions tend to also yield fewer readability violations. Some studies propose that women may be more likely to exhibit traits associated with meticulousness and attention to detail [170, 171]. We argue this inherent quality could translate into a collective tendency towards writing more readable snippets. However, deeper investigation might be needed to ascertain the nature of these differences. Furthermore, we observed that states with a higher proportion of females tend to output more performant code. This observation is notable considering that code performance is often neglected in the face of more tangible factors such as usability and design patterns [172, 173, 174]. Our results contribute to the existing body of knowledge by adding another perspective that neglecting performance is not as widespread as prior works previously hinted. Finally, previous research indicates that both genders can indeed contribute to more secure coding practices, albeit through different approaches [175]. For example, males may favour techniques like obfuscation, cryptography, and secure privilege management, while females might lean towards secure networking, library utilisation, and data minimisation practices [175]. This is also reflected in our results, where the gender composition of a state appears to have no noticeable impact on its level of security violations, unlike the other three dimensions. That is, our findings suggest that states with higher female composition exhibit similar levels of security violations as those with higher male composition.

5.4

Code Snippet Themes (RQ4)

In this section, we discuss the latent themes inherent in each regions’ code snippets as outlined in RQ4: What themes are prevalent in code quality violations, and how do these themes differ across US regions? Looking at users’ code snippets across regions, we found that users of California tend to contribute snippets that lean towards complexity and implicitness, reducing intuitiveness for new programmers in the corresponding languages. The majority of their reliability errors seem to align with this observed complexity, such as unclosed resources and race conditions. One potential explanation for this finding could be the established presence of major tech hubs and a thriving tech ecosystem within California. We postulate that developers of such environments might be more likely to produce code that reflects professional practices, which may differ from beginner-oriented approaches. In fact, California held the highest number of tech jobs (1,487,864) in 2023, almost twice that of the secondhighest, Texas (867,278) [146]. Our findings add another nuance to prior RQ2 results. Previously, we found that states with broader tech ecosystems (e.g., California and New York) exhibited moderate densities of reliability and readability violations. Yet a closer examination of the actual code snippets

from these states reveals a trend towards more complex violations, contrasting the lower overall frequency. These findings thus indicate that developers in these areas may prioritise professional coding practices that can lead to more intricate and less explicit, yet less error-prone code. This pattern corroborates Bowen et al. [176], who argued that enterprise software systems need to be reliable. However, snippets from California were found to exhibit slight performance overheads or employed less efficient methods compared to faster alternatives. This trend might be linked to the fast-paced startup culture prevalent in California where developers are expected to deploy software products rapidly [177]. We posit that this behaviour translates to Stack Overflow where users in such environments may prioritise functionality over minor performance optimisations, as long as these overheads are not detrimental to the software’s functionality. Additionally, the pressure to deliver software quickly might lead to the adoption of less efficient but readily implementable solutions, as opposed to investing time in complex yet marginally faster alternatives [178]. Certain specific violations were also notable. For instance, code snippets from California were found to yield high degrees of redundant stylistic choices, which might again be attributed to the state’s large number of tech companies (55,868 in 2023) [146], inadvertently leading to a diversity of coding conventions. In such dynamic environments, it may be unlikely that all developers will strictly adhere to a single coding standard, leading to the stylistic deviations as observed in our results. User contributions from Utah exhibited the lowest frequency of performance-related violations. This finding might be explained by the state’s industrial landscape, which is centred around performancecritical sectors like mining, manufacturing, and petroleum production, alongside information technology. The prominence of these industries, exemplified by the Bingham Canyon Mine which is one of the world’s largest [179], could foster a culture of prioritising code efficiency within the state’s developer community. Certain initiatives also exist to bridge the gap between IT and manufacturing practices, such as the century-old Utah Manufacturers Association45. At the other end of the spectrum, user contributions from North Dakota primarily exhibited simpler reliability violations, such as syntax errors and generic raw exception handling. This observation aligns with the state’s nascent tech ecosystem as it only hosted 1,219 tech business establishments and 13,272 net tech employment in 2023 [146]. However, the presence of more complex violations, like misusing built-in methods, suggests a growing sophistication within North Dakota’s developer community and hints that this state could catch up with others. In fact, in 2022, it experienced a net growth of +2.5% in tech employment, which is relatively higher than other less rural states such as Massachusetts (+1.6%) and District of Columbia (+0.6%) [146]. Interestingly, our analysis revealed a similar tendency for users in both California and North Dakota to contribute code with inadvertent file openings via insecure connections. This aspect highlights another key finding of our qualitative analyses where, despite the vast differences in the technological landscapes between states, developers seem to exhibit a degree of ‘hive mind,’ such that they tend to have similar thought patterns regardless of where they reside. These results align with Salminen et al. [180] who outlined the uniformity in software developer thinking, suggesting knowledge transfer or common pitfalls regardless of where users reside. Other coding themes were also found to exhibit consistent patterns across all three states. One example is the low prevalence of storing passwords in plaintext or unsalted hashes, which hints that users across all three states may already be aware regarding the dangers of such practices.

5.5

Implications

Our study holds significance for both software development teams, Stack Overflow administrators, and other CQA platforms that foster collaboration. Firstly, different programming languages have varying inherent language features that may influence the types of violations users make. For example, Java snippets exhibited a higher prevalence of reliability errors, potentially due to the language’s stricter syntax compared to Ruby, which showed a bias towards readability errors given its more flexible conventions. Software development teams should consider the inherent complexity of their chosen languages when building their tech stack. Additionally, implementing standardised coding rulesets can help mitigate errors stemming from these language-specific characteristics. For example, off-the-shelf tools such as Codacy, Semgrep, and SonarQube provide extensible rulesets such that teams can add their own standards on top of existing ones. Next, regions with more established tech industries tend to exhibit more intricate coding practices, leading to more complex errors. We underscore the importance of 45

https://www.linkedin.com/company/utah-manufacturers-association

implementing effective onboarding mechanisms for geographically dispersed software development teams. Such mechanisms can help bridge potential knowledge gaps between team members from different regions, ensuring everyone has a solid understanding of the team’s coding conventions, best practices, and what constitutes good quality code. Furthermore, there is also a need for targeted educational initiatives in regions with less established tech sectors, such as coding-related workshops. We believe such programs could address the prevalence of simpler errors, and in turn equip these regions’ tech workforce with stronger programming fundamentals. Policy makers can allocate more investment in education as our results hinted that such endeavours may improve code quality in rural states. In terms of gender, our results revealed contrasting coding styles. For instance, females were associated with more readable code, whilst males were associated with higher reliability. Software development teams can leverage these insights to integrate diverse strengths and perspectives within the codebase, ultimately leading to a more well-rounded and robust product. For Stack Overflow (and similar CQA) site administrators, we advocate for a built-in linting feature that automatically assesses code snippets for reliability, readability, performance, and security issues before users submit them in their answers. We believe such a feature could promote higher-quality code examples within the platform and reduce inadvertent errors for users who integrate these snippets into their codebases.

THREATS TO VALIDITY

This section outlines the limitations to our work. We classify threats into internal, external, and construct validity, in line with prior literature [23].

6.1

Internal Validity

Firstly, 14,174,843 users were dropped from the dataset due to missing or invalid location data, and another 2,986,774 were excluded due to their location data pointing to other countries. Among these excluded users, some may have been from the United States, even if not explicitly stated. We acknowledge that these actions taken to ensure representativeness of our analyses may inadvertently omit valuable information. Conversely, there is also the possibility of users entering false locations. Secondly, while our work is influenced by Meldrum et al. [11] as we used roughly similar quality dimensions and linting tools, we omit the calculation of specific data attributes for each snippet other than LOC and LLOC (e.g., code length, code spaces, or snippet-per-answer ratio), as was done in their study. The rationale behind this is that programming languages vary in their complexity, and therefore such attributes may not be comparable. For instance, a “Hello World” program would be more concise in JavaScript than in Java. Third, our calculation of LLOC for each snippet incorporated regular expressions to comprehensively capture multiline comments, mindful of the variation in comment identifiers across different languages. While we acknowledge the potential for some comments to evade detection, we have crafted and rigorously tested our regular expressions to minimise this risk to the greatest extent possible. This approach ensures a high degree of confidence in the overall accuracy of our LLOC measurements. We also acknowledge potential reporting discrepancies with respect to the Ruby Readability dimension, with the top ten most frequent violations encompassing only 55.99% of the total, while other languages and dimensions exceeded 75% coverage. Thus, we might have missed some important details regarding how Ruby readability varies across different regions. Lastly, some tools employed priority indices such as PMD, while other tools did not. We postulate that including these indices in the calculation process may introduce an element of unfairness, even when employing weighting mechanisms. Consequently, we opted to exclude such indices from our results and treat all violations across all languages as having equal priority.

6.2

External Validity

Our study is focussed only on Stack Overflow users inside the US, excluding users from other countries. Our findings may therefore not be generalisable towards users from countries outside the US, or towards other CQA sites like Cross Validated or Ask Ubuntu. Secondly, Stack Overflow is a dynamic platform where coding practices may evolve over time due to changes in technology trends and user demographics. To ensure results align with the latest trends, we have used the June 2022 release of the Stack Exchange Data Dump, being the latest at the time of our investigation. While our findings may not accurately capture the current state of software development in the US, prior work has noted that Stack Overflow data are consistent across time [11, 181]. Other threats still loom, however, such as the

evolution of code quality due to the emergence of generative models like Google’s Gemini or OpenAI’s ChatGPT [26]. Over half of GPT-generated answers contain errors [26], underlining Stack Overflow’s continued relevance as a source of reliable solutions. In fact, both of our quantitative and qualitative explorations reveal that reliability errors are primarily program flow alterations, not critical issues that prevent the code from functioning. Therefore, these errors can likely be addressed when users integrate the snippets into their projects. Thirdly, our selection of five languages poses a limitation to generalisability by excluding other popular languages like C++ or C#, and emerging languages like Rust or Go. Inferences drawn from our analyses may not be applicable to these other languages. For instance, TypeScript snippets might not necessarily exhibit the same security violations as JavaScript snippets, despite the syntactic similarity of both languages. Our language selection also only includes two of the top ten most popular languages in the 2022 Stack Overflow Developer Survey [81], namely Python and SQL. Fourth, across all languages, we observe a substantial degree of unparsable and uncompilable snippets, impeding our ability to assess them. Consequently, our results may not be a representative depiction of these five languages. However, the identified trends largely align with prior literature [11, 122], so we believe our work still provides sufficient insights for SE research and practice. Our fifth threat to external validity pertains to the small size of Stack Overflow snippets. These snippets are often provided in isolation, lacking full context of the software projects they belong to (if any). While literature has established snippets’ presence in large software repositories [15, 16], inferences from this study might not extend beyond the specific context of our work. Finally, our study samples California, Utah, and North Dakota to conduct a series of inductive content analyses, mirroring prior research [12, 13]. This selection serves as a representative proxy due to the impracticality of analysing content from all states. Consequently, our findings may not be universally applicable to other regions, and there is a possibility of encountering distinct regional trends if content analysis were to be conducted on other states.

6.3

Construct Validity

To confirm whether concepts are operationalised sufficiently [182], we initially evaluated the accuracy of Guesslang, and our assessment indicates that the tool performs satisfactorily (as described in Section 3.3). This evaluation aimed to reduce the risk of mislabelling languages with near-identical syntax (e.g., distinguishing between C/C++, or JavaScript/TypeScript/CoffeeScript). Despite our efforts, some threats persist, as illustrated by the embedded SQL snippet shown in Figure 5. Afterwards, we noted that linting tools were able to identify unparsable snippets, mitigating this particular threat to a minimal level. The classification of the 54 technologies predicted by Guesslang may also be subject to errors. For instance, some developers may not consider SQL to be a programming language as it cannot be used to create standalone applications. However, by aligning our categorisations with existing literature, we ensure that this threat is minimal. With regards to our quality dimensions, we acknowledge that other researchers and practitioners may have different interpretations for these constructs. In other words, what constitutes reliable, readable, performant, or secure code may vary from person to person. While our four quality dimensions were formulated based on prior literature, it remains probable that these four facets may not fully capture the complex nuance of code snippet quality. For instance, there may exist additional dimensions that were not accounted for in our study. After all, measuring code quality remains a challenging endeavour [63]. In terms of our inductive content analyses, we achieved inter-coder reliability with κ values between 0.825 and 0.869, signifying a good level of agreement [115]. While the level of agreement is strong [183], caution is warranted when interpreting these results as the values fall into the range of 64–81% reliability [183]. In other words, 19–36% of coded exchanges may not be as reliable [183]. Additionally, the final coding themes represent a refinement process where some potential themes may not have been included. However, all disagreements between coders were addressed through consensus to ensure the final coding scheme’s robustness, and we believe this iterative process minimises this threat.

CONCLUSIONS AND FUTURE WORK

This study examines code snippet quality within Stack Overflow answers, focussing on SQL, JavaScript, Python, Ruby, and Java. While prior research has examined related concepts, most of these studies have focussed on only one or two programming languages, and have typically assessed only one quality dimension. Our study aims to advance the conceptualisation of code snippet quality by encompassing a

broader spectrum of programming languages and evaluating it across four distinct quality dimensions: reliability, readability, performance, and security. Moreover, we also investigate how code quality may relate to diversity complications that are widespread in SE-related practices. We found evidence that readability violations are the most prevalent across all languages, followed by reliability, performance, and security. While readability violations largely pertain to issues like code clutter and improper whitespace, more nuanced violations also exist, such as excessively long code lines that induce confusion due to prolonged lateral eye movements. In terms of reliability, violations typically manifest either as critical errors leading to execution interruptions, or as subtle alterations in program behaviour. Performance-related violations vary in their severity, ranging from small memory allocations to catastrophic backtracking when constructing regular expressions [138]. Finally, security violations mainly revolve around unsanitised string inputs (posing risks of code injection and XSS attacks), along with less common but still severe violations such as Unicode bidirectional attacks (leading to trojans) and mutable static fields. We calculated violation densities for each dimension to ensure fair representation when analysing results across states and cities of the US. The US was selected as a case study given its role as the birthplace of Stack Overflow, its status as the largest English-speaking country (with English being the lingua franca of SE), its global leadership in technology, and its position as host to the highest number of Stack Overflow users. Firstly, states with a large tech presence (e.g., California, Texas, and New York) tend to generate more parsable snippets, which may be attributed to the size of their tech workforce, translating to higher code production and increased presence on Stack Overflow. Contrary to expectations, the abundance of parsable snippets does not directly lead to a higher incidence of violations across different languages and dimensions. Notably, such states displayed only moderate violation densities in reliability and readability dimensions, and even less for performance and security. We propose such variability stems from the combined influence of a diverse tech workforce (i.e., senior and junior programmers), emerging industries, and the ubiquitous nature of technology within these regions. Users from rural areas exhibit a lower propensity for generating reliability violations, which may arise from the prioritisation of reliable functionality over rapid release cycles (time-to-market), with the latter often characteristic of startup environments in high-tech states. Our analysis reveals a predominantly top-down trend propagation, where state-level patterns are largely reflected at the city level (albeit not as stark). Cities renowned for their R&D clusters such as San Francisco, CA and Seattle, WA generated moderate levels of reliability and readability violations, and even fewer under security. The trend diverges for performance violations, however, with large, non-tech-focussed cities having more – which could be due to the specific industries in those areas. In terms of diversity indicators, regions with more equitable wealth distribution, higher education investment, and higher coverage of domestic technology exhibited fewer code violations. Interestingly, the racial makeup of a state itself was not a strong indicator, but rather the socioeconomic factors often associated with predominantly white populations, such as lower investment in rural areas. Gender also played a role: states with a higher proportion of females had fewer readability and performance violations, while those with a higher proportion of males had fewer reliability violations. Security showed an interesting trend where both genders contributed to better code, but through different approaches. Qualitative content analyses confirmed that developers from California, a state with a large tech hub, tended to write more complex code and prioritised professional practices over readability for beginners. This leads to errors that are also equally complex. Their fast-paced startup culture that emphasises time-to-market over optimisation also gives rise to minor performance inefficiencies (albeit not detrimental). In contrast, Utah, which hosts performancecritical industries, had users who contributed code with a higher emphasis on efficiency. North Dakota, whose tech industry is nascent, exhibited simpler errors but also certain intermediate-level ones, suggesting a maturing developer base. Interestingly, some coding patterns were consistent across all regions, such as the avoidance of storing passwords in plaintext or unsalted hashes, highlighting a shared knowledge base among developers regardless of location. Nevertheless, our findings suggest that users contributing answers on Stack Overflow may not always fully grasp the broader reliability, readability, performance, and security implications of their given solutions. Thus, we advocate developers to exercise caution when integrating such code into their software repositories. Although the majority of identified violations are not of critical concern, their unchecked presence has the potential to adversely impact the repository’s overall quality.

Our study has illuminated potential directions for future studies. Subsequent investigations could, for example, leverage the insights gained from our study to employ Bayesian inference techniques to complement our frequentist approaches. It is also possible to triangulate results from all four dimensions to determine whether there is a trade-off between each quality dimension. For instance, researchers may validate whether there is any causal relationship between performance and readability, or whether performance and security are inversely proportional [52]. Notably, an additional avenue of research could involve the development of a composite measure that integrates all four operationalised constructs, resulting in an all-encompassing metric that holistically conveys the idea of code snippet quality. There is also the need to conduct a full-scaled qualitative investigation to ascertain the unearthed patterns from our content analysis. Furthermore, leveraging our facets of quality, future research could employ modelling techniques to quantify the extent of user contribution within the platform. We contend these research venues hold significant value in enhancing our understanding of the coding practices prevalent on the platform, thereby elucidating their implications for broader software development practices. Finally, we assert that our findings should not, under any circumstances, be utilised to profile the superiority of one region over another.

REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

Riemer, K., Schellhammer, S., & Meinert, M. (2018). Collaboration in the Digital Age: How Technology Enables Individuals, Teams and Businesses (K. Riemer, S. Schellhammer, & M. Meinert Eds.). Cham: Springer. doi:10.1007/978-3-319-94487-6 Le, L. T., & Shah, C. (2018). Retrieving people: Identifying potential answerers in Community QuestionAnswering. Journal of the Association for Information Science and Technology, 69(10), 1246-1258. doi:10.1002/asi.24042 Barua, A., Thomas, S. W., & Hassan, A. E. (2014). What are developers talking about? An analysis of topics and trends in Stack Overflow. Empirical Software Engineering, 19(3), 619-654. doi:10.1007/s10664-012-9231-y Ahmad, A., Feng, C., Ge, S., & Yousif, A. (2018). A survey on mining Stack Overflow: question and answering (Q&A) community. Data Technologies and Applications, 52(2), 190-247. doi:10.1108/DTA07-2017-0054 Asaduzzaman, M., Mashiyat, A. S., Roy, C. K., & Schneider, K. A. (2013, 18-19 May 2013). Answering questions about unanswered questions of Stack Overflow. Paper presented at the 2013 10th Working Conference on Mining Software Repositories (MSR). doi:10.1109/MSR.2013.6624015 Trienes, J., & Balog, K. (2019). Identifying Unclear Questions in Community Question Answering Websites, Cham. doi:10.1007/978-3-030-15712-8_18 Ponzanelli, L., Mocci, A., Bacchelli, A., Lanza, M., & Fullerton, D. (2014, 29 Sept.-3 Oct. 2014). Improving Low Quality Stack Overflow Post Detection. Paper presented at the 2014 IEEE International Conference on Software Maintenance and Evolution. doi:10.1109/ICSME.2014.90 Subramanian, S., & Holmes, R. (2013, 18-19 May 2013). Making Sense of Online Code Snippets. Paper presented at the 2013 10th Working Conference on Mining Software Repositories (MSR). doi:10.1109/MSR.2013.6624012 Bafatakis, N., Boecker, N., Boon, W., Salazar, M. C., Krinke, J., Oznacar, G., & White, R. (2019, 25-31 May 2019). Python Coding Style Compliance on Stack Overflow. Paper presented at the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). doi:10.1109/MSR.2019.00042 Campos, U. F., Smethurst, G., Moraes, J. P., Bonifácio, R., & Pinto, G. (2019, 25-31 May 2019). Mining Rule Violations in JavaScript Code Snippets. Paper presented at the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). doi:10.1109/MSR.2019.00039 Meldrum, S., Licorish, S. A., Owen, C. A., & Savarimuthu, B. T. R. (2020). Understanding stack overflow code quality: A recommendation of caution. Science of Computer Programming, 199, 102516. doi:10.1016/j.scico.2020.102516 Zolduoarrati, E., Licorish, S. A., & Stanger, N. (2024). Harmonising Contributions: Exploring Diversity in Software Engineering through CQA Mining on Stack Overflow. ACM Transactions on Software Engineering and Methodology. doi:10.1145/3672453 Anonymised. (2024). [Reserved for Contributions Metric work]. [Reserved for Contributions Metric work]. doi:10.1145/RESERVED Tickoo, A., Chauhan, S., & Gupta, G. R. (2022). Friendliness Of Stack Overflow Towards Newbies. arXiv preprint arXiv:2208.10488. doi:10.48550/arXiv.2208.10488 Fischer, F., Böttinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., & Fahl, S. (2017, 22-26 May 2017). Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security. Paper presented at the 2017 IEEE Symposium on Security and Privacy (SP). doi:10.1109/SP.2017.31

[16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26]

[27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39]

An, L., Mlouki, O., Khomh, F., & Antoniol, G. (2017, 20-24 Feb. 2017). Stack Overflow: A code laundering platform? Paper presented at the 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). doi:10.1109/SANER.2017.7884629 Atwood, J. (2008). None of Us is as Dumb as All of Us. Retrieved from https://blog.codinghorror.com/stack-overflow-none-of-us-is-as-dumb-as-all-of-us/ Yeh, A. (2022). Teaching English as a Foreign Language in the Philippines. In Philippine English (pp. 353-362): Routledge. Lutz, B. (2009). Linguistic challenges in global software development: lessons learned in an international SW development division. Paper presented at the 2009 Fourth IEEE International Conference on Global Software Engineering. doi:10.1109/ICGSE.2009.33 Haner, J., & Garcia, D. (2019). The artificial intelligence arms race: Trends and world leaders in autonomous weapons development. Global Policy, 10(3), 331-337. doi:10.1111/1758-5899.12713 Broughel, J., & Thierer, A. D. (2019). Technological innovation and economic growth: A brief report on the evidence. Mercatus Research Paper. doi:10.2139/ssrn.3346495 Zolduoarrati, E., & Licorish, S. A. (2021). On the value of encouraging gender tolerance and inclusiveness in software engineering communities. Information and Software Technology, 139, 106667. doi:10.1016/j.infsof.2021.106667 Ford, D., Harkins, A., & Parnin, C. (2017). Someone like me: How does peer parity influence participation of women on stack overflow? Paper presented at the 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). doi:10.1109/VLHCC.2017.8103473 Abdulkareem, S. A., & Abboud, A. J. (2021). Evaluating Python, C++, JavaScript and Java Programming Languages Based on Software Complexity Calculator (Halstead Metrics). IOP Conference Series: Materials Science and Engineering, 1076(1), 012046. doi:10.1088/1757-899X/1076/1/012046 Brass, S., & Goldberg, C. (2004, 8-9 Sept. 2004). Semantic errors in SQL queries: a quite complete list. Paper presented at the Fourth International Conference onQuality Software, 2004. QSIC 2004. Proceedings. doi:10.1109/QSIC.2004.1357967 Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024). Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions. Paper presented at the Proceedings of the CHI Conference on Human Factors in Computing Systems. doi:10.1145/3613904.3642596 Anonymised. (2024). Does Location Influence Coding Practices? A Cross-Regional Study on Stack Overflow Code Quality [Data set]. doi:10.5281/zenodo.13622420 Ahmad, M. O., & Gustavsson, T. (2024). The Pandora's box of social, process, and people debts in software engineering. Journal of Software: Evolution and Process, 36(2), e2516. doi:10.1002/smr.2516 Luan, K., Ling, C.-D., & Xie, X.-Y. (2016). The nonlinear effects of educational diversity on team creativity. Asia Pacific Journal of Human Resources, 54(4), 465-480. doi:10.1111/1744-7941.12078 Olla, P., & Atkinson, C. (2004). Developing a wireless reference model for interpreting complexity in wireless projects. Industrial Management & Data Systems, 104(3), 262-272. doi:10.1108/02635570410525807 Kankanhalli, A., Tan, B. C. Y., & Wei, K.-K. (2006). Conflict and Performance in Global Virtual Teams. Journal of Management Information Systems, 23(3), 237-274. doi:10.2753/MIS0742-1222230309 Morin, J., & Ghosh, K. (2021). Linguistic Analysis of Stack Overflow Data: Native English vs Non-native English Speakers. Paper presented at the ECML PKDD 2021. Communications in Computer and Information Science, Cham. doi:10.1007/978-3-030-93733-1_9 Lai, J., & Widmar, N. O. (2021). Revisiting the Digital Divide in the COVID-19 Era. Applied Economic Perspectives and Policy, 43(1), 458-464. doi:10.1002/aepp.13104 Zolduoarrati, E., Licorish, S. A., & Stanger, N. (2023). Secondary studies on human aspects in software engineering: A tertiary study. Journal of Systems and Software, 200, 111654. doi:10.1016/j.jss.2023.111654 Blincoe, K., Springer, O., & Wrobel, M. R. (2019). Perceptions of Gender Diversity's Impact on Mood in Software Development Teams. IEEE Software, 36(5), 51-56. doi:10.1109/MS.2019.2917428 Hidellaarachchi, D., Grundy, J., Hoda, R., & Madampe, K. (2022). The Effects of Human Aspects on the Requirements Engineering Process: A Systematic Literature Review. IEEE Transactions on Software Engineering, 48(6), 2105-2127. doi:10.1109/TSE.2021.3051898 Rodríguez-Pérez, G., Nadri, R., & Nagappan, M. (2021). Perceived diversity in software engineering: a systematic literature review. Empirical Software Engineering, 26(5), 102. doi:10.1007/s10664-02109992-2 Rodríguez, P., Mäntylä, M., Oivo, M., Lwakatare, L. E., Seppänen, P., & Kuvaja, P. (2019). Chapter Four - Advances in Using Agile and Lean Processes for Software Development. In A. M. Memon (Ed.), Advances in Computers (Vol. 113, pp. 135-224): Elsevier. doi:10.1016/bs.adcom.2018.03.014 Chatley, R., Donaldson, A., & Mycroft, A. (2019). The Next 7000 Programming Languages. In B. Steffen & G. Woeginger (Eds.), Computing and Software Science: State of the Art and Perspectives (pp. 250282). Cham: Springer International Publishing. doi:10.1007/978-3-319-91908-9_15

[40] [41] [42] [43] [44] [45]

[46] [47] [48]

[49] [50] [51]

[52]

[53] [54] [55] [56] [57] [58] [59] [60]

Jiménez, R. C., Kuzak, M., Alhamdoosh, M., Barker, M., Batut, B., Borg, M., . . . Crouch, S. (2017). Four simple recommendations to encourage best practices in research software. F1000Res, 6. doi:10.12688/f1000research.11407.1 Haoues, M., Sellami, A., Ben-Abdallah, H., & Cheikhi, L. (2017). A guideline for software architecture selection based on ISO 25010 quality related characteristics. International Journal of System Assurance Engineering and Management, 8(2), 886-909. doi:10.1007/s13198-016-0546-8 Al-Kilidar, H., Cox, K., & Kitchenham, B. (2005, 17-18 Nov. 2005). The use and usefulness of the ISO/IEC 9126 quality standard. Paper presented at the 2005 International Symposium on Empirical Software Engineering, 2005. doi:10.1109/ISESE.2005.1541821 Izurieta, C., Griffith, I., & Huvaere, C. (2017, 9-10 Nov. 2017). An Industry Perspective to Comparing the SQALE and Quamoco Software Quality Models. Paper presented at the 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). doi:10.1109/ESEM.2017.42 Nugroho, A., Visser, J., & Kuipers, T. (2011). An empirical model of technical debt and interest. Paper presented at the Proceedings of the 2nd Workshop on Managing Technical Debt, Waikiki, Honolulu, HI, USA. doi:10.1145/1985362.1985364 Biase, M. d., Rastogi, A., Bruntink, M., & Deursen, A. v. (2019, 26-26 May 2019). The Delta Maintainability Model: Measuring Maintainability of Fine-Grained Code Changes. Paper presented at the 2019 IEEE/ACM International Conference on Technical Debt (TechDebt). doi:10.1109/TechDebt.2019.00030 Scalabrino, S., Linares-Vásquez, M., Oliveto, R., & Poshyvanyk, D. (2018). A comprehensive model for code readability. Journal of Software: Evolution and Process, 30(6), e1958. doi:10.1002/smr.1958 Buse, R. P. L., & Weimer, W. R. (2010). Learning a Metric for Code Readability. IEEE Transactions on Software Engineering, 36(4), 546-558. doi:10.1109/TSE.2009.70 AlOmar, E., Mkaouer, M. W., & Ouni, A. (2019, 28-28 May 2019). Can Refactoring Be Self-Affirmed? An Exploratory Study on How Developers Document Their Refactoring Activities in Commit Messages. Paper presented at the 2019 IEEE/ACM 3rd International Workshop on Refactoring (IWoR). doi:10.1109/IWoR.2019.00017 Peruma, A., Simmons, S., AlOmar, E. A., Newman, C. D., Mkaouer, M. W., & Ouni, A. (2021). How do I refactor this? An empirical study on refactoring trends and topics in Stack Overflow. Empirical Software Engineering, 27(1), 11. doi:10.1007/s10664-021-10045-x Zhang, H., Wang, S., Li, H., Chen, T. H., & Hassan, A. E. (2022). A Study of C/C++ Code Weaknesses on Stack Overflow. IEEE Transactions on Software Engineering, 48(7), 2359-2375. doi:10.1109/TSE.2021.3058985 Selvaraj, M., & Uddin, G. (2022). Does Collaborative Editing Help Mitigate Security Vulnerabilities in Crowd-Shared IoT Code Examples? Paper presented at the Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Helsinki, Finland. doi:10.1145/3544902.3546235 Firouzi, E., Sami, A., Khomh, F., & Uddin, G. (2020). On the use of C# Unsafe Code Context: An Empirical Study of Stack Overflow. Paper presented at the Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Bari, Italy. doi:10.1145/3382494.3422165 Treude, C., & Robillard, M. P. (2017, 17-22 Sept. 2017). Understanding Stack Overflow Code Fragments. Paper presented at the 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). doi:10.1109/ICSME.2017.24 Van Rossum, G., Warsaw, B., & Coghlan, N. (2001). PEP 8 – Style Guide for Python Code. In Python.org (Vol. 1565, pp. 28). Ahmad, M., & Cinnéide, M. Ó. (2019, 25-31 May 2019). Impact of Stack Overflow Code Snippets on Software Cohesion: A Preliminary Study. Paper presented at the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). doi:10.1109/MSR.2019.00050 Counsell, S., Swift, S., & Crampton, J. (2006). The Interpretation and Utility of Three Cohesion Metrics for Object-Oriented Design. ACM Trans. Softw. Eng. Methodol., 15(2), 123–149. doi:10.1145/1131421.1131422 Zerouali, A., Mens, T., & De Roover, C. (2021). On the usage of JavaScript, Python and Ruby packages in Docker Hub images. Science of Computer Programming, 207, 102653. doi:10.1016/j.scico.2021.102653 Geremia, S., Bavota, G., Oliveto, R., Lanza, M., & Penta, M. D. (2019, 30 Sept.-1 Oct. 2019). Characterizing Leveraged Stack Overflow Posts. Paper presented at the 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM). doi:10.1109/SCAM.2019.00025 Ebert, C., Cain, J., Antoniol, G., Counsell, S., & Laplante, P. (2016). Cyclomatic Complexity. IEEE Software, 33(6), 27-29. doi:10.1109/MS.2016.147 Pantiuchina, J., Lanza, M., & Bavota, G. (2018, 23-29 Sept. 2018). Improving Code: The (Mis) Perception of Quality Metrics. Paper presented at the 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). doi:10.1109/ICSME.2018.00017

[61]

[62]

[63] [64] [65] [66]

[67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81]

González, C. A., Zumel, L. A., Acero, J. A., Lenarduzzi, V., Martínez-Fernández, S., & Rodríguez, S. R. (2021, 6-10 Dec. 2021). A preliminary investigation of developer profiles based on their activities and code quality: Who does what? Paper presented at the 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). doi:10.1109/QRS54544.2021.00103 Lopez, T., Tun, T. T., Bandara, A., Levine, M., Nuseibeh, B., & Sharp, H. (2018). An investigation of security conversations in stack overflow: perceptions of security and community involvement. Paper presented at the Proceedings of the 1st International Workshop on Security Awareness from Design to Deployment, Gothenburg, Sweden. doi:10.1145/3194707.3194713 Jones, C., & Bonsignour, O. (2011). The Economics of Software Quality. Boston, MA: Addison-Wesley Professional. Boogerd, C., & Moonen, L. (2009, 16-17 May 2009). Evaluating the relation between coding standard violations and faultswithin and across software versions. Paper presented at the 2009 6th IEEE International Working Conference on Mining Software Repositories. doi:10.1109/MSR.2009.5069479 Santos, J. A. M., Rocha-Junior, J. B., Prates, L. C. L., Nascimento, R. S. d., Freitas, M. F., & Mendonça, M. G. d. (2018). A systematic review on the code smell effect. Journal of Systems and Software, 144, 450-477. doi:10.1016/j.jss.2018.07.035 Mitra, D., Arora, M., Rakhra, M., Kumar, C. R., Reddy, M. L., Reddy, S. P. K., . . . Shabaz, M. (2021). A Hybrid Framework to Control Software Architecture Erosion for Addressing Maintenance Issues. Annals of the Romanian Society for Cell Biology, 2974–2989-2974–2989. Retrieved from http://www.annalsofrscb.ro/index.php/journal/article/view/2839 Bhatia, S., & Malhotra, J. (2014). A survey on impact of lines of code on software complexity. Paper presented at the 2014 International Conference on Advances in Engineering & Technology Research (ICAETR - 2014). doi:10.1109/ICAETR.2014.7012875 Bartish, A., & Thevathayan, C. (2002). BDI Agents for Game Development. Paper presented at the Proceedings of the first international joint conference on Autonomous agents and multiagent systems: part 2, Bologna, Italy. doi:10.1145/544862.544901 Yang, D., Martins, P., Saini, V., & Lopes, C. (2017, 20-21 May 2017). Stack Overflow in GitHub: Any Snippets There? Paper presented at the 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). doi:10.1109/MSR.2017.13 Abdalkareem, R., Shihab, E., & Rilling, J. (2017). On code reuse from StackOverflow: An exploratory study on Android apps. Information and Software Technology, 88, 148-158. doi:10.1016/j.infsof.2017.04.005 Ndukwe, I. G., Licorish, S. A., Tahir, A., & MacDonell, S. G. (2023). How have views on Software Quality differed over time? Research and practice viewpoints. Journal of Systems and Software, 195, 111524. doi:10.1016/j.jss.2022.111524 Sedano, T. (2016, 5-6 April 2016). Code Readability Testing, an Empirical Study. Paper presented at the 2016 IEEE 29th International Conference on Software Engineering Education and Training (CSEET). doi:10.1109/CSEET.2016.36 Bhat, T., & Nagappan, N. (2006). Building Scalable Failure-proneness Models Using Complexity Metrics for Large Scale Software Systems. Paper presented at the 2006 13th Asia Pacific Software Engineering Conference (APSEC'06). doi:10.1109/APSEC.2006.25 Ryoo, J., Kazman, R., & Anand, P. (2015). Architectural Analysis for Security. IEEE Security & Privacy, 13(6), 52-59. doi:10.1109/MSP.2015.126 Chan, J. C.-W., & Paelinckx, D. (2008). Evaluation of Random Forest and Adaboost tree-based ensemble classification and spectral band selection for ecotope mapping using airborne hyperspectral imagery. Remote Sensing of Environment, 112(6), 2999-3011. doi:10.1016/j.rse.2008.02.011 Warren, P., Gaskell, C., & Boldyreff, C. (2001, 10-10 Nov. 2001). Preparing the ground for Website metrics research. Paper presented at the Proceedings 3rd International Workshop on Web Site Evolution. WSE 2001. doi:10.1109/WSE.2001.988789 Fritchey, G. (2018). SQL Query Performance Tuning. In G. Fritchey (Ed.), SQL Server 2017 Query Performance Tuning: Troubleshoot and Optimize Query Performance (pp. 1-22). Berkeley, CA: Apress. doi:10.1007/978-1-4842-3888-2_1 Beasley, R. E. (2020). Database Design, SQL, and Data Binding. In R. E. Beasley (Ed.), Essential ASP.NET Web Forms Development: Full Stack Programming with C#, SQL, Ajax, and JavaScript (pp. 359-394). Berkeley, CA: Apress. doi:10.1007/978-1-4842-5784-5_20 Cherfi, A., Nouira, K., & Ferchichi, A. (2018). Very Fast C4.5 Decision Tree Algorithm. Applied Artificial Intelligence, 32(2), 119-137. doi:10.1080/08839514.2018.1447479 Palli, A. S., Jaafar, J., Hashmani, M. A., Gomes, H. M., & Gilal, A. R. (2022). A Hybrid Sampling Approach for Imbalanced Binary and Multi-Class Data Using Clustering Analysis. IEEE Access, 10, 118639-118653. doi:10.1109/ACCESS.2022.3218463 Stack Overflow. (2022). Stack Overflow Developer Survey 2022. Retrieved from https://survey.stackoverflow.co/2022

[82] [83] [84] [85] [86] [87]

[88] [89] [90] [91] [92] [93] [94]

[95] [96] [97] [98]

[99] [100] [101] [102]

[103]

Paulson, L. D. (2007). Developers Shift to Dynamic Programming Languages. Computer, 40(2), 12-15. doi:10.1109/MC.2007.53 Tratt, L. (2009). Chapter 5: Dynamically Typed Languages. In Advances in Computers (Vol. 77, pp. 149184): Elsevier. doi:10.1016/S0065-2458(09)01205-4 Ali, T. O. M., Awadelseed, O. S. A., & Eldewahi, A. E. W. (2016, 20-23 Feb. 2016). Random multiple layouts: Keylogger prevention technique. Paper presented at the 2016 Conference of Basic Sciences and Engineering Studies (SGCAC). doi:10.1109/SGCAC.2016.7457997 Yang, D., Hussain, A., & Lopes, C. V. (2016). From Query to Usable Code: An Analysis of Stack Overflow Code Snippets. Paper presented at the Proceedings of the 13th International Conference on Mining Software Repositories, Austin, Texas. doi:10.1145/2901739.2901767 Imbugwa, G. B., Araújo, L. J. P. d., Khazeev, M., Enombe, E., Saliu, H., & Mazzara, M. (2021). A case study comparing static analysis tools for evaluating SwiftUI projects. Journal of Physics: Conference Series, 2134(1), 012022. doi:10.1088/1742-6596/2134/1/012022 Bogner, J., & Merkel, M. (2022). To Type or Not to Type? A Systematic Comparison of the Software Quality of JavaScript and Typescript Applications on GitHub. Paper presented at the Proceedings of the 19th International Conference on Mining Software Repositories, Pittsburgh, Pennsylvania. doi:10.1145/3524842.3528454 Anand, T., Cimmer, D., Heymann, E., Kiser, R., Miller, B., Ruh, I., . . . Zage, J. (2021). Open OnDemand Engagement Final Report. Retrieved from http://hdl.handle.net/2022/26590 Fluri, J., Fornari, F., & Pustulka, E. (2023, 14-15 May 2023). Measuring the Benefits of CI/CD Practices for Database Application Development. Paper presented at the 2023 IEEE/ACM International Conference on Software and System Processes (ICSSP). doi:10.1109/ICSSP59042.2023.00015 Dintyala, P., Narechania, A., & Arulraj, J. (2020). SQLCheck: Automated Detection and Diagnosis of SQL Anti-Patterns. Paper presented at the Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, Portland, OR, USA. doi:10.1145/3318464.3389754 Rafnsson, W., Giustolisi, R., Kragerup, M., & Høyrup, M. (2020). Fixing Vulnerabilities Automatically with Linters. Paper presented at the International Conference on Network and System Security, Cham. doi:10.1007/978-3-030-65745-1_13 Kapil, S. (2019). Debugging and Testing Python Code. In Clean Python: Elegant Coding in Python (pp. 221-251). Berkeley, CA: Apress. doi:10.1007/978-1-4842-4878-2_8 Peng, S., Liu, P., & Han, J. (2019). A Python Security Analysis Framework in Integrity Verification and Vulnerability Detection. Wuhan University Journal of Natural Sciences, 24(2), 141-148. doi:10.1007/s11859-019-1379-5 Beller, M., Bholanath, R., McIntosh, S., & Zaidman, A. (2016, 14-18 March 2016). Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software. Paper presented at the 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). doi:10.1109/SANER.2016.105 Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Gall, H. C., & Zaidman, A. (2020). How developers engage with static analysis tools in different contexts. Empirical Software Engineering, 25(2), 1419-1457. doi:10.1007/s10664-019-09750-5 Zampetti, F., Mudbhari, S., Arnaoudova, V., Di Penta, M., Panichella, S., & Antoniol, G. (2021). Using code reviews to automatically configure static analysis tools. Empirical Software Engineering, 27(1), 28. doi:10.1007/s10664-021-10076-4 Trautsch, A., Herbold, S., & Grabowski, J. (2023). Are automated static analysis tools worth it? An investigation into relative warning density and external software quality on the example of Apache open source projects. Empirical Software Engineering, 28(3), 66. doi:10.1007/s10664-023-10301-2 Wicklund, N., & Östlund, L. (2022). It Is Never Too Early to Learn About Code Quality : Analyzing Code Quality of First-Year Programming Students and the Difference Between TA Groups. (Independent thesis Basic level (degree of Bachelor) Student thesis). Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-320034 DiVA database. (2022:506) Kent, P. (1995). SQL Joins -- The Long and The Short of It. In: SAS Institute Inc. Trautsch, A., Herbold, S., & Grabowski, J. (2020). A longitudinal study of static analysis warning evolution and the effects of PMD on software quality in Apache open source projects. Empirical Software Engineering, 25(6), 5137-5192. doi:10.1007/s10664-020-09880-1 Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2023). How to Design and Evaluate Research in Education (11th ed.). San Francisco, CA: McGraw Hill. Licorish, S. A., & MacDonell, S. G. (2013). Adopting softer approaches in the study of repository data: a comparative analysis. Paper presented at the Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, Porto de Galinhas, Brazil. doi:10.1145/2460999.2461035 Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107-115. doi:10.1111/j.1365-2648.2007.04569.x

[104] [105]

[106] [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] [123]

[124] [125] [126] [127]

Anandarajan, M., Hill, C., & Nolan, T. (2019). The Fundamentals of Content Analysis. In Practical Text Analytics: Maximizing the Value of Text Data (pp. 15-25). Cham: Springer International Publishing. doi:10.1007/978-3-319-95663-3_2 Vourvachis, P. (2007). On the use of content analysis (CA) in corporate social reporting (CSR): Revisiting the debate on the units of analysis and the ways to define them. Paper presented at the British Accounting Association Annual Conference 2007. Retrieved from https://eprints.kingston.ac.uk/id/eprint/4129 Jang, M., & Kim, S. (2024). Key traits of top answerers on Korean Social Q&A platforms: insights into user performance and entrepreneurial potential. Humanities and Social Sciences Communications, 11(1), 744. doi:10.1057/s41599-024-03239-3 Computing Technology Industry Association. (2023). Cyberstates 2023: State of the Tech Workforce. Retrieved from https://comptiacdn.azureedge.net/webcontent/docs/default-source/researchreports/comptia-state-of-the-tech-workforce-2023.pdf Tonkin, J. D., Heino, J., Sundermann, A., Haase, P., & Jähnig, S. C. (2016). Context dependency in biodiversity patterns of central German stream metacommunities. Freshwater Biology, 61(5), 607-620. doi:10.1111/fwb.12728 Alharthi, H., Outioua, D., & Baysal, O. (2016). Predicting questions’ scores on Stack Overflow. Paper presented at the Proceedings of the 3rd International Workshop on CrowdSourcing in Software Engineering. doi:10.1145/2897659.2897661 United States Government Accountability Office (1996). Content Analysis: A Methodology for Structuring and Analyzing Written Material. In. Jordan, A., Kunkel, D., Manganello, J., & Fishbein, M. (2010). Media messages and public health: A decisions approach to content analysis. New York City, NY: Routledge. Rane, N., Choudhary, S., & Rane, J. (2024). Gemini versus ChatGPT: applications, performance, architecture, capabilities, and implementation. Journal of Applied Artificial Intelligence, 5(1), 69-93. doi:10.48185/jaai.v5i1.1052 Neuendorf, K. A. (2017). The Content Analysis Guidebook (2nd ed.). Thousand Oaks, CA: SAGE. Danielson, W. A., & Mullen, J. J. (1965). A Basic Space Unit for Newspaper Content Analysis. Journalism Quarterly, 42(1), 108-110. doi:10.1177/107769906504200114 McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276-282. Retrieved from https://hrcak.srce.hr/89395 Wang, Y., Wang, C., Li, X., Yun, S., & Song, M. (2014). How are identifiers named in open source software? About popularity and consistency. arXiv:1401.5300. doi:10.48550/arXiv.1401.5300 Pechtchanski, I., & Sarkar, V. (2002). Immutability specification and its applications. Paper presented at the Proceedings of the 2002 joint ACM-ISCOPE conference on Java Grande, Seattle, Washington, USA. doi:10.1145/583810.583833 Karwin, B. (2010). SQL Antipatterns: Avoiding the Pitfalls of Database Programming. Raleigh, NC: Pragmatic Bookshelf. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.): L. Erlbaum Associates. doi:10.1016/C2013-0-10517-X Leahey, E. (2005). Alphas and Asterisks: The Development of Statistical Significance Testing Standards in Sociology. Social Forces, 84(1), 1-24. doi:10.1353/sof.2005.0108 Holzmann, G. J. (2019). Does Not Compute. IEEE Software, 36(3), 14-16. doi:10.1109/MS.2019.2896053 Reid, B., Treude, C., & Wagner, M. (2020). Optimising the Fit of Stack Overflow Code Snippets into Existing Code. Paper presented at the Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, Cancún, Mexico. doi:10.1145/3377929.3398087 Regulwar, G. B., Thakare, V., & Tugnayat, R. (2020). Implementation of Code Cleanup and Code Standard Refactoring with PMD and UCD Tool. Paper presented at the ICDSMLA 2019: Proceedings of the 1st International Conference on Data Science, Machine Learning and Applications. doi:10.1007/978981-15-1420-3_48 Coblenz, M., Nelson, W., Aldrich, J., Myers, B., & Sunshine, J. (2017). Glacier: Transitive class immutability for Java. Paper presented at the 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). doi:10.1109/ICSE.2017.52 Akam, T., Lustig, A., Rowland, J. M., Kapanaiah, S. K. T., Esteve-Agraz, J., Panniello, M., . . . Walton, M. E. (2022). Open-source, Python-based, hardware and software for controlling behavioural neuroscience experiments. eLife, 11, e67846. doi:10.7554/eLife.67846 Yannakoudakis, E. J., & Cheng, C. P. (1989). A domain-oriented approach to improve the userfriendliness of SQL. Computer Standards & Interfaces, 9(2), 127-141. doi:10.1016/09205489(89)90005-6 Mithani, F., Machchhar, S., & Jasdanwala, F. (2016, 15-17 Dec. 2016). A novel approach for SQL query optimization. Paper presented at the 2016 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC). doi:10.1109/ICCIC.2016.7919717

[128] [129] [130] [131] [132] [133] [134] [135] [136] [137] [138] [139] [140] [141] [142] [143] [144] [145] [146] [147] [148] [149] [150]

[151]

Dageville, B., Das, D., Dias, K., Yagoub, K., Zait, M., & Ziauddin, M. (2004). Automatic SQL tuning in Oracle 10g. Paper presented at the Proceedings of the Thirtieth International Conference on Very Large Data Bases, Toronto, Canada. doi:10.5555/1316689.1316784 Hunter-Zinck, H., de Siqueira, A. F., Vásquez, V. N., Barnes, R., & Martinez, C. C. (2021). Ten simple rules on writing clean and reliable open-source scientific software. PLOS Computational Biology, 17(11), e1009481. doi:10.1371/journal.pcbi.1009481 Schorsch, T. M. (1990). Increasing the readability and comprehensibility of programs. (Master's Thesis). University of Colorado, Retrieved from https://apps.dtic.mil/sti/citations/ADA231805 Flanagan, D., & Matsumoto, Y. (2008). The Ruby Programming Language: Everything You Need to Know. Sebastopol, CA: O'Reilly Media, Inc. Duchnicky, R. L., & Kolers, P. A. (1983). Readability of Text Scrolled on Visual Display Terminals as a Function of Window Size. Human Factors, 25(6), 683-692. doi:10.1177/001872088302500605 Wolf, D., & Henley, A. J. (2017). What Is a Database? In Java EE Web Application Primer: Building Bullhorn: A Messaging App with JSP, Servlets, JavaScript, Bootstrap and Oracle (pp. 5-10). Berkeley, CA: Apress. doi:10.1007/978-1-4842-3195-1_2 Lyu, Y., Alotaibi, A., & Halfond, W. G. J. (2019, 29 Sept.-4 Oct. 2019). Quantifying the Performance Impact of SQL Antipatterns on Mobile Applications. Paper presented at the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). doi:10.1109/ICSME.2019.00015 Mosses, P. D. (2006). Formal Semantics of Programming Languages: — An Overview —. Electronic Notes in Theoretical Computer Science, 148(1), 41-73. doi:10.1016/j.entcs.2005.12.012 Matsumoto, Y. (2002). Ruby in a nutshell: a desktop quick reference. Sebastopol, CA: O'Reilly Media, Inc. Inden, M. (2022). Strings. In Java Challenges: 100+ Proven Tasks that Will Prepare You for Anything (pp. 141-209). Berkeley, CA: Apress. doi:10.1007/978-1-4842-7395-1_4 Tovarnák, D., Vaeková, A., Novák, S., & Pitner, T. (2013, 9-12 Dec. 2013). Structured and Interoperable Logging for the Cloud Computing Era: The Pitfalls and Benefits. Paper presented at the 2013 IEEE/ACM 6th International Conference on Utility and Cloud Computing. doi:10.1109/UCC.2013.30 Shar, L. K., & Tan, H. B. K. (2013). Defeating SQL Injection. Computer, 46(3), 69-77. doi:10.1109/MC.2012.283 Ferrari, D., Carminati, M., Polino, M., & Zanero, S. (2020). NoSQL Breakdown: A Large-scale Analysis of Misconfigured NoSQL Services. Paper presented at the Proceedings of the 36th Annual Computer Security Applications Conference, Austin, USA. doi:10.1145/3427228.3427260 Shahriar, H., & Zulkernine, M. (2011, 18-22 July 2011). Injecting Comments to Detect JavaScript Code Injection Attacks. Paper presented at the 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops. doi:10.1109/COMPSACW.2011.27 James, F. (1990). A review of pseudorandom number generators. Computer Physics Communications, 60(3), 329-344. doi:10.1016/0010-4655(90)90032-V Kelsey, J., Schneier, B., Wagner, D., & Hall, C. (1998). Cryptanalytic Attacks on Pseudorandom Number Generators. Paper presented at the Fast Software Encryption, Berlin, Heidelberg. doi:10.1007/3-54069710-1_12 Tenkanen, T., Kallio, H., & Poikolainen, J. (2018). Security Challenges of IoT-Based Smart Home Appliances. In M. Lehto & P. Neittaanmäki (Eds.), Cyber Security: Power and Technology (pp. 271288). doi:10.1007/978-3-319-75307-2_16 Boucher, N., & Anderson, R. (2023). Trojan source: Invisible vulnerabilities. Paper presented at the 32nd USENIX Security Symposium (USENIX Security 23). Retrieved from https://www.usenix.org/conference/usenixsecurity23/presentation/boucher Computing Technology Industry Association. (2023). Cyberstates 2023: The Definitive Guide to the US Tech Industry and Tech Workforce. Retrieved from https://www.cyberstates.org/pdf/CompTIA_Cyberstates_2021.pdf Messina, J., & Dvorkin, E. (2019). Expanding Tech Apprenticeships in New York City. Center for an Urban Future. Retrieved from https://eric.ed.gov/?id=ED601220 Besker, T., Martini, A., Lokuge, R. E., Blincoe, K., & Bosch, J. (2018, 23-29 Sept. 2018). Embracing Technical Debt, from a Startup Company Perspective. Paper presented at the 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). doi:10.1109/ICSME.2018.00051 Shah, H., Gorg, C., & Harrold, M. J. (2010). Understanding Exception Handling: Viewpoints of Novices and Experts. IEEE Transactions on Software Engineering, 36(2), 150-161. doi:10.1109/TSE.2010.7 Kim, E.-J., & You, Y.-Y. (2022). A Study on the Impact of Startups of Security Awareness on the Sustainability of Corporate Growth. In S. R. Samanta, P. K. Mallick, P. K. Pattnaik, J. R. Mohanty, & Z. Polkowski (Eds.), Cognitive Computing for Risk Management (pp. 115-124). Cham: Springer International Publishing. doi:10.1007/978-3-030-74517-2_8 Gupta, V., Fernandez-Crehuet, J. M., & Hanne, T. (2020). Fostering Continuous Value Proposition Innovation through Freelancer Involvement in Software Startups: Insights from Multiple Case Studies. Sustainability, 12(21), 8922. doi:10.3390/su12218922

[152] [153] [154] [155] [156]

[157] [158] [159] [160] [161] [162] [163] [164] [165] [166] [167]

[168] [169] [170] [171] [172] [173] [174] [175]

Klinger, J., Mateos-Garcia, J., & Stathoulopoulos, K. (2018). Deep learning, deep change? Mapping the development of the Artificial Intelligence General Purpose Technology. arXiv preprint arXiv:1808.06355. doi:10.48550/arXiv.1808.06355 Lohr, S. (2015). Data-ism: Inside the Big Data Revolution. New York City, NY: Simon and Schuster. Stayton, J., & Mangematin, V. (2016). Startup time, innovation and organizational emergence: A study of USA-based international technology ventures. Journal of International Entrepreneurship, 14(3), 373409. doi:10.1007/s10843-016-0183-y U.S. Bureau of Labor Statistics. (2022). Occupational Employment and Wage Statistics: Web Developers [Data set]. Retrieved from https://data.bls.gov/oes/#/home Crawford, T. (2017). A Comparison of Server Side Scripting Technologies. Paper presented at the Proceedings of the International Conference on Software Engineering Research and Practice (SERP), Athens. Retrieved from https://www.proquest.com/openview/0caec45c2e3b5393fb097383d96285a5/1?pqorigsite=gscholar&cbl=1976341 City of Irving Texas. (2022). City of Irving, Texas: Annual Comprehensive Financial Report 2021-22. Retrieved from https://www.cityofirving.org/ArchiveCenter/ViewFile/Item/4286 Xiong, Q., Lian, S., Zeng, Z., He, R., Zhu, B., & Yang, X. (2023). An empirical analysis of vulnerability information disclosure impact on patch R&D of software vendors. Journal of Intelligent & Fuzzy Systems, 44, 839-853. doi:10.3233/JIFS-221316 Swart, J. (2020). Gini Index: Conceiving Inequality in One Single Number. In W. Leal Filho, A. M. Azul, L. Brandli, A. Lange Salvia, P. G. Özuyar, & T. Wall (Eds.), No Poverty (pp. 1-13). Cham: Springer International Publishing. doi:10.1007/978-3-319-69625-6_76-1 Wilterdink, N. (2016). American exceptionalism? The growth of income and wealth inequality in the United States and other Western societies. Human Figurations, 5(2). Retrieved from http://hdl.handle.net/2027/spo.11217607.0005.204 Illinois State University. (2023). State Fiscal Support for Higher Education, Fiscal Years 2020-21 [Data set]. Data set. Retrieved from https://education.illinoisstate.edu/downloads/grapevine/tables/fy21/GPV_Table1_FY21.pdf Kemmerer, R. A. (2003, 3-10 May 2003). Cybersecurity. Paper presented at the 25th International Conference on Software Engineering, 2003. Proceedings. doi:10.1109/ICSE.2003.1201257 Cerny, T., Svacina, J., Das, D., Bushong, V., Bures, M., Tisnovsky, P., . . . Huang, J. (2020). On Code Analysis Opportunities and Challenges for Enterprise Systems and Microservices. IEEE Access, 8, 159449-159470. doi:10.1109/ACCESS.2020.3019985 Mihelič, A., Vrhovec, S., & Hovelja, T. (2023). Agile Development of Secure Software for Small and Medium-Sized Enterprises. Sustainability, 15(1), 801. doi:10.3390/su15010801 Silva, E. F., Dembogurski, B. J., & Semaan, G. S. (2023). A literature review of computational thinking in early ages. International Journal of Early Years Education, 31(3), 753-772. doi:10.1080/09669760.2022.2107491 Mattingly, M. J., & Schaefer, A. (2021). Education in Rural America. In Serving Gifted Students in Rural Settings (1 ed., pp. 53-70). New York City, NY: Routledge. doi:10.4324/9781003237938-4 Wu, L., Looi, C.-K., Multisilta, J., How, M.-L., Choi, H., Hsu, T.-C., & Tuomi, P. (2020). Teacher’s Perceptions and Readiness to Teach Coding Skills: A Comparative Study Between Finland, Mainland China, Singapore, Taiwan, and South Korea. The Asia-Pacific Education Researcher, 29(1), 21-34. doi:10.1007/s40299-019-00485-x Liao, S.-M. (2023). SCRATCH to R: Toward an Inclusive Pedagogy in Teaching Coding. Journal of Statistics and Data Science Education, 31(1), 45-56. doi:10.1080/26939169.2022.2090467 Vasilescu, B., Capiluppi, A., & Serebrenik, A. (2012, 14-16 Dec. 2012). Gender, Representation and Online Participation: A Quantitative Study of StackOverflow. Paper presented at the 2012 International Conference on Social Informatics. doi:10.1109/SocialInformatics.2012.81 Nochlin, L. (2017). Why Are There No Great Women Artists? In D. Goldblatt, L. B. Brown, & S. Patridge (Eds.), Aesthetics (4 ed., pp. 46-51). New York City, NY: Routledge. doi:10.4324/9781315303673-12 Maharani, A., Sulaiman, H., Saifurrohman, Aminah, N., & Rosita, C. D. (2019). Analyzing the student’s cognitive abilities through the thinking levels of geometry van hiele reviewed from gender perspective. Journal of Physics: Conference Series, 1188(1), 012066. doi:10.1088/1742-6596/1188/1/012066 Bulka, D., & Mayhew, D. (1999). Efficient C++: performance programming techniques. Reading, MA: Addison-Wesley. Spinellis, D. (2006). Code Quality: The Open Source Perspective (Effective Software Development Series). Reading, MA: Addison-Wesley Professional. Weinberger, R., Springel, V., & Pakmor, R. (2020). The AREPO Public Code Release. The Astrophysical Journal Supplement Series, 248(2), 32. doi:10.3847/1538-4365/ab908c Pikulin, V., Kubo, D., Nissanka, K., Bandara, S., Shamsiemon, M. A., Yasmin, A., . . . Kanij, T. (2023, 11-15 Sept. 2023). Towards Developer-Centered Secure Coding Training. Paper presented at the 2023

[176] [177] [178] [179]

[180]

[181] [182] [183]

38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW). doi:10.1109/ASEW60602.2023.00008 Bowen, P. L., Heales, J., & Vongphakdi, M. T. (2002). Reliability factors in business software: volatility, requirements and end-users. Information Systems Journal, 12(3), 185-213. doi:10.1046/j.13652575.2002.00128.x Röhl, K.-H. (2019). Entrepreneurship: a comparative study of the interplay of culture and personality from a regional perspective. Journal of Small Business & Entrepreneurship, 31(2), 119-139. doi:10.1080/08276331.2018.1462621 Ribbers, P. M. A., & Schoo, K. C. (2002, 10-10 Jan. 2002). Designing complex software implementation programs. Paper presented at the Proceedings of the 35th Annual Hawaii International Conference on System Sciences. doi:10.1109/HICSS.2002.994374 Moore, J. R., Pankow, K. L., Ford, S. R., Koper, K. D., Hale, J. M., Aaron, J., & Larsen, C. F. (2017). Dynamics of the Bingham Canyon rock avalanches (Utah, USA) resolved from topographic, seismic, and infrasound data. Journal of Geophysical Research: Earth Surface, 122(3), 615-640. doi:10.1002/2016JF004036 Salminen, J., Santos, J. M., Kwak, H., An, J., Jung, S.-g., & Jansen, B. J. (2020). Persona Perception Scale: Development and Exploratory Validation of an Instrument for Evaluating Individuals’ Perceptions of Personas. International Journal of Human-Computer Studies, 141, 102437. doi:10.1016/j.ijhcs.2020.102437 Lotter, A., Licorish, S. A., Savarimuthu, B. T. R., & Meldrum, S. (2018, 26-30 Nov. 2018). Code Reuse in Stack Overflow and Popular Open Source Java Projects. Paper presented at the 2018 25th Australasian Software Engineering Conference (ASWEC). doi:10.1109/ASWEC.2018.00027 Westen, D., & Rosenthal, R. (2003). Quantifying construct validity: Two simple measures. Journal of Personality and Social Psychology, 84(3), 608-618. doi:10.1037/0022-3514.84.3.608 McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276-282. doi:10.11613/BM.2012.031

Record · ID 155363 · SHA-256 c7a0baef26c4e976
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.