Feature Article: Software Testing
All Green, Still Broken: Real-Flow Verification Lessons from an LLM-Integrated, Multi-Market Web Application Muhammad Bilal, Technical University of Munich, Germany
arXiv:2606.22475v1 [cs.SE] 21 Jun 2026
Ali Hassaan Mughal, Independent Researcher, USA
Abstract—Modern web applications increasingly combine three ingredients that are hard to test: output from large language models, multi-market internationalization, and browser-driven front-ends over external data sources. We report on a production rental-search assistant whose automated suite grew to 1,553 test cases in six weeks. The suite passed continuously, yet user-facing defects continued to reach production. We studied all 252 bug-fix commits in the project and classified each by the boundary, or seam, it escaped through. About 44 percent of the fixes fall in four seams that component-level unit tests cannot observe: the live browser runtime, the non-default market, the end-to-end flow, and the whole-system level. A fix without a guard at the seam let one defect ship twice. We present the four-seam framework, the measured defect distribution, and the practices we adopted, including a simple way for a team to find the seam that carries the most fixes.
with a passing run required before deployment.
KEY INSIGHTS › A large, passing test suite can miss whole classes of defect. A component test replaces the uncontrolled side of a boundary with a stand-in to run deterministically. That stand-in hides the seam. In the system studied, about 44 percent of bug fixes landed in such seams, so test count is a weak measure of quality. › A single end-to-end pass through the running product, performed as a user in a non-default market, observes seams that component-level tests cannot. › When the same defect recurs, fix the whole class behind it. Add a build check that fails on any repeat, so the same defect cannot ship again.
T
he system studied here is a production web application developed over a six-week period. Its automated test suite grew to 1,553 test cases across 144 files and was run on every change,
This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Preprint version; submitted to IEEE Software. Preprint — not the published version
During the same period the project recorded 252 bug-fix commits. Six user-facing defects reached production. One recurred after an earlier fix. All six passed the existing suite before release. This article examines why a large passing suite did not prevent these defects. We classified all 252 bugfix commits in the project. Most escaped defects sat at boundaries the components do not control: the browser runtime, the non-default market, and the end-to-end user flow. We call these boundaries seams. We then measure how poorly component-level tests observe them. Prior work has already shown that coverage correlates weakly with how many defects a suite catches. [1] This article makes three contributions: i) a four-seam framework for the under-studied setting of one system that combines model output, multi-market internationalization, and a browser runtime; ii) a full-census classification of 252 defects by the seam through which each escaped; and iii) a reproducible procedure that identifies the seam carrying the most fixes in a repository. Figure 1 summarizes the argument. IEEE Software
1
FEATURE ARTICLE
All green, still broken A continuously passing suite leaks defects at four boundaries it never observes.
Reaches users 6 escaped the suite 1 a verbatim recurrence
✓ Automated suite
1,553 tests, every run green
Runtime 29 fixes
Market 23 fixes
Flow 35 fixes
System 23 fixes
110 of 252 bug fixes (44%) escaped at these four seams, none of them observable by a component-level unit test
THE SYSTEM AND ITS TESTS The system is a rental-search assistant. It collects listings from several portals. It screens each listing against a user’s criteria with a large language model. A second model drafts an outreach message. The results appear on a dashboard of server-rendered HTML with small client-side libraries on top. The system serves two markets. The first is the original market, with one European city, one currency, and one language. The second was added later, with a different country, currency, and language. Figure 2 shows how the suite grew. It reached 1,364 test functions, which pytest expands to 1,553 test cases, in 43 days. The curve flattens after about three weeks. All defects discussed in this report occurred after that point, while the suite was already large and passing. The suite is broad. About 30 percent of the tests cover data ingestion and parsing, 21 percent configuration and utilities, 10 percent operations, 10 percent internationalization, 9 percent the dashboard, 8 percent the screening pipeline, 6 percent authentication, and 6 percent the drafter. The exercised code paths and the actual product behaviors are not the same set. The defects below fall in that gap.
THE FOUR SEAMS A seam is a boundary where our code meets something it does not control. We saw four of them, shown in Figure 3. 2
IEEE Software
Test functions (cumulative)
FIGURE 1. The paradox this article examines. An automated suite of 1,553 tests ran green on every change, yet defects escaped at four boundaries that component-level tests never observe. A full-census study of 252 bug-fix commits places 110 of them (44 percent) at these four seams (Runtime, Market, Flow, and System), and six such defects reached users, one of them a verbatim recurrence.
1,600
All reported defects shipped after week 3, when the suite was already large and green
1,200
1,364
800
400
0 0
7
14
21
28
35
42
Days since first commit FIGURE 2. Growth of the test suite over 43 days, counted as test functions. The suite reached 1,364 functions, which pytest expands to 1,553 cases. Growth plateaus after about three weeks; every defect in this report shipped in the shaded region that follows, when the suite was already large and passing.
The runtime seam is where server output meets a browser that runs scripts. Our tests checked the HTML we sent. They did not run the browser that turns that HTML into behavior. The market seam is where default-market assumptions meet a user in another market. Our tests and our manual checks used our own account, which always lived in the first market. The flow seam is where individually correct parts meet a real user interaction. A handler and a template can each be correct on their own. The assembled interaction can still fail. 2026
ALL GREEN, STILL BROKEN
Runtime browser scripts 29 fixes
98
Backend Data
System whole-system rule 23 fixes
Component code unit-tested
Market non-default user 23 fixes
Flow user interaction 35 fixes
44 35
Flow
Market
23
System
23 0
FIGURE 3. The shaded dashed region shows what a component-level unit test observes: one component, with the rest stubbed out. The four seams lie just outside it, where the code meets conditions it does not control, and the orange arrows show defects escaping across that boundary. Counts are bug fixes per seam from the census of Figure 4.
The system seam is where a local change meets a whole-system rule. One component can pass every test while a system-level guard misreads the new behavior. The four seams share one mechanism. A component test swaps the uncontrolled side of each boundary for a stand-in: a stored HTML string for a live browser, the default configuration for the market axis, a stub for an adjacent component, a fixed constant for the wholesystem baseline. The swap makes the test repeatable. It also removes the seam from view. A seam is the slice of behavior a test gives up to gain determinism. Adding more such tests buys confidence inside the stubbed world. It does not move the boundary of what gets observed. That is why the suite stayed large and green while every defect in this report shipped.
A STUDY OF 252 DEFECTS We classified every bug-fix commit in the project against the four seams.
Method The classification followed four steps. i) Sampling. We took one snapshot of the repository: 740 commits across six weeks. From these we selected every commit whose message begins with the conventionalcommit prefix “fix.” That gave 252 bug-fix commits against 272 feature commits, nearly one fix per feature. ii) Taxonomy derivation. We defined the four seams from the six initial defects (Table 1), then applied them unchanged to the full census. iii) Classification. A short list of priority-ordered keyword rules assigned each 2026
110 fixes (44%, unitinvisible)
29
Runtime
20
40
60
80
100
120
Bug-fix commits (of 252) FIGURE 4. All 252 bug-fix commits classified by seam, using the generic published rules and sorted by frequency. The four seam categories (Flow, Runtime, Market, System) sum to 110 fixes, or 44 percent, none observable by a componentlevel unit test. “Backend” (ordinary server logic) and “Data” (external sites changing) are where component tests are already strong.
fix to one category. A fix reaches a seam only when its message uses that seam’s vocabulary; everything else falls into a general backend group. The rules are deterministic and contain no project-specific names.1 We did not hand-edit the output. A deterministic classifier has no inter-rater variance, so the threat here is rule validity, not reliability. We treat the four-seam share as a conservative lower bound, for the reasons in Study Limitations. iv) Guard proxy. We recorded whether each fix also changed a test file, as a proxy for adding a regression guard.
Results Figure 4 shows the distribution. About 110 fixes, or 44 percent, fall in the four seams that unit tests cannot see: 35 in the flow seam, 29 in the runtime seam, 23 in the market seam, and 23 in the system seam. Data-source fixes are 17 percent (44 fixes). The largest single group, 39 percent, is ordinary backend logic, where component tests are strong. No single seam dominates. Together the four seams total 110 fixes, more than double the data-source churn that teams already treat as routine work. Of the 252 fixes, 107 (42.5 percent) also changed a test file. That is an upper bound on real regression
1 Replication package (classifier rules and aggregate counts) archived at Zenodo: https://doi.org/10.5281/zenodo. 20780007.
IEEE Software
3
FEATURE ARTICLE
guards, since touching a test is not the same as adding the right one. One defect still shipped twice. A quoting error in a client-side component broke a form step. It was visible only in a running browser. The first fix left behind no test that could catch it again. The guard lived in the same browser-blind harness that had missed the defect, so it could not see the repeat. The 42.5 percent figure counts how often a fix shipped with a test. It does not show whether that test sits where the defect can return.
Four Defects Up Close One defect from each seam follows. In the runtime seam, a feature resolved a pasted listing link. It passed every server-side test but failed in the browser. The search box sent its request dynamically, and the handler replied with an ordinary redirect instead of the header the client library expected. The page did not update. In the market seam, a user in the second market tried to save a search and got an error. A validator still encoded a rule from the first market and rejected a value that was valid in the second. The branch had a passing unit test. No run had exercised it as a user in the second market. In the flow seam, the budget preset controls did not respond when selected. Each part was correct in isolation. No test exercised the interaction. In the system seam, ingestion volume was deliberately increased. A monitor compared volume against a fixed baseline. It read the higher numbers as a broken pipeline and raised a false alarm. Every test of the monitor passed, because each still assumed the old baseline. Table 1 lists all six, with at least one in each seam.
4
iii) We turn recurring defects into build checks. One guard renders every screen for the second market and fails the build on any leak of the wrong currency, the wrong city, or a first-language string. A companion check confirms that the first market is unchanged. A refactor routed all market differences through one place, so the guard has a single point to watch. The defect that had shipped twice can no longer recur. iv) System guards must expect intentional change. We taught our throughput watchdog about deliberate volume changes. A healthy change no longer trips an alarm meant for a broken one. Table 2 collects these habits as a checklist that other teams can apply. Each row pairs a seam with what component tests miss and the lowest-cost check we adopted, and points to the matching incidents in Table 1. The checklist is not specific to this system. Run the classifier from the replication package on a project’s commit history. It reports which seam carries the most fixes. That seam is where additional testing is most likely to reduce escaped defects.
FACTORS SPECIFIC TO THIS SETTING
WHAT WE CHANGED
Three properties of this class of system intensify the problem. Output from a language model is hard to unit test. It is not deterministic, and its correctness is fuzzy, a version of the test oracle problem. [3] Developers compensate with many narrow assertions, [4], [5], [6] even as the model keeps changing. [7] Internationalization multiplies every flow by a market axis that stays hidden unless someone walks it. Browser-driven front-ends move real logic into a runtime the backend test harness never starts. Each property pushes effort toward isolated tests and away from the seams. This project had all three.
We did not respond by writing more unit tests. We changed our definition of done and where we spend test effort. We now treat an untested seam as technical debt to track and pay down. [2] i) Real-flow verification is now part of done. A change is finished only after we run the real user action in the real runtime, across its states: a dynamic request and a plain one, zero rows and many, each data source and an unsupported one, the success path and each error path. In practice this is a short scripted pass through the running interface in a browser. ii) We test along the market axis. Every marketaware screen is now exercised end to end as a user in the non-default market. That configuration is the one our routine development left uncovered.
The weak link between test coverage and bug finding is established. Inozemtseva and Holmes showed that coverage is not strongly correlated with test-suite effectiveness. [1] That fits a suite that is large but blind to these boundaries. A related limit is the test oracle problem: deciding whether a given output is correct. [3] The seam problem studied here is upstream of the oracle. At a seam the test environment never produces the production condition. The input that would expose the defect is never generated, and the oracle is never consulted. Coverage and oracle quality both presuppose that the triggering condition arises; at a seam it
IEEE Software
RELATED WORK
2026
ALL GREEN, STILL BROKEN
TABLE 1. Six user-facing defects, grouped by the seam each exposed. All passed the test suite before release; one recurred after an earlier fix.
#
What shipped broken
Seam
Why the green suite missed it
1
Pasted listing URL resolved server-side but failed over the dynamic search box (it needed an HX-Redirect header, not a 303 redirect) A real second-market user received HTTP 422 saving criteria; the validator rejected a valid local listing type Onboarding overwrote a second-market user’s chosen search type A quoting error in a client-side component broke a form step from initializing, a verbatim recurrence Budget preset buttons were silently ignored in both markets A false “pipeline broken” page appeared after we widened ingestion volume
Runtime
Validation never drove the real browser
Market
Verification walked only a default-market user
Market
Same blindness, a different surface
Runtime
Visible only in a live browser
Flow
Caught only by real interaction
System
Unit tests did not model a system-level baseline shift
2 3 4 5 6
TABLE 2. Seam-coverage checklist. Each row is backed by a practice we adopted and the incidents that prompted it (Table 1).
Seam
What component tests miss
The lowest-cost check we adopted
Example
Runtime / browser
Behavior that appears only when the browser runs scripts, such as dynamic-request handling and client-side initialization Branches that a default-market user never reaches
One scripted real-browser pass for each changed screen, across its request and error states
Inc. 1, 4
Walk each market-aware screen end to end as a user in a non-default market Drive the real end-to-end user interaction, not its parts in isolation Make system guards aware of intended changes and re-baseline on deliberate volume changes Promote the defect class to a build-level check that fails on any repeat
Inc. 2, 3
Market / i18n Flow / UX System / ops Recurrence
The wiring between individually correct parts Whole-system shifts caused by an intended local change A fixed defect returning because the guard was not at the seam
does not. The four seams map where that precondition fails in one modern stack. A second line of work studies the link between a test and the behavior it protects. Traceability methods recover which test exercises which code, [8] and regression analysis traces which change introduced a bug. [9] Curated fault datasets such as Defects4J [10] and BugsInPy [11] pair each real bug with a triggering test. Assembling those links by hand is costly. Our recurrence showed the same thing in the field: a fix without a guard at the seam protects nothing. An untested seam is a form of technical debt to track and pay down. [2] Internationalization testing has been studied mainly at the presentation layer. Empirical work catalogs internationalization layout and string-resource defects in mobile interfaces [12] and detects layout and config2026
Inc. 5 Inc. 6 Inc. 4
uration failures in web pages, while explicitly setting aside non-presentation locale defects. [13] We found no prior work that treats locale-dependent validation or business logic as a defect seam. That is the marketseam class this report documents. The escape mechanism also resembles variability bugs: defects that appear only under certain build configurations, which configuration-oblivious analyses miss. [14] The market seam is a runtime-locale analogue.
STUDY LIMITATIONS This is an experience report on a single application, developed and maintained by its two authors. The incidents are self-reported. We count the defects we found, so silent escapes are not measured. The seam classification is automated from commit-message keyIEEE Software
5
FEATURE ARTICLE
words, with no hand-editing. Classification from commit messages is known to mislabel some commits, [15] so the result is approximate. A fix is placed in a seam only when its message uses that seam’s vocabulary. Fixes worded differently fall into a general backend group, which holds 39 percent of fixes. The fourseam share is therefore a conservative lower bound. Whether the four seams generalize to other systems that combine model output, internationalization, and a browser runtime remains untested.
CONCLUSION Counting passing tests says little about product quality. What matters is which seams between the code and the user have been exercised at all. A large share of defects occur at these seams, where component tests cannot see them. A low-cost check that does see them is a single end-to-end pass through the running product. Run it in the configuration least represented during development, such as a non-default market. The spread of defects across seams shows a team where to put its testing effort.
REFERENCES 1. L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 435–445, doi: 10.1145/2568225.2568271. 2. P. Kruchten, R. L. Nord, and I. Ozkaya, “Technical debt: From metaphor to theory and practice,” IEEE Softw., vol. 29, no. 6, pp. 18–21, 2012, doi: 10.1109/MS.2012.167. 3. E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Trans. Softw. Eng., vol. 41, no. 5, pp. 507–525, 2015, doi: 10.1109/TSE.2014.2372785. 4. C. E. Jimenez et al., “SWE-bench: Can language models resolve real-world GitHub issues?,” in Proc. Int. Conf. Learn. Representations, 2024, arXiv:2310.06770. 5. X. Hu et al., “Assessing and advancing benchmarks for evaluating large language models in software engineering tasks,” ACM Trans. Softw. Eng. Methodol., 2025, doi: 10.1145/3786771. 6. X. Chen, C. Gao, C. Chen, G. Zhang, and Y. Liu, “An empirical study on challenges for LLM application developers,” ACM Trans. Softw. Eng. Methodol., accepted 2025, arXiv:2408.05002. 7. M. Azanza, B. Pérez Lamancha, and E. Pizarro, “Tracking the moving target: A framework for continuous evaluation of LLM test generation in industry,” in
6
IEEE Software
Proc. 29th Int. Conf. Eval. Assessment Softw. Eng., 2025, arXiv:2504.18985.
8. R. White and J. Krinke, “TCtracer: Establishing testto-code traceability links using dynamic and static techniques,” Empirical Softw. Eng., vol. 27, no. 3, art. 67, 2022, doi: 10.1007/s10664-021-10079-1.
9. M. Maes-Bermejo, A. Serebrenik, M. Gallego, F. Gortázar, G. Robles, and J. M. González-Barahona, “Hunting bugs: Towards an automated approach to identifying which change caused a bug through regression testing,” Empirical Softw. Eng., vol. 29, 2024, doi: 10.1007/s10664-024-10479-z.
10. R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for Java programs,” in Proc. Int. Symp. Softw. Testing Anal., 2014, pp. 437–440, doi: 10.1145/2610384.2628055.
11. R. Widyasari et al., “BugsInPy: A database of existing bugs in Python programs to enable controlled testing and debugging studies,” in Proc. 28th ACM ESEC/FSE, 2020, pp. 1556–1560, doi: 10.1145/3368089.3417943.
12. C. Escobar-Velasquez, M. Osorio-Riaño, J. Dominguez-Osorio, M. Arevalo, and M. LinaresVásquez, “An empirical study of i18n collateral changes and bugs in GUIs of Android apps,” in Proc. IEEE Int. Conf. Softw. Maintenance Evol., 2020, pp. 581–592, doi: 10.1109/ICSME46990.2020.00061.
13. A. Alameer and W. G. J. Halfond, “An empirical study of internationalization failures in the web,” in Proc. IEEE Int. Conf. Softw. Maintenance Evol., 2016, pp. 88–98, doi: 10.1109/ICSME.2016.55.
14. A. Mordahl, J. Oh, U. Koc, S. Wei, and P. Gazzillo, “An empirical study of real-world variability bugs detected by variability-oblivious tools,” in Proc. 27th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng., 2019, pp. 50–61, doi: 10.1145/3338906.3338967.
15. K. Herzig, S. Just, and A. Zeller, “It’s not a bug, it’s a feature: How misclassification impacts bug prediction,” in Proc. 35th Int. Conf. Softw. Eng., 2013, pp. 392–401, doi: 10.1109/ICSE.2013.6606585. 2026
ALL GREEN, STILL BROKEN
Muhammad Bilal is an AI and Digitalization Consultant in the German industrial sector. He holds a Master of Science in Management from the Technical University of Munich, Germany, and has previously worked as a Software Engineer, Business Analyst, and Product Owner. His research interests include the impact of technology on business performance, product quality analytics, the automation of industrial pipelines, large language models, and agentic AI systems. He is the corresponding author.
Ali Hassaan Mughal has worked as a Senior Software Developer and Team Lead at Xpressdocs, and earlier at Paycom and a stealth robotics company. He is pursuing an Applied MBA in Data Analytics at Texas Wesleyan University, USA, and holds an M.Sc. in Computer Science from Kansas State University. His research interests include automated software testing, web application quality assurance, applied machine learning, large language models, and agentic AI systems.
2026
IEEE Software
7