arXiv:2607.29557v1 [cs.SE] 31 Jul 2026
Alteron: A Tool for Behavioral Regression Testing Across NLP Classifier Versions Shazzad Hossain
Proma Chowdhury
Mridha Md. Nafis Fuad
University of Dhaka [email protected]
University of Dhaka [email protected]
University of Dhaka [email protected]
Abstract—Evaluating evolving Natural Language Processing (NLP) models is important for ensuring reliable behavior across updates, but standard benchmark metrics do not fully capture how model behavior changes across versions. Existing work has focused mainly on testing models in isolation rather than comparing successive versions in continuous integration workflows. We present Alteron, a tool for detecting behavioral regressions across NLP model versions with metamorphic testing. Alteron constructs a test corpus from labeled source examples and compares model versions on metamorphically transformed inputs. In an evaluation spanning 10 metamorphic relations (MRs), 4 model versions, and 3 model-update transitions, Alteron identified 16 behavioral regressions, 11 of which were releaseblocking. The results show that common model updates can preserve overall task performance while still introducing undesirable behavior changes, and that behavioral checks across model versions can reveal failures that aggregate benchmark metrics alone do not capture. The tool is open-source and available at https://github.com/shazzad5709/alteron. A screencast demonstration is available at https://youtu.be/szwiWW5O4do. Index Terms—metamorphic testing, behavioral regression, continuous integration, NLP, Software Engineering for AI
I. I NTRODUCTION Natural Language Processing (NLP) classifiers are widely used in modern software systems, making reliable evaluation essential as these models evolve over time. These models are frequently updated after deployment [1]. Before release, each updated version must be validated to ensure it behaves as intended. Standard evaluation methods typically rely on labeled benchmarks and aggregate metrics such as accuracy. These metrics summarize performance over a test set, but they do not capture how individual predictions change between model versions [2]. As a result, an updated model may perform better on some inputs while performing worse on others, even when aggregate metrics remain similar or improve. This can hide meaningful behavior changes on individual inputs. Detecting behavior changes in NLP model updates is challenging as standard ML metrics summarize performance at the aggregate level and do not show how predictions shift on individual inputs. For example, a spam detection model may classify “Win a free iPhone now” as spam, but after an update misclassify “Win a free smartphone now,” although both messages express the same intent. We refer to this issue as behavioral regression. This motivates the need for evaluation methods that go beyond aggregate metrics and
check whether updated models preserve consistent predictions on semantically equivalent inputs. Recent work shows that Metamorphic Testing (MT) [3] provides a practical way to expose behavioral failures in NLP systems without requiring labeled follow-up data [4]. Tooloriented efforts such as LLMORPH [5] and MDPMORPH [6] further demonstrate the practicality of automated MR-based testing for NLP and ML models. However, these approaches primarily focus on testing or analyzing a model in isolation. They do not address the continuous integration (CI) setting in which a new version must be compared against the previous accepted version to determine whether system evolution has introduced release-relevant behavioral regressions. ALTERON addresses this problem by treating metamorphic testing as a version-to-version behavioral regression check for NLP classifiers in continuous integration workflows. ALTERON applies a predefined set of metamorphic relations to labeled source examples, stores the resulting source and follow-up pairs as a reusable test corpus, and records the predictions and MR outcomes for both the new version and the previous accepted version. It then compares the two versions only on source examples that both versions classify correctly. On that shared subset, ALTERON measures how much the MR pass rate changes from one version to the next. The resulting reports distinguish blocking from non-blocking regressions and can be consumed directly in release pipelines through a machine-readable CI outcome. We present a pilot evaluation of ALTERON on 9 dataset profiles spanning sentiment analysis, natural language inference, and generic robustness settings. The pilot uses 4 model versions connected by retraining, distillation, and quantization transitions. The evaluation measures whether versionto-version decision behavior is preserved across controlled model updates over a fixed metamorphic test suite. The pilot identified 16 behavioral regressions, 11 of which were releaseblocking, showing that ALTERON can detect release-relevant regressions that are not apparent from source-side accuracy alone, particularly in generic robustness profiles. The paper contributes an operational tool based on metamorphic testing for NLP model evolution, a behavioral regression testing workflow for continuous integration, and a pilot demonstration of its use across realistic update scenarios.
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Input Labeled source data MR Registry
1
Input
Corpus Generation
Initial deployed version
Corpus Generator
Automated Validator
Output
Initial snapshot generator
Test corpus
Generated candidate pairs
Output
Initial snapshot
Test corpus
Family
Representative MRs
Expected Relation
Tasks
Sentiment
Negation insertion, adverb insertion, uppercasing Gender substitution, hypothesis negation Space injection, capitalization change, keyboard typo
Flip / monotonic Flip / invariant
SA
Invariant
SA, NLI, topic
NLI Generic
Test corpus
Input New candidate version
TABLE I MR FAMILIES IN A LTERON .
Initial 2 Snapshot Generation
3 Candidate snapshot generator
Continuous Integration Run New candidate snapshot Gate Allows Release
source labels for subset selection
Previous accepted snapshot
Output
Regression Differencer
Severity-based evaluator
Regression Report
CI Summary
Gate Blocks Release
NLI
substitution, cross-gender substitution, and hypothesis negation. The generic robustness family includes space injection, capitalization change, and keyboard typo simulation. Baseline artifacts. For CI regression checking, ALTERON requires a fixed test corpus of validated MR test pairs and a behavioral snapshot of the previous accepted version over that corpus. The fixed test corpus is generated once and reused across model updates. The stored snapshot is reused until a new version is accepted, at which point the snapshot of the new version becomes the next baseline for comparison.
Fig. 1. Logical workflow of A LTERON.
B. Output II. A LTERON ALTERON is a behavioral regression testing tool for versionto-version evaluation of NLP classifiers in CI workflows. Figure 1 illustrates the logical workflow of A LTERON. Given a previously accepted model as the baseline, a new candidate model, a selected set of MRs, and a fixed test corpus, it produces behavioral snapshots, regression reports, and a machinereadable CI summary that indicates whether any detected regressions should block the candidate model from progressing through continuous integration. A. Input ALTERON takes inputs across four categories: Model versions. ALTERON evaluates an NLP classifier through a framework-agnostic interface. Two versions of the model are required. The baseline represents the previously accepted model, and the candidate represents the new version under evaluation. The model must return a predicted label and a confidence score for each input. Labeled source data. ALTERON requires labeled source examples for the NLP task under test. Source labels are used to build the matched subset, which retains only examples that both versions classify correctly. They are also used by taskspecific metamorphic relations to determine input eligibility. Metamorphic relations (MRs). The user selects which MRs to apply from a registry. The registry draws on the MR catalog of Cho et al. [4], a systematic review of 191 MRs across 24 NLP tasks. ALTERON currently implements ten MRs across three families. These families are summarized in Table I. The sentiment-specific family includes negation insertion, period-to-exclamation replacement, emphasizing adverb insertion, and noun/adjective uppercasing. The natural language inference (NLI)-specific family includes same-gender
ALTERON produces four categories of output. Fixed test corpus. ALTERON produces per-MR CSV files that store validated source and follow-up pairs. It also creates a SHA-256 manifest, a rejection log explaining skipped inputs, and sample artifacts for manual review. The fixed test corpus is a verified set of source and follow-up pairs that is reused across model versions. This ensures that any observed differences come from changes in the model rather than the test data. Behavioral snapshots. For each model version, ALTERON records per-MR predictions on source and follow-up inputs, confidence scores, MR pass or fail outcomes, and fairnessregression flags where applicable. Regression reports. For each MR, ALTERON reports how model performance changes on the original input examples, how many matching cases were evaluated, and how the MR pass rate changes on that subset. It also indicates whether the observed change is considered a behavioral regression, the MR’s severity level in the registry, and whether the regression is severe enough to fail the CI check. CI result. ALTERON emits a machine-readable summary of the CI run together with an exit code. The summary records the overall outcome, including the selected profile, compared model versions, regression threshold, number of blocking regressions, and the generated report paths, while detailed MRlevel reasons for failure are provided in the regression report. ALTERON returns exit code 0 when no blocking regression is found and 1 otherwise, making it directly usable as a CI gate. C. Process ALTERON operates in three stages: Corpus generation. For each source input and selected MR, ALTERON applies the transformation function to produce a follow-up input and then runs MR-specific automated checks
to validate the pair. Invalid or ineligible pairs are recorded in a rejection log. Accepted pairs are written to a fixed test corpus with a SHA-256 hash manifest for integrity checks. Initial snapshot generation. Each model version is evaluated on the fixed test corpus produced during corpus generation. For every source and follow-up pair, ALTERON records the predicted labels, confidence scores, and whether the model’s outputs satisfy the expected metamorphic relation. CI run. In each CI run, ALTERON first evaluates the new candidate model on the fixed test corpus to generate a new snapshot. It then compares that snapshot against the stored snapshot of the previous accepted version through regression differencing. For each MR, this comparison uses only source examples that both versions classify correctly, so that the measured change reflects behavioral differences rather than regular classification error on the source inputs. ALTERON flags a behavioral regression when the change in MR pass rate on that subset falls below a configurable threshold. Finally, ALTERON evaluates the resulting regression signals under a configured CI profile to determine the reported outcome. ALTERON provides built-in execution profiles to balance speed and coverage. The pr-fast profile samples a small number of examples per MR to provide rapid pull-request feedback, whereas the release-full profile evaluates the complete fixed test corpus for a full release-time check. Both profiles can be customized. If a blocking regression is detected under the active profile, ALTERON returns a failing result for the CI check; otherwise, it returns a passing result. D. Implementation Details ALTERON is implemented as a Python package exposing two CLI entry points, alteron for corpus and snapshot operations and alteron-ci for CI execution. The implementation uses spaCy for dependency parsing and POS tagging, NLTK for real-word collision checks in typo generation, and YAML configuration for MR metadata and CI profiles. All generated artifacts are stored as plain files, so the tool does not depend on any external database and its outputs remain easy to inspect. Each MR specifies how follow-up inputs are generated, how transformed pairs are validated, and how the expected metamorphic relation is checked. MR transformations are rulebased and seeded rather than LLM-generated, which makes corpus generation deterministic and reproducible. For example, negation MRs use dependency parses to insert negation at controlled syntactic positions, gender-substitution MRs use a constrained lexicon to avoid partial or agreement-breaking swaps, and keyboard-typo MRs use QWERTY adjacency with real-word collision checks. Tokenizer-sensitive MRs such as capitalization changes are checked for applicability during corpus generation and skipped when the tokenizer would make the test invalid. MR metadata is stored in a YAML registry containing task applicability, expected relation, known failure modes, implementation references, and CI handling categories such as hard-fail and soft-warning. These categories allows
the same MR library to support different release policies. The behavioral-regression threshold can be customized globally, per CI profile, or per run. The tool is publicly available and designed for reuse and extension through its CLI, MR registry, and framework-agnostic model loader interface. III. T OOL U SAGE ALTERON is available from the project repository and can be used through its command-line interface. A. Installation ALTERON requires Python 3.10+ together with the dependencies listed in the repository, including spaCy, the en_core_web_sm model, and the NLTK words corpus. B. Running the Tool ALTERON exposes two CLI entry points: alteron for corpus and snapshot operations, and alteron-ci for CI execution. A typical workflow consists of three commands. Corpus generation. To generate a fixed test corpus from labeled source data and a selected set of metamorphic relations, execute alteron corpus generate. Initial snapshot generation. To generate the initial snapshot from the initial deployed version, execute alteron snapshot baseline. ALTERON loads models through a framework-agnostic model-loader interface that returns the prediction interface expected by the tool; example loaders are provided in the repository. CI run. The continuous integration check is run with alteron-ci. It uses a selected CI profile, such as pr-fast or release-full, and returns a pass or fail result that can be consumed directly by CI. C. Reading the Outputs The main outputs for inspection are the regression report and the CI summary. Regression report. The regression report contains one entry per MR, with the following fields: • mr_id: the MR identifier. • source_accuracy_delta: the change in source-side accuracy between the baseline and candidate. • n_matched: the size of the matched subset used for behavioral comparison. • matched_pass_rate_delta: the change in MR pass rate on the matched subset. • behavioral_regression_flag: whether the MR crossed the regression threshold. • release_blocked: whether the MR contributes to a blocking release decision. CI summary. The CI summary records the overall result of the run, including whether a blocking regression was detected and the final exit status returned by alteron-ci. The artifact directory also contains the fixed test corpus, the behavioral snapshots, and the rejection logs for further inspection when needed.
IV. E XPERIMENTAL S ETUP We evaluate four BERT-family model versions namely v1_base, v2_retrain, v3_distilled, and v4_quantized. BERT-family encoders were chosen because they remain a standard architecture for supervised NLP classification [7]. All models are cased encoders, enabling capitalization-sensitive MRs. These versions correspond to three common model update operations, namely retraining on the original task data, knowledge distillation [8], and post-training 8-bit quantization [9], which reflect practical updates used to improve task performance or reduce model size and deployment cost. The evaluation spans nine dataset profiles across three task families. For sentiment analysis, we use SST-2 [10] and IMDb [11]. For natural language inference, we use SNLI [12] and MultiNLI [13]. For robustness evaluation, we apply generic MRs to SST-2, IMDb, SNLI, MultiNLI, and AG News [14]. Task-specific profiles are evaluated on their respective datasets, while generic profiles are applied across all datasets to measure robustness under lexical and syntactic perturbations across classification tasks. Overall, the setup includes 9 dataset profiles, 10 MRs, 4 model versions, and 3 model transitions. V. E VALUATION We performed a preliminary system evaluation of A LTERON based on the experimental setup in Section IV to assess its ability to identify behavioral regression. Additionally, we conducted an exploratory user study to assess the usability and usefulness of A LTERON. A. System Evaluation of Alteron We evaluated ALTERON on NLP model updates to assess whether it can identify behavioral regressions that are not reflected in standard benchmark metrics. For each dataset, we built a fixed test set by applying MRs to labeled examples and keeping only valid transformed pairs that pass automated checks. We also logged rejected cases for later inspection. The final test set is stored with a hash and manifest to ensure integrity and reproducibility, and it is reused across all model versions to ensure fair comparison. We then evaluated all four model versions on this fixed test set and stored their outputs as snapshots. Each snapshot includes predictions for both original and transformed inputs, confidence scores, and whether each MR is satisfied. Finally, we compare three sequential model updates: v1_base -> v2_retrain, v2_retrain -> v3_distilled, and v3_distilled -> v4_quantized. For each MR, we compute how often both versions correctly classify the original input and measure changes in MR behavior on the transformed inputs. A behavioral regression is flagged when this change falls below a predefined threshold. The decision to block a release depends on MR severity and the active CI configuration. Table II summarizes the results across the nine dataset profiles.
TABLE II P ILOT SUMMARY BY PROFILE . Profile
Task
MR Family
Flags
Block
sa_sst2 sa_imdb nli_snli nli_multinli gen_sst2 gen_imdb gen_snli gen_multinli gen_agnews
SA SA NLI NLI SA SA NLI NLI Topic
Sentiment Sentiment NLI NLI Generic Generic Generic Generic Generic
0 0 0 0 4 3 3 6 0
0 0 0 0 3 3 1 4 0
∆ Pass-Rate −0.0483 −0.0299 −0.0207 −0.0354 −0.2596 −0.1905 −0.0741 −0.2444 −0.0313
In total, ALTERON produced 27 regression reports (9 profiles × 3 transitions) across 81 MR-level comparisons and flagged 16 behavioral regressions, 11 of which were releaseblocking hard failures. As shown in Table II, all flagged regressions occurred in the generic robustness profiles, with gen_multinli contributing the largest number of flags, followed by gen_sst2 and gen_imdb. A representative case appears in the gen_multinli profile for v1_base -> v2_retrain. On space injection invariance MR, source accuracy increased from 0.4647 to 0.7259, yet the matched MR pass rate dropped from 0.6909 to 0.4466 over 7,513 matched examples, yielding a matched_pass_rate_delta of -0.2444 and a blocking CI outcome. This shows that a new version can perform better overall while still failing more often on inputs that should preserve the same decision. This was not an isolated case; similar regressions appeared across the generic robustness settings. Across the three transitions, the pilot suggests that different update types affect behavioral robustness in different ways. Retraining improved source accuracy in some profiles while reducing robustness to surface perturbations. Distillation produced the largest number of flagged regressions, suggesting that compression can preserve top-line performance while weakening stability under transformed inputs [8]. By contrast, quantization produced only one flagged regression in this pilot, suggesting a smaller behavioral impact [9]. This matters in practice because these transitions are common production updates, and gains in accuracy, size, or efficiency can still be accompanied by behavioral regressions across model versions. B. Exploratory User Study We conducted a controlled user study to evaluate how participants interpret behavioral regression results produced by A LTERON in a CI setting. The study included eight participants, comprising three ML engineers, one DevOps engineer, and four academic researchers. Participants were given a prerun model comparison scenario consisting of a fixed test corpus, a stored snapshot of a baseline model, and a candidate model under evaluation. A LTERON compares these models using MRs, which test whether the candidate model preserves consistent behavior under controlled input transformations. Participants then inspect the outputs of a CI pipeline, which produces two artifacts: a CI summary and a regression report
as mentioned in Section III-C. Participants use these outputs to identify which MRs caused regressions, understand how model behavior changed compared to the baseline, and judge whether the changes are severe enough to block deployment. To evaluate usability and perceived usefulness, participants completed a post-study questionnaire using a 5-point Likert scale ranging from 1 (Strongly disagree) to 5 (Strongly agree). The questionnaire assessed clarity of the system purpose, ease of following instructions, ease of setup, navigability of generated artifacts, interpretability of results, and perceived usefulness of A LTERON in CI/CD workflows. Results show that all eight participants completed the study run. Seven participants correctly identified the baseline and candidate model versions as well as the overall CI outcome, and seven also correctly identified capitalization invariance MR as the blocking MR. Most participants relied primarily on the regression report to determine the source of the failure, and it was consistently rated as the most useful artifact for understanding the CI result. In the usability and usefulness evaluation, six out of eight participants reported that they would be able to use A LTERON again with similar guidance, and six participants agreed that the system would be useful in a model maintenance or CI/CD workflow. VI. R ELATED W ORK Traditional NLP evaluation relies on labeled benchmarks and aggregate metrics such as accuracy. Prior work on behavioral testing has shown that such evaluation can miss important robustness and linguistic failures [2]. Recent tools show that metamorphic testing can be automated for ML systems. Metamorphic testing has also been used to examine the internal consistency and linguistic properties of deep NLP models [15] and to support automated behavioral testing in machine translation [16]. LLMORPH automates MR-based testing for large language models across NLP tasks [5], while MDPMORPH applies automated metamorphic testing to deep reinforcement learning agents [6]. These approaches primarily focus on testing or analyzing a model in isolation. They do not address version-to-version comparison in continuous integration. Alteron is designed for this setting. VII. C ONCLUSION Alteron addresses the problem of behavioral regression in evolving NLP classifiers by treating metamorphic testing as a version-to-version check in continuous integration workflows. It combines a fixed test corpus, per-version snapshots, and matched-subset regression differencing to detect changes that aggregate benchmark metrics alone may miss. In a pilot evaluation spanning nine dataset profiles, four model versions, and three update transitions, Alteron identified release-relevant regressions that were concentrated in generic robustness settings and were not apparent from source-side accuracy alone. These results suggest that common model updates such as retraining, distillation, and quantization can preserve overall task performance while still introducing undesirable behavioral
change. Future work includes expanding the MR library, broadening evaluation to more NLP tasks and model families, and refining the tool’s usability in larger-scale continuous integration settings. R EFERENCES [1] C. Renggli, B. Karlaš, B. Ding, F. Liu, K. Schawinski, W. Wu, and C. Zhang, “Continuous integration of machine learning models with ease.ml/ci: Towards a rigorous yet practical treatment,” in Proceedings of Machine Learning and Systems (MLSys), vol. 1, pp. 322–333, 2019. [2] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with CheckList,” in Proc. 58th Annu. Meeting Assoc. Comput. Linguistics (ACL), pp. 4902–4912, Jul. 2020, doi: 10.18653/v1/2020.acl-main.442. [3] T. Y. Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. H. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys, vol. 51, no. 1, Art. no. 4, pp. 1– 27, Jan. 2018, doi: 10.1145/3143561. [4] S. Cho, S. Ruberto, and V. Terragni, “Metamorphic testing of large language models for natural language processing,” in Proc. 41st IEEE Int. Conf. Softw. Maint. Evol. (ICSME), 2025, pp. 174–186, doi: 10.1109/ICSME64153.2025.00025. [5] S. Cho, S. Ruberto, and V. Terragni, “LLMORPH: Automated metamorphic testing of large language models,” in Proc. 40th IEEE/ACM Int. Conf. Autom. Softw. Eng. (ASE), 2025, pp. 4102–4105, doi: 10.1109/ASE63991.2025.00385. [6] J. Li, Z. Zheng, Y. Xing, D. Ren, S. Cho, and V. Terragni, “Metamorphic testing of deep reinforcement learning agents with MDPMORPH,” in Proc. 40th IEEE/ACM Int. Conf. Autom. Softw. Eng. (ASE), 2025, pp. 4086–4089, doi: 10.1109/ASE63991.2025.00381. [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional transformers for language understanding,” in Proc. 2019 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Language Technologies (NAACL-HLT), 2019, pp. 4171–4186, doi: 10.18653/v1/N19-1423. [8] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” in Proc. EMC2 Workshop, co-located with NeurIPS, 2019, arXiv:1910.01108. [9] Z. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, “ZeroQuant: Efficient and affordable post-training quantization for largescale transformers,” in Advances in Neural Information Processing Systems, vol. 35, 2022. [10] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in Proc. 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018, pp. 353– 355, doi: 10.18653/v1/W18-5446. [11] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proc. 49th Annu. Meeting Assoc. Comput. Linguistics: Human Language Technologies, 2011, pp. 142–150. [12] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” in Proc. 2015 Conf. Empirical Methods Natural Language Processing (EMNLP), 2015, pp. 632–642. [13] A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in Proc. 2018 Conf. North Amer. Chapter Assoc. Comput. Linguistics: Human Language Technologies (NAACL-HLT), 2018, pp. 1112–1122, doi: 10.18653/v1/N18-1101. [14] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in Advances in Neural Information Processing Systems, vol. 28, 2015, pp. 649–657. [15] E. Manino, J. Rozanova, D. Carvalho, A. Freitas, and L. Cordeiro, “Systematicity, compositionality and transitivity of deep NLP models: A metamorphic testing perspective,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 2355–2366, doi: 10.18653/v1/2022.findings-acl.185. [16] J. Ferrando, M. Sperber, H. Setiawan, D. Telaar, and S. Hasan, “Automating behavioral testing in machine translation,” in Proc. Eighth Conf. Machine Translation (WMT), 2023, pp. 1014–1030, doi: 10.18653/v1/2023.wmt-1.97.