Evaluating quality in synthetic data generation for large tabular health datasets Jean-Baptiste Escudié1,2,3 , Benjamin Barnes2 , Stefan Meisegeier2 , Klaus Kraywinkel2 , Fabian Prasser3 , and Nils Körber∗1
arXiv:2604.15961v1 [cs.LG] 17 Apr 2026
1
Centre for Artificial Intelligence in Public Health Research, Robert Koch Institute 2 German Centre for Cancer Registry Data, Robert Koch Institute 3 Medical Informatics, Berlin Institute of Health at Charité - Universitätsmedizin Berlin
Abstract
the fidelity of the datasets to the original data. From classical statistical methods (copulas, Bayesian networks, SMOTE) tabular data synthesis evolved to deep generative models with the adaptation of VAEs and GANs for table (medGAN, table-GAN, CTGAN), capable of handling mixed-type columns and complex joint distributions, embeddings for categoricals and conditioning to stabilize discrete/continuous generation (Choi et al., 2017; Park et al., 2018; Xu et al., 2019). These deep models have then seen refined architectures and objectives proposed (CTAB-GAN, VAE variants, (Zhao et al., 2021, 2023)), as well as differential privacy (DP) and Private Aggregation of Teachers Ensemble (PATE) based privacy variants introduced (Yoon et al., 2019). More recent diffusion/score-based models and transformer/LLM-style sequence encodings have also been adapted for tabular synthesis (Borisov et al., 2023; Kotelnikov et al., 2023; Truda, 2023). Given the growing number of methods for synthesizing data that are being published, a comparison of these methods is a challenging task. The evaluation of synthetic data is a multifaceted process and the absence of consensus regarding the most suitable evaluation methods is noteworthy (Lautrup et al., 2025). The prevalence of metrics with minimal overlap in publications illustrates this phenomenon. A categorization of the facets commonly evaluated is possible. It is common practice to differentiate between metrics that assess the quality (also called fidelity or resemblance) of the synthetic data, its utility, and its privacy. However, within each of these branches, a multitude of metrics exists. Taxonomies have been proposed (Kaabachi et al., 2025, pp. ; Hernandez et al., 2025; Drechsler & Haensch, 2023; Vallevik et al., 2024) and if they differ in structure and naming, they are designed to encompass the diverse metrics found in the literature in these three accepted categories of quality, utility and privacy. In many cases,
There is no consensus in the field of synthetic data on concise metrics for quality evaluations or benchmarks on large health datasets, such as historical epidemiological data. This study presents an evaluation of seven recent models from major machine learning families. The models were evaluated using four different datasets, each with a distinct scale. To ensure a fair comparison, we systematically tuned the hyperparameters of each model for each dataset. We propose a methodology for evaluating the fidelity of synthesized joint distributions, aligning metrics with visualization on a single plot. This method is applicable to any dataset and is complemented by a domain-specific analysis of the German Cancer Registries’ epidemiological dataset. The analysis reveals the challenges models face in strictly adhering to the medical domain. We hope this approach will serve as a foundational framework for guiding the selection of synthesizers and remain accessible to all stakeholders involved in releasing synthetic datasets.
1
Introduction
Synthetic tabular data is seeing a growing interest in conjunction to the increasing need for privacy preserving access to sensitive health data. The two primary applications for synthetic health data are privacy and data augmentation. In the context of privacy preserving synthetic data, the promise is to be able to perform statistical analysis with similar outcome on an adjacent dataset that does not allow - or with a limited probability - to reindentify individuals, reconstruct or infer individual attributes from the original dataset. The quality of the synthesized datasets is a critical factor in both applications, as it is essential to ensure ∗ Corresponding author: [email protected].
1
Figure 1: Quality evaluation methodology overview. tic datasets comprising a greater number of samples and/or features. Indeed, as the dataset under consideration grows, the resource requirements to apply a given method (runtime, memory) may become unrealistic. Furthermore, the hyperparameters that demonstrate efficacy on a specific dataset may exhibit suboptimal performance on a substantially distinct dataset.
a custom compound metric is constructed - e.g. by means of a weighted sum of seleted metrics (Adams et al., 2025; Chundawat et al., 2024; Du & Li, 2025; Yan et al., 2022) - to generate a single score. Also, software libraries, e.g. (Qian et al., 2023; Synthetic Data Metrics, 2025), collect large numbers of metrics implementations in one place. While this provides readyto-use implementations, the interpretation of the results remains complex and context dependent. In the context of searching for a privacy-preserving synthesizer suitable for a specific use case, it is imperative to evaluate the trade-off between maximizing privacy guarantees and ensuring the quality and utility of the synthesizer. The acceptability of particular choices is contingent upon the particulars of the context. In an ideal scenario, this process would entail the involvement of all relevant stakeholders, including patients, healthcare providers, public health agencies, epidemiologists, and researchers, irrespective of their statistical literacy. Consequently, there is a necessity for a straightforward method to assess quality, thereby providing guidance in determining the privacy guarantee tradeoff.
We examine the synthesis of epidemiological data from the German Cancer Registries, which commenced in the 1970s and currently encompasses over 13 million cases. The data encompasses variables that encode clinical diagnoses in the International Classification of Diseases 10th Revision German Modification (ICD-10-GM). The dimensionality of medical classifications, such as ICD, often poses a significant challenge for machine learning models. The present work focuses on the evaluation of quality from small to large tabular datasets. This benchmark is a comparative analysis of seven models representing recent major machine learning (ML) families. The models are evaluated on four different datasets, each with increasing dimensions in terms of number of rows and columns. The analysis has a particular focus on categorical data. The objective of this study is to establish a definitive foundation for the discussion of quality when selecting a method for synthesizing a privacy-preserving synthetic dataset. To this end, hyperparameter optimization (HPO) was performed on each model and dataset to ensure a fair comparison. Subsequently, a simplified visual evaluation is
A further limitation in the comparison of published models stems from the heterogeneity of the datasets utilized, which frequently exhibit limited size, rarely exceeding 100,000 samples and a modest number of features, as seen in public machine learning datasets. This prompts the inquiry into whether a reported method exhibits equivalent efficacy on more realis2
Dataset Name
Categorical N Variables Categories
Abalone Adult EpiCancerGER USCensus1990
1 9 10 68
Numerical N Variables
N Variables
Total N Samples
Vector Size
8 6 0 0
9 15 10 68
4175 48800 1000000 2458200
11 112 718 396
3 106 718 396
Table 1: Datasets characteristics. Vector size denotes the size of a vector for a sample in the dataset after one-hot encoding the categorical variables. presented, employing a limited yet sufficient number 2.1 Overview of metrics to rank the models and assess their perforGiven a dataset Dreal , a model M is trained and then mance. used to generate a synthetic dataset Dsynthi . The The main contributions of this study are: quality of Dsynthi compared to Dreal is then measured. Subsequently, another set of the model’s hyperparam• a. We conduct a comprehensive scaling analysis, eters is then tested and the process is reiterated until evaluating model performance across datasets of the tuning budget is depleted. up to several million entries, which exceeds the Upon completion of the independent tuning proscales typically reported. cess for each combination of Dreal and M , the dataset Dsynthbest is considered for the final ranking of mod• b. Addressing the common issue of relying on els. A separate ranking of the models was performed default settings, this work provides an extensive for each dataset Dreal . empirical study on the impact of systematic tuning on the ability of models to converge to optimal 2.2 Datasets hyperparameters. The benchmark includes four datasets, as detailed • c. Evaluation and visualisation methods that are in Table 1. Abalone (Nash et al., 1994) and Adult concise yet enough to rank models, and can be (Becker & Kohavi, 1996) are two public datasets frequently utilized in machine learning with modest diextended for domain specific analysis. mensions and heterogeneous types of variables (categorical and numerical). EpiCancerGER is a subset of historical epidemiological data from all the cancer 2 Methods registries in Germany (Institute, 2023), spanning from 1970 to 2019. This subset includes a total of one milAfter an initial overview of the hyperparameter opti- lion cases. The list of included variables can be found mization and ranking, we present in the following sec- in the Appendix. The USCensus1990 (Meek et al., tions the datasets, the models, the particularities of 2001) is a large-scale public dataset that contains exthe HPO settings, the visualisations and metrics used clusively categorical data. for evaluation and ranking the models. We then conThis selection exhibits the range from 4k to 2.5M clude with the domain specific analysis extending the samples, and the encoded sample dimensions range evaluation for the German Cancer Registries dataset. from 11 to 540. It shoud be noted that missing valModel
Family
Differential Privacy ability
Reference
TVAE CTGAN PGM PrivSyn TabDDPM TableDiffusion GReaT
Variational Auto Encoder GAN Probabilistic graph model Distribution free model Diffusion model Diffusion model Fined-tuned LLM
No No Yes Yes No Yes No
Xu et al., 2019 Xu et al., 2019 Mckenna et al., 2019 Zhang et al., 2021 Kotelnikov et al., 2023 Truda, 2023 Borisov et al., 2023
Table 2: Models included in the benchmark. 3
Dataset USCensus1990 EpiCancerGER Adult Abalone
HPO budget N completed trials Timeout 150 250 350 350
HPO metrics Categorical Numerical
10h + 10h 5h + 5h 5h + 5h 5h + 5h
MAE2 MAE2 MAE2
Hist_IoU2 Hist_IoU2
Table 3: HPO settings. Timeout in HPO budget is split into training and synthesizing steps. ues were considered to be an inherent category to be HPO optimizer remained unaware of the failure. The reproduced in the synthetic dataset. For this reason, motivation for this approach stems from the recognithe missing data is not reported seperately. tion that the causes of failures are heterogeneous. For instance, an invalid set of hyperparameters could have been sampled due to the presence of highly specific 2.3 Models model requirements, such as the necessity of a batch 7 models were included in the benchmark as presented size that is a multiple of two and of the PAC size. in Table 1. GReaT (Borisov et al., 2023) was included Furthermore, the implementation of a set of hyperpafor reference, however due to its substantial computa- rameters has the potential to result in a training or tion costs, no HPO was conducted on it, and only a generation process that exhausts available resources, limited subset of 10k samples was utilized and gener- leading to errors such as timeout or out-of-memory exceptions. For the purpose of this benchmark, a selecated. A number of models - PGM (Mckenna et al., 2019), tion of NVIDIA GPUs was utilized, including H100s, PrivSyn (Zhang et al., 2021), TableDiffusion (Truda, A100s, and L40s. Ultimately, it was not possible to 2023) - have differential privacy capabilities integrated attribute the failure entirely to the model implementainto their design. For these models, the epsilon value tions without compromising the fairness of the benchwas set to 1e+8 to simulate a privacy budget that is mark. To address this, the opportunity to complete nearly infinite. This was done to ensure a fair com- the same number of trials with an unlimited number of failed trials was extended to every model. parison against all models. Implementations were retrieved from SynMeter (Du & Li, 2025), which aggregates either the original repos2.5 Evaluation metrics and visualisaitories or the SynthCity (Qian et al., 2023) versions tion and provides default hyperparameters after SynMeter performed HPO in a different setting. For categorical variables, we primarily employed
2.4
marginal distributional similarity, encompassing onedimensional, pair-wise and n-way marginals. For numerical variables, for the main metric was the histogram’s intersection over union for single and pairs of variables.
Hyperparameters Optimization
Table 3 details the characteristics of the HPO process that differed for each of the 4 datasets. The ranges of hyperparameters are presented for each model in the [Appendix]. The rest of the HPO process remained the same for each combination of model and dataset. Namely, the Tree Parzen Estimator (TPE) (Bergstra et al., 2011) was employed as optimizer for categorical only datasets (EpiCancerGER and USCensus1990). The HPO optimizer was the Multi Objective TPE (Ozaki et al., 2020) for mixed datasets (Abalone, Adult). The implementations of this methods were based on the Optuna library (Akiba et al., 2019). A salient feature of our HPO configuration was the stipulation that the number of trials should equal the number of completed trials. Consequently, in the event of a failed training or generation attempt, the model’s budget did not account for the trial, and the
2.5.1
Metrics for categorical variables
The primary metric for categorical variables was derived from the pairwise marginals. This metric was employed in two distinct ways: first, to calibrate the HPO tuning, and second, to determine the final ranking of the models. In this study, the mean absolute error of all pairwise marginals, denoted M AE2 , was utilized as the primary metric for analysis.
M AE2 =
1 2 |VCAT |
2 VCAT
∗
X
|preal (vi , vj ) − psynth (vi , vj )|
vi ,vj
(1) 4
2 where VCAT is the set of unordered pairs of categorical variables, preal and psynth are their empirical probabilities in the real and synthetic datasets respectively. As secondary metrics, we also measured the single marginals and their corresponding mean absolute error M AE1 were measured. Two new metrics were defined: Coverage of categories for each variable and variables pairs, and the proportion of Invented relationships. Coverage is defined per variable as the proportion of categories in Dreal that are also generated at least once in Dsynthi . To illustrate, if Dsynthi comprises solely samples with SEX = f emale, yet Dreal encompasses both SEX = male and SEX = f emale categories, then Coverage1SEX value is determined to be 50\Consequnetly, we defined Coverage2 as the coverage applied to every pair of variables. Invented is the proportion of generated samples for which the value is not present in Dreal . Once more the average Invented1 for the dataset is the average for every variables. Invented was measured only pairwise (Invented2 ) as every model tested generates only valid categories (either because of the use of one-hot encoding or the application of filtering mechanisms to discard invalid values).
2.5.2
2.5.4
Visualizing a set of 1D or 2D histograms is a non trivial task. Instead, quantile-quantile (QQ) plots are presented for all variables in a single plot, with each colored line representing a distinct variable.
2.6
Domain violation
To complement the evaluation of the EpiCancerGER dataset, an assessment of domain violation in the synthetic dataset was conducted. This analysis is distinct from the other metrics in that it is dataset-specific. In order to execute this analysis, it is necessary to manually define the domain. This analsysis sought to define domain violation for three pairs of variables involving ICD codes. In EpiCancerGER, both full and 3-digit ICD codes are included. This enables the establishment of a primary domain that is unambiguously delineated. A violation is identified when the two codes do not correspond to the ICD classification system. Specifically, this occurs when the ICD’s three-digit code does not serve as a prefix to the entire code. The synthesizer should be capable of recovering this strict matching, as it is strictly adhered to in the real dataset. However, given that both codes are represented as independent variables, it is necessary to assess whether the models respect the ICD structure strictly. The second case of domain validation analysis was defined for the pair of variables sex and ICD 3-digits. Again, there exists a strict exclusion verified in the real dataset for 15 ICD chapters covering genital organs neoplasms listed in the [Appendix]. The third and final case of domain validation involved a comparison of age groups with ICD 3-digits codes. For this analysis, the minimum and maximum ages observed in the real dataset for each ICD chapter were determined. Subsequently, it was verified whether the synthetic dataset respects these established ranges. This is often referred as out-of-range detection.
Visualisations for categorical variables
Conveniently all these metrics can be repsented visually on two scatter plots: one for marginals, the second for the pair-wise marginals. M AE1 and M AE2 are the mean distance of every point from the diagonal line. The points falling on the x and y axis contributes to Coverage and Invented respectively. 2.5.3
Visualisation for numerical variables
Metrics for numerical variables
The primary metric for numerical variables was a binning distance on pairs of variables. This metric was employed both to calibrate the hyperparameters tuning and to determine the final ranking of the models. Specifically we discretized the numerical variable using 10 bins of equal sizes and computed the average intersection over union (IOU) of the corresponding histograms. A similar metric was measured for variables instead of pairs. We’ll refer to these metrics as Hist_IoU1 and Hist_IoU2 . For visual interpretation, they conveniently correspond to the average L1 distance in all the 1D and 2D histograms constructed from the sets of variables and pairs of variables respectively. Theese values correspond as well to the mean absolute error of empirical probabilities following formula (1) with VN2 U Mbin as the set of unordered pairs of binned numerical variables.
3
Results
3.1
Main results - best models after HPO
The comparison of the top-performing models following HPO reveals that PrivSyn emerged as the leading option across all datasets in terms of pairwise categorical metrics. This phenomenon is illustrated in the accompanying plots, where the points align more closely with the diagonal line, indicating that the real 5
Figure 2: Categorical (denoted CAT): scatter plots of pair-wise marginals for the best models after HPO. On each plot, a point is drawn for a pair-wise level. E.g. the point for SEX=MALE and ICD10=C20, is drawn at x=0.47 and y=0.55, if 47% and 55% are the observed percentages in the real and synthetic datasets respectively. Numerical (denoted NUM): QQ plots of every variable with the real quantiles on x-axis and the synthetic quantiles on the y-axis. marginals equal the synthetic ones. This objective was accomplished with minimal disruption to existing pairwise relationships, while ensuring a large coverage of all existing relationships. PrivSyn also demonstrated optimal performance in the joint distribution of numerical variables in the Adult dataset. The QQ plot manifests a stepping effect, attributable to the optimal number of bins as determined by HPO, which in turn drives the binning performed internally by PrivSyn. The introduction of the artifact by the model is reflected in the conspicuously substandard Hist_IoU1 metric.
rics Invented2 and Coverage with those of PrivSyn revealed no significant disparities. However, a notable divergence was observed in the primary joint distribution metrics, specifically M AE2 and Hist_IoU2 .
The following models in the ranking were TVAE and CTGAN. It is noteworthy that the performance of the two models was comparable, with the exception of EpiCancerGER. On the two primary joint distribution metrics, M AE2 and Hist_IoU2 , the models exhibited divergent behaviors. In fact, the TVAE model exhibited a tendency to overestimate relationships as the prevalence increased. Conversely, the CTGAN The second-best model demonstrated a consistent model demonstrated a propensity to underestimate repattern of performance across all datasets, with Tabb- lationships, a phenomenon that is evident in the scatdpm exhibiting the second highest level of perfor- ter plots. Additionally, TVAE demonstrated a prefmance. The presence of a systematic artifact, such as erence for low Invented2 over high Coverage2 , while PrivSyn, on the numerical variables was not observed. CTGAN exhibited the inverse preference. AdditionConsequently, the optimal alignment of 1D histograms ally, TVAE outliers, defined as variables synthesized with real data was achieved. A comparison of the met- with reduced quality, exhibit a greater disparity from 6
MAE1 ↓
MAE2 ↓
Categorical Coverage1 ↑ Coverage2 ↑
Invented2 ↓
dataset
model
USCensus1990
PrivSyn TabDDPM CTGAN TVAE PGM GReaT TableDiffusion
0.0002 0.0034 0.0121 0.0129 0.0000 0.0174 0.0385
0.0002 0.0019 0.0065 0.0070 0.0095 0.0101 0.0224
1.0000 1.0000 1.0000 0.9931 1.0000 0.9517 1.0000
0.9828 0.9916 1.0000 0.9275 0.9981 0.8184 1.0000
0.0000 0.0002 0.0053 0.0005 0.0265 0.0000 0.0225
EpiCancerGER
PrivSyn TabDDPM TVAE PGM TableDiffusion CTGAN GReaT
0.0000 0.0019 0.0041 0.0000 0.0073 0.0087 0.0170
0.0000 0.0006 0.0015 0.0020 0.0028 0.0031 0.0073
0.9795 0.9975 0.9881 1.0000 0.6950 1.0000 0.8904
0.9398 0.9373 0.9234 0.9670 0.4545 0.9813 0.7339
0.0001 0.0003 0.0068 0.0401 0.0062 0.0163 0.0000
Adult
PrivSyn TabDDPM TVAE CTGAN PATE-GAN PGM TableDiffusion GReaT
0.0005 0.0030 0.0125 0.0137 0.0148 0.0000 0.0283 0.0239
0.0003 0.0009 0.0043 0.0043 0.0052 0.0061 0.0077 0.0077
1.0000 0.9974 0.9974 1.0000 0.9259 1.0000 0.9071 0.9277
0.9416 0.9283 0.9559 0.9889 0.8293 0.9758 0.8085 0.7961
0.0002 0.0001 0.0011 0.0035 0.0092 0.0116 0.0003 0.0001
Abalone
PrivSyn TabDDPM GReaT TVAE CTGAN TableDiffusion PGM
0.0010 0.0110 0.0096 0.0762 0.0089 0.1851 0.0000
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000
Numerical Hist_IoU1 ↑ Hist_IoU2 ↑
0.0337 0.9010 0.6580 0.6148 0.6131 0.4004 0.6903 0.8438
0.9830 0.8931 0.6976 0.7286 0.7482 0.4990 0.6019 0.8099
0.0659 0.8927 0.8004 0.7478 0.7454 0.4636 0.6941
0.9622 0.9142 0.7984 0.7711 0.5691 0.4440 0.3547
Table 4: Main results, best models after HPO. ↓ denotes lower is better. ↑ denotes higher is better. The M AE2 and Hist_IoU2 metrics rank the models. Bold metrics values denotes the best in dataset. the original data distribution. In contrast, CTGAN demonstrates a more uniform performance across variables. A more thorough examination determined that TVAE exhibited unambiguous artifacts, particularly evident in the USCensus1990 data. These artifacts manifested as lines that deviated from the primary body of points, suggesting a propensity to either overor underestimate specific values in proportion to their support in the original data. PGM and TableDiffusion demonstrated suboptimal performance in this benchmark, with PGM exhibiting superior performance compared to TableDiffusion. PGM demonstrated a notable capacity to accurately estimate the marginal probabilities; however, its performance was suboptimal in terms of capturing the joint distributions, particularly in scenarios where prevalence levels were lower.
3.2
ity of cases, the HPO led to a significant improvement in quality for both numerical and categorical variables. On the two large datasets USCensus1990 and EpiCancerGER, for every models except PrivSyn, the default hyperparameters result in suboptimal performance. The most significant case is TabDDPM which demonstrated the most substantial enhancement of 85,797%. In the absence of HPO, the model would have been considered substandard. However, it demonstrated a second-best performance in our primary ranking, indicating its reliance on hyperparameters choice. As demonstrated in Figure 3, the maximum perfomance of most models was typically attained during the first 150 HPO trials. PGM and PrivSyn exhibited minimal variations in performance across the various trials and demonstrated reduced reliance on their hyperparameters.
HPO improvement compared to 3.3 default hyperparameters
Table 5 compares the main metrics M AE2 and Hist_IoU2 with the default hyperparameters provided in SynMeter (Du & Li, 2025) (with the exception of relaxed differential privacy) and after the HPO tuning performed in this study. In the vast major-
Impact of dataset size
Due to the dataset-dependence of our metrics, we employed models ranking, and visual interpretation to estimate the impact of dataset sizes on model performance. Consequently, it is challenging to evaluate the impact of dataset size on behavior based 7
Figure 3: Evaluation metrics over HPO tuning. The y-axis on the left represents M AE2 and Hist_IoU2 on the right. Cross marks locate when the default hyperparameters were outperformed.
8
Figure 4: Evaluation pair-wise scatter plots over the HPO tuning for USCensus1990.
9
Categorical MAE2 ↓ default HPO improvement %
dataset
model
USCensus1990
PrivSyn TabDDPM CTGAN TVAE PGM GReaT TableDiffusion
0.0002 0.0891 0.0100 0.1094 0.0095 0.0101 0.0780
0.0002 0.0019 0.0065 0.0070 0.0095 N/A 0.0224
0.0000 -0.0872 -0.0035 -0.1024 -0.0000 N/A -0.0556
10% -98% -35% -94% -0% N/A -71%
EpiCancerGER
PrivSyn TabDDPM TVAE PGM TableDiffusion CTGAN GReaT
0.0000 0.0475 0.0400 0.0020 0.0504 0.0044 0.0073
0.0000 0.0006 0.0015 0.0020 0.0028 0.0031 N/A
-0.0000 -0.0469 -0.0385 -0.0000 -0.0476 -0.0013 N/A
-41% -99% -96% -0% -94% -30% N/A
Adult
PrivSyn TabDDPM TVAE CTGAN PATE-GAN PGM TableDiffusion GReaT
0.0006 0.0007 0.0091 0.0071 0.0254 0.0061 0.0288 0.0077
0.0003 0.0009 0.0043 0.0043 0.0052 0.0061 0.0077 N/A
-0.0003 0.0003 -0.0048 -0.0028 -0.0202 -0.0000 -0.0211 N/A
-53% 39% -53% -39% -79% -1% -73% N/A
Abalone
PrivSyn TabDDPM GReaT TVAE CTGAN TableDiffusion PGM
Numerical Hist_IoU2 ↑ default HPO improvement %
0.8955 0.8901 0.5705 0.6806 0.3405 0.2723 0.1551 0.8099
0.9830 0.8931 0.6976 0.7286 0.7482 0.4990 0.6019 N/A
0.0876 0.0031 0.1271 0.0480 0.4076 0.2267 0.4468 N/A
10% 0% 22% 7% 120% 83% 288% N/A
0.5563 0.0011 0.7984 0.7250 0.4894 0.2262 0.2953
0.9622 0.9142 N/A 0.7711 0.5691 0.4440 0.3547
0.4058 0.9131 N/A 0.0461 0.0797 0.2179 0.0594
73% 85,797% N/A 6% 16% 96% 20%
Table 5: HPO improvement. Comparing metrics after HPO to default hyperparameters. ↓ denotes lower is better. ↑ denotes higher is better. on dataset size for PrivSyn and TabDDPM as these two models consistently achieved the top two rankings. However, a more substantial enhancement was observed for P rivSyn following HPO tuning for the smaller datasets, Adult and Abalone; this was in contrast to the observations for USCensus1990 and EpiCancerGER. The significance of HPO for TabDDPM appears to be independent of dataset size. However, an exception was observed for the Adult dataset, where TabDDPM demonstrated a relative decline in performance following HPO.
tional to this coverage of invalid code space: TabDDPM exhibited a mere 1.07% of the samples to be in violation of the ICD code chapters structure, yet still encompassed 8.22% of the potential violations. Over the 3 ICD domain violations we assessed, PrivSyn exhibited the lowest number of affected samples, while PGM demonstrated the highest.
4 3.4
Discusion
Domain violation
As demonstrated in Table 7, all models exhibited domain violations in the EpiCancerGER dataset, though to varying extents. It was evident that all models generated invalid pairs of values; however, the coverage of the invalid possible space varied significantly. For instance, the range of ICD vs. ICD 3-digits varied from 598 (1.01%) of the 59,492 possible violations (60,048 possible pairs - 556 pairs valid and present in the real dataset) for TableDiffusion to 39,379 (60.2%) for CTGAN. The proportion of the 1 million synthesized samples affected was not systematically propor10
We performed a quality evaluation of the datasets generated by 7 recently published models of different major ML families, after a systematic hyperparameters tuning to each of the four datasets of different scales. We have developed a novel evaluation method that utilizes a concise set of metrics, aligned with a condensed visual representation. This method was sufficiently comprehensive for the purpose of model ranking. We have demonstrated on the EpiCancerGER dataset how it can be extended by additional domainspecific analysis.
GReaT PrivSyn TabDDPM TVAE TableDiffusion CTGAN PGM
ICD vs ICD 3-digit 60,048 possible levels (556 x 108) 556 levels observed in real data
Sex vs ICD 3-digit 216 possible levels (2 x 108) 201 levels observed in real data
Age group vs ICD 3-digit 1,944 possible levels (18 x 108) 1,745 levels observed in real data
Domain violations n distinct % samples
Domain violations n distinct % samples
Domain violations n distinct % samples
0 761 4,889 2,559 598 39,379 27,953
0.00% 0.08% 1.07% 21.13% 24.93% 46.41% 93.48%
0 8 15 11 3 15 15
0.00% 0.00% 0.02% 0.19% 0.02% 0.57% 7.97%
0 46 56 49 0 84 92
0.00% 0.01% 0.01% 0.01% 0.00% 0.03% 0.06%
Table 7: Domain violations on EpiCancerGER dataset. Number of distinct level pairs outside of domain and percentage of samples affected in the synthetic data. placed on categorical data. However, it was demonstrated that M AE can be adapted to numerical variIn order to address the paucity of consensus on eval- ables through the process of binning, resulting in the uation metrics in the discipline of synthetic data, Hist_IoU metric. while hopefully providing an approach that is sufficiently simple to be comprehensible to all stakeholders involved in the process of releasing a synthetic 4.2 Visualization dataset, regardless of their statistical literacy, we em- Scatter plots can serve as a graphical method for comployed marginal-based metrics M AE, Coverage, and paring the distributions of categorical variables in a Invented. Marginal statistics are frequently employed single plot. However, they are seldom employed agfor quality assessment, yet they lack the capacity to gregated at the level of the dataset for all variables, measure joint distributions. Relationships constitute and this is typically only done for the purpose of vian indispensable component of the data to be repro- sualizing independent distributions (Baowaly et al., duced in the synthesized datasets. Consequently, the 2019). We introduced scatter plots on every pair of metrics were defined on the independent variable (de- variables, which are utilized for the visual assessment gree 1) and every pair of variables (degree 2). of the joint distributions. The scatterplots demonThe closest existing metric to M AE1 is the total strate a strong correlation with the M AE, Coverage, variation distance (TVD), with a value that is half and Invented metrics, each of which has a readily inof M AE1 . Being an average, it may be insufficient terpretable graphical representation. M AE is the avto analyze the case when differences with the orig- erage distance to the identity line which corresponds inal marginals are the most extreme. To this end, the ideal case where all synthetic marginals match the we defined Coverage and Invented. We observed a real ones and mimimize the metric to zero. Coverage tradeoff between Coverage on one end and M AE and is the proportion of points falling the x-axis. And Invented on the other end. The precise mechanism Invented is the proportion of generated samples with is contingent upon the internals of each model; how- the corresponding marginals falling on the y-axis. ever, a parallel can be drawn between this tradeoff In the case of numerical variables, a graphical inand an exploration-exploitation tradeoff in the con- terpretation of Hist_IoU is also applicable. It is text of GAN models. A mode collapse occurs when the the average of the L1 distances of all 1D histograms model exclusively generates the most prevalent cate- (Hist_IoU1 ) or 2D histograms (Hist_IoU2 ). gories, also known as the mode. This phenomenon The visualization in one plot of the comparison of constitutes a local minimum that is nonetheless an ef- all numerical independent or joint distributions in the fective strategy, wherein the generator utilizes a salient dataset remains a challenging endeavor, necessitataspect of the original data to confound the discrimina- ing further investigation and refinement of analytitor. This scenario would result in a reduction of the cal methods. A common approach relies on plotting metrics related to coverage. Conversely, in the sce- two heatmaps side by side of the correlation matrices, nario of exhaustive exploration, the model would not as illustrated in (Moazemi et al., 2024). It requires acquire the capacity to restrict the extent of gener- choosing appropriate correlation measures and does ated value to the existing pairs of categories, thereby not scale for large number of variables. We instead augmenting the Invented2 metric. opted to present QQ plots, which have the capacity to In this particular benchmark, the emphasis was represent all independent variables (one line per vari-
4.1
Evaluation metrics choice
11
able) in a single plot. However, this approach is not did not invent any relationship on the EpiCancerGER without its limitations, as it does not extend to joint dataset - thus the absence of domain violations - but at distributions. the cost of significantly lower Coverage2 and M AE2 . It is noteworthy that, in our experiments, domain violations increase with the cardinality of the variable 4.3 Relation to other metrics pair. This aligns with (Pilgram et al., 2025) findings, Although covering joint distributions, our marginal despite their different definition. They observed that based metrics yielded rankings of the models that may the main factor driving hallucination rates was dataset differ if other metrics were to be used. cardinality. Depending on the application, strategies A comparison was made between the rankings ob- to mitigate domain violations could include the filtratained using the Jensen-Shannon distance (JSD) and tion of invalid samples. However, this approach is not the Wasserstein distance (WD), which are also com- without its drawbacks, as it may result in interaction monly used in synthetic data evaluation. If the rela- with the measured marginals and the potential introtionships between any pair of the M AE2 , JSD or WD duction of underepresentation bias if the filtered out measurements is not strictly monotonic, the ranking incoherences are not independent of other patterns in would still have been unchanged for the first 4 best the data. Therefore, such mitigation requires careful models and the least performing as illustrated in Fig- consideration. ure 5 in the Appendix.
4.4
4.5
Domain violations analysis
The generic quality evaluation methods proposed herein have the capacity to be complemented in order to address the specifics of the dataset. In the case of the EpiCancerGER dataset, the structure and semantics of the International Classification of Diseases (ICD) were leveraged to define domains with clearly delineated boundaries of invalid values. Domain driven definitions for exact boundaries can be challenging, and two alternative approaches were identified in our literature review. (Yan et al., 2022) leveraged a data-driven approach identifying diseases present only for one gender in the real data. Rather than being a external definition of the domain, it is in fact closer to the notion of fidelity to the original data and to the Invented metric, and does not guarantee domain correctness. A typical counter example would be breast and prostate cancers: men with breast cancer could be absent of the original data but still be valid if synthesized, whereas women with prostate cancer would be invalid. (Pilgram et al., 2025) delineates clearly these two cases in their definition of hallucinations in synthetic tabular data and propose in their work another data-driven approach to define the source of gound truth from the data by creating variant of the data population and training the models on only a subset. A record is considered not hallucinated if present in the source population, whether present or not in the training data. Our preliminary investigation indicated a challenge faced by all models except GReaT in strictly adhering to a medically coherent domain. To varying extents, each model yielded full ICD codes that did not align with the ICD chapters, or ICD chapters that were incompatible with the sex or age group attributed to the synthesized sample. In the case of GReaT, the model 12
Effect of HPO
Given the models’ high sensitivity to their hyperparameters, with the exception of PGM, it is recommended to tune their values for each dataset. This recommendation aligns with the approach outlined in (Du & Li, 2025), which advocates for the utilization of an HPO algorithm, albeit with three key diffrerences. Firstly, the objective of tuning differs because their emphasis is on the application preserving privacy synthetic data, rather than focusing quality. The researchers employed a multifaceted approach, leveraging a weighted sum of membership disclosure for privacy, query error for utility, and Wasserstein distance for fidelity score. The weighting of these three components was empirically set to equal weights after observing similar ranges for the three components. The optimization of such a compound objective may result in the prioritization of one objective over the other objectives. A similar challenging situation was encountered in our study with the Adult dataset, which has both numerical and categorical variables and necessitated the implementation of a Pareto front to ascertain the optimal trials. This approach is outlined in Optuna’s multi-objective feature (Akiba et al., 2019). For the other datasets, our objective was more straightforward: we sought to identify a single metric (either M AE2 or Hist_IoU2 ). A second key aspect of the methodology that was implemented differently was the use of a time budget and allowance for unlimited trials, but only counting completed trials in the models HPO budget. As delineated in the Methods section, it is posited that this approach engenders a more equitable comparison. The rationale underlying this assertion is that subjecting the models to a broad spectrum of parameters and dataset scales, which may not have been encountered
during their development, can precipitate either the manifestation of bugs or the exhaustion of resources (e.g., RAM, GPU RAM, runtime). By not penalizing the implementations we tested for not being error proof on our datasets, or having trials failing because they ran on the most modest of our mix of GPUs, our unlimited trials HPO settings actively compensated for these phenomenons. The third different key aspect was the number of trials conducted for the HPO. According to the published source code of SynMeter, 50 trials were utilized. We used 350 trials for the datasets we share with their experiments (Adult and Abalone), and 250 and 150 for our larger datasets (EpiCancerGER and USCensus1990, respectively). In our experiments this proved to be beneficial as metrics kept improving up to the 240th trial for TabDDPM. Although this step of last observed improvement varied with models and datasets. It is also noteworthy that the change in hyperparameters had a negligible impact on PGM performance. Finally, one may consider the cost versus marginal improvement of longer HPO.
4.6
Bests models
Coverage was satisfactory, though not the optimal values were observed. Furthermore, the invented relationships were found to be at a notably low level. This outcome is in stark contrast with the optimal Coverage2 values observed for CTGAN (on Adult and EpiCancerGER) and TableDiffusion (on USCensus1990). However, this is achieved at the expense of a substantial increase in Invented2 and a deterioration in M AE2 .
5
Conclusion
In this study focusing on quality of synthetic data, we benchmarked 7 models on four datasets of varying scales in both number of samples and number of variables. In order to ensure a fair comparison, the challenge of models sentivity to hyperparameters was overcome by means of a systematic tuning of each model to each dataset. The best models after tuning were compared and ranked using a limited set of metrics. The proposed visualization method facilitates the interpretation of distances to pair-wise margins, coverage of joint distributions, and the propensity of a model to invent relationships absent in real data, all within a single plot. A demonstration was conducted on a German Cancer Registries dataset to illustrate how application-specific analysis can complement the proposed generic quality assessment of synthetic tabular data. We hope these methods provide a clear foundation for discussing quality when choosing a method for synthesizing a privacy preserving synthetic dataset.
PrivSyn was found to consistently yield optimal quality metrics (M AE2 , Hist_IoU2 ) across datasets in the context of the benchmark, while concomitantly maintaining adequate coverage of pairwise joint categorical distributions (Coverage2 ). Depending on the application (e.g., data augmentation, privacy), concerns about generating a synthetic dataset too close to the original may be raised. In such cases, a dediReferences cated analysis would be required. PrivSyn was devel- 6 opped with builtin differential privacy for the privacy preserving synthetic data application. In the context References of our experimental setup, the epsilon parameter was configured to 1e+8. This configuration was imple- Adams, T., Birkenbihl, C., Otte, K., Ng, H. G., Rieling, J. A., Näher, A.-F., Sax, U., Prasser, F., mented with the objective of facilitating a meaning& Fröhlich, H. (2025). On the fidelity versus ful comparison in terms of quality with other models privacy and utility trade-off of synthetic pawhile concurrently minimizing the impact of the diftient data. iScience, 28 (5), 112382. https:// ferential privacy feature. How the quality degrades as doi.org/10.1016/j.isci.2025.112382 the epsilon is lowered to acceptable values in privacy application could guide the decision process regarding Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A Next-generation Hyperthe quality/privacy tradeoff. parameter Optimization Framework. ProceedPrivSyn’s capacity to generate data close to the ings of the 25th ACM SIGKDD International original without inventing relationships between variConference on Knowledge Discovery and Data ables is also an important aspect in the application of Mining. data augmentation. But our quality analysis should be complemented with other aspects such as measuring Baowaly, M. K., Lin, C.-C., Liu, C.-L., & Chen, K.-T. (2019). Synthesizing electronic health the capacity to generate close but non existing samrecords using improved generative adversarples, quantified with the Distance to Closest Record ial networks. Journal of the American Medi(DCR) for example. cal Informatics Association, 26 (3), 228–241. TabDDPM emerged as the second most effective https://doi.org/10.1093/jamia/ocy142 model across all datasets. As with PrivSyn, the 13
Becker, B., & Kohavi, R. (1996). Adult [Published: UCI Machine Learning Repository]. Bergstra, J., Bardenet, R., Bengio, Y., & Kégl, B. (2011). Algorithms for Hyper-Parameter Optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems (Vol. 24). Curran Associates, Inc. https : / / proceedings . neurips . cc / paper _ files / paper / 2011 / file / 86e8f7ab32cfd12577bc2619bc635690 - Paper . pdf Borisov, V., Sessler, K., Leemann, T., Pawelczyk, M., & Kasneci, G. (2023). Language Models are Realistic Tabular Data Generators. The Eleventh International Conference on Learning Representations. https://openreview.net/ forum?id=cEygmQNOeI Choi, E., Biswal, S., Malin, B., Duke, J., Stewart, W. F., & Sun, J. (2017, August). Generating Multi-label Discrete Patient Records using Generative Adversarial Networks. In F. Doshi-Velez, J. Fackler, D. Kale, R. Ranganath, B. Wallace, & J. Wiens (Eds.), Proceedings of the 2nd Machine Learning for Healthcare Conference (pp. 286–305, Vol. 68). PMLR. https://proceedings.mlr.press/v68/ choi17a.html Chundawat, V. S., Tarun, A. K., Mandal, M., Lahoti, M., & Narang, P. (2024). A Universal Metric for Robust Evaluation of Synthetic Tabular Data. IEEE Transactions on Artificial Intelligence, 5 (1), 300–309. https : / / doi . org / 10 . 1109/TAI.2022.3229289 Drechsler, J., & Haensch, A.-C. (2023, April). 30 Years of Synthetic Data [arXiv:2304.02107 [cs]]. https://doi.org/10.48550/arXiv.2304. 02107 Du, Y., & Li, N. (2025). Systematic Assessment of Tabular Data Synthesis. Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, 2414– 2428. https : / / doi . org / 10 . 1145 / 3719027 . 3765067 Hernandez, M., Osorio-Marulanda, P. A., Catalina, M., Loinaz, L., Epelde, G., & Aginako, N. (2025). Comprehensive evaluation framework for synthetic tabular data in health: Fidelity, utility and privacy analysis of generative models with and without privacy guarantees. Frontiers in Digital Health, 7, 1576290. https: //doi.org/10.3389/fdgth.2025.1576290 Institute, R. K. (2023). Cancer in Germany 2019/2020 [Publisher: Robert Koch-Institut]. https : / / doi.org/10.25646/11842
14
Kaabachi, B., Despraz, J., Meurers, T., Otte, K., Halilovic, M., Kulynych, B., Prasser, F., & Raisaro, J. L. (2025). A scoping review of privacy and utility metrics in medical synthetic data. npj Digital Medicine, 8 (1), 60. https : //doi.org/10.1038/s41746-024-01359-3 Kotelnikov, A., Baranchuk, D., Rubachev, I., & Babenko, A. (2023). TabDDPM: Modelling tabular data with diffusion models. Proceedings of the 40th International Conference on Machine Learning. Lautrup, A. D., Hyrup, T., Zimek, A., & SchneiderKamp, P. (2025). Syntheval: A framework for detailed utility and privacy evaluation of tabular synthetic data. Data Mining and Knowledge Discovery, 39 (1), 6. https://doi.org/10. 1007/s10618-024-01081-4 Mckenna, R., Sheldon, D., & Miklau, G. (2019, June). Graphical-model based estimation and inference for differential privacy. In K. Chaudhuri & R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning (pp. 4435–4444, Vol. 97). PMLR. https://proceedings.mlr.press/v97/ mckenna19a.html Meek, C., Thiesson, B., & Heckerman, D. (2001). US Census Data (1990) [Published: UCI Machine Learning Repository]. Moazemi, S., Adams, T., Ng, H. G., Kühnel, L., Schneider, J., Näher, A.-F., Fluck, J., & Fröhlich, H. (2024, August). NFDI4Health Workflow and Service for Synthetic Data Generation, Assessment and Risk Management. In R. Röhrig, N. Grabe, U. H. Hübner, K. Jung, U. Sax, C. O. Schmidt, M. Sedlmayr, & A. Zapf (Eds.), Studies in Health Technology and Informatics. IOS Press. https : / / doi . org / 10 . 3233/SHTI240834 Nash, W., Sellers, T., Talbot, S., Cawthorn, A., & Ford, W. (1994). Abalone [Published: UCI Machine Learning Repository]. Ozaki, Y., Tanigaki, Y., Watanabe, S., & Onishi, M. (2020). Multiobjective tree-structured parzen estimator for computationally expensive optimization problems. Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 533–541. https://doi.org/10.1145/ 3377930.3389817 Park, N., Mohammadi, M., Gorde, K., Jajodia, S., Park, H., & Kim, Y. (2018). Data synthesis based on generative adversarial networks. Proceedings of the VLDB Endowment, 11 (10), 1071–1083. https : / / doi . org / 10 . 14778 / 3231751.3231757
Pilgram, L., El Kababji, S., Liu, D., & El Emam, conference / usenixsecurity21 / presentation / K. (2025). Magnitude and Impact of Hallucizhang-zhikun nations in Tabular Synthetic Health Data on Zhao, Z., Kunar, A., Birke, R., & Chen, L. Y. Prognostic Machine Learning Models: Valida(2021, November). CTAB-GAN: Effective Tation Study. Journal of Medical Internet Reble Data Synthesizing. In V. N. Balasubramasearch, 27, e77893. https://doi.org/10.2196/ nian & I. Tsang (Eds.), Proceedings of The 77893 13th Asian Conference on Machine LearnQian, Z., Cebere, B.-C., & van der Schaar, M. (2023). ing (pp. 97–112, Vol. 157). PMLR. https:// Synthcity: Facilitating innovative use cases of proceedings.mlr.press/v157/zhao21a.html synthetic data in different data modalities. Zhao, Z., Kunar, A., Birke, R., Van der Scheer, H., & https://doi.org/10.48550/ARXIV.2301.07573 Chen, L. Y. (2023). Ctab-gan+: Enhancing Synthetic Data Metrics. (2025, November). DataCebo, tabular data synthesis [Publisher: Frontiers Inc. https://docs.sdv.dev/sdmetrics/ Media SA]. Frontiers in big Data, 6. https : Truda, G. (2023, August). Generating tabular datasets / / doi . org / https : / / doi . org / 10 . 3389 / fdata . under differential privacy [arXiv:2308.14784 2023.1296508 [cs]]. https://doi.org/10.48550/arXiv.2308. 14784 Vallevik, V. B., Babic, A., Marshall, S. E., Elvatun, S., Brøgger, H. M., Alagaratnam, S., Edwin, B., Veeraragavan, N. R., Befring, A. K., & Nygård, J. F. (2024). Can I trust my fake data – A comprehensive quality assessment framework for synthetic tabular data in healthcare. International Journal of Medical Informatics, 185, 105413. https : / / doi . org / 10 . 1016 / j . ijmedinf.2024.105413 Xu, L., Skoularidou, M., Cuesta-Infante, A., & Veeramachaneni, K. (2019). Modeling Tabular data using Conditional GAN. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 32). Curran Associates, Inc. https : / / proceedings . neurips . cc / paper _ files / paper / 2019 / file / 254ed7d2de3b23ab10936522dd547b78- Paper. pdf Yan, C., Yan, Y., Wan, Z., Zhang, Z., Omberg, L., Guinney, J., Mooney, S. D., & Malin, B. A. (2022). A Multifaceted benchmarking of synthetic electronic health record generation models. Nature Communications, 13 (1), 7609. https://doi.org/10.1038/s41467- 02235295-1 Yoon, J., Jordon, J., & Schaar, M. v. d. (2019). PATEGAN: Generating Synthetic Data with Differential Privacy Guarantees. International Conference on Learning Representations. https : //openreview.net/forum?id=S1zk9iRqF7 Zhang, Z., Wang, T., Li, N., Honorio, J., Backes, M., He, S., Chen, J., & Zhang, Y. (2021). PrivSyn: Differentially Private Data Synthesis. 30th USENIX Security Symposium (USENIX Security 21), 929–946. https://www.usenix.org/
15
7
Appendix
Contents .1 .2 .3 .4
Comparison of rankings from different metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . Variables included in EpiCancerGER dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sex vs ICD 3-digit domain violations (EpiCancerGER dataset) . . . . . . . . . . . . . . . . . . Hyperparameters ranges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
17 18 19 20
.1
Comparison of rankings from different metrics
Figure 5: Comparison of rankings from different metrics.
17
.2
Variables included in EpiCancerGER dataset
Variable
Label
Description
SEX
Gender
Gender details: 1 = male 2 = female
ALTGRP
Age group at diagnosis
The age at diagnosis categorized in 5-year age groups based on the (imputed) age at diagnosis: a00b04 - from 0 to 4 years a05b09 - from 5 to 9 years . . . a85plus - 85 years and above
ICDGM10
ICD-10-GM (GM=German ification)
ICD code for the tumor diagnosis (4-digit, 10th Revision, WHO) Valid codes: C00.0 - C97.0 (excluding C77.0 - C79.9) (Malignant neoplasms without metastases) D00.0 - D09.9 (In situ neoplasms) D32.0 - D33.9 (Benign brain tumors) D37.0 - D48.9 (Neoplasms of uncertain or unknown behavior)
ICDGM10DREI
ICD-10-GM Chapter
ICD chapter for the tumor diagnosis (3-digit, 10th Revision, WHO)
UICC
TNM to UICC
Conversion of TNM classifications to UICC stages (for TNM editions 6 and 7)
DIG
Dignity
Indication to the dignity: 0 = Benign neoplasm 1 = Neoplasm of uncertain or unknown behaviour 2 = In situ neoplasm 3 = Malignant neoplasm
GRAD
Grading
Histopathological grading for solid tumors (1-digit): 1 = well differentiated 2 = moderately differentiated 3 = poorly differentiated 4 = undifferentiated/anaplastic 5 = low grade (low malignant) 6 = intermediate grade 7 = high grade (highly malignant)
DSICH
Type of diagnostic confirmation
GROBST
Stage (internal use)
Indication of type of diagnostic confirmation (1 digit): 0 = autopsy 1 = clinical without spec. diagnostics 2 = clinical diagnosis 3 = death certificate (DCO) 4 = spec. tumor maker 5 = cytology 6 = Histology metastasis 7 = Histology primary tumor 8 = Other 1=in situ 2=local 3=regional 4=remote metastases 5=systemic disease -999=unknown
ZTYP
Cell Type
Mod-
Cell type of haemato-oncological malignancies (1-digit): 5 = T cell type 6 = B cell type 7 = Zero cell type 8 = Natural killer cell type
Table 8: Variables included in EpiCancerGER dataset.
18
.3
Sex vs ICD 3-digit domain violations (EpiCancerGER dataset) Female C51 C52 C53 C54 C55 C56 C57 C58 D06 D39
Male
ICD Chapter
C60 C61 C62 C63 D40
Malignant neoplasm of vulva Malignant neoplasm of vagina Malignant neoplasm of cervix uteri Malignant neoplasm of corpus uteri Malignant neoplasm of uterus, part unspecified Malignant neoplasm of ovary Malignant neoplasm of other and unspecified female genital organs Malignant neoplasm of placenta Carcinoma in situ of cervix uteri Neoplasm of uncertain or unknown behaviour of female genital organs Malignant neoplasm of penis Malignant neoplasm of prostate Malignant neoplasm of testis Malignant neoplasm of other and unspecified male genital organs Neoplasm of uncertain or unknown behaviour of male genital organs
Table 9: ICD-10 Chapters considered for the domain violations analysis Sex vs ICD 3-digit.
19
.4
Hyperparameters ranges Processed as
min
max
optimized optimized stratified optimized optimized optimized optimized stratified optimized
8 1e-07 (256, 256) or (512, 512) 1e-06 32 10 1e-07 (128, 128) or (256, 256) 1e-06
2048 0.1
batch_size compress_dims decompress_dims embedding_dim epochs l2scale loss_factor
optmized stratified stratified optimized optimized optimized optimized
32 (128, 128) or (256, 256) (128, 128) or (256, 256) 32 10 1e-07 1.0
4096
2_cliques 3_cliques bi_nums delta epsilon max_bins num_iters tri_nums
fixed fixed fixed fixed fixed optimized optimized fixed
30 30 30 0.5000 10000000.0 10 1000 30
delta epsilon max_bins update_iterations
fixed fixed optimized optimized
0.5 100000000.0 10 10
batch_size d_layers dropout lr num_timesteps steps weight_decay
optimized stratified optimized optimized optimized optimized optimized
8 (256 x 7, 128) or (1024 x 7, 128) 1e-06 1e-05 10 10 1e-06
batch_size d_layers diffusion_steps epoch_target epsilon_target lr predict_noise
optimized stratified optimized optimized fixed optimized optimized
32 (128, 128) or (256, 256) 1 1 100000000.0 1e-05 False
Model
Hyperparameter
CTGAN
batch_size discriminator_decay discriminator_dim discriminator_lr embedding_dim epochs generator_decay generator_dim generator_lr
TVAE
PGM
PrivSyn
TabDDPM
TableDiffusion
Values
0.1 4096 1000 0.1 0.1
1024 200 0.1 10.0
1000 10000
1000 1000 4096 0.7 0.1 10000 100000 0.2 400 16 16 0.1 True
Table 10: Details of hyperparameters ranges. ’optimized’ denotes hyperparameters tuned during HPO. ’stratified’ means different HPO were run (only the bests are presented in the results). 20