Beyond Statistical Similarity: A Multidimensional Framework for Evaluating Synthetic Tabular Data

Abstract

This research examines the quality of synthetic tabular datasets generated by two deep generative models, CTGAN and TVAE. The evaluation uses three benchmark datasets representing different application areas: Adult Income, Bank Marketing, and German Credit. The study goes beyond conventional assessments based solely on statistical similarity. Each model is evaluated from three perspectives. The first examines how closely the statistical characteristics of the synthetic data resemble those of the original data. The second determines whether important relationships between variables, particularly their correlations, are retained. The third focuses on practical machine learning usefulness by training predictive models with synthetic data and testing them on an independent test set containing real, unmodified observations. To bring these evaluation aspects together, the study introduces the Insight Preservation Score (IPS). This composite measure combines distributional similarity, correlation preservation, and machine learning performance into one continuous score while maintaining the information provided by each individual component. The results indicate that CTGAN and TVAE each perform well in different aspects, meaning that neither model is consistently superior when individual metrics or datasets are considered separately. However, when the three dimensions are evaluated jointly using IPS, TVAE obtains the strongest overall performance across all three datasets. Its advantage is particularly noticeable for the German Credit dataset. These findings highlight an important limitation of evaluating synthetic data through statistical similarity alone. Matching the distributions of real and synthetic data does not necessarily mean that the synthetic dataset preserves meaningful relationships or remains useful for predictive modeling. A combined evaluation of distributional fidelity, correlation structure, and machine learning utility therefore provides a broader and more informative way to assess synthetic tabular data.