Synthetic Data for Machine Learning: How to Evaluate It
Evaluate synthetic ML data with real holdouts, baselines and leakage checks. Distinguish training data from random QA records without invented accuracy claims.
By Vincent RuanPublished June 25, 2026Last updated September 6, 20265 min read
Synthetic data for machine learning consists of generated observations used to train or evaluate a model. Its value depends on whether it retains the relationships needed by that task. A realistic-looking row is insufficient: a plausible customer name says nothing about how that customer would churn.
This guide focuses on evaluating tabular synthetic training data. For software fixtures and database seeding, use the test data generation guide. The distinction matters because a pipeline can run successfully on random inputs while learning nothing useful.
Choose the data source by the job
| Source | Useful job | What remains unproven |
|---|---|---|
| Rule-based fictional records | Schema, import, batching and UI tests | Real-world predictive relationships |
| Resampled training data such as SMOTE | A candidate treatment for class imbalance | Whether it improves the selected metric on real data |
| A fitted tabular generator such as CTGAN | A candidate source of training rows with learned relationships | Utility, subgroup behavior and disclosure risk |
| A domain simulator | Scenarios with an explicit mechanism and labels | How closely the simulation matches deployment |
CTGAN’s original paper evaluates a model for mixed categorical and continuous tabular data [ctgan]. That paper supports testing CTGAN as one candidate; it does not establish that every GAN outperforms a statistical sampler. Compare techniques on your own prediction problem using the same protocol.
Split before fitting the generator
- Split the real data into training, validation and final test partitions. Use entity or time splits when repeated customers or temporal deployment would make a random row split misleading.
- Fit imputers, encoders, scalers, oversamplers and the synthetic generator using training data only.
- Generate candidate training datasets and fit the downstream models.
- Select generator settings, model settings and any real/synthetic mix using validation data.
- Evaluate the selected setup once on the untouched real test partition and report uncertainty and subgroup results.
In cross-validation, fit the generator or oversampler again inside each training fold. The imbalanced-learn documentation demonstrates why resampling the full dataset before splitting produces leakage and overly optimistic evaluation [resampling]. An apparently strong score can come from that mistake rather than better training data.
Compare three experiments on the same real holdout
| Experiment | Downstream training data | Evaluation data | Question |
|---|---|---|---|
| Real baseline | Real training partition | Untouched real test partition | How well does the available real data work? |
| TSTR | Synthetic rows generated from real training data | The same real test partition | Does the generated data retain useful signal? |
| Augmented candidate | Real training rows plus a selected synthetic mix | The same real test partition | Does adding synthetic data improve the target outcome? |
Keep model families and tuning effort comparable, and report generated row counts and seeds. For rare-event tasks, choose a metric tied to the cost of mistakes, such as recall at a fixed false-positive rate, rather than accepting overall accuracy alone. Record subgroup sample sizes so a score based on a handful of cases is not presented as stable.
Training and testing on one synthetic release can yield misleading model-selection conclusions. The ICML paper Synthetic Data, Real Errors studies this evaluation problem [synthetic-errors]. A real holdout measures utility for its own population and time period; it still does not guarantee performance after deployment shifts.
Set acceptance criteria before looking at the score
| Dimension | Evidence to retain | Limit of the evidence |
|---|---|---|
| Task utility | Baseline, candidate score, metric, uncertainty and repeated-run variation | No universal acceptable gap such as “within a few points” |
| Subgroup utility | Per-group metric and sample count | An overall improvement can conceal a regression |
| Data fidelity | Missingness, category frequencies and task-relevant joint relationships | Matching marginal distributions does not prove predictive utility |
| Copying or disclosure | Exact-match checks and attacks with explicit capabilities | No successful attack in this setup is not proof of anonymity |
| Operational reproducibility | Source partitions, package versions, seeds and generator configuration | Changing the release or input data can change the result |
Do not treat PSI 0.1, an attacker AUC near 0.5, or a small nearest-neighbor distance as a universal pass/fail standard. Distance depends on scaling and feature types; attack performance depends on attacker access and evaluation design. Define the decision rule for the actual use case and retain its limitations.
Utility and privacy need separate evidence
A model can perform well while revealing training information. Carlini and colleagues demonstrated extraction of training examples from a language model [carlini]. That is evidence that memorization is possible, not a measured leakage rate for every tabular generator. NIST SP 800-188, finalized in 2023, discusses de-identification techniques and governance rather than treating a synthetic label as automatic anonymity [nist].
When claiming differential privacy, document the mechanism, the protected unit, parameters such as epsilon and delta, and composition across releases. An ordinary generator followed by an attack test does not acquire a differential-privacy guarantee. Generated names or masked IDs also do not settle whether other fields reveal source records.
Where a fake-profile generator fits
Fakenamely’s bulk exporter produces fictional profile fixtures for development. It does not fit a statistical model to your production dataset, preserve a measured label distribution or validate ML training utility. Use it to check an ingestion pipeline, feature shape or batch size. Evaluate a domain dataset separately before making predictive-performance claims.
References & sources
- Modeling Tabular Data using Conditional GAN (2019) — NeurIPS
- Common pitfalls: data leakage when resampling — imbalanced-learn
- Synthetic Data, Real Errors: How (Not) to Publish and Use Synthetic Data (2023) — ICML / PMLR
- Extracting Training Data from Large Language Models (2021) — USENIX
- SP 800-188: De-Identifying Government Datasets: Techniques and Governance (2023) — NIST
Frequently asked questions
Is synthetic data as accurate as real data for machine learning?
There is no universal accuracy gap. Results depend on the dataset, generator, prediction task, metric and evaluation split. Compare a real-data baseline and a synthetic-data candidate on the same untouched real holdout, including important subgroups.
Can I train a useful model on random names and addresses?
They can test parsing, batching, feature pipelines and training-job execution. Random profile fields do not supply a validated relationship to a business outcome, so success on those rows does not establish real-world predictive performance.
What is Train-Synthetic-Test-Real?
Train a downstream model on generated training rows, then evaluate it on held-out real rows. Compare it with the same model family trained on real training rows. Keep the final real test set outside generator fitting, preprocessing and model selection.
Should I apply SMOTE before splitting the dataset?
No. Split first and resample only the training data. During cross-validation, resampling belongs inside each training fold. Resampling the whole dataset can leak information into the evaluation set.
Does passing a privacy attack test prove anonymity?
No. A failed attack only describes that attacker and test setup. It does not prove that every possible attack fails. Privacy review needs the data provenance, release context and threat model; formal privacy claims require a correctly specified mechanism and accounting.