Benefits of Performance Testing with AI-Generated Synthetic Data¶

Software systems need close-to-real-world data to test against. Teams typically choose between three approaches:

Approach	Description	Primary Benefits	Primary Challenges
Real Data	Random sample of the data in the production database.	Representative of real-world data	Contains highly sensitive information and access is usually locked down
Fake Data	Generate fake data from scratch using the column types and some general attributes (e.g. value ranges) from the production data	Easy to generate large volumes of fake data	Not very representative of real-world data
Synthetic Data	Train an AI model on a sample of the production data	Representative of real-world data and less sensitive to share inside an organization	Requires expertise in model training and data quality evaluation

In this tutorial, we'll generate fake data with the DayZSynthesizer and synthetic data with the GaussianCopulaSynthesizer, then compare both against the real data — first statistically, then through a simulated fraud detection scenario.

When to use this cookbook: You're evaluating whether to use fake or synthetic data in place of production data — for example, when populating a dev/test environment or stress-testing a downstream service like a fraud detection pipeline.

Prerequisite: DayZSynthesizer is part of SDV Enterprise. The GaussianCopulaSynthesizer used for the synthetic-data sections is available in SDV Community.

import warnings

warnings.filterwarnings('ignore')

Loading and Exploring the Data¶

We'll work with the credit_card_transactions dataset. Here's an overview of its columns:

User: user ID linking transactions together
Card: card issuer, encoded as an integer
Year: year the transaction occurred
Month: month the transaction occurred
Day: day the transaction occurred
Time: time the transaction occurred
Amount: transaction amount
Use Chip: if an EMV chip card was used, or if the transaction was online
Merchant Name: name of the merchant
Merchant City: registered city for the merchant
Merchant State: registered state for the merchant
Zip: registered zip code for the merchant
MCC: merchant category code, a 4-digit number identifying the type of transaction
Errors?: any errors experienced as part of the transaction
Is Fraud?: whether the transaction was ultimately fraudulent

from sdv.datasets.demo import download_demo

datasets, full_metadata = download_demo(
    modality='multi_table',
    dataset_name='credit_card_transactions'
)

full = datasets['credit_card_transactions-ibm_v2']
full["Amount"] = full["Amount"].str.replace("$", "").astype(float)
full["Zip"] = full["Zip"].astype(str).str.replace(".0", "", regex=False)
full["Time"] = full["Time"].str.split(":").str[0].astype(int)

real_data = full.sample(frac=0.1, random_state=0).reset_index(drop=True)
del datasets, full


metadata = full_metadata.tables['credit_card_transactions-ibm_v2']
for col, sdtype in [
    ("User", "categorical"),
    ("Card", "categorical"),
    ("Amount", "numerical"),
    ("Merchant City", "categorical"),
    ("Merchant State", "categorical"),
    ("Time", "numerical"),
]:
    metadata.update_column(column_name=col, sdtype=sdtype)

Let's preview the data:

real_data.head()

	User	Card	Year	Month	Day	Time	Amount	Use Chip	Merchant Name	Merchant City	Merchant State	Zip	MCC	Errors?	Is Fraud?
0	1264	0	2015	10	20	19	59.63	Swipe Transaction	2301046522991844568	Westlake	OH	44145	5813	NaN	No
1	1693	3	2011	8	19	17	41.63	Swipe Transaction	6661973303171003879	North Hollywood	CA	91605	5211	NaN	No
2	763	2	2009	2	10	17	16.84	Swipe Transaction	-8288218236188507426	Pomeroy	OH	45769	5812	NaN	No
3	1133	1	2006	3	16	13	13.32	Swipe Transaction	-3150043395187512028	Edinburgh	IN	46124	5921	NaN	No
4	550	0	2020	2	25	3	8.90	Chip Transaction	-337314228625997565	Parkton	NC	28371	5921	NaN	No

Now let's explore the dataset to understand its key properties — how many transactions and users it contains, where the values fall, and how the categorical columns relate. Each of the following questions is designed to surface something important about the data that the synthesizers will need to reproduce.

How large is the dataset and how many unique users does it have?

print(f'Sample rows: {len(real_data):,}')
print(f'Columns: {real_data.shape[1]}')
print(f'Unique users: {real_data["User"].nunique():,}')

Sample rows: 2,438,690
Columns: 15
Unique users: 2,000

The dataset is large enough to exercise statistical modeling — over a million transactions across a couple thousand unique users. That gives the synthesizer plenty of signal to learn distributions and correlations from, and gives us enough rows to see meaningful differences between real, fake, and synthetic data downstream.

Which columns have missing values?

# Check for missing values
missing = real_data.isnull().sum()
missing[missing > 0]

Merchant State     272248
Errors?           2399915
dtype: int64

Only Merchant State and Errors? have missing values. We'll configure our fake data generator to match these proportions later.

What does the distribution of transaction amounts look like?

The vast majority of transactions are small — clustered near \$0 — with a long right tail extending past \$1,500. This heavy skew is typical of real-world spending behavior, where everyday purchases dominate and large transactions are rare.

Breaking it down into ranges reveals more detail:

Negative amounts represent refunds or chargebacks — they peak around -\$75. The \$0–\$250 range shows a steep decline from very small purchases, with periodic spikes at round-dollar amounts. Transactions above \$250 are sparse, forming a long tail out to \$1,500+.

What years are represented in this dataset?

Transaction volume grows steadily from the 2000s through 2020, reflecting increasing credit card adoption over time. The drop visible in 2020 reflects that the dataset only covers the first few months of that year (as the dataset was created at the beginning of 2020), not a real decrease in transaction volume.

How balanced is the fraud label?

The fraud label is highly imbalanced — only a tiny fraction of transactions are flagged as fraud. This is realistic: real-world fraud rates are typically below 1% of transactions. The imbalance matters for any downstream modeling task: a naive model that always predicts "not fraud" would already be ~99% accurate, so we'll need to be careful about how we evaluate performance later.

How do the key categorical columns relate to each other?

The chart below shows the flow between payment method, transaction errors, and fraud status.

The Sankey chart shows how transactions flow across payment method, error status, and fraud label. The thick band on the right indicates that most transactions have no recorded errors — the Errors? column is blank (i.e. null) for the vast majority of rows. The smaller bands break out the rare error types: bad PIN, insufficient balance, technical glitches, etc. Fraud is rare across all payment methods and is concentrated among transactions with no recorded error.

Generating Fake Data with DayZSynthesizer¶

The DayZSynthesizer generates fake data by randomly sampling values from user-defined ranges and categories. Unlike a statistical model, it does not learn any patterns or correlations from the real data.

To make the fake data as realistic as possible, we configure it step by step with information from the real dataset.

First, we instantiate a DayZSynthesizer from the metadata. The synthesizer starts in a default state where it knows the column types but nothing about the actual values, ranges, or null rates in the real data — those are the things we configure in the next steps.

from sdv.single_table import DayZSynthesizer

fake_synthesizer = DayZSynthesizer(metadata)

Next, we tell the synthesizer the missing-value proportions for each column that has nulls. By default, DayZSynthesizer assumes every column is fully populated, which would produce fake data without any nulls. Real credit card data has nulls in Merchant State and Errors?, so we explicitly set those proportions to match — otherwise the fake data would look unrealistically complete.

# Set missing value proportions to match real data
for col in ['Merchant State', 'Errors?']:
    fake_synthesizer.set_missing_values(
        column_name=col,
        missing_values_proportion=round(
            real_data[col].isnull().sum() / len(real_data), 10
        )
    )

Next, we tell the synthesizer the valid categories for each categorical column. Without this, the synthesizer doesn't know what values are legal for Merchant Name, Use Chip, etc., and would either fail to generate them or produce placeholders. We pass in the unique values from the real data so the fake data uses the same vocabulary.

cat_columns = [
    'User', 'Card', 'Use Chip', 'Merchant Name',
    'Merchant City', 'Merchant State', 'Zip', 'MCC',
    'Errors?', 'Is Fraud?'
]
for col in cat_columns:
    fake_synthesizer.set_category_values(
        column_name=col,
        category_values=real_data[col].dropna().unique().tolist()
    )

Finally, we set the numerical and datetime bounds for each non-categorical column. Without these, the synthesizer has no idea what range to sample from for Amount, Year, Month, or Day. We use the real data's min/max for each — this gives the fake data realistic ranges, even though the values within those ranges are sampled uniformly at random.

for col in ['Amount', 'Year', 'Month', 'Day', 'Time']:
    cast = float if col == 'Amount' else int
    fake_synthesizer.set_numerical_bounds(
        column_name=col,
        min_value=cast(real_data[col].min()),
        max_value=cast(real_data[col].max())
    )

    if col != 'Amount':
        fake_synthesizer.set_rounding_scheme(
            column_name=col,
            num_decimal_digits=0
        )

Now let's generate fake data with the same number of rows as our real dataset.

num_sample_rows = len(real_data)
fake_data = fake_synthesizer.sample(num_sample_rows)
fake_data[['Year', 'Month', 'Day', 'Time']] = fake_data[['Year', 'Month', 'Day', 'Time']].astype(int)
fake_data.head()

	User	Card	Year	Month	Day	Time	Amount	Use Chip	Merchant Name	Merchant City	Merchant State	Zip	MCC	Errors?	Is Fraud?
0	1451	1	1995	2	3	15	2718.94	Chip Transaction	3533508019517891227	Bradley Beach	NaN	1840	3405	NaN	No
1	429	2	2002	11	12	11	691.59	Online Transaction	6655367386832070574	Airville	Algeria	80016	7276	NaN	No
2	843	5	2002	10	28	2	4568.73	Online Transaction	5979176342209983088	Ore City	Uganda	97470	5311	NaN	Yes
3	1402	4	2015	5	20	12	2811.24	Swipe Transaction	-5140159998192253928	Arverne	AL	22935	5651	NaN	Yes
4	644	4	1992	7	23	8	2437.87	Online Transaction	686332575408057196	Bridgeport	NJ	7823	5499	NaN	Yes

Why does the fake data look uniform within each range?

By design. DayZSynthesizer is built for cases where you have no real data to learn from — only schema and ranges — so it samples uniformly within the bounds you provide. Its only configuration knobs are bounds (set_numerical_bounds, set_datetime_bounds), categories (set_category_values), null proportions (set_missing_values), and rounding (set_rounding_scheme); there is no distribution-shaping parameter. To capture real-world shapes (skews, modes, correlations), you need a synthesizer that learns from real data — which is exactly what GaussianCopulaSynthesizer does next.

Generating Synthetic Data with GaussianCopulaSynthesizer¶

The GaussianCopulaSynthesizer learns the statistical distributions and correlations in the real data, then generates new rows that preserve those patterns. This is the key difference from fake data — synthetic data is modeled, not randomized.

Notice that we don't need any of the manual configuration we did for DayZ — no category lists, no numerical bounds, no datetime ranges. The synthesizer learns all of these properties directly from the data.

from sdv.single_table import GaussianCopulaSynthesizer

synthetic_synthesizer = GaussianCopulaSynthesizer(metadata)
synthetic_synthesizer.fit(real_data)

synthetic_data = synthetic_synthesizer.sample(num_sample_rows)
synthetic_data.head()

	User	Card	Year	Month	Day	Time	Amount	Use Chip	Merchant Name	Merchant City	Merchant State	Zip	MCC	Errors?	Is Fraud?
0	1594	1	2012	9	5	14	35.35	Swipe Transaction	-4500542936415012428	Breaux Bridge	IL	2145	5499	NaN	No
1	1798	0	2012	7	2	12	134.92	Swipe Transaction	-4334232547381218591	Pensacola	OH	97504	5499	NaN	No
2	332	2	2011	2	13	7	36.44	Swipe Transaction	-5581123930363301609	ONLINE	WA	nan	4121	NaN	No
3	1888	2	2018	7	20	15	95.10	Swipe Transaction	3639123430068731390	Detroit	OK	47129	5814	NaN	No
4	1283	1	2008	11	27	3	12.94	Swipe Transaction	-5162038175624867091	Los Angeles	PA	10280	5812	NaN	No

Evaluating Data Quality¶

SDV provides a built-in quality evaluation that scores how well the generated data matches the real data's statistical properties. Scores range from 0 to 1, where 1 means a perfect statistical match.

How does the fake data compare to the real data?

from sdv.evaluation.single_table import evaluate_quality

fake_quality_report = evaluate_quality(
    real_data=real_data,
    synthetic_data=fake_data,
    metadata=metadata,
    verbose=False
)

print(f"Overall Score (Average): {float(fake_quality_report.get_score()):.2%}")

Overall Score (Average): 29.41%

The fake data lands a low overall quality score. This isn't a bug — it's expected. DayZ samples uniformly within the bounds you specify, so it preserves the schema and value ranges but loses the column shapes (skews, modes) and the column-pair correlations that the quality score rewards.

How does the synthetic data compare?

synthetic_quality_report = evaluate_quality(
    real_data=real_data,
    synthetic_data=synthetic_data,
    metadata=metadata,
    verbose=False
)

print(f"Overall Score (Average): {float(synthetic_quality_report.get_score()):.2%}")

Overall Score (Average): 70.98%

Key Takeaway: Synthetic data achieves a significantly higher quality score than fake data because it learns the statistical patterns in your real data rather than generating random values.

Comparing Transaction Amount Distributions¶

The quality score gives us an overall number, but let's look at specific columns to understand where the differences are. We break down transaction amounts into five fine-grained ranges and compare the distributions across all three datasets.

Fine Grained Distributions for Real Transaction Amounts

The real data shows distinct patterns in each range: refunds cluster around -\$75, small purchases (\$1–\$10) are extremely common, the \$10–\$100 range has a steep decline with periodic spikes at round-dollar amounts, \$100–\$500 drops off sharply, and transactions above \$500 are rare with a long tail.

Fine Grained Distributions for Fake Transaction Amounts

The fake data produces nearly flat, uniform distributions across every range. The characteristic peaks and patterns from the real data are completely absent — the \$500+ range even shows values in the micro-range (150µ), far from realistic.

Fine Grained Distributions for Synthetic Transaction Amounts

The synthetic data reproduces the overall shape of each range — the refund cluster, the steep decline in small purchases, and the right skew in higher amounts. While individual spikes are smoothed out, the general patterns are preserved.

The synthetic data closely mirrors the shape of the real data's distributions across all amount ranges. The fake data, by contrast, shows flat uniform distributions that don't capture any of the real spending patterns.

Simulating a Fraud Detection Scenario¶

Quality scores and distribution plots tell us about statistical similarity, but do they translate to real-world performance? Let's test with a practical scenario.

Credit card companies process millions of transactions daily. Their fraud detection systems assess each transaction and return a risk score, and the processing time depends on the transaction amount — larger transactions trigger more complex checks (additional verification, higher-risk review queues, etc.) and take longer.

To test whether real, fake, and synthetic data behave the same way under this workload, we simulate the runtime for each transaction. The simulation isn't measuring actual fraud-detection runtime — instead, for each transaction we draw a runtime from a normal distribution whose mean increases with the transaction amount. Concretely:

Amounts under \$5 → mean runtime 0.1s
\$5–\$20 → mean runtime 0.2s
\$20–\$40 → mean runtime 0.3s
\$40–\$80 → mean runtime 0.5s
\$80 and up → mean runtime 0.7s

Because the simulated runtime is a function of Amount, the shape of the resulting runtime distribution depends entirely on the shape of the Amount distribution in each dataset. If a dataset preserves the real distribution of transaction amounts, its simulated runtimes will match the real runtimes; if it doesn't (like fake data with uniform amounts), the runtime distribution will diverge.

import numpy as np

# Amount ranges and their corresponding (mean_runtime, std) values
ranges = [
    ((-float('inf'), 5), (0.1, 0.01)),
    ((5, 20), (0.2, 0.01)),
    ((20, 40), (0.3, 0.05)),
    ((40, 80), (0.5, 0.05)),
    ((80, float('inf')), (0.7, 0.05))
]


def calculate_fraud_risk(data, ranges):
    """Simulate a fraud check that takes longer for larger amounts."""
    amount = data['Amount']
    for (lower, upper), (loc, scale) in ranges:
        if lower <= amount < upper:
            return np.random.normal(loc=loc, scale=scale)


def collect_runtimes(data, sample_sizes, ranges):
    """Run the simulation at multiple sample sizes."""
    results = {}
    for size in sample_sizes:
        sample = data.sample(size)
        results[size] = [
            calculate_fraud_risk(row, ranges)
            for _, row in sample.iterrows()
        ]
    return results

Let's run the simulation on multiple sample sizes for each data type.

sample_sizes = [100_000, 1_000_000, len(real_data)]

real_runtimes = collect_runtimes(
    real_data, sample_sizes, ranges
)
fake_runtimes = collect_runtimes(
    fake_data, sample_sizes, ranges
)
synthetic_runtimes = collect_runtimes(
    synthetic_data, sample_sizes, ranges
)

Here are the runtime distributions at each sample size:

Runtime Distributions for Real Data¶

The real data produces a multi-modal runtime distribution with distinct peaks corresponding to the amount-based processing tiers. The pattern is consistent across all three sample sizes, becoming sharper with more data.

Runtime Distributions for Fake Data¶

The fake data produces a completely different shape — dominated by a single large peak around 0.7, with a small spike near 0.1. This is because the fake data's uniform amount distribution sends most transactions to the highest processing tier.

Runtime Distributions for Synthetic Data¶

The synthetic data reproduces the same multi-modal pattern as the real data — with modes appearing at similar positions, although there is a deviation in the frequency. This confirms that preserving the amount distribution translates directly to realistic downstream behavior.

The synthetic data produces runtime distributions that closely match the real data at every sample size. The fake data, by contrast, produces a fundamentally different shape — because its uniform Amount distribution doesn't reflect the real spending pattern, its runtimes also don't match.

Quantifying Runtime Distribution Similarity¶

Using KSComplement, we can put a number on how similar the runtime distributions are. A score of 1.0 means the distributions are identical; a score of 0.0 means they are completely different.

Fake vs. Real Data:

import pandas as pd
from sdmetrics.single_column import KSComplement

full_size = sample_sizes[2]

KSComplement.compute(
    real_data=pd.Series(real_runtimes[full_size]),
    synthetic_data=pd.Series(fake_runtimes[full_size])
)

np.float64(0.31777675719341114)

The fake data scores poorly on KSComplement — far below 1.0 — confirming numerically what the runtime distribution plots showed visually: the fake data's runtime distribution is fundamentally different from the real data's because its underlying Amount distribution is uniform, not skewed.

Synthetic vs. Real Data:

KSComplement.compute(
    real_data=pd.Series(real_runtimes[full_size]),
    synthetic_data=pd.Series(synthetic_runtimes[full_size])
)

np.float64(0.8682300743431924)

Key Takeaway: The synthetic data achieves a higher KSComplement score than the fake data, confirming that synthetic data produces runtime distributions closer to the real data. This validates that the downstream behavior of synthetic data better reflects real-world conditions.

Comparing Total Runtimes¶

Finally, let's compare the total processing times across all three datasets and sample sizes.

total_runtimes_df = pd.DataFrame({
    "Sample Size": sample_sizes * 3,
    "Total Runtime": [
        sum(real_runtimes[s]) for s in sample_sizes
    ] + [
        sum(fake_runtimes[s]) for s in sample_sizes
    ] + [
        sum(synthetic_runtimes[s]) for s in sample_sizes
    ],
    "Dataset Type": (
        ['Real'] * 3 + ['Fake'] * 3 + ['Synthetic'] * 3
    )
})

pd.options.display.float_format = '{:.2f}'.format
total_runtimes_df

	Sample Size	Total Runtime	Dataset Type
0	100000	36297.75	Real
1	1000000	363089.30	Real
2	2438690	885148.19	Real
3	100000	63683.73	Fake
4	1000000	637485.83	Fake
5	2438690	1554482.13	Fake
6	100000	39254.25	Synthetic
7	1000000	393030.40	Synthetic
8	2438690	958315.48	Synthetic

All three datasets scale linearly with sample size, but at different rates. The fake data (blue) has the highest total runtime because its uniform amounts overrepresent the expensive processing tiers. The synthetic data (teal) tracks closer to the real data (dark), confirming that realistic amount distributions lead to realistic aggregate performance.

Conclusion¶

In this tutorial, we compared real, fake, and synthetic data using a credit card transaction dataset.

Metric	Fake Data	Synthetic Data
Quality Score	Low	High
Distribution Match	Poor	Strong
Runtime Similarity (KSComplement)	~0.30	~0.86

Key Takeaway: Synthetic data is a far better alternative to fake data for testing, development, and simulation. It preserves the statistical patterns in your real data, which means downstream tasks produce realistic results.

To learn more, visit the SDV documentation.