Synthetic data for AML: why we reject datasets that train a perfect model

We reject synthetic AML datasets when the model trained on them scores too well. It sounds backwards. But a near-perfect score doesn’t mean you’ve built a great model. Instead, it usually means the synthetic criminals are easy to tell apart from everyone else in the dataset. 

Essentially, the model has picked up quirks in how the data was generated, and it won’t find any of those quirks inside a real bank. 

That’s why we treat an easy win as a warning. In fact, in one recent development cycle for a pig butchering typology, we tested 32 candidate datasets and rejected 8. Five of those failed because the model performed unrealistically well.

Good synthetic data should make the model work for its score. In this post, we walk through the two quality gates every Consilient dataset has to pass. We also cover what we do when a dataset fails, and what banks should also ask before trusting a strong result. 

Why synthetic AML data needs to make detection difficult

Say you’re building synthetic training data for a structuring model. Every suspicious account in the dataset makes cash deposits above $10,000. Train a model on that data, and it may perform exceptionally well.

But real bank data is far messier. 

A restaurant can legitimately deposit large amounts of cash after a busy weekend. A business owner can inject a substantial sum into their company. A contractor can receive frequent cash payments. Each can share characteristics with an activity an AML model has been trained to investigate:

This is where synthetic data quality starts to show.

If only suspicious accounts exhibit a particular characteristic, the model can use that characteristic to predict the label. Give legitimate accounts credible versions of the same behavior and the decision becomes harder.

Of course, deposit size isn’t enough. Transaction frequency may not be enough either. The model has to learn how several behaviors occur together, how they develop over time, and how the suspicious account differs from legitimate customers with similar activity.

That’s the standard synthetic AML data needs to meet. A model should have to learn the financial behavior associated with the typology, rather than a convenient feature of the dataset.

Gate one: Distributional overlap

Creating that kind of ambiguity takes deliberate work. It’s easy to generate suspicious behavior. Yet much harder to make sure the legitimate population gives the model enough credible alternatives.

Santander, for example, says its synthetic datasets must meet predefined quality thresholds before use, with each dataset evaluated across fields, relationships, and temporal behavior. Its wider framework considers fidelity, utility, and privacy. 

For synthetic AML data, we also need to know whether suspicious and legitimate behavior has become too easy for a model to separate. 

That’s why we got to work at Consilient on a way to test for it.

One of the quality checks we use is distributional overlap. For every feature available to the model, we examine whether legitimate and suspicious accounts occupy credible overlapping ranges. When one group consistently behaves differently, the model may have found an easy route to the label.

You can see how quickly this becomes a problem:

The aim isn’t to remove useful information from the data. A large cash deposit can still contribute to a suspicious profile. So can transaction frequency or a high-value transfer.

The key point is that the model needs more context before it can reach a useful conclusion.

Once legitimate customers occupy some of the same ranges, individual features become less revealing on their own. The model has to consider how behaviors occur together, how they develop over time, and what else is happening across the account.

When one feature separates the populations too neatly, the dataset goes back for further work.

Often, that means spending more time on the legitimate customers. And that turned out to be one of the most important parts of building useful synthetic AML data.

Why legitimate behavior deserves the same attention

This led us to another lesson: building the suspicious accounts is only half the work.

For each typology, you need credible customers whose legitimate financial activity resembles some of the behavior you’re trying to detect. Otherwise, the background population becomes too predictable and the model has an easier job than it will have inside a bank.

So when Consilient develops synthetic AML data, we deliberately include different levels of difficulty:

Take the cash example again. A useful dataset needs the suspicious account making repeated cash deposits. It also needs the restaurant depositing its weekend takings. Then it needs harder cases where transaction value alone won’t settle the decision.

The same discipline applies to the suspicious population. If every criminal follows the textbook version of a typology, the model can simply learn that version. Different amounts, timing, customer types, and methods of concealment force it to deal with variation.

This is why legitimate behavior takes so much work. The background population determines how difficult the suspicious activity is to identify.

Get that population right and a high model score starts to mean considerably more. But there’s still a second test to run: train a model on the dataset and see just how easy you’ve made its job.

Gate two: Model non-triviality

Once the suspicious and legitimate populations have enough overlap, we put the dataset through another test.

Consilient trains the same fixed-specification model on every candidate dataset. Keeping the model consistent means changes in performance can be traced back to the data, rather than changes made to the model between tests.

We’re looking for two types of failure:

#1. When the model struggles

If performance is too weak, the dataset may not contain enough learnable information about the typology. We’ve made the distinction between suspicious and legitimate behavior so difficult that the model can’t identify useful differences. This tells us the dataset needs more work.

#2. When the model finds it too easy

The other result is more interesting. An unusually strong score can mean the suspicious and legitimate populations are still too easy to separate. There may be a feature, combination of features or characteristic of the generated data that gives the model an unintended shortcut.

The purpose of model non-triviality testing is to find a credible middle ground. The typology needs enough learnable behavior for a model to identify it, alongside enough overlap and ambiguity to make the classification difficult.

When a dataset falls outside that range, we don’t make the test easier.

We go back and improve the financial behavior.

What happens when synthetic AML data fails a quality check

A failed quality check gives the team somewhere specific to look. Say cash deposit value is separating the suspicious and legitimate populations too easily. We ask what credible legitimate activity could produce similar values. This might mean adding seasonal cash businesses with periods of higher takings.

If incoming transfer size is doing too much of the model’s work, the dataset may need more businesses receiving legitimate commercial payments, or capital injections. If transaction frequency is the giveaway, we need customers whose ordinary activity produces a comparable volume.

The response depends on what the model has found:

Then we run the quality checks again and retrain the same fixed-specification model. This can take several rounds. 

There’s an important discipline here. We keep the quality test consistent and improve the financial behavior that failed it.

That approach can mean adding considerably more legitimate activity to fix a dataset built to detect crime. But those legitimate customers are what stop the model taking shortcuts. And once a model can perform well under those conditions, the score becomes much more useful.

What should banks ask about synthetic data quality for AML?

For banks assessing synthetic data for AML, a strong model score needs some context; a few questions can tell you a lot:

1. Do legitimate and suspicious customers occupy overlapping behavioral ranges?
   If transaction values, frequency or other features separate the 2 populations neatly, the model may have an easy route to the label.
2. Which legitimate customers reproduce the characteristics associated with the typology?
   A dataset designed around suspicious activity should also contain credible businesses and customers that create similar financial behavior for legitimate reasons.
3. How are candidate datasets tested consistently?
   Changing the model alongside the data makes it harder to understand what caused a change in performance. A consistent test gives you a better view of the dataset itself.
4. What happens when the model scores unusually well?
   This is a revealing one. An exceptional result should prompt investigation into what the model learned and whether the populations are too easy to separate.
5. How often does synthetic data fail the quality process?
   Rejection is useful evidence. Consilient rejected 8 of 32 candidate datasets, including 5 because model performance was unrealistically strong.

A synthetic data provider should be able to explain what its models are learning, which datasets didn’t make it through testing, and what changed as a result. For AML teams, that’s far more informative than being shown a near-perfect score and asked to admire it.

What 32 attempts at one AML typology tell us about synthetic data quality

Of course, this quality process means candidate datasets don’t always make it through. One Consilient development cycle for a pig-butchering typology gives a useful example. 

• 🟣32 candidate dataset runs went through the quality gates. 
• 🟣Eight were rejected. 
• 🟣Five of those eight failed because the model scored unrealistically well.

That’s a 25% rejection rate in this particular development cycle. It isn’t a benchmark for every typology Consilient develops, but it shows what happens when high model performance is treated as something to investigate.

Those 5 datasets had produced the kind of result that could easily look impressive in a model evaluation. Our testing indicated that the distinction between suspicious and legitimate behavior had become too easy to learn.

So they didn’t pass.

The work continued beyond the model tests. The pig-butchering package went through 9 rounds of analyst review before it met the release criteria. That review examined whether the financial behavior itself made sense to people experienced in financial crime.

Two pieces of evidence stand out here. The first is the final model performance. The second is what had to be rejected along the way.

For synthetic data, the rejection record can tell you a great deal about the standard behind the finished dataset.

Good synthetic AML data should make the model work

Financial crime detection depends on distinctions that are often difficult to make. Legitimate customers can make large cash deposits, receive substantial transfers, and transact at high volumes. Synthetic training data needs to give the model those difficult cases too.

That means testing for overlap, building credible legitimate behavior and investigating results that look unusually strong. Sometimes it means rejecting a dataset and starting another round of work.

We’ve done exactly that at Consilient. In the pig-butchering development cycle covered here, 5 candidate datasets were rejected because the model performed unrealistically well.

A model score becomes more meaningful when you know it had to work for it.

If you’re exploring how synthetic data could support your AML model development or testing, talk to us, we’d genuinely love to help.