Consilient Bank Trials Demonstrate Federated Machine Learning Improves Effectiveness and Efficiency for Financial Crime Detection

FinTech innovator Consilient recently concluded six months of bank trials that produced the first known empirical test showing that federated machine learning can help detect financial crime. 

Over the past two years, Dozer™ has demonstrated increases in both effectiveness and efficiency over existing rules-based AML/CFT tools. We first proved that machine learning (ML) improves detection of financial crimes over extant rules-based systems. We then moved an ML model between banks to show the potential of federated learning. In the first such test, for example, within one bank, Dozer reduced false positives from above 90% down to 12%. In this study, Dozer alerted on 50 bank customers in a random stratified sample population of over 5,000,000. Human review confirmed 44 out of 50 true positives, or a 12% false-positive rate. In addition, 43 of the 44 true positives were previously known by the bank, so Dozer discovered one false negative from the bank’s incumbent system, and the bank BSA team filed one new SAR following this experiment. 

We initially deployed Dozer at a single large financial institution (“Institution 1”). At Institution 1, we built a model to identify money services businesses (MSBs) based on behavioral features. We deployed the model on a stratified random sample of over 5,000,000 accounts, alerting on 50 bank customers operating like MSBs who had not identified themselves as such.  Analyst teams at Institute 1 worked these cases and confirmed 44 of 50 were, in fact, MSBs. This was a  massive reduction in false positives, from above 90% with existing systems down to 12%. This model performed across the entire data set of accounts at Institution 1 with an AUC-ROC score of 0.82 (18% inefficiency/ineffectiveness) at identifying MSBs. More importantly, Dozer discovered one false negative from the bank’s incumbent system, and Institution 1’s BSA team filed one new SAR following this experiment. That’s an 87% reduction in the false-positive rate with a small increase in discovery.

Next, we transferred this algorithm to a second financial institution (“Institution 2”). At Institution 2, we used predictions from the algorithm, which was trained only on external data. These predictions (with no training at Institution 2 at all, known as 0-shot transfer learning) resulted in an AUC-ROC score of 0.83 at identifying MSBs. Note that these banks serve different customers, and their customers thus represent totally different samples. While the measure of 0.83 AUC-ROC at Institution 2 isn’t directly comparable to 0.82 at Institution 1, it shows just how much was learned at Institution 1 and, more significantly, the ability to transfer that learning to a second institution. 

We then fit a model to identify MSBs specifically in Institution 2’s customer base. This model resulted in 0.88 AUC-ROC. Of course, the model trained on the actual data at Institution 2 outperformed the model fitted on Institution 1’s data and applied at Institution 2. But the performance difference was small, suggesting the model from Institution 1 was picking up on some features not learned in Institution 2’s data (e.g., because there is a class of MSBs present in large numbers at Institution 1 and not at Institution 2). The goal of transfer learning in such settings is to combine models fitted in different settings to outperform models fitted only on a single data set.

To measure transfer learning, DOZER combined both the local model (fit at Institution 2) and the transferred algorithm from Institution 1. This combined algorithm resulted in a 0.90 AUC-ROC, exceeding the model trained at Institution 2 by 0.02 AUC-ROC, or 17% (0.02 difference out of 1.00-0.88 or 0.12) of remaining AUC.  

This process demonstrates the power of federated learning. By moving just weights and parameters between two data sets with identical feature DEFINITIONS but with different and non-i.i.d. feature DISTRIBUTIONS, we improved the performance on Institution 2’s data by 0.02 AUC-ROC points. While many approaches to applying machine learning at Institution 2 would be able to attain performance equivalent to the 0.88 measure, a federated learning approach (which is implemented in Dozer) that leverages information from other non-i.i.d. data sources was proven to improve that single-bank performance and show promise for improving efficiency and effectiveness.

These results show that the global financial regime can successfully use federated learning technology to fight criminal activity, preserve privacy and expand banking products to more communities.

For more information, see our white paper, “Federated Learning through Revolutionary Technology,” which can be downloaded here.