Case Study - Fraud Detector Audit
Problem
Our client was a multi-national, organically growing, medium-sized company with an already existing fraud detector system. They have identified a set of very rare outlier cases that were posing a threat to their system, but due to the scarcity of examples of certain kinds of fraud, they faced great difficulty making their system more resilient. They had an initial idea that if samples more similar to the few known problematic ones could be artificially generated, adding those to the training could improve their system’s resilience.
Solution
TechnoLynx has started off according to the initial scope and managed to develop a system that were generating new samples out of thin air, however, these did not exhibit the specific fraudulent properties we were looking for, and then later, a system that could augment originally correct/incorrect samples with minor changes making them fail/pass on the current detector, highlighting that it has more generic resilience issues than the one that was already known.
After concluding that generative approaches are also limited by the lack of data the same way as classification was, we tightened our cooperation with the client’s team, and after reviewing their data acquisition pipeline and accuracy metrics, we ended up delivering a set of recommendations which could improve the overall resilience of their existing system.
We mainly worked with a specific version of Keras to meet the customer’s requirements, and admittedly, a considerable amount of time has been spent working around Keras’ bugs related to the training of generative systems; hence, we also recommended the client gradually move away from using Keras for future projects.
Result
During the scope of this project our final recommendations have been implemented, resulting in a more dependable way of measuring accuracy of the model and a dramatic 20%+ improvement in accuracy as well.
Our best-practice recommendations were implemented at a process level at the client’s organization, delivering further benefit to the client’s other projects.
