AI and Banking: Where AI Actually Earns Its Keep

Banking AI pays back where high-volume scoring meets tight latency. Here is when GPUs beat CPUs for fraud models, and how to keep workloads portable.

AI and Banking: Where AI Actually Earns Its Keep
Written by TechnoLynx Published on 20 Feb 2024

Banking AI repays its compute most clearly when a model scores a very large number of events against a tight latency budget. Fraud and anti-money-laundering screening is the plain example, and UK firms say it is one of the areas with the highest perceived current benefit. The headline savings figures for AI and banking, such as those Backbase passes on from Accenture and McKinsey, are consultancy and vendor estimates made across a whole portfolio. They tell you little about whether one use case will cover its own serving cost. The engineering questions that decide the return are narrower: which model family you serve, whether a GPU or a CPU scores it more cheaply at your volume, and whether the workload can move between cloud and on-premise hardware without being rebuilt.

Adoption is no longer the open question. The Bank of England and FCA’s 2024 survey of UK financial services found that 75% of responding firms already use AI, with a further 10% planning to within three years. In the 2022 survey the figures were 58% and 14%. For the wider picture across underwriting, claims and fraud, our overview of AI use in insurance and finance covers the sector. This article stays with banking compute.

Where Does AI Deliver the Clearest Operational Value in Banking Today?

Fraud and financial-crime detection is among the clearest cases. The same 2024 survey lists data and analytical insights, AML and combating fraud, and cybersecurity as the areas of highest perceived current benefit. Cybersecurity (37%) and fraud detection (33%) are among the most common current uses, and a further 31% of respondents expect to use AI for fraud detection within three years. JPMorgan Chase tells a similar story: it initially focused its AI on risk management, such as fraud detection.

Fraud pays back for structural reasons, not fashion. The volume spreads the fixed cost of serving across a huge number of decisions. Visa’s figures show what that looks like at network scale: more than 127 billion transactions on VisaNet in a year, each analysed in about one millisecond, against more than 500 risk attributes. Visa estimates about $25 billion of fraud prevented a year. That is Visa’s own estimate. It shows the shape of the return, not a figure another bank should expect.

The most common model family across all use cases reported by UK financial-services firms is a familiar one. Gradient boosting models make up 32% of all AI use cases reported in the survey, well ahead of transformer-based models at 10%, with random forests and decision trees close behind. Tree ensembles also suit audited decisions, because it is easier to see which variables drive a decision forest than a deep network. How to design the fraud and AML scenarios those models serve is a separate problem, covered in our piece on fraud/AML scenario design.

Operational automation is the other strong contender. JPMorgan reports that AI and machine learning cut unit costs in its corporate and investment bank’s know-your-customer processes by nearly 40%. That is a self-reported figure from one bank. Even so, it fits the pattern: repetitive, high-volume work with a cost per case you can already measure.

Are the Headline AI Savings Figures for Banking Realistic?

As portfolio-level estimates, they are plausible. As forecasts for any single deployment, they are not reliable. Backbase, a banking software vendor, writes that financial firms that moved AI from pilots into production report 15 to 20% operational cost reductions at moderate adoption. It also relays Accenture’s finding that AI-leading banks gained 125 basis points of return on equity and 452 basis points of cost-to-income improvement over laggards. The same post cites a McKinsey estimate of a 30 to 50% reduction in manual workload once AI agents in finance run at scale. These are consultancy figures passed on by a vendor with a platform to sell, and the Accenture figures compare leaders with laggards.

The more useful warning in that source concerns cost structure. Backbase argues that when every use case is built on its own data model and integration path, a bank running 15 pilots carries integration overhead greater than the savings of any one of them. Whatever you make of the vendor’s remedy, the mechanism is real: a fifteenth bespoke pipeline does not get cheaper to run because the first fourteen exist.

Serving cost is part of the same arithmetic. The Hummingbird paper from Microsoft researchers cites an Amazon Web Services analysis: model scoring accounts for 45 to 65% of the total cost of ownership of data science solutions, because models are trained once and then used many times. A use case that scores often but has little value per decision can therefore lose money on inference alone. The survey’s materiality ratings bear on this. Of all reported use cases, 62% were rated low materiality, 22% medium and 16% high.

What Infrastructure Does Production Banking AI Actually Need?

For fraud, one production pattern pairs graph embeddings with a low-latency serving layer for tree ensembles. NVIDIA’s fraud detection blueprint shows that pattern. A graph neural network turns transaction data into embeddings, an XGBoost model scores those embeddings, and both are served through Triton Inference Server. Its training container hides the work of building a feature store for tabular data and a graph store for structural data. NVIDIA’s case for adding the GNN is that it looks at how transactions connect rather than at each one alone, which it says improves accuracy and cuts false positives.

In NVIDIA’s fraud example, Triton’s Forest Inference Library (FIL) backend on a DGX-1 with eight V100 GPUs kept p99 latency under 2 ms while serving over 400,000 inferences per second, about 20 times the throughput of CPU execution. That implies a CPU baseline of roughly 20,000 inferences per second on the same workload (400,000 divided by 20). NVIDIA’s own conclusion is about model complexity rather than raw speed: within any given latency budget, the GPU lets you deploy a more complex and accurate model than the CPU can.

Credit risk is harder to size from evidence. We would not quote a figure for it. So the GPU argument above transfers to credit only once you profile your own volumes. Market-facing inference is a different regime again. STAC’s audit of an NVIDIA GH200 system measured 99th-percentile latencies in microseconds on LSTM models used for trading workloads. That belongs with trading latency constraints, not retail credit.

Diagnosing whether the stack is compute-bound, memory-bound or waiting on data for bank workloads is covered in diagnosing GPU utilisation.

When GPU Inference Pays off for Tree Models, and When CPU Wins

A GPU pays off for tree-model inference when the forest is medium or large and the batches are big. It does not pay off when data loading becomes the bottleneck. A 2023 study comparing decision forest inference platforms found GPUs cheaper than CPUs for medium and large forests with large batches, and CPUs better wherever data loading was the bottleneck. On a fraud dataset with 10-tree models, data transfer accounted for 90 to 97% of total latency, depending on the CPU-side platform. The authors put it plainly: small datasets cannot keep a GPU busy, and the compute saved does not cover the cost of moving data between CPU and GPU.

Workload shape Cheaper option in the evidence What the evidence shows
Small forest (about 10 trees), features loaded per request CPU, ideally close to the data CPU platforms cut cost by 99% versus GPU in the forest study
Medium forest (about 500 trees) CPU in the study’s end-to-end setup; GPU can win with large batches CPU platforms cut cost by 44% versus GPU at 500 trees; GPU was cheaper for medium and large forests with large batches
Large forest (about 1,600 trees) GPU GPU reached better overall cost at 1,600 trees
High-volume fraud scoring with a strict p99 budget and a complex model GPU serving (Triton FIL) Under 2 ms p99 at over 400,000 inferences per second on eight V100s
Existing Xeon fleet running XGBoost or LightGBM Test optimised CPU libraries first Intel reports faster inference with oneDAL on AVX-512 Xeon at unchanged prediction quality (vendor benchmark)
Very sparse inputs Avoid tensor-compiled routes without testing Hummingbird’s authors note poor sparse support on DNN runtimes

Price per hour misleads in the same way. In the forest study, the GPU instance (AWS g4dn.2xlarge with a T4) cost $0.752 an hour against $0.532 for the CPU instance (r4.2xlarge), so about 41% more per hour. Whether that premium pays off depends entirely on which row of the table your workload sits in.

The sources also disagree, and the disagreement is worth spelling out. The Hummingbird paper found CPU cost 10 to 120 times higher than GPU across its experiments, while the forest study found CPU far cheaper for small models. The forest study counted data loading end to end, and in fraud detection features are typically pulled from transaction databases. For a production scoring service the end-to-end measurement is the more relevant one, because the service pays for the data path whether or not the benchmark counts it. NVIDIA’s account of PayPal, in which CPU-only servers could not meet a real-time global fraud requirement and T4 GPUs lowered server capacity by nearly 8x, sits at the high-volume end of the table.

The practical rule follows from TechnoLynx’s general stance: profile the actual serving path before choosing hardware, because advice like “GPUs are faster for inference” only makes sense in a context that profiling supplies.

Why Is AI-Driven Credit Decisioning Slower to Deploy than AI-Driven Fraud Detection?

Credit decisioning faces explainability and audit demands, and its returns take 12 to 24 months to measure. Backbase suggests measuring credit-decisioning returns by tracking cost per funded loan, approval-to-funding cycle time and portfolio quality over 12 to 24 months. A model whose value takes that long to prove takes that long to earn trust.

The explanation burden is also formal. In the 2024 survey, 81% of firms using AI employ some explainability method, with feature importance (72%) and Shapley additive explanations (64%) the most common. One vendor of on-premise banking AI argues that credit models must be fully explainable, auditable and free from discriminatory bias under fair lending rules. We would read that as a fair description of the regulatory pressure, while noting that the vendor uses it to argue for on-premise deployment. Variable-level explanation is also easier for decision forests than for deep networks.

Third-party models add a further gap in understanding. Of firms using or planning to use AI, 46% report only partial understanding of the AI they use, largely because of third-party models. A credit committee cannot sign off what nobody can explain. How probability-of-default models are built, validated and monitored is covered in our guide to PD credit-risk models. What a regulator expects to see recorded for an AI-assisted workflow is set out in what an AI workflow evidence pack looks like.

Keeping Banking AI Portable Across Cloud and On-Premise Hardware

Keeping a banking AI workload portable is engineering work, because a model that performs in training is a different artefact from one that performs on the deployment target. Triton Inference Server illustrates the serving side. It can run models on either CPU or GPU, it ships as a Docker container that works with Kubernetes platforms such as EKS, GKE, AKS and OpenShift, and it runs on bare metal as well as virtualised on-premise environments. NVIDIA’s fraud blueprint shows one cloud route: the model is deployed with Triton on a SageMaker or Amazon EKS endpoint.

The model format is the other half. Traditional ML toolkits are optimised for training, and supporting every operator in every target environment is expensive, which is why their portability is limited. Hummingbird gets around this by compiling tree models and scikit-learn operators into tensor operations, with output to PyTorch/TorchScript, ONNX and TVM. The conversion has costs to plan for:

  • Output drift: in Hummingbird’s tests, most converted models differed from the original output on fewer than 0.1% of records, with a worst case of 0.8% (a random forest on the Covtype dataset). A bank needs to validate that difference against its own decision thresholds.
  • Conversion time: converting a random forest to TVM for GPU execution took up to 40 minutes, and compilation-based tools such as TreeLite and lleaves needed more than a day for a 1,600-tree model.
  • Memory: TVM used two to three times the memory of scikit-learn, and Hummingbird is limited to the memory of a single GPU.

The reason to bother is concentration risk, not engineering tidiness. The 2024 survey found that the top three cloud providers accounted for 73% of named cloud providers, and the top three model providers for 44%, up from 18% in 2022. Respondents ranked critical third-party dependencies as the second-highest systemic risk. The Financial Stability Board’s report names third-party dependencies and service provider concentration among the AI vulnerabilities that could raise systemic risk. A workload that can only run on one provider’s stack turns that systemic concern into an operational one for the bank.

Latency pulls in the same direction. One on-premise AI vendor states that cloud inference typically adds 50 to 200 ms of round-trip time, against single-digit milliseconds on a local network. That is a vendor claim, and it depends on distance and network conditions. But measured against Visa’s roughly one millisecond per transaction, any added network hop is significant for authorisation-path scoring. JPMorgan, for one, reports that about 50% of its applications now run on public or private cloud.

TechnoLynx’s position on AI engineering for finance is that portability work pays off when it is scoped against a measurable trigger, such as a performance ceiling, a constraint of the deployment target or a lock-in cost.

Who Can Move Our Fraud or Credit-Risk Models Between Cloud and On-Premise Hardware?

This work does not suit every bank. If training and production already share one stack and nothing forces a move, there is nothing to port. Fair-lending judgements and model-risk sign-off stay with your risk function and its validators. Two other routes are realistic. Where the bank is content with one provider’s stack, its managed serving platform is a reasonable choice. Where an in-house platform team already runs Triton or ONNX Runtime at scale, that team is a realistic route.

How Much Weight Should a Bank Put on Published GPU and CPU Figures?

Two gaps should shape how much weight any bank puts on published numbers. First, no independent measurement here covers credit-risk inference at production scale, so sizing for credit rests on your own profiling, not on fraud benchmarks. Second, several GPU-versus-CPU figures come from vendors (NVIDIA, Intel), while the academic forest study shows the answer flipping with forest size and data path. The decision that matters is which row of the table your scoring service is in today, and which row it will be in once the model grows.