Bias-controlled testing finds no LLM trading strategy that reliably beats the market. In the large robo-advice study, the gains were lower costs and lower risk, not market-beating returns. In robo-advice, a large study reports lower fees and better risk-adjusted returns, and Mastercard’s own analysis claims fewer false alarms on genuine card payments. Trading is the weak spot. Language-model trading edges that look strong in short backtests tend to shrink or disappear once the tests run over longer periods with bias controls. So expect AI fintech to deliver better plumbing, not a money machine.
Where AI fintech earns its keep today
AI use in finance is already routine, and most of it is operational. In the Bank of England and FCA survey of UK financial services firms (2024), 75% of respondents said they already use AI and a further 10% plan to within three years, up from 58% using it in the 2022 survey. Those firms rated data and analytical insights, anti-money laundering and fraud, and cybersecurity as the areas where AI currently brings the most benefit. The Financial Stability Board (2024) describes current use as mainly improving internal operations and regulatory compliance, with generative AI adding document summarisation, information retrieval and code generation.
The SEC chair, speaking to FSOC in December 2023, listed call centres, account opening, compliance programmes, claims processing, trading algorithms and sentiment analysis, along with fast change among robo-advisers and brokerage apps. NVIDIA’s 2025 survey of financial services, a vendor survey and best read as one, reports that generative AI use for customer experience through chatbots and assistants rose from 25% to 60% of respondents. The same survey reports that trading and portfolio optimisation topped respondents’ list of generative AI use cases by ROI, at 25% of responses.
Fully autonomous decision-making is still rare. In the BoE/FCA survey, 55% of AI use cases involved some automated decision-making, but fully autonomous decision-making accounted for only 2% of applications. For the banking side of this picture, our piece on where AI actually earns its keep in banking looks at the retail and operations use cases in more depth.
What the evidence says, task by task
The table below summarises which finance tasks AI handles well, based on the sources this article draws on.
| Task | What the evidence shows | What limits it |
|---|---|---|
| Card and payment fraud detection | Gradient-boosted models reach an F1 score of 0.85 on a customer-level benchmark; Mastercard’s own initial modelling claims detection rates improved by an average of 20% | Precision against recall: every false positive blocks a genuine customer |
| Credit scoring | High-risk under the EU AI Act | Bias testing, human oversight, and explaining why an application was denied |
| Robo-advice | Sharpe ratio up from -0.014 to 0.115 at six months; expense ratios down from 19 to 9 basis points | Gains come from lower risk, not higher returns |
| LLM trading signals | 50-58% directional accuracy in one simulated study; no significant alpha in a 20-year bias-controlled study | Non-stationary markets, low signal-to-noise, backtest bias |
| Questions answered from financial documents | Best tested configuration was correct in 79% of cases on FinanceBench | Hallucinations that look plausible and are hard to catch |
| Risk calculation (HPC) | Vendor-reported node count cut from 100 to 4 for CVA on eight-GPU servers | Vendor figure, separate from the STAC benchmark audit |
Fraud detection works, and the false-positive bill is the real metric
Firms place fraud detection near the top of the areas where AI brings them the most benefit today. Visa told CNBC in 2024 that it scores more than 500 attributes on each transaction across roughly 300 billion transactions a year, and that it prevented $40 billion in fraudulent activity between October 2022 and September 2023. Mastercard’s February 2024 announcement claims its generative-AI enhancements raise fraud detection rates by 20% on average and cut false positives by more than 85% in its own analysis. Those are company claims from the operators themselves, not independent tests.
Academic benchmark results also point to strong fraud models. On the Customer-level Fraud Detection Benchmark, which restructures transaction data around customer behaviour, XGBoost achieved an F1 score of 0.85 with precision of 0.85 and recall of 0.84. The authors stress precision because false positives are expensive in fraud work.
The trade-off is the engineering decision, not the model choice. Tighten the threshold and you catch more fraud while declining more legitimate cards. Loosen it and you let losses through. Which side to favour depends on the cost of a blocked customer against the cost of a missed fraud, and that is a business number the model cannot supply. The pressure is also rising: a report cited by CNBC in 2024 estimated that generative AI could push US fraud losses to $40 billion by 2027, from $12.3 billion in 2023.
Credit decisions are where regulation bites hardest
Under the EU AI Act, AI used to evaluate creditworthiness or credit scores is high-risk under Annex III point 5(b), while AI intended to detect financial fraud is expressly excluded from that category. The binding date for standalone Annex III credit and insurance systems is 2 December 2027. Two duties stand out in practice. Article 10 requires training and validation data to be examined for bias, including proxies such as a postcode standing in for ethnicity. Article 14 requires human oversight from someone who can actually understand and override the score. Banks that buy a scoring model rather than build one are still deployers and must carry out a Fundamental Rights Impact Assessment before use.
The US position is just as plain. The CFPB’s September 2023 guidance says lenders using AI must give consumers accurate and specific reasons for a credit denial, and that “there is no special exemption for artificial intelligence”. In the UK, the FCA puts the responsibility for explainability and governance on the institutions themselves.
For engineering, this decides the architecture before any accuracy figure does. A model whose decisions cannot be traced to stated reasons is a liability in lending, so explainability (feature attribution such as SHAP, reason codes, documented bias tests) belongs in the design, not added at audit time. SHAP is one established technique for showing which features drove a given prediction.
Robo-advice: better portfolios, mostly through lower risk
Robo-advice has a large measured study of an advice service, which also involved human advisors, improving individual investors’ outcomes. A study of more than 80,000 previously self-directed investors who signed up for advice between 2015 and 2018 found their six-month annualised Sharpe ratio rose from -0.014 before advice to 0.115 after, both statistically significant. The share of wealth in index funds rose from 47% to 83%, and average expense ratios fell from 19 to 9 basis points.
The mechanism is less exciting than “the algorithm beats the market.” The authors found the improvement came mainly from lower portfolio risk, not higher average returns. Robo-advice diversified people, moved them out of individual stocks and costly active funds, and removed home bias. The investors who gained most were the ones holding a lot of cash, trading heavily or paying high fees. The authors also found that automatic implementation of the advice was crucial to its effect.
This is AI carrying out a repeatable, disciplined task automatically, which is the kind of problem where we expect measurable value. That matches our standing view that projects aiming at realistic, repeatable work are far more likely to pay back than projects that need the machine to outthink the market.
Can an AI trading system reliably make money, or is that hype?
In FINSABER’s bias-controlled tests, neither LLM agent produced significant alpha, and earlier reported gains were likely artefacts of narrow evaluations. The optimistic case is real but narrow. In a simulated backtest of LLM-generated signals on 50 large-cap equities over 2023, GPT-4 Turbo produced an annualised return of 31.2% and a Sharpe ratio of 1.42, ahead of the S&P 500. The same authors say those figures were produced under idealised simulation and should not be read as predictions of live trading. Their test ignores market impact, partial fills and microstructure, all of which tend to reduce realised returns. Signal accuracy across models was roughly 50-58%, against a 33% random baseline for a three-class buy/sell/hold task. The study covered a single year and one market regime, and the authors call durability across regimes the key unanswered question.
The bias-controlled counter-evidence is broader. The FINSABER study tested LLM investing strategies over two decades and more than 100 symbols, controlling for survivorship, look-ahead and data-snooping bias, and found that previously reported LLM advantages deteriorated significantly. Neither LLM agent produced statistically significant alpha, with every p-value above 0.34, and a plain buy-and-hold baseline significantly outperformed both in the bias-mitigated setups. The regime analysis is the most telling part: the strategies were too conservative in bull markets and too aggressive in bear markets, and in the 2008 crisis one agent’s drawdown on a single stock approached -75%, against -50% for the index. Larger models did not reliably beat smaller ones, and simple models such as ARIMA often beat both on risk-adjusted terms.
Two further findings complete the picture. Lopez-Lira and Tang showed that GPT-4 scores on news headlines predict subsequent price drift, yet strategy returns decline as LLM adoption rises, which fits markets pricing the signal in. Glasserman and Lin showed that backtesting an LLM on news from inside its training window is biased, because the model may already “know” what happened next.
Why markets defeat models that work elsewhere
Financial markets reject several assumptions machine learning relies on. Their data-generating process is non-stationary: relationships change as the market environment changes, sometimes permanently. Signal-to-noise is low by design, because arbitrageurs remove inefficiencies quickly. Finance is also a small-data field, with usually one limited history per variable, which makes overfitting and spurious patterns more likely. Flexible models will find patterns whether or not they are real. Crises are rare, severe and each one is different, which leaves training data sparse on them.
None of this is an unavoidable obstacle to ML in markets. Gu, Kelly and Xiu found large gains from trees and neural networks in measuring asset risk premia, in some cases doubling the performance of leading regression-based strategies, and traced the gains to nonlinear predictor interactions. That is the measurement of asset risk premia across stocks, a different task from an LLM calling tomorrow’s price.
Checklist for reading an AI trading claim
Before trusting any reported trading edge, check these points:
- Horizon: is the test long enough? The FINSABER authors, citing Bailey et al., recommend at least three years of data for daily strategies, and 10 to 20 years or more for weekly or monthly ones.
- Universe: does it include delisted and failed stocks, or only today’s winners such as TSLA and AMZN?
- Training overlap: does any backtest period fall inside the model’s training data?
- Validation: are training, validation and test periods split contiguously in time, with embargo gaps considered to prevent leakage?
- Multiple testing: has the Sharpe ratio been deflated for the number of strategies tried?
- Execution: are market impact, partial fills and spreads modelled, or only a flat cost per trade?
- Metric: is performance reported with risk-adjusted metrics rather than as headline return?
What goes wrong when you rely on AI for financial decisions
The first risk is wrong answers that look right. On FinanceBench, which asks factual questions about public companies from their SEC filings, GPT-4-Turbo with a retrieval system answered incorrectly or refused to answer 81% of questions. Even the best configuration tested was correct only 79% of the time, which the authors judge too low to use with confidence in live industry settings. Many failures were coherent, well-argued answers with detailed calculations that were still wrong, which makes them harder to catch than a refusal. The trading study above reports numerical hallucination and degradation in sideways markets as persistent failure modes, and concludes that LLMs belong inside human-in-the-loop systems, not acting as independent decision-makers.
The second risk is not knowing what you have bought. In the BoE/FCA survey, 46% of firms reported only partial understanding of the AI technologies they use, largely because of third-party models, and a third of use cases are now third-party implementations, up from 17% in 2022.
The third risk is systemic. The same remarks warn that AI could promote herding and monocultures as actors get the same signal from a base model or data aggregator. The FSB (2024) names third-party concentration, market correlations, cyber risk, and model risk with data quality as vulnerabilities that stand out for their potential to raise systemic risk. The trading study above gives a concrete version: if one model systematically misreads a type of earnings release and thousands of systems act on it together, the market impact could feed itself.
The last risk is plain overclaiming. SEC orders against two investment advisers, Delphia and Global Predictions, found misleading statements about AI capabilities they did not have, including Delphia’s claim to predict “which companies and trends are about to make it big”. Our position is that most AI R&D projects fail, AI projects more often than most, and anyone commissioning financial AI deserves to hear that before the work starts, with the risk named and scoped instead of hidden.
What GPU and HPC compute fintech AI needs, and what drives the cost
Compute cost depends on which of two very different workloads you run. The first is classical quantitative HPC: risk calculations such as CVA and Greeks. Vendors report large gains on GPUs, from fewer nodes and faster runs. Dell and NVIDIA report that dense eight-GPU servers cut the node count for CVA-style calculations from 100 to 4 (a result separate from their STAC-A2 audit), and that the eight-GPU system ran 2.5 times faster than a four-GPU configuration on the large Greeks benchmark. J.P. Morgan moved more than half its equity-derivatives risk computation to hybrid GPU/CPU systems on NVIDIA Tesla M2070 GPUs, with a 40x speedup and over 80% savings. Both are vendor-reported. They show the size of the gain reported on parallel risk maths, not what your code will get. Our write-up on GPU acceleration for quantitative finance workloads covers how that porting is done.
The second workload is model-driven: LLM inference over news, filings and customer conversations. Here cost is counted in tokens and latency, and it shapes design choices. In the trading study, Gemini 1.5 Pro had lower signal accuracy but cost $0.007 per 1,000 tokens with 3.1 seconds of latency, which the authors present as the better fit for high-volume use. The same authors note that the parameter-efficient fine-tuned FinGPT gave competitive or superior performance on structured financial tasks at a fraction of the parameter count and operating cost of larger models. FINSABER found large-scale LLM backtesting expensive and argues that API cost should be built into risk-adjusted metrics such as the Sharpe ratio. Training is harder still: retraining an LLM on a rolling window to get clean out-of-sample backtests is described as computationally infeasible for the foreseeable future.
Decide first whether the task is a realistic AI problem at all, measure the cost per decision with a smaller or fine-tuned model, and only then choose GPUs, because a trading signal that cannot survive its own inference bill has no edge to protect.
The question to settle before you build
The honest version of “can machines make you a millionaire” becomes a scoping question. Is the task you want to automate one where the evidence already shows AI performing, such as scoring transactions, rebalancing portfolios or summarising documents with a human checking the output? Or does it require a model to see what thousands of well-funded competitors have missed, in a market that changes once the edge is found? The first kind of project can be planned, measured and costed. The second is research with a high chance of a null result, and that is fine to attempt as long as everyone involved has priced it that way from the start.
