Coinbase Finds Newer AI Models Fall Behind Predecessors In Fraud Detection Benchmark

Coinbase has printed findings displaying that the most recent technology of frontier AI fashions performs worse than their predecessors at detecting fraudulent transactions, prompting the trade to maneuver its risk-control operations towards open-weight fashions with domain-specific post-training.

In the second installment of its “Owning Intelligence” collection, launched on October 7, Coinbase described evaluating six giant language fashions on a hard and fast benchmark of 16,140 transactions spanning 7,293 customers. The dataset was drawn from 9 weeks of manufacturing information previous to the deployment of the corporate’s LLM-powered danger agent, making certain that the agent’s personal interventions wouldn’t bias observable fraud outcomes. The cohort contained 813 confirmed fraudulent transactions.

The analysis in contrast Opus 4.5 in opposition to Opus 5, Sonnet 4.6 in opposition to Sonnet 5, and GPT-5.4 in opposition to GPT-5.6. Across all three pairs, the newer variations recorded decrease recall, F1 scores, and dollar-weighted recall — the share of whole fraudulent transaction worth efficiently recognized, a metric Coinbase considers significantly necessary for cost danger.

The magnitude of regression different significantly. Sonnet 5’s recall fell by 22.2 share factors relative to Sonnet 4.6, with dollar-weighted recall dropping 22.9 factors; each precision and recall declined, indicating a weaker total means to differentiate fraud from respectable exercise relatively than a easy trade-off between the 2. Opus 5 confirmed a extra modest recall decline of 0.8 factors. GPT-5.6 adopted a special sample: precision improved by 11.5 factors, however recall fell by 20.7 factors and dollar-weighted recall by 21.8, in step with extra conservative classification conduct.

Implications for Model Selection and Coinbase’s Response

Coinbase cautioned that its benchmark demonstrates a regression with out establishing its trigger, noting that with out entry to suppliers’ coaching information or goals, explanations stay speculative. Plausible elements embrace shifts in resolution boundaries and immediate conduct, trade-offs launched by post-training processes, and modifications in training-data composition and process priorities.

The findings reshape how the corporate approaches mannequin upgrades. Rather than asking which launch is latest, Coinbase argues that groups ought to consider whether or not a candidate improves the general trade-off between precision and recall, whether or not noticed variations stem from the mannequin itself or from modifications to surrounding prompts and resolution insurance policies, and whether or not the mannequin meets latency, reliability, and value necessities for stay cost flows.

The outcomes motivated a strategic shift towards mannequin possession. Coinbase has moved the risk-control agent for its fiat deposit service, Onramp, towards open-weight fashions and domain-specific post-training. According to the collection’ third installment, the corporate post-trained Qwen3.5-9B utilizing SkyRL on Anyscale, producing a mannequin that outperformed Opus 4.5 on all 4 fraud-detection metrics whereas decreasing end-to-end serving latency by 55%.

The transfer builds on earlier outcomes: in Part 1 of the collection, Coinbase reported that layering an LLM evaluate agent onto its current machine-learning mannequin and guidelines engine — deployed with out changing the prevailing danger stack — lowered fraudulent transactions by 30% and fraud worth by 22% in a web-based A/B experiment.

The submit Coinbase Finds Newer AI Models Fall Behind Predecessors In Fraud Detection Benchmark appeared first on Metaverse Post.