In early 2024, a finance officer at Arup—the British engineering firm behind the Sydney Opera House—joined what looked like a routine video call. The CFO walked through an urgent transfer. Colleagues nodded along in their tidy little squares. The request was unusual but plausible, the kind that shows up late in a deal when everyone just wants to finish cleanly.

About $25 million went out the door in a handful of transfers.

Every person on that call, except the finance officer, was synthetic. The faces were generated. The voices were cloned. The whole meeting was a movie, staged in real time.

That story gets told as a warning about AI. And ever since, the response across the industry has followed a predictable script: if criminals have AI, we need AI. Deepfake detectors. Voice analysis. Anomaly engines. An arms race, with vendors happy to sell weapons to both sides.

Some of those tools are useful. But before your organization commits its security budget to fighting AI with AI, it's worth asking a more uncomfortable question.

What actually failed at Arup?

AI fraud statistics tell a different story than the headlines

Last year, for the first time in its 26-year history, the FBI's Internet Crime Complaint Center added "AI" as a crime descriptor. The 2025 numbers: roughly $900 million in AI-attributed losses, out of $20.9 billion in total reported cybercrime losses.

Read that again. Even in the year deepfakes dominated the headlines, AI was formally identified in about 4% of reported losses. The other 96% was largely the same fraud that has worked for decades—fraudulent instructions, compromised email threads, invoice swaps, a trusted voice asking for something urgent.

Verizon's Data Breach Investigations Report has tracked this for years, and the figure barely moves: roughly 60% of confirmed breaches involve a human element. A click. A call. An instruction that looked right. Not malware defeating a firewall—a person being persuaded.

The threat isn't that machines got smarter. It's that the oldest attack in finance—convincing a human to move money—just got cheaper to run at scale.

That's the part the arms-race framing misses.

How AI is changing payment fraud (and how it isn't)

Here's the thing about the Arup case: nothing about the structure of that fraud was new. An authority figure made an urgent request. The timing created pressure. The target trusted what they recognized. Criminals have run that play over the phone, over email, and over fax machines for as long as money has moved.

What changed is the cost of performing familiarity. A convincing voice used to require a skilled impersonator. A convincing face was impossible. Now a few minutes of source material and ordinary tools are enough.

AI didn't create new vulnerabilities in transactions. It accelerated the ones that were already there. It took gaps that existed quietly—reliance on recognition, compressed decision-making, verification that collapses under deadline pressure—and made them show up faster and more convincingly.

That distinction matters more than it might seem. Because if the problem hasn't changed shape, the solution doesn't need to either.

Why AI fraud detection tools can't save you

Think about what a deepfake detector actually does. It tries to determine whether a face or voice is real. It's fighting perception with perception—and it's guessing, probabilistically, against an adversary that improves every month.

Now consider this: a perfect deepfake detector would have done nothing to stop most of the wire fraud that happened last year. The fraudulent instruction that drains a bond closing usually doesn't arrive by synthetic video. It arrives from a real, compromised email account, in a real thread, with formatting and tone that are genuine because the attacker has been reading the conversation for weeks. There's nothing synthetic to detect.

Detection tools answer the question "is this person who they appear to be?" But that was always the wrong question. The right question is "has this transaction been verified, regardless of who appears to be asking?"

When you can no longer trust your eyes and ears, you'd better be able to trust your process.

The human element in fraud, reconsidered

It's tempting to read the 60% figure and conclude that people are the weak link. That's the wrong lesson, and organizations that act on it—more training, more warnings, more blame—keep getting the same results.

The finance officer at Arup wasn't careless. They did what their system asked of them: they recognized their colleagues, took instructions from authority, and executed under deadline. The failure wasn't human. The failure was a system that placed the entire weight of a $25 million decision on one person's ability to spot a fake in real time.

No amount of vigilance fixes that, and no detection tool does either. What fixes it is structure:

  • Classification that treats high-risk transactions differently by design—the amount, the irreversibility, and the context set the controls, not who's asking. That removes discretion at exactly the moment pressure is highest.
  • Staging that spreads verification across time. Synthetic performances work best in short bursts; staged systems demand consistency over days, across channels. A deepfake can survive one meeting. It struggles to survive a process.
  • Separation of duties so initiation, approval, and execution live with different people. An attacker now has to impersonate several individuals, in several ways, without contradiction. Possible in theory. Expensive in practice.
  • Account validation that asks what's been true over time—how long the account has existed, whether the change fits the counterparty's history—instead of what's being said on a call.

None of this depends on outguessing the next generation of AI. That's the point. Recognition-based security gets weaker every time the technology improves. Evidence-based security doesn't care how good the fake is, because it never asked the fake a question in the first place.

Where AI actually belongs in fraud prevention

None of this means AI has no place on the right side of the ledger. Used well, it's good at the work that exhausts people: reconciling versions, flagging anomalies, comparing instructions against history, preparing a transaction so thoroughly that funding becomes a mechanical step instead of a judgment call.

That's the realistic division of labor: AI prepares, humans approve, and structure decides. The tools assist the framework. They don't replace it. An organization that buys detection software but still lets a single convincing phone call redirect a wire hasn't reduced its risk. It has automated its false confidence.

The criminals who took $25 million from Arup understood something the finance team didn't yet: in an era of synthetic media, recognition is dead as a security control. The organizations that internalize that lesson won't win by deploying smarter AI than the attackers. They'll win by building transactions that prove themselves before they execute—so that it no longer matters how convincing the request sounds.

Threats come and go. Tools rise and fall. Techniques have a shelf life.

Structure lasts.

Frequently asked questions

How is AI being used in payment fraud?

Mostly to lower the cost of old techniques: cloned voices for fraudulent payment requests, deepfake video calls impersonating executives, and AI-written phishing that mimics the tone of real email threads. The underlying play—convincing a person to authorize or redirect a payment—hasn't changed.

Can AI fraud detection tools stop deepfake fraud?

They can help flag synthetic media, but they're probabilistic and they only address one channel. Most wire fraud arrives through legitimate, compromised accounts where there's nothing synthetic to detect. Verification processes that don't rely on recognition—staged approvals, separation of duties, account validation—stop both.

What percentage of fraud involves the human element?

Roughly 60% of confirmed breaches involve a human action, according to Verizon's Data Breach Investigations Report—a figure that has held steady for years. By contrast, the FBI attributed about $900 million of $20.9 billion in 2025 cybercrime losses specifically to AI.