Agents that argue their way to a defensible answer
A multi-agent investigation practice built on competing personas, adversarial debate, and an LLM judge. It turns ad-hoc fraud questions into auditable runs, and once caught an agent fabricating a reconciliation.
Business Analyst, AWS Payments & Fraud Prevention
Turned ad-hoc fraud questions into dozens of auditable, leadership-ready reports backed by 800+ scripts and 900+ outputs, reconciled to the cent, and the adversarial loop caught an LLM fabricating a reconciliation before it shipped.
LLM agents are fast and confident, which in fraud work is dangerous: a fluent, wrong reconciliation that lands in a stand-up is worse than no answer. I wanted the speed without trusting any single model's say-so.
I built a methodology, not just prompts. An 'arena' runs competing personas at a question and has a judge synthesize consensus, disagreements, and the best-supported answer. A 'debate' mode has two agents argue across rounds to a converged verdict, and an 'investigation' mode runs params-driven ring deep-dives. Every run is a self-contained folder with numbered scripts, raw outputs, and a graded report, so any number traces back to the query that produced it. Headline figures get reconciled to the cent across independent reproductions, and a second agent re-checks the first.
A near-daily cadence over a month-plus produced dozens of leadership-ready reports backed by 800+ query scripts and 900+ outputs, the majority reconciled to the cent. The adversarial structure earned its keep: in one debate, an agent invented an impossible reconciliation and a wrong headline. The loop caught it and drove the agents to a corrected, converged answer instead of shipping the fabrication.
A single confident agent ships the fabricated reconciliation and its wrong headline straight into a stand-up, and the month's investigation throughput collapses back to one slow manual case at a time.
The failure mode of an LLM in fraud analysis isn’t being slow. It’s being fluently wrong in a stand-up. So I built the speed in but never let a single model’s confidence be the evidence.
The methodology has three shapes. An arena runs competing personas at the same question and has a judge synthesize where they agree, where they don’t, and which answer is best-supported. A debate puts two agents in an argument across rounds until they converge on a verdict. An investigation mode runs structured, params-driven ring deep-dives. The common thread is that every run is a self-contained folder of numbered scripts, raw CSVs, and a graded report, so any figure can be clicked back to the exact query behind it.
Discipline lives on top of that: headline numbers reconciled to the cent across independent reproductions, and a second agent re-checking the first before anything is called done. Over a month-plus this sustained a near-daily cadence, with dozens of leadership reports backed by 800+ scripts and 900+ outputs.
The clearest proof it was worth the overhead came in one debate, where an agent invented a reconciliation that couldn’t be true and attached a wrong headline to it. A single-agent pipeline would have shipped it. The adversarial loop caught the fabrication and drove both agents to a corrected, converged answer. That’s the whole reason the structure exists.