Cutting a fraud pipeline from a coffee break to a glance
A refund-analysis Spark job that set the pace of every leakage question asked through it, cut from ~34 minutes to minutes without a bigger cluster.
Senior Product Analyst, Risk & Abuse, Zalando
Cut a core refund-analysis pipeline from ~34 minutes to minutes, mostly by reading only the needed columns and filtering on partitions before any data moved.
- Column pruning 100
- Early partition filters 68
- Broadcast joins 47
- Selective caching 29
- Delta materialization 21
Show the data table
| Category | Value (relative contribution) | Note |
|---|---|---|
| Column pruning | 100 | Reading only the needed columns. A handful were used while the whole binary path was being opened. |
| Early partition filters | 68 | Filtering before the read means pruned partitions are never opened, not opened and discarded. |
| Broadcast joins | 47 | A dimension table small enough to ship to every executor costs a copy instead of a shuffle. |
| Selective caching | 29 | Caching a DataFrame read once pays memory for nothing, and the shuffle-partition default was sized for a volume this job never sees. |
| Delta materialization | 21 | Only where the same raw Parquet was re-read often enough to amortize the write. |
A core refund and Salesforce-case analysis on Databricks ran around 34 minutes. At that length a pipeline stops being a tool and becomes a tax: you batch questions to amortize the wait instead of following the thread, which is backwards for fraud work.
- I treated the runtime as a data-layout problem, not a cluster-size one.
- I replaced the brute-force reads of whole binary warehouse paths with selective partitioned loads that pull only the columns the analysis uses.
- I pushed date and partition filters to the front so less data ever moves.
- I broadcast the small dimension tables, such as customer-extended, instead of shuffling them across the cluster.
- I cached only the DataFrames that are genuinely reused, tuned the shuffle-partition count to the real volume, and materialized hot raw Parquet into Delta where it paid for itself.
The same analysis returns in a fraction of the time, and the cadence of the work changed with it. Refund and leakage questions could be iterated inside one session instead of costing one expensive run per sitting.
Every refund question keeps costing a half-hour brute-force read from the data lake, and the iterative leakage investigations behind Remaining Fraud Damage stay far slower to produce.
A 34-minute pipeline is not a performance footnote. It sets the cadence of every question asked through it, and at half an hour a run, the question you cannot quite justify is the question you never ask.
The obvious request is a bigger cluster. Compute was never the problem, and the shape of the read said so before any tuning did. The job opened whole binary data-warehouse paths in order to use a handful of columns, so most of the work it did was work nobody had asked for. That ratio is what made column pruning the dominant lever rather than a guess: the cheapest honest version of this job still had to read the columns the analysis used, and everything above that floor was waste being paid for on every single run. The levers that followed push the same principle further down the job. Less data enters the pipeline at all, less of it crosses the network, and less memory goes to holding things that get read once.
A pipeline you schedule your day around is a different instrument from one you run mid-thought. Refund and leakage questions arrive in chains, where the answer to one suggests the next, and a chain only gets followed when asking is cheap.