Backtest 63.7%, production AUC 0.51: a postmortem

What we built

PropLab scores NBA and WNBA player props through seven independent blocks: player form, matchup, game context, market line, teammate synergy, analysis quality and external signals. Each returns a 0-100 score with a confidence index, and a confidence-weighted mean produces the final score that decides whether a pick is published.

On top of that engine we fitted a logistic meta-learner. It takes the seven block scores plus the pick direction and outputs a single calibrated probability that the pick hits. The idea was sound: a weighted mean treats every block as positively predictive, while signed coefficients let an anti-predictive block subtract instead.

Offline it looked excellent. The top tier came out at a 63.7% hit rate with an AUC of 0.56. We deployed it in shadow mode, writing its probability next to every saved pick without letting it influence anything, and planned the switch for the following release.

What went wrong

The switch never happened, because the backtest never reproduced.

Measured against clean live-only data, the deployed model scored an AUC of about 0.51. That is a coin flip. Worse than the flat result, the ordering was inverted: the band the model was most confident about hit 53%, while the band immediately below it hit 67.6%. A model whose top tier underperforms its second tier is not a weak model, it is a broken one.

The obvious suspect is the serving path, so we checked it first. We recomputed the probability for saved picks using the production code and compared it against the training-time computation. The difference was 2e-16, which is floating point noise. The code was correct.

The actual cause: a CSV nobody documented

The training set had been exported once, into a file, from a database snapshot. The export query was never recorded. Neither was the date window, which turned out to be a narrow slice of spring. Nobody wrote down how the file had been produced, so nobody could check whether it matched what production actually served.

It did not match. The training population and the serving population were different, which is the textbook train-serve mismatch. What made it survive review for weeks was not the mismatch itself but its invisibility: with no provenance recorded, there was nothing to compare against, and the impressive offline numbers had no way of being challenged.

The lesson is uncomfortable and cheap to state. The failure was not in the model, the features or the serving code. It was a file whose origin nobody wrote down.

What we changed

First, the model is parked. It is still computed and stored next to every pick, and it still drives nothing user-facing.

Second, provenance is now mandatory. Any dataset that informs a decision has to carry its exact query, its date window and its reason in the changelog, in the same commit that uses it. If it is not written down, it did not happen.

Third, the replacement candidate is a gradient boosting combiner, and it is only allowed to graduate on live out-of-sample picks it has never influenced. It runs as a frozen forward shadow, writing predictions to its own table, and it will replace the current engine only if its volume-matched top tier stays above the live engine over several hundred forward-settled picks. No backtest can promote it.

Fourth, calibration is monitored daily. Brier score, log loss, AUC and per-tier hit rates are computed over a trailing window, with the window floored at the model finalize date so training data cannot leak into the metric that is supposed to police it.

If you build these things

A public record that shows only the good parts is marketing, not evidence. We publish the misses next to the hits for the same reason we published this postmortem: a claim nobody can falsify is worth nothing.

If you are fitting a meta-learner on top of your own scoring stack, the cheapest insurance is not a better model. It is writing down, in the commit that uses it, exactly which rows your training file contains and how you got them.