Making probabilistic model forecasts tamper-evident (and why it changes evaluation)
Quantitative Model
A recurring problem in evaluating trading and forecasting models is that the record of what the model actually said — in what exact form, at what time — is editable after the outcome is known. The winning calls get screenshotted, the losing ones quietly disappear, and what's left looks like skill but is often survivorship bias with a clean UI.
Ordinary timestamps don't fix this: a database row can be updated, a log regenerated. So we've been experimenting with a simple discipline — before the event a forecast describes, we reduce the forecast file to a SHA-256 hash and anchor it via OpenTimestamps (Bitcoin). The content never leaves our machine; what's published is a compact proof that the exact forecast existed before that block was mined. Backdating would require re-mining the chain.
Two effects on model evaluation: (1) it forces pre-registration — you can't revise the wording or drop the misses, so the track record becomes an adversarial witness to your own model; (2) it makes proper scoring rules meaningful — once a probabilistic forecast is locked, Brier score / log loss / CRPS against the realized outcome can't be gamed by hindsight selection.
We run this publicly across crypto, sports, and prediction markets (Polymarket/Kalshi), wins and losses both on the board, partly as a forcing function on ourselves. Curious how others here handle forecast provenance and out-of-sample honesty in live algo models — do you pre-register signals, or rely on locked backtest configs + walk-forward?
— Alex Malinowski, NeuPortal (neuportal.ai). Educational, not financial advice.
