PRACTITIONER FIELD NOTE 02 · 4 MIN READ

Why Generative LLMs Cheat in SEC Filing Backtests (And Why Deterministic Parsers Win).

Passing 10-K and 8-K filings into an LLM produces extraordinary simulated Sharpe ratios right up until live capital is deployed. Here are the three structural defects that caused Starling Quantitative to ban generative LLMs from our signal pipeline.

AUTHOR: STARLING QUANTITATIVE RESEARCH PUBLISHED: NOVEMBER 16, 2026

Over the past three years, wiring frontier or open-weight LLMs to read corporate filings has become the default pitch across emerging quant startups. Early in our own research, we audited that exact workflow across management discussion text, Form 10-K Item 1A risk factor diffs, and Form 8-K event disclosures. On historical backtests, the numbers looked effortless. Once we audited where the simulated edge was actually coming from, we removed every generative LLM from the stack.

1. Pre-Training Cutoff Leakage Cannot Be Redacted Away

Any LLM trained between 2024 and 2026 has already ingested news articles, post-mortems, and Wikipedia summaries covering every corporate bankruptcy, fraud, and 10x winner from 2019 through 2025. When you feed a 2022 Form 10-K from Silicon Valley Bank or Bed Bath & Beyond into that model and ask for a distress or sentiment score, the weights already know how the story ended.

A common defense in published "LLM alpha" papers is entity masking: stripping out the ticker symbol, company name, and executive names before prompting the model. In practice, entity masking fails completely on financial filings. If an anonymized 2022 filing mentions a $91 billion held-to-maturity agency mortgage portfolio alongside venture-backed technology deposit concentration in Santa Clara, the model resolves the issuer to Silicon Valley Bank instantly from context. You cannot run an honest 2019 to 2025 walk-forward backtest using a model whose weights memorized 2019 to 2025 outcomes.

2. Temperature Zero Is Still Non-Deterministic

Institutional factor attribution requires strict functional reproducibility: given identical filing bytes, f(x) today must equal f(x) six months from now to twelve decimal places. Even with temperature = 0, sparse Mixture-of-Experts routing, GPU batch-size floating-point non-associativity, and silent vendor API checkpoint updates cause LLM output probabilities to drift between runs.

When you are running 5-fold Purged Walk-Forward Cross-Validation across 3,607 Russell 3000 CIKs and debugging why an Information Coefficient shifted by 40 basis points after a risk-model change, having a black box in the feature layer whose historical outputs drift quietly makes rigorous attribution impossible.

Audit Dimension Generative LLM Text Scoring Starling Deterministic Parsers
Historical Lookahead Contaminated by pre-training knowledge of post-filing bankruptcies and winners, even with ticker masking. Zero learned world-knowledge weights; strictly causal against down-to-the-second SEC acceptanceDateTime.
Replay Reproducibility Drifts across GPU batch sizes and API model checkpoints even at temperature = 0. Bitwise IEEE-754 identical across every historical replay and live intraday daemon sweep.
Vulnerability to IR Spin High; corporate counsel actively sanitizes MD&A prose tone during deteriorating quarters. Immune to prose tone; parses mandatory XBRL tags, statutory item codes, and insider share settlements.

3. The Alpha Lives in Unspinnable Mechanics, Not Prose

Once we demeaned forward stock returns against point-in-time GICS Sector ETFs and regressed out five standard OHLCV style factors (Market, Size, 12-1 month Momentum, 5-day Short-Term Reversal, and 14-day ATR Volatility), pure prose sentiment carried virtually zero residual edge. Public companies pay experienced securities lawyers and investor relations teams specifically to keep MD&A prose calm when fundamentals deteriorate.

What corporate counsel cannot rewrite are mandatory statutory mechanics and structured schedules: XBRL working-capital bloat vs. operating cash flow, non-routine C-suite cash Form 4 cluster purchases, accelerated share repurchase dealer-covering contracts, USPTO forward-citation velocity normalized by R&D spend, and hard statutory distress filings such as Form 8-K Item 4.02 (auditor non-reliance on prior financial statements) or Form NT (inability to file a 10-K or 10-Q on time).

Replacing raw LLM confidence scores (which produced an out-of-sample residual Rank IC of -0.0083) with deterministic parsers normalized via Out-of-Sample Blom rank quantiles and zero-anchored Median Absolute Deviation lifted 5-fold out-of-sample active-event residual Rank IC to +0.0901 across 794 admitted events, with strictly monotonic net-of-cost terciles (+24.27 bps < +121.35 bps < +208.39 bps).

OPEN-SOURCE TOOLS & WORKING PAPER

Inspect the Deterministic Calibration Math & PIT Clock

Read Research Note RN-02 (DOI: 10.5281/zenodo.24222814) for the complete Out-of-Sample Blom and MAD calibration equations, or install our MIT-licensed Python SEC timestamp router (pip install edgar-pit-clock):