One question per study, frozen when it is answered.
Each study has a permanent page. Once its findings are final, the study is frozen: its evidence is pinned by SHA-256 at a git tag, its write-ups stop changing, and a check fails if any pinned artifact moves. A number later found wrong is corrected in the repository’s errata against the frozen artifact. New experiments go into the next study.
The rule exists because the alternative — a repository whose numbers quietly drift as the interesting result changes — is exactly what the first claim audit found 29 instances of.
- Study001
Frozen 13 Sept 2026 · evidence at study-001
Better memory made it worse at saying “I don’t know”.
Post-training gemma-4-E2B-it with an external memory system: LoRA SFT, DPO and on-policy distillation, and the abstention regression that arrived with better recall
Hybrid retrieval moved answerable recall from 0.16% to 15.13% and cut the unanswerable-abstention rate from 13.75% to 8.75%: the model started answering more and abstaining less. Supervising abstention explicitly restored 70.0% abstention and drove false-abstention on answerable probes to 32.1%. All 688 probes, 76 hash-pinned artifacts, 29 audit findings, and the numbers each one came from.
- Probes
- 688
- Pinned artifacts
- 76
- Audit findings
- 29
- Seeds
- 1
- Study002
In progress · no results yet
The crossover, multi-seed replication, and what Study 001 never measured
2B + scaffold against an 8B with no memory, ≥3 seeds per headline number, PMB scoring for the quantized and DPO checkpoints
Everything after the Study 001 freeze. No results yet: each candidate is pre-registered before it runs, and every one is reported against the frozen Study 001 checkpoints rather than in place of them.
Study 001’s manifest pins 76 artifacts across 7 groups — the benchmark, every training corpus, the evaluation metrics and the raw per-probe responses they were recomputed from, the run configs, the results write-ups and the public surfaces — at the commit recorded as 4d9e5a0, with hashes taken over LF-normalised bytes so they verify on a fresh clone. Six living documents are recorded for provenance and deliberately excluded from the check, because the errata is expected to keep growing.
The frozen results themselves: the best full-PMB system, E-distill, reaches 18.59% answerable recall at 71.25% abstention — measured once, on one seed.