Better memory made it worse
at saying “I don’t know”.

The ladder below is the whole study in one chart. A is the raw model, B adds SFT with no memory, D adds memory with no SFT, E adds both, and E-distill adds on-policy distillation. Each rung isolates one change — which is exactly what the audit found missing from the first write-ups of the distillation result (errata E2).

Two of the three points in that trajectory are numbers in a document, not in a JSON artifact. The runs behind them had their metrics.json files overwritten by the final rebalanced run, so the committed record of the first two is docs/proper_scale_results.md itself. This page parses that table rather than retyping it, and says where each figure lives (guardrail G7). The honest description of the final state is: 32.1% false abstention is a real cost, above where the broken run started at 9.9%, and a genuinely optimal ratio was never searched for.

1. BUG 01
   A dedup step was collapsing 227 intended examples to 1.

   The data generator deduped by exact assistant-response text. Abstention and irrelevant-retrieval examples use a fixed template response by design, so after the first one every subsequent example looked like a duplicate and was dropped — in both SFT v0 and SFT v1. The datasheets recorded the casualty honestly: "abstention": 1, "irrelevant_retrieval": 1. Fixed by deduping only the memory-relevant category by response text.

   model_quirks #16
2. BUG 02
   The eval harness could not recognise the right answer.

   After the dedup fix, the model began producing the exact abstention phrase it had been trained on — and the rule-based detector’s phrase list did not contain it, so correct abstention scored as non-abstention. UAR appeared to fall to 2.5%. Rescoring the saved responses with the phrase list fixed gave 96.25%.

   model_quirks #17
3. NOT A BUG
   Over-correction, which is the actual finding.

   With the intended proportion of templated abstention and irrelevant-retrieval examples restored (~17% of the corpus), the model over-generalises the canned hedge as a low-risk default — 69.2% of answerable probes. The project’s own explanation is that short fixed-string targets are easy wins for the loss function next to generating a correct, specific, longer answer. Reducing the ratios and adding four paraphrases per template recovered most of it; it did not remove the cost.

   the finding

The lesson the project wrote down as a guardrail. A rule-based grader silently drifts out of sync with the training-data phrasing it is meant to detect, and a dedup step applied to templated data is a silent dataset-size bug. Both were caught only because a number looked wrong in a specific direction — which is why the project keeps a “suspiciously good or bad is a bug signal” discipline.

E29 is the one finding the audit did not reach by reading. It came from verifying the freeze against a clean checkout of the commit that introduced the files — and what it found was that four of the six integrity pins the repository published had never verified under any algorithm the auditor tried (the ledger’s own wording is quoted in the card above). The cause is still undetermined and is recorded as such; the fix was to recompute all 9 corpora with a documented algorithm, create the one pin that was missing, and add a regression test so a hash cannot rot silently again. The page counts the 9 pins at the tag — 4 benchmark and 5 training — rather than quoting a number, because a typed count is what went stale in the first place.

The rest of the record

The discipline, not the checkpoint, is the reusable artifact. The corrections above were not applied silently: the findings ledger records each one, the errata records what the artifacts show instead, and the guardrails file turns each mistake into a rule for the next pass. findings.json ↗ resolutions.json ↗

The 13 rules that came out of it

Read out of docs/GUARDRAILS.md by the build script, so the list cannot drift from the file. G9b is here because the E29 correction above rewrote G9 and split the partial-verification failure out of it.

1. G1
   A result's label comes from the script that produced it, never from the write-up

   Findings #1 and #2 — the highest-severity items in the audit — both came from trusting prose over code. The 70.0% UAR was credited to the DPO checkpoint for months because a table said "SFT+DPO+memory", while `run_system_e_v1.py:35` loaded `outputs/sft/v1/merged` and named the system `E_sft_memory_v1`. The one-line check that would have caught it is reading `HFEngine(...)` in the generating script. *Check:* for every number in a write-up, name the script and the checkpoint path that produced it. If the artifact filename and the prose label disagree, the filename wins.

2. G2
   A two-stage comparison must isolate one stage

   Finding #2: distillation was credited with a +3.3pp gain measured between `outputs/sft/v1/merged` and `outputs/distill/v1/merged` — but the distilled model was trained *from the DPO output*, so the delta included DPO. The isolating measurement was the pairwise run that held DPO constant on both sides. *Check:* for any "X improved Y by Z" claim, list every stage that differs between the two systems. If more than one differs, report it as a combined effect or run the isolating comparison.

3. G3
   Absence of a measurement is stated as absence

   There is no full-PMB `pra_lenient`/UAR measurement for any DPO checkpoint. A blank cell in a results table reads as "not yet run", but a borrowed number reads as a result. *Check:* an em dash for a metric that was never computed, plus one sentence saying so, and never a number carried over from a neighbouring row.

4. G4
   Counts are generated, not typed

   Every stale count in this audit (#3, #14, #17, #21, #27) was a hand-written number that drifted: 40 personas vs 8, 451 tests vs 473, 21 bugs vs 27, "15k lines" vs 66,383, two GGUF size sets in one file. *Check:* where a count can be derived from an artifact, generate it or say when it was last verified. Datasheets are generated — fix the generator, not the generated file (finding #19 was caused by a hardcoded title in `scripts/build_pmb.py`).

5. G5
   "Done" and "not done" live in one place

   Finding #8: `docs/research_questions.md` still described imatrix quantization as an undone next step while `docs/quantization_results.md` documented it as completed. Finding #25: three files called `dpo-v1-scale` current-best while four others named `distill-v1`. *Check:* a project-state claim appears in exactly one document and is linked from anywhere else. When state changes, the stale claims are found by grepping the old state's name.

6. G6
   Configuration claims are verified per file, not per pattern

   Finding #4: "base model revision is pinned by commit SHA in each config" was true for 2 of 6 configs; the other four used `"main"`. *Check:* a claim about "all configs" is backed by a command that reads all of them.

7. G7
   A number with no committed artifact is labelled as reported, not measured

   Finding #6: every quantization figure — throughput, perplexity, file sizes, parameter count — is restated from run logs that are not in the repository, and `*.gguf` is gitignored so nothing can be recomputed. *Check:* the docs carry an evidence note naming what is not reproducible from a fresh clone.

8. G8
   A referenced artifact must exist

   Findings #16 and #26: `make figures` was part of the stated reproducibility chain and `scripts/make_figures.py` never existed; a results file was referred to as "if it exists". A command in a reproducibility chain that cannot run invalidates the claim. *Check:* every path and command in a doc is executed or resolved during the same pass that writes it.

9. G9
   A hash is recomputed from a clean checkout before it is trusted

   This rule was written wrong the first time. It originally said "all six `hash.txt` files verify, once CRLF is normalised to LF". Four of them do — the SFT/DPO datasets — and **four do not, under any algorithm**, which was only established by testing them against a clean checkout of the commit that introduced them (finding #29). The original claim came from a partial check that was generalised to all six. *Check:* recompute every hash, from a clean checkout, before writing that they verify. Record the algorithm and the line-ending assumption next to the hash; this repo had two undocumented and incompatible algorithms in use. Where a hash cannot be reproduced, name the artifact that supersedes it rather than leaving a broken pin in place, and remove the claim that it is a pin.

10. G9b
    A partial verification is not a verification

    The specific failure above: four hashes were confirmed and the sentence was written about six. Nothing about the four that passed said anything about the other two. *Check:* a claim about "all N" is backed by a check that covered N, and the check's coverage is stated.

11. G10
    Published cards are corrected by re-push, and errata in the meantime

    Six model cards were already on the Hub carrying the misattributed figure. The repository cannot fix a published card by committing. *Check:* after correcting a card source, re-push; if a number is already public and wrong, add it to `reports/ERRATA.md` so the repository's own record supersedes it.

12. G11
    Contradictory status text is worse than no status text

    Finding #11: the SFT datasheets said contamination was unchecked, the distill datasheet said it was checked clean, and it cited the SFT datasheet as its source. The data was clean the whole time; only the documentation was wrong. *Check:* a claim that cites another file must match that file. Where two files state the same fact, generate one from the other or state it once.

13. G12
    A field documented as populated must be populated

    Finding #22: the README said probes ship with "acceptable alternatives"; the field is empty in 688/688, which is precisely why `pra_strict` is ~0 and `pra_lenient` is the reported metric. *Check:* every documented schema field is either populated or documented as unpopulated with the consequence spelled out.

The living documents are excluded on purpose. The errata grows as mistakes are found and the guardrails gain rules; hashing them into the enforced set would mean every future correction “breaks the freeze”, which is backwards. The hash is computed over LF-normalised bytes so it verifies on a fresh clone — the naive sha256sum fails on all of them because the working tree is CRLF, which is the mistake guardrail G9 exists to prevent.

The commit is resolved with git, not read from the manifest. A generated field is not evidence of the state it names: the manifest’s own git_commit is stale here, and taking it at face value would have put the wrong commit on this page. Both values are shown.

The freeze was re-taken, and each re-take is logged

The freeze was re-taken. Each entry records what moved and why it did not change a Study 001 finding. An unrecorded re-freeze makes the manifest meaningless, so this list is part of the artifact.