The crossover, replication, and what Study 001 never measured.

small-mind-companion · study 002 · in progress

Study 002 is everything after the Study 001 freeze. It has no results. Each experiment below is pre-registered before any GPU time is spent, and its results will be reported against the frozen Study 001 checkpoints rather than in place of them: a re-measurement that contradicts a frozen number is a Study 002 result, and the frozen number stays where it is.

No results yet14 candidates0 experiments run0 numbers on this page

Eight lines of work, none of them run.

Ordered by how much of Study 001 each one unblocks. Numbers only where a Study 001 artifact already holds one — a pre-registration states what will be measured and against what threshold, not what the answer will be.

Needs a GPU · 8 items
  1. 01
    The crossover experiment (H16/H17)

    H16 predicted a +15 to +35pp recall advantage; H17 predicted the larger model still wins by a wide margin on multi-step reasoning. Neither was tested.

    `run_crossover_baseline.py` is written and unrun: 2B + scaffold vs the 8B `gemma-4-E4B-it` with no memory. This is Study 001's own thesis question and the one experiment that answers it.

    unblocks: Study 001’s thesis question
    Script ready · not run
  2. 02
    Multi-seed replication

    The harness computes bootstrap intervals only for pra_strict and UAR, so pra_lenient needs one added first. The repo already has bootstrap_ci, paired_bootstrap_diff, holm_bonferroni and minimum_detectable_effect.

    Every Study 001 number is a single seed and a single run. The retrieval finding in particular — recall up 0.16% → 15.13%, abstention *down* 13.75% → 8.75% — is the kind of claim that needs ≥3 seeds and a paired bootstrap before it should be relied on. The repo already has `bootstrap_ci`, `paired_bootstrap_diff`, `holm_bonferroni` and `minimum_detectable_effect`; nothing new is needed but compute.

    unblocks: Every headline number
    Not started
  3. 03
    A full-PMB measurement of the DPO stage

    (E1). No `pra_lenient` or UAR figure exists for any DPO checkpoint; `dpo-v1-scale` appears only in pairwise comparisons.

    unblocks: Any DPO result at all
    Not started
  4. 04
    Fix the DPO lineage before measuring it

    (E13). The preference pairs were generated *before* SFT v1's dedup fix and ratio rebalance: 2263/2277 user turns match the current `sft/v1/train.jsonl` but **0/2277 system texts** do, so the retrieval context differs from the corpus the SFT stage trained on. Regenerate the pairs, then measure the stage. The cause of the divergence was never pinned down and should be.

    unblocks: Any DPO result at all
    Not started
  5. 05
    H22 abliteration

    (24 probes built, unrun) and **H24 emotional range** (27 probes across 9 registers, unrun).

    unblocks: Two pre-registered hypotheses
    Probes built · not run
  6. 06
    Image-derived memory tiers

    (RQ13, second half) — the base model is a VLM, but nothing in the memory tier comes from an image.

    unblocks: The multimodal claim
    Not started
  7. 07
    Score the quantized models on PMB

    (E6). The quantization work was `llama-bench` plus a coherence check; no quality benchmark was ever run against a quantized checkpoint, so "Q3_K_S is the smallest verified-coherent level" rests on manual generation inspection.

    unblocks: The deployment claim
    Not started · GPU and artifact policy needed
  8. 08
    Resolve the two GGUF size sets

    (E5). `README.md` and the cards' files tables say 8.64 / 4.61 / 3.18 GiB; `hf_readmes/gguf_README.md`'s own evaluation table says 8.62 / 4.59 / 3.17 GiB. No artifact backs either. Re-run `llama-bench` against the GGUF files, commit the output, and correct whichever set is wrong.

    unblocks: The quantization numbers
    Not started
No GPU needed · 6 items
  1. 09
    Commit the quantization evidence

    (E6). No `llama-bench`, `llama-imatrix` or `llama-quantize` output is committed (`*.gguf` is gitignored and `results/` holds no quantization directory), so every size, throughput and perplexity figure is restated from run logs a reader cannot check. Even without re-running the benchmarks, a `results/quantization/` directory holding the original stdout would close most of this.

    unblocks: Every quantization figure
    No GPU needed
  2. 10
    Populate `acceptable_alternatives`, or drop the field

    (E22). It is present on all 688 probes and empty in 688/688, which is why `pra_strict` is ~0 throughout and `pra_lenient` is the reported metric. Populating it makes `pra_strict` meaningful; dropping it removes the need to explain it away.

    unblocks: pra_strict as a usable metric
    No GPU needed
  3. 11
    Widen the k-sweep

    It covers 120 of 688 probes (15 per category, seed 1337) and is reported as an inverted U peaking at k=8. At n=120 the curve's shoulders are not distinguishable from noise; a full-688 run costs inference, not training.

    unblocks: The k=8 optimum
    No GPU needed
  4. 12
    Measure `mur`

    It is defined and implemented in the harness and is **0.0 in every saved run** — because it is computed but not called, not because the models score zero. Either wire it up or remove it from the reported metric set.

    unblocks: H5’s stated metric
    No GPU needed
  5. 13
    Pin `base_model_revision` in the DPO and distillation configs

    (E4). `distill_v1.yaml` (base and teacher), `dpo.yaml`, `dpo_v1_more_epochs.yaml` and `dpo_v1_scale.yaml` all reference `"main"`. Until they pin a SHA, no run launched from them can be called reproducible.

    unblocks: The reproducibility claim
    No GPU needed
  6. 14
    A small blinded human evaluation

    No human review exists anywhere: every datasheet says "not human-reviewed", and no reviewer log or teacher transcript was retained, so the `gpt-5.6-luna` teacher identity is self-reported and unverifiable from the repository. A few dozen probes scored by a human would bound how much the judge-based metric can be trusted.

    unblocks: Trust in the judge
    No GPU needed
What Study 002 starts from
  • Study 001’s checkpoints, corpora and evaluations, frozen at the tag study-001. None of them is modified, relabelled or replaced, and no Study 002 number may be reported in place of a Study 001 one.
  • The best full-PMB system in Study 001 is E-distill at 18.59% pra_lenient and 71.25% UAR — one seed, one run, 688 probes. It is the baseline a Study 002 result has to beat, and the bar for calling anything an improvement is a paired comparison, not a difference of two point estimates.
  • The false-abstention cost was reduced, not removed: 32.1% of answerable probes still hedge, above the 9.9% the broken run started at. Any Study 002 fix has to move both numbers, and a null result is publishable.
  • The audit ledger still has 2 findings marked OPEN and 2 PARTIAL. Correcting a Study 001 claim is not Study 002 work — those go into reports/ERRATA.md against the frozen artifact.