small-mind-companion · study 002 · in progress

The crossover, replication, and what Study 001 never measured.

Study 002 is everything after the Study 001 freeze. It has no results. Each experiment below is pre-registered before any GPU time is spent, and its results will be reported against the frozen Study 001 checkpoints rather than in place of them: a re-measurement that contradicts a frozen number is a Study 002 result, and the frozen number stays where it is.

NO RESULTS YET8 candidates0 experiments run0 numbers on this page
SCOPE

Eight lines of work, none of them run.

Ordered by how much of Study 001 each one unblocks. Numbers only where a Study 001 artifact already holds one — a pre-registration states what will be measured and against what threshold, not what the answer will be.

  1. 01
    The crossover experiment (H16/H17)

    run_crossover_baseline.py is written and unrun: 2B + scaffold against the 8B gemma-4-E4B-it with no memory. This is Study 001’s own thesis question — can a small model plus a scaffold substitute for scale — and the one experiment that answers it. H16 predicted a +15 to +35pp recall advantage; H17 predicted the larger model still wins by a wide margin on multi-step reasoning. Neither was tested. Needs a GPU.

    unblocks: Study 001’s thesis question
    Script ready · not run
  2. 02
    Multi-seed replication

    Every Study 001 number is a single seed. Finding 1 in particular — the 5.0pp UAR drop from adding memory, and the false-abstention cost that followed — is the kind of claim that needs at least three seeds and a paired bootstrap before it should be relied on. The harness currently computes bootstrap intervals only for pra_strict and UAR, so pra_lenient needs one added first.

    unblocks: Every headline number
    Not started
  3. 03
    Populate acceptable_alternatives, or drop the field

    The field is empty in 688/688 probes, which is exactly why pra_strict is 0.0% throughout and pra_lenient is the reported metric (errata E22, guardrail G12). Populating it makes pra_strict meaningful; dropping it means the metric is chosen deliberately rather than by omission.

    unblocks: pra_strict as a usable metric
    Not started
  4. 04
    Score the quantized models on PMB

    The quantization work was llama-bench plus a manual coherence check. No quality benchmark was ever run against a quantized checkpoint, and no quantization figure in the repository is backed by a committed artifact — *.gguf is gitignored and results/ holds no quantization output (errata E5/E6). That is why no quantization number appears anywhere on this site.

    unblocks: The deployment claim
    Not started · GPU and artifact policy needed
  5. 05
    Fix the DPO lineage

    No full-PMB measurement of the DPO checkpoint exists, and the preference pairs were generated before SFT v1’s final regeneration: 2,263/2,277 user turns match but 0/2,277 system texts do, because the retrieval contexts diverged and the cause was never root-caused (errata E1/E13). Regenerate the pairs and measure the stage properly.

    unblocks: Any DPO result at all
    Not started
  6. 06
    H22 abliteration and H24 emotional range

    Both harnesses were built and neither was run: 24 abliteration probes and 27 emotional-range probes across 9 registers. They are pre-registered and idle.

    unblocks: Two pre-registered hypotheses
    Probes built · not run
  7. 07
    Image-derived memory tiers

    RQ13’s second half. The base model is a vision-capable model — the processor loads as Gemma4Processor and the multimodal path is live at evaluation time — but nothing in the memory tier has ever come from an image. The benchmark has a vision category and the bake-off never exercised it.

    unblocks: The multimodal claim
    Not started
  8. 08
    Pin base_model_revision in the DPO and distillation configs

    Four of the six training configs still use "main" (dpo.yaml, dpo_v1_scale.yaml, dpo_v1_more_epochs.yaml, distill_v1.yaml), and distill_v1.yaml also uses "main" for its teacher, so runs launched from them are not revision-pinned and should not be described as reproducible (errata E4, guardrail G6). No re-run may be called reproducible before this lands.

    unblocks: The reproducibility claim
    Not started · blocks the others’ wording
WHAT STUDY 002 STARTS FROM
  • Study 001’s checkpoints, corpora and evaluations, frozen at the tag study-001. None of them is modified, relabelled or replaced, and no Study 002 number may be reported in place of a Study 001 one.
  • The best full-PMB system in Study 001 is E-distill at 18.59% pra_lenient and 71.25% UAR — one seed, one run, 688 probes. It is the baseline a Study 002 result has to beat, and the bar for calling anything an improvement is a paired comparison, not a difference of two point estimates.
  • The false-abstention cost was reduced, not removed: 32.1% of answerable probes still hedge, above the 9.9% the broken run started at. Any Study 002 fix has to move both numbers, and a null result is publishable.
  • The audit ledger still has 2 findings marked OPEN and 2 PARTIAL. Correcting a Study 001 claim is not Study 002 work — those go into reports/ERRATA.md against the frozen artifact.