Not this: Claude/Codex API fine-tune · Centaur replacement · production persona. Curriculum order (Rust → CLI → eval → SFT): Post-Training Path.
1 · Why — how I started loving local models
I already live in API agents — Claude, Codex, whatever is sharp this month. They're great at harness work: tools, long context, coding, “just get it done.”
What they can't give me is a clean answer to:
Did our washed facts change a model we own?
That itch is how Polar Lab started. Not “replace Claude.” More like: keep a small open model on the desk (Mac MPS / CPU), teach it with LoRA, and force an honest holdout score.
When the score sat at 0.200, the lab was telling the truth. When short-fact v4 and later packs hit 1.000, I could see why — data shape and adapter capacity, not vibes.
Loving local models, for me, means loving ownership + measurement. The romance is the scoreboard.
Why Polar exists in one line: prove owned data → owned weights → holdout score.
- API brains are rented. You can prompt them; you cannot LoRA “our facts” into their weights.
- Memory and registries can retrieve truth, but retrieval is not a weight update.
- Only a model you own can answer whether this washed JSONL changed this behavior.
Why early low scores were still good news: a flat 0.200 on small packs meant the pipeline and eval were honest. A fake high score from leakage would have taught the wrong lesson. Polar did its job when it refused to lie.
2 · Then a small video of the cloud loop
I don't only run this on the laptop. Cursor cloud agents pull the same repo, train on CPU, write reports, open PRs. That trail is the video storyboard:
- Wash JSONL (Machina / space)
./run_next.shorPOLAR_CONFIG=…- Holdout table: LoRA vs base
- Miss → boost golds → retrain
- PR merge · scoreboard at 1.000
That's enough for a short cloud clip: terminal → report → GitHub. The point of the video is the loop, not a chat UI skin. Cloud doesn't invent capacity. It records the same lab rules at scale while you're away.
3 · Where this model sits — vs the harness
| Layer | What it is | Uses local LoRA? |
|---|---|---|
| Harness (Centaur-style agent runtime) | Rented / API brain + tools + orchestration | No — Polar is not a Centaur replacement |
| Memory (files, Postgres, Redis, second brain) | Retrieve truth | No — retrieval ≠ weight update |
| Product (/tools registry, live URLs / status, contact) | Live site facts | No — registry wins for users |
| Polar Lab | Gated LoRA playground | Yes — only here by default |
So: the 0.5B + adapter is not plugged into the harness as the daily brain. Harness keeps using strong API models + memory. Polar is the side lab where we prove post-training hygiene before anyone talks about persona export.
If someone later wants a local model inside a harness, that is a separate product decision with an explicit export gate — not “smoke run = ship.”
4 · Scenarios — when each piece wins
| Scenario | Use | Why |
|---|---|---|
| Live tool URL / status / hub tags that change often | Product registry | Facts change in git tomorrow morning; LoRA would lag |
| Guest lists, emails, allowlist codes, API secrets | Memory / never train | Compliance + freshness; must not enter JSONL |
| Long second-brain trees, SKU tables, news feeds | Retrieve, don’t memorize | Too big / too mutable for exact-match LoRA |
| “Did these washed short facts stick in our weights?” | Polar holdout | Only base-vs-LoRA exact_match answers that |
| Coding, multi-tool ops, long diligence jobs | API harness | Capacity and tooling a 0.5B adapter doesn’t have |
| Offline / edge / own-weights literacy / merge gate | Local LoRA (Polar) | Cheap, owned, measurable |
| Overnight CPU train + report + PR while you’re away | Cloud agent | Same SPEC; good process video; not magic capacity |
Decision line: if a wrong answer is fixed by editing one source file, it belongs in product memory — not weights. If you need the number to prove a weight change, use Polar + holdout.
5 · Which layers need changing
| Layer | Touch it? | What to change |
|---|---|---|
| Product / registry | Yes (product work) | Live status / URLs / tags — edit registry / source files, not LoRA |
| Memory (files / DB / Redis) | Yes when wrong | Hotfix truth, PII, long docs; wrong answer → fix the source, don’t bump max_steps |
| Harness (Centaur / API agent) | Don’t wire Polar yet | Keep the API brain + tools; don’t make the 0.5B adapter the daily brain |
| Polar Lab (LoRA) | Lab only | Wash short golds; run holdout; keep adapter behind the gate |
| Cloud agent | No architecture change | Keep using it for train / report / video trail; same lab rules |
One-liner: wrong answer in production → fix registry / memory. Need proof that weights changed → change Polar data / gates. Harness does not take the adapter yet.
6 · What we adjusted — pack by pack
6.1 What failed early. On small Machina packs (36–79 rows), holdout exact_match stuck at 0.200 — same as base. Loss was falling; strings were not sticking.
| Failure mode | Example | Why it hurt |
|---|---|---|
| Paraphrase | gold path → “In the Aileena memory store.” | Meaning-ish, string fail |
| Ignored “reply with only” | short gold → long waffle | Format not pinned |
| Near-miss hallucination | paradigmxyz/centaur → wrong org URL | Plausible fake |
| Wrong numeral / concept | 90 → 100, sandbox pods → wrong story | Identifier not learned |
Root causes: too little data; harsh golds (paths/names); strong base priors toward fluent essays; train/eval paraphrases that tiny LoRA doesn't transfer; more steps alone didn't fix strings.
What was not broken: the Transformer, the eval script, Mac-vs-CPU as the main story, or “must have an H100 to learn the loop.”
6.2 Machina — short-fact v4.
| Before | Adjustment | After |
|---|---|---|
| 36–79 rows, LoRA 0.200 = base 0.200 | 450 train rows of identical short golds for paths/names; ~400 steps | LoRA 1.000 vs base 0.200 (CPU + Mac MPS) |
Progress: volume of identical short answers was the missing piece. Same holdout, same metric — behavior moved.
6.3 Space — miss-boost + full-attn LoRA.
| Before | Adjustment | After |
|---|---|---|
| Number misses (−132, 0.086, 102.3, 725…) | Wave-1 miss-boost → 1002 train / 10 eval | Intermediate 0.600 |
| Over-boost to 1338 rows | Hit-set flip / overfit | Discarded (0.400) |
| q,v-only LoRA on wave-1 data | Expand LoRA targets to q/k/v/o | 1.000 vs base 0.000 |
Progress: more of the right short golds helped; dumping more of the wrong kind hurt; adapter capacity (full attention projections) closed the rest.
6.4 What we deliberately did not do
- Dump Slack / whole
aileena_second_brain/**into JSONL - Treat smoke adapters as production persona
- Jump
0.5B→ larger base before holdout moved - Wire the adapter into Centaur/harness without an export gate
- Start DPO/RL before SFT exact-match was honest and moving
7 · Scoreboard — progress vs base
| Pack | Train/eval | Config | LoRA | Base | Δ | Useful ≥0.60? | Clear win ≥ base+0.20? |
|---|---|---|---|---|---|---|---|
| Machina | 450/10 | machina_sft.yaml | 1.000 | 0.200 | +0.800 | YES | YES |
| Space | 1002/10 | space_sft.yaml (q/k/v/o) | 1.000 | 0.000 | +1.000 | YES | YES |
Earlier checkpoints (for the story arc):
| Checkpoint | Score | Note |
|---|---|---|
| Arithmetic smoke | 0.667 | Short answers can stick |
| Machina v1–v3 | 0.125–0.200 | Honest flatline |
| Machina v4 | 1.000 | Short-gold volume unlock |
| Space mid | 0.600 → 0.400 → 1.000 | Miss-boost good; over-boost bad; q/k/v/o finish |
Both packs meet and clear the lab gates on CPU.
8 · Eval — is it up to standard?
Metric (utils/eval.py):
pred.strip().lower() == gold.strip().lower()
- Disjoint train vs holdout prompts (
scripts/check_data.py→ overlap = 0) - Always report base vs LoRA on the same holdout
temperature=0for scoring- Washed English short golds only in this repo (per SPEC)
- Never commit
outputs/adapters as the product brain
| Gate | Rule | Machina | Space |
|---|---|---|---|
| Beat baseline / noise | LoRA clearly above ~0.20 floor when base is weak | Pass | Pass |
| Useful | LoRA ≥ 0.60 | Pass (1.000) | Pass |
| Clear win | LoRA ≥ base + 0.20 | Pass (+0.80) | Pass (+1.00) |
| Hygiene | Train/eval disjoint; no holdout leakage | Pass | Pass |
| Honesty | Don’t edit eval golds mid-comparison with an old baseline | Pass (by process) | Pass |
Verdict: yes — for these washed short-fact packs, holdout exact_match is in standard and at the top of the bar.
What “in standard” does not mean: it does not mean the adapter should answer live product status/URLs for users (registry wins). It does not mean Polar replaces Centaur or Claude. It does not mean CI “proves 1.000” on every PR — CI is hygiene + dry-run; pack scores come from real train/eval runs. Exact-match is brittle on purpose: paraphrases count as zero so we don't congratulate ourselves with vibes.
9 · Where each piece wins
| Piece | Best at | Weak at |
|---|---|---|
| API harness (Claude etc.) | Reasoning, tools, coding, live ops | You don’t own weights; can’t LoRA “our facts” into it |
| Memory / registry | Fresh truth, PII boundaries, hotfix in git | Doesn’t change model defaults; needs retrieval every time |
| Local 0.5B + LoRA (Polar) | Cheap proof: data → behavior → exact_match; offline literacy | Tiny model; brittle strings; stale when facts move; not a harness brain |
| Cloud agent runs | Long CPU trains, reports, PRs while you’re away; process video | Still the same lab rules; not magic capacity |
One-liner: harness + memory for shipping answers; Polar for proving weight change; cloud for recording the proof at scale.
10 · Closing
I didn't fall in love with local models because they're smarter than the harness. I fell in love because they're mine to change, and Polar makes the change visible.
- Why: own the weights and measure them.
- Scenarios: registry/memory for live truth; Polar for proof; harness for shipping work.
- Progress: short identical golds, miss polish, and full-attn LoRA moved Machina and space from flat or mid scores to 1.000.
- Eval: disjoint holdout
exact_match— meets useful (≥0.60) and clear-win (≥ base+0.20) gates on Machina and space.
Cloud runs turn that into a small video: wash → train → miss → fix → 1.000. Harness stays the sharp rented brain. Registry stays the live truth. LoRA stays behind the gate until we deliberately decide otherwise.
That's the setup. That's enough to publish — and enough to keep building.
Lab: polar-lab · Back to Dispatch