vinny/CMOnet
Watch
1
0
Fork
You've already forked CMOnet
 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Build the laboratory corpus with the Scenario Builder #7

New issue
Open
opened 2026-10-13 05:49:32 -04:00 by vinny · 1 comment
vinny commented 2026-10-13 05:49:32 -04:00
Owner
Copy link

All 14 scenario descriptors in docs/laboratory/corpus.json still wait for save files. That is why these STATUS gates were closed as "decided" rather than passed:

  • 12-reproducible
  • 12-held-out
  • 12-cross-scenario
  • the human-parity gates 13-non-inferiority and 13-superiority depend on ready saves too

The Scenario Builder can now generate forces on empty scenarios: 2,000 units spawned and read back in 26.8 s live (docs/verification/SCENARIO-BUILDER-LIVE-2026-10-05.md).

Proposal:

  1. Use the builder to make the corpus saves, one per family (air, surface, subsurface, mixed; offensive and defensive; constrained tanking and magazines; neutral contacts; mid-scenario takeover).
  2. Record their SHA-256 in the corpus.
  3. Run tools/lab.py capture/replay and the held-out evaluation against them.

Found in the 2026-10-13 review (PR #1).

All 14 scenario descriptors in `docs/laboratory/corpus.json` still wait for save files. That is why these STATUS gates were closed as "decided" rather than passed: - `12-reproducible` - `12-held-out` - `12-cross-scenario` - the human-parity gates `13-non-inferiority` and `13-superiority` depend on ready saves too The Scenario Builder can now generate forces on empty scenarios: 2,000 units spawned and read back in 26.8 s live (`docs/verification/SCENARIO-BUILDER-LIVE-2026-10-05.md`). Proposal: 1. Use the builder to make the corpus saves, one per family (air, surface, subsurface, mixed; offensive and defensive; constrained tanking and magazines; neutral contacts; mid-scenario takeover). 2. Record their SHA-256 in the corpus. 3. Run `tools/lab.py` capture/replay and the held-out evaluation against them. Found in the 2026-10-13 review (PR #1).
vinny commented 2026-10-17 19:58:38 -04:00
Author
Owner
Copy link

Lab run status, 2026-10-17

Steps 1 and 2 are done. All 14 corpus scenarios are built and ready, with their SHA-256 in docs/laboratory/corpus.json.

Step 3 is partly done. I ran live trials with tools/lab_run.py on CMOnet 0.7.87–0.7.97, using the expert-commander flow r72. Nine of the 14 scenarios have a completed trial:

Scenario Side Ended by Losses own / enemy FER Failed Invalid Unknown Fault recovery
air-defensive-cap NATO time budget 28 / 8 0.30 0.015 0 0 –
surface-defensive-escort (2026-10-16, earlier version) Coalition wall-clock cap 1 / 3 2.33 0 0.798 0.054 –
surface-offensive-sag China time budget 0 / 0 1.0 0 0 0 –
subsurface-asw-defensive NATO time budget 2 / 0 0.2 0 0 0 –
subsurface-offensive-patrol Allies time budget 12 / 0 0.04 0.017 0.034 0 –
mixed-dispersed-worldwide Allies time budget 2 / 0 0.2 0 0 0 –
mixed-midscenario-takeover NATO time budget 34 / 11 0.33 0 0 0 –
mixed-takeover-heldout Taiwan time budget 100 / 68 0.68 0.017 0 0 –
resilience-heldout-reload DPRK time budget 2 / 0 0.2 0.007 0 0 87 s

Five scenarios have no completed trial yet:

  • resilience-reload-alternate: stopped at its restart_cmonet fault. The runner now connects the new process (#149).
  • resilience-provider-socket: stopped twice:
    • once by hand, after a fault restored stale settings (#150);
    • once because CMO, busy, refused the lifecycle barrier (#151).
  • air-offensive-strike: earlier attempts on 2026-10-16 did not finish.
  • scale-500-high-compression and scale-500-development: not started. The run was stopped after resilience-heldout-reload at the operator's request.

Fixes that came out of the runs: #146–#155, plus tester fixes ef27991 (tester issue 2) and bfcbde0. Tester issue 6 is still open.

This issue stays open until the five remaining trials have run on the current version.

🤖 Generated with Claude Code

## Lab run status, 2026-10-17 Steps 1 and 2 are done. All 14 corpus scenarios are built and `ready`, with their SHA-256 in `docs/laboratory/corpus.json`. Step 3 is partly done. I ran live trials with `tools/lab_run.py` on CMOnet 0.7.87–0.7.97, using the expert-commander flow r72. Nine of the 14 scenarios have a completed trial: | Scenario | Side | Ended by | Losses own / enemy | FER | Failed | Invalid | Unknown | Fault recovery | |---|---|---|---|---|---|---|---|---| | air-defensive-cap | NATO | time budget | 28 / 8 | 0.30 | 0.015 | 0 | 0 | – | | surface-defensive-escort (2026-10-16, earlier version) | Coalition | wall-clock cap | 1 / 3 | 2.33 | 0 | 0.798 | 0.054 | – | | surface-offensive-sag | China | time budget | 0 / 0 | 1.0 | 0 | 0 | 0 | – | | subsurface-asw-defensive | NATO | time budget | 2 / 0 | 0.2 | 0 | 0 | 0 | – | | subsurface-offensive-patrol | Allies | time budget | 12 / 0 | 0.04 | 0.017 | 0.034 | 0 | – | | mixed-dispersed-worldwide | Allies | time budget | 2 / 0 | 0.2 | 0 | 0 | 0 | – | | mixed-midscenario-takeover | NATO | time budget | 34 / 11 | 0.33 | 0 | 0 | 0 | – | | mixed-takeover-heldout | Taiwan | time budget | 100 / 68 | 0.68 | 0.017 | 0 | 0 | – | | resilience-heldout-reload | DPRK | time budget | 2 / 0 | 0.2 | 0.007 | 0 | 0 | 87 s | Five scenarios have no completed trial yet: - **resilience-reload-alternate**: stopped at its `restart_cmonet` fault. The runner now connects the new process (#149). - **resilience-provider-socket**: stopped twice: - once by hand, after a fault restored stale settings (#150); - once because CMO, busy, refused the lifecycle barrier (#151). - **air-offensive-strike**: earlier attempts on 2026-10-16 did not finish. - **scale-500-high-compression** and **scale-500-development**: not started. The run was stopped after resilience-heldout-reload at the operator's request. Fixes that came out of the runs: #146–#155, plus tester fixes ef27991 (tester issue 2) and bfcbde0. Tester issue 6 is still open. This issue stays open until the five remaining trials have run on the current version. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
vinny/CMOnet#7
No description provided.