LLM order tests on the operator's controller and its flow (issue #8) #26

Manually merged

vinny merged 0 commits from vinny/flow-order-tests into master 2026-10-13 22:25:41 -04:00 AGit

Conversation 0 Commits - Files changed -

vinny commented 2026-10-13 22:17:24 -04:00

Owner

Copy link

Closes #8. tools/llm_order_tests.py --controller URL --controller-data DIR runs the order cases on a running CMOnet as the operator set it up: its imported flow, its model and its settings. It refuses while AI Stop is on, writes no report when it refuses, and reads the operator's store read-only to judge.

• Run of 2026-10-13 (flow expert-commander revision 72, Qwen3.8-27B, AI TEST SCENARIO at 1x): 2 of 4 passed after re-judging. move_destroyer: route 20.0 nm north at 15 kn, both read back. launch_pair: two F-35C from CVN 81 on a new patrol 41 nm east of DDG 91. readiness_report and patrol_setup failed in the flow itself: a schema-invalid campaign reply, followed by reviews writing under a change-nothing order; and a brief claiming a patrol that its compile step found infeasible. Evidence: docs/verification/llm/order-tests-2026-10-13-flow.json, summarized in docs/llm/PROMPT_ANATOMY.md. FL-8 is recorded as the partial qualification flows-live-model.
• Harness fixes from the run: a flow's library calls pass bare GUIDs, create reference points in one areas.create_reference_points call, and launch through an active mission. The write set is CMOnet's own WRITE_OPERATIONS. The last case's order is now cancelled, so the flow stops acting on it.
• The flow faults are the tester's (vinny/cmo-llm-tester#1): its cbf8613 makes decide optional in decision points and has the brief say what was ordered rather than claim effects. A human still rebuilds, imports and approves the flow. CMOnet's side is #24 (a structured "no game changes" order).
• Stacked on #25.

🤖 Generated with Claude Code

Closes #8. `tools/llm_order_tests.py --controller URL --controller-data DIR` runs the order cases on a running CMOnet as the operator set it up: its imported flow, its model and its settings. It refuses while AI Stop is on, writes no report when it refuses, and reads the operator's store read-only to judge. - **Run of 2026-10-13** (flow expert-commander revision 72, Qwen3.8-27B, AI TEST SCENARIO at 1x): 2 of 4 passed after re-judging. move_destroyer: route 20.0 nm north at 15 kn, both read back. launch_pair: two F-35C from CVN 81 on a new patrol 41 nm east of DDG 91. readiness_report and patrol_setup failed in the flow itself: a schema-invalid campaign reply, followed by reviews writing under a change-nothing order; and a brief claiming a patrol that its compile step found infeasible. Evidence: `docs/verification/llm/order-tests-2026-10-13-flow.json`, summarized in `docs/llm/PROMPT_ANATOMY.md`. FL-8 is recorded as the partial qualification `flows-live-model`. - **Harness fixes from the run:** a flow's library calls pass bare GUIDs, create reference points in one `areas.create_reference_points` call, and launch through an active mission. The write set is CMOnet's own `WRITE_OPERATIONS`. The last case's order is now cancelled, so the flow stops acting on it. - The flow faults are the tester's (vinny/cmo-llm-tester#1): its cbf8613 makes `decide` optional in decision points and has the brief say what was ordered rather than claim effects. A human still rebuilds, imports and approves the flow. CMOnet's side is #24 (a structured "no game changes" order). - Stacked on #25. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

vinny added 7 commits 2026-10-13 22:17:25 -04:00

Flight plan editor for the General: the tester's Phase 46 tools (issue #5) ... 6e13243fa0

missions.create_flight_plans makes a strike mission's flight plans, and
missions.edit_flight_plan inserts, moves, retypes (Initial Point, Refuel,
Target, Hold, and so on) and deletes one flight's waypoints, with CMO's
altitude and throttle presets. tester.check_flight_plan reads each one
back. They are synced from the tester at 8cf8406 (library revision
changes).

Both refuse an active mission (operator 2026-10-13: inactive missions
only): an active strike whose aircraft can't form a flight makes CMO pop
a modal "cannot launch" dialog in the player's game. The live probe
measured the API: insert and delete count from 1, waypoint wrappers take
position, type and presets, and exact per-leg altitude and speed don't
exist on flight-plan waypoints. Takeoff time and TOT stay with
missions.set_schedule.

tools/qualify_library.py gains a flightplans phase. It makes the strike
inactive before assigning two aircraft of one base and loadout, sets one
aircraft per flight, then generates, edits and removes. The live run
follows. A tools.search test now expects the exact name first. G2 goldens
are regenerated, and the tester copy is in its 9a38504.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Version 0.7.11 (flight-plans) 3961e6f96a

LLM order tests on a running controller: the operator's flow with its own model (issue #8) ... 87d242f60c

--controller runs the order cases on a controller already running as its
operator set it up: the imported flow, its real model and stored key,
which the tool never sees. Nothing in its settings changes. Its AI Stop
must be released by the operator on a disposable save, and the judge
reads that controller's store read-only (--controller-data, checked
against its data_directory_id). The case loop and the clean-up are shared
with the scratch-controller mode, and a refused controller leaves no
report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Order tests judge a flow's library calls: bare GUIDs, plural reference points, mission launches ... fcd5e52248

An imported flow calls the tester's library writes: they pass bare GUIDs (the store
keys units as u:GUID), create reference points in one areas.create_reference_points
call, launch through an active mission rather than hosting.launch, and draw or
remove the side's own no-navigation zones while routing. The write set is now
CMOnet's own WRITE_OPERATIONS. Run of 2026-10-13: move_destroyer was judged failed
although DDG 91's route and speed were read back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Flow order tests on the operator's controller: 2 of 4 after re-judging (issue #8) ... ccc497e184

The run of 2026-10-13 with flow expert-commander revision 72 and the operator's
model: move_destroyer and launch_pair met their orders; readiness_report and
patrol_setup failed in the flow itself (a schema-invalid campaign reply followed by
reviews writing under a change-nothing order; a brief claiming a patrol its compile
step found infeasible). The harness now also cancels the last case's order, which
otherwise stayed active for the flow to act on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

FL-8 recorded: the imported flow with a real model, 2 of 4 (issue #8) ... c9d40accd2

Qualification flows-live-model (partial) on runtime tuple rt-1900.20-6ea83b7ddca3,
with its report docs/verification/llm/order-tests-2026-10-13-flow.json; the
generated status documents are rendered again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Version 0.7.12 (flow-order-tests) ee458a12e0

vinny referenced this pull request 2026-10-13 22:20:28 -04:00

Decision packet around the order, stable prefix and working memory (issues #3, #4) #27

vinny manually merged commit ee458a12e0 into master 2026-10-13 22:25:41 -04:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

vinny/CMOnet!26

No description provided.