vinny/cmo-llm-tester
Watch
1
0
Fork
You've already forked cmo-llm-tester
 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Expert commander: report-only and forceless orders, and a brief from compile results (CMOnet FL-8) #1

New issue
Closed
opened 2026-10-13 22:12:10 -04:00 by vinny · 1 comment
vinny commented 2026-10-13 22:12:10 -04:00
Owner
Copy link

CMOnet's FL-8 run of 2026-10-13 (CMOnet docs/verification/llm/order-tests-2026-10-13-flow.json) gave four orders to expert-commander revision 72 with Qwen3.8-27B. Two failures came from the flow:

  1. readiness_report ("Report how many aircraft are on CVN 81's flight deck ... Do not change anything in the game."): node n18 campaign_orders failed with "'decide' is a required property". One decision point had no decide text. The next review cycles then ran as campaign cycles and sent 20 writes: side doctrine and EMCON, and CAP, AEW, strike and ASW missions. No cycle answered the question (30 aircraft parked on the deck). Fixed in the builder (cbf8613): decide is optional. Still open: a report-only order should keep a cycle from ordering anything. Today the briefing shows the President's words, but a campaign wake still plans a campaign.
  2. patrol_setup ("Create a reference point about 30 nm east of CVN 81, then create an anti-air patrol mission named TEST-CAP that uses that point ... Do not assign or launch any aircraft."): compile returned infeasible: order.forces must hold 1-8 items. The zone verbs need forces, so an empty mission cannot be ordered. Meanwhile the brief, written before compile ran, said "I have established TEST-CAP". The brief wording is fixed in the builder (cbf8613): say what was ordered, never that it took effect. Still open: a zone verb with no forces (mission only), and a brief that reflects compile results rather than the plan.

The builder fixes need a rebuild of projects/expert-commander, then a human import and approval in CMOnet.

CMOnet's FL-8 run of 2026-10-13 (CMOnet `docs/verification/llm/order-tests-2026-10-13-flow.json`) gave four orders to expert-commander revision 72 with Qwen3.8-27B. Two failures came from the flow: 1. **readiness_report** ("Report how many aircraft are on CVN 81's flight deck ... Do not change anything in the game."): node n18 `campaign_orders` failed with "'decide' is a required property". One decision point had no `decide` text. The next review cycles then ran as campaign cycles and sent 20 writes: side doctrine and EMCON, and CAP, AEW, strike and ASW missions. No cycle answered the question (30 aircraft parked on the deck). **Fixed** in the builder (cbf8613): `decide` is optional. **Still open:** a report-only order should keep a cycle from ordering anything. Today the briefing shows the President's words, but a campaign wake still plans a campaign. 2. **patrol_setup** ("Create a reference point about 30 nm east of CVN 81, then create an anti-air patrol mission named TEST-CAP that uses that point ... Do not assign or launch any aircraft."): compile returned `infeasible: order.forces must hold 1-8 items`. The zone verbs need forces, so an empty mission cannot be ordered. Meanwhile the brief, written before compile ran, said "I have established TEST-CAP". The **brief wording is fixed** in the builder (cbf8613): say what was ordered, never that it took effect. **Still open:** a zone verb with no forces (mission only), and a brief that reflects compile results rather than the plan. The builder fixes need a rebuild of `projects/expert-commander`, then a human import and approval in CMOnet.
vinny commented 2026-10-14 18:52:42 -04:00
Author
Owner
Copy link

Fixed in 989312a, on top of 1d26909:

  • Report-only orders: the flow reads CMOnet's structured order.scope.game_changes (CMOnet issue #24, 0.7.26), never the wording. A report-only order wakes a report tier: one reporter request that answers from the briefing. Compile makes no programs under it and keeps the queue. The briefing has a table of the aircraft not in the air (by host and condition, ready now, readying, on a mission) for questions like the flight-deck one. CMOnet's write gate also refuses every write under such an order.
  • Zone missions with no forces: zone verbs accept empty forces. The mission and its area are made with no units assigned, and the mission is kept as unmanned until units are added. An optional name covers orders like TEST-CAP.
  • Brief from compile results: when compile refuses one of the General's orders, the reporter rewrites the decision brief from compile's results before anything is sent. Schema refusals now count as the General's.

Rebuilt and imported into CMOnet as expert-commander revision 74, inactive. Revision 72 still runs. It needs a person to approve the changed algorithms/expert_commander.py, then a live FL-8 rerun.

Fixed in 989312a, on top of 1d26909: - **Report-only orders:** the flow reads CMOnet's structured `order.scope.game_changes` (CMOnet issue #24, 0.7.26), never the wording. A report-only order wakes a `report` tier: one `reporter` request that answers from the briefing. Compile makes no programs under it and keeps the queue. The briefing has a table of the aircraft not in the air (by host and condition, ready now, readying, on a mission) for questions like the flight-deck one. CMOnet's write gate also refuses every write under such an order. - **Zone missions with no forces:** zone verbs accept empty `forces`. The mission and its area are made with no units assigned, and the mission is kept as `unmanned` until units are added. An optional `name` covers orders like TEST-CAP. - **Brief from compile results:** when compile refuses one of the General's orders, the `reporter` rewrites the decision brief from compile's results before anything is sent. Schema refusals now count as the General's. Rebuilt and imported into CMOnet as expert-commander revision 74, inactive. Revision 72 still runs. It needs a person to approve the changed `algorithms/expert_commander.py`, then a live FL-8 rerun.
vinny closed this issue 2026-10-14 18:52:42 -04:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
vinny/cmo-llm-tester#1
No description provided.