LLM workflow tester

A separate, locally hosted program for designing and testing LLM workflows, including the tools the model may use. It does not import or connect to CMOnet. The design brief is docs/llm_tester/PLAN.md.

The tester has saved file formats, an expression language, a JSON API, and a page for projects, flow graphs, nodes, agents, tools, tests, runs and comparisons.

Start

Double-click start.bat, or run it from a terminal. The first start creates .venv and installs the pinned requirements, which needs internet access. The tester then opens http://127.0.0.1:8765/ in your browser. Stop it with Ctrl+C.

Option	Effect
start.bat --port 9000	Use another port
start.bat --no-browser	Do not open the browser
start.bat --setup	Create or update .venv, then exit
start-reload.bat	Restart the service when tester Python changes
start-reload.bat --port 9000	Reload on changes using another port

The reload launcher watches Python files in src/llm_tester/ and algorithms/. It is for development: restarting interrupts requests in progress. Refresh an open browser tab to see HTML changes; the server serves the current file on each request.

Requirements are reinstalled automatically when requirements.txt changes. Delete .venv to rebuild the environment from scratch.

The page

• Projects. The button with the open project's name, at the left of the top bar, opens the project list. There you can open a project, create one, or duplicate or delete one. A project whose file can't be read is listed with the reason, and can only be deleted. Deleting moves a project to the trash (see Trash).
• Renaming. The Flow tab shows the open project's name and description. Change the name there and save; the project's ID and folder stay the same.
• The flow graph. The Flow tab's canvas shows the open project's graph.
  • Add a node by dragging its type from the palette onto the canvas, or by clicking the type to add it in the middle of the view.
  • Connect two nodes by dragging from a node's right-hand port onto the other node. A branch has one port per route, plus else.
  • Select a node or a connection and press Delete to remove it. The start node can't be removed.
  • Drag the background or turn the mouse wheel to pan; hold Ctrl to zoom. The buttons in the corner zoom, and Fit shows the whole flow.
  • Nodes snap to a 10-pixel grid.
• Node properties. Selecting a node shows its title, key, purpose and settings on the right; × or a click on empty canvas shows the project again. Drag the panel's left edge, or press the arrow keys on it, to widen or narrow it; the page remembers the width.
  • Other nodes read a node's output by its key. A new node's key follows its title until you save or edit the key.
  • Each node type has a form for its settings. A value comes from a path (completed from start, memory and the nodes before this one), a constant, or a template, whose Insert a value menu adds a {{path}}. A constant is Text, kept as typed, or JSON, for numbers, true, false, null, lists and objects.
  • A tool node's tool comes from the library, and its arguments follow the tool's parameters: choosing a tool adds a row for each argument it requires, and buttons add the others.
  • A branch's routes are rows of conditions, checked in order. Renaming a route, or moving it, keeps its connections; removing a route removes them, and the status strip says how many.
  • A value that breaks the rules, such as a name with spaces, shows what is wrong and isn't used; leaving the box puts back the last good value.
  • Edit as JSON, under each form, shows the same settings as JSON, for copying them between nodes. JSON typed there is checked by the same rules as a save and used once it's valid. Revert goes back to the last valid settings, and Save waits until they're fixed.
• Agents. With no node selected, the Agents list under the project shows the project's agents. New agent asks for a name, which also makes the agent's ID; renaming later changes only the name. Select an agent to edit its prompt, model settings, tool access and the other settings.
  • Tool access includes the tools with any of the included tags, except those with an excluded tag, plus the tools listed by ID. The editor says how many tools that gives the agent.
  • An agent node's form picks its agent, or creates one, and Edit agent opens it. Deleting an agent that nodes use asks first; they then show "Missing agent" until you choose another.
  • The Tools tab shows which of the open project's agents can use a tool.
• Tool nodes show their tool's ID, with a red write badge when the tool is a write, or a note when the tool isn't in the library.
• Problems. The Flow tab's Problems panel lists what is wrong with the flow as it is on the screen, checked again a moment after each edit. Errors, such as a node that can't be reached, a setting not chosen yet or a path to a node that doesn't run first, will stop runs; warnings, such as reading a node that runs on only one route of a branch, won't. Saving still works, and says how many errors remain.
  • Click a problem to select its node or agent. Nodes and agents with problems have a red (or, for warnings only, amber) mark, and a selected node or agent lists its own problems at the top of its properties.
  • A path or template may read start, memory, or a node that always runs before the node reading it. In {{path?}} placeholders and branch conditions, a node that may not have run is fine: they allow for a missing value.
• Edges that can't be drawn. If a project file has an edge the canvas can't draw, such as one from a route that no longer exists, it stays in the project and is listed in the Problems panel, where Remove connection removes it.
• The Tools tab lists the tool library. Search it, or filter it by class or tag, and click a tool's ID to open it.
  • Edit its title, description (the text the model sees), class, tags, parameters (a JSON Schema for the arguments), result schema, result example and, for writes, the result a logged write returns. The JSON boxes are checked as you type; a tool with an error isn't saved.
  • Saving a change to the description, parameters, result schema or class raises the tool's version. Imported tools show where in CMOnet they came from.
  • New tool asks for the tool's ID, such as tester.my_tool. Delete tool moves the tool, and its Lua file if it has one, to the trash.
• The Tests tab lists the open project's tests. Create, rename, duplicate or delete a test there. Each test can hold several steps, each with an instruction, optional event, mock read results and expected tool calls. A run starts from the test's starting memory. The editor accepts unfinished tests; warnings identify missing tools, schema mismatches and duplicate mock entries. The first exact mock row wins, then the first any-arguments row. Imported tools have no result examples, so a new mock result stays null with a warning until you supply data.
• Saving. Save, or Ctrl+S, saves every unsaved change: the open project, then the open tool and test. While there are any, Save shows a dot and the browser tab's title starts with •. The page asks before you open, create or duplicate another project, or open another tool, with unsaved changes, and the browser warns before you leave the page.
• Two tabs. If the project was saved in another tab or window after you opened it, saving asks what to do. Reload drops this page's edits and loads the latest version. Save as copy keeps them in a new project, with the project's tests. A tool changed in another tab, or on disk, offers Reload; a tool deleted elsewhere can be created again with your edits.
• The page remembers the last opened project and tab.
• Undo and redo. ↶ and ↷ beside the zoom buttons, or Ctrl+Z and Ctrl+Y (or Ctrl+Shift+Z), undo and redo changes to the open project: the graph, its nodes' settings and its agents. The last 50 states are kept while the page is open; a burst of edits, such as typing a title, is one step. In a text box, the keys undo typing there instead.
• Keyboard. Ctrl+S saves, Ctrl+Enter runs the chosen test, and Delete removes the selected node or connection.
• Long runs. The trace shows 200 events a page, with Earlier, Later and Newest; a live run follows its newest events.

The tool library

Tools are files in tools/: <tool id>.json, plus <tool id>.lua for a tool that runs against CMO. The first set is imported once from CMOnet's operation list, in the CMOnet repository next to this one:

.venv\Scripts\python -m llm_tester.import_tools

Option	Effect
path	Read another operation list instead of ..\CMOnet\src\cmonet\resources\operations.json
--source-commit SHA	Record this CMOnet commit, instead of asking git
--skip-application-writes	Leave out CMOnet's 18 application writes
--overwrite	Replace tool files that already exist

The command only reads that file; no CMOnet code runs. Game reads and writes are tagged cmo, game-read or game-write; application reads app-read; application writes app-write and log-only; and every tool family:<CMOnet family>. CMOnet's 11 blocked operations are never imported. Existing tool files are kept unless you pass --overwrite. The report is printed and saved to the data folder's import-reports/.

After the import the tester owns these files: edit them in the Tools tab. Hand edits work too, and the page refuses to save over a tool that changed on disk since it was opened.

API

The service has a JSON API under /api/ for projects, their tests, the tool library and the service settings. http://127.0.0.1:8765/docs lists every request and can send them; that page loads its scripts from the internet. Changes are only accepted from the tester's own page, or from tools that send no Origin header, such as curl. Request bodies must be JSON.

A save sends the whole project, test or settings file with the revision it was loaded at. If it was saved again in between, the save is refused with 409, so two tabs never silently overwrite each other's edits.

The page reads the node types from /api/node-types, and checks a node's settings with /api/nodes/check, which applies the same rules as a save but saves nothing. /api/projects/{id}/validate lists the problems of the project it is sent, such as the page's unsaved version, and /api/agent-template holds the settings a new agent starts with.

A tool is read with its content hash, and a save sends the hash back as ?content_hash=: a tool changed since it was read gets 409. /api/tools/check checks a tool without saving it, and /api/tools/resolve lists the exact tools an agent's tool access gives it.

Mock runs (API)

POST /api/runs with {"project_id":"demo","test_id":"smoke", "step_index":0,"mode":"mock"} returns a run ID. The flow must pass graph validation. GET /api/runs/{id} returns the record and events; GET /api/runs/{id}/events streams stored events followed by live ones. Use GET /api/runs?project_id=demo for history, POST /api/runs/{id}/stop to stop, and DELETE /api/runs/{id} to remove a run. The event stream ends with a done event, carrying the run's final status, once the run has ended and every event has been sent. Runs live under %LOCALAPPDATA%\llm_tester\runs, outside the repository. Mock reads use the selected test step; writes are logged and return the tool's mock write result. Each call gets its own copy of the mock result.

The run bar starts a selected test step in Mock mode and saves pending edits first. The Flow tab shows live node states and a searchable trace with inputs, outputs, tool calls and timing. In the trace, a tool result that found no mock data, a logged write, a failed call and a model server's error reply are marked. The Runs tab lists saved runs, displays each run's saved graph, and can delete individual runs or finished runs older than a chosen number of days.

Settings now configures an Unsloth or OpenAI-compatible model endpoint, default model, and request concurrency. Enter an API key there and save it; the page then shows only whether a key is saved for that endpoint. Test connection lists the model IDs returned by /models so you can choose a default. Keys are stored in the data folder's credentials.json, outside projects and run records. The model client uses a queue, never follows redirects, and keeps request bodies and model replies available for later agent traces without including authorization headers. An error reply is traced with its status and body, and the node's error gives the server's reason. A reply without a reasoning field is traced as "not returned". If the settings were saved in another tab, Save offers to reload them.

Agent nodes now run in Mock mode using their saved prompt and model settings. In prompt JSON mode, the model sees only the tools selected for that agent; it can call them over several rounds and receives their mock results before returning a final answer. The run trace records model requests and replies, tool calls and results, timing, errors, and the server-reported model. A node that expects JSON needs an object as its final answer, in either mode; a reply may be wrapped in one ```json fence. A node that expects text keeps a plain-text reply as it is, and writes an object answer, which the default reply instructions ask for, as JSON text.

Agents can also offer the same resolved tools through the provider's native function-calling field. The tester maps dotted tool IDs to valid function names, stores that mapping with the run, and sends each native tool result back using its call ID. Set Tool offering on an agent to compare this mode with prompt JSON on the same flow.

In prompt JSON mode, How the prompt lists each tool chooses a full entry (ID, description and parameter schema) or a compact one (ID, class, description and arguments, such as entity_id: string, refresh?: boolean). A compact signature shows a nested object's fields in braces, naming its optional ones without their kind, | between alternatives, and a closed set of values as "a"|"b", so center: {entity_id: string} | {area_id: string} | {latitude_deg: number, longitude_deg: number} shows the three ways to give a center. List only these tools in the prompt names the tools the prompt lists, in order; the agent can still call its other tools, and the prompt says how many there are. Give such an agent tools.search and tools.describe: they answer from the tool library (algorithms/tool_catalog.py, leaving out the tester's own tools), so the model can find a tool and read its whole schema. A search ranks tools by the query's words, ignoring case: a word matches when a word of a tool's name, purpose or family starts with it, so mission finds missions.create, and common words such as "the" don't count. (CMOnet's search finds only the whole query as written, so a query of several words seldom finds anything.) Describing a name that isn't a tool is an error, as in CMOnet. When an agent calls them, they find only the tools that agent is offered in the run, and a search names under unavailable the matching tools it has that can't run there. When the agent's instructions or first message name one of its tools that can't run in the run, the system message says so, and not to look for it.

An agent always knows its tool budget. The system message (in prompt JSON mode) gives its Most tool rounds, and each round's results end with Tool rounds left: <n> of <max>.; when none are left, the line goes on to say the next reply must have no tool calls and give the final answer. A reply that calls tools with none left is an invalid reply: its calls aren't run, and it gets the repair message while the agent has Repair attempts left; otherwise the node fails with "tool round limit reached". A call whose arguments don't fit the tool's parameters gets, with its invalid_arguments error, an argument_contract: the tool's whole argument signature, with every field's kind, every value and the bounds of numbers, once per tool in a round (later ones point to it). The repair message for a reply that isn't valid JSON quotes the text where it breaks, marked <<HERE>>, such as a brace the model left out.

In a dry-run or live run, an agent is offered only the tools that can run there: a read with Lua, or with mock data in the test step; a Python tool; a write (in a live run, only with Lua or tagged log-only); and a composite whose steps all can. The others are left out of its tool list, its native tools and the tool catalog, and a note in the trace names them; a call to one gets not_available. A mock run offers every resolved tool, since mock data answers them.

Algorithm nodes run documented public functions from algorithms/*.py with def name(inputs: dict, params: dict) -> dict. The node form lists available files and functions; Refresh files rescans after an external edit. Files reload when their modification time changes, so the next run uses the new code. Each used file's SHA-256 is saved in the run record. Loading a changed file and calling the function happen in a worker thread with a 30-second timeout, which covers the file's top-level code; a timed-out thread may continue in the background. Functions get copies of their inputs and parameters. Anything a function raises fails its node, even SystemExit from a stray sys.exit(). Files may define dataclasses. The bundled algorithms/examples.py calculates great-circle distance and bearing.

To implement a tool in Python, set its Python implementation to file.py:function in the Tools tab. Its inputs are the tool arguments and params is empty. It runs before mock data in every mode and has no game access. Algorithm functions must return a JSON-compatible dictionary.

The Code tab edits top-level algorithms/*.py and tools/*.lua files. Choose a file, edit it, and use the top bar's Save button. The editor shows line numbers. A Python syntax error is shown after saving; the file stays saved so you can repair it. A tool Lua file is checked when it is opened and saved: the tester's rules for tool Lua (see tools/README.md) and Lua 5.4's own syntax check, with the line and column; nothing runs. A Lua file whose name matches no tool says it is never sent. New from template creates an algorithm file, or, in the Tool Lua folder, asks which tool the file is for, offering the tools without one, and starts <tool id>.lua with the Lua contract and that tool's arguments. Delete moves a file to the data folder's trash, and Reload opens the file as saved on disk. If another editor changes a file after you open it, Save refuses to overwrite it and offers Reload, which drops your edits; cancel to copy them first. Listing the files, as the Flow tab's algorithm form does, loads them, so it runs their top-level code.

The file API lists with GET /api/files?area=algorithms|tool_lua, and reads, creates, saves and trashes files at /api/files/{area}/{name}. A PUT includes the loaded SHA-256 as ?content_hash= and sends {"content":"..."}. Paths stay in their selected top-level folder. Saving code never runs it.

Composite tools in the Tools tab call existing tools in order. Turn on Composite tool, add and reorder steps, choose each tool, and map its arguments using paths, constants or templates. args holds the composite call's arguments, and steps[0], steps[1] and so on hold earlier results. For example, step 2 can read steps[0].id from a mock write that returned a made-up ID. Composites cannot contain composites. Their class follows their steps: a composite with any write step is a write. Mock writes inside a composite are logged and never sent. The run record fixes the composite and each step tool's version and hash at run start. Trace entries show each step under its parent call. An agent can call the composite through its own parameter schema like any other tool. The tool list marks composites, and a composite's step picker leaves them out. Expected tool calls count a composite call once; its steps are part of it.

Repeated runs and comparisons

Settings → Runs can delete finished runs older than a number of days each time the tester starts; empty keeps every run. The Runs tab shows how much space the project's runs, and all runs, take (GET /api/runs/storage?project_id=...).

Set Count in the run bar to 1–50. A count above one creates a batch that runs the selected test step sequentially. Every run of a batch uses the project, test and tools as they were when the batch started, so edits saved meanwhile wait for the next run or batch; algorithm files are read when each run uses them, and their hashes are recorded. The Runs tab shows each batch's status counts, durations, outputs and tool-call sequences. Stop ends the current run and the remaining batch. A deleted run stays in its batch's list as deleted, and deleting a batch's last run deletes the batch. Batches and run records stay in the data folder outside the repository.

In a test step, list expected tool calls in order. Each listed argument must equal the corresponding actual argument; actual calls may have additional arguments. The calls checked are the flow's own: a composite counts once, and its steps aren't counted separately. A run shows matched, missing and unexpected calls, and a batch shows how many of its checked runs matched. A step with no expected calls has no expected-call score.

Open a finished run in the Runs tab and select a node in its saved graph to Rerun from that node. The new run uses the current saved project and reuses the earlier nodes' outputs and memory. Its trace copies each reused node's tool calls, model requests and replies, marked "reused", so its expected-call check and comparisons cover the whole flow. The rerun is refused when an earlier node is missing or its key changed, and while another run is going.

The Compare section accepts two runs or two batches. It shows what differs in the flow (nodes, connections, agents' prompts and settings), the tools, the algorithm files and the test step, then node status, values read, output and timing, line diffs of rendered prompts and model replies, and tool calls in sequence. Moving a node on the canvas isn't a difference. The APIs are /api/batches, /api/runs/{run_id}/rerun, and /api/compare/runs or /api/compare/batches.

CMO socket

The tester has its own client for CMO's socket, 127.0.0.1:7777 by default, which the socket proxy inside CMO serves. Don't run CMOnet against the same game while the tester uses it. Only a saved tool's Lua file is sent there, inside the tester's fixed wrapper, plus the proxy's fixed HEALTH request, which runs no Lua.

• The top bar shows the socket's state: idle, busy or paused. The dot is amber when the last request failed and red while paused. Click it for the details: why it is paused, errors in a row, the wait before the next request, and the last problem with its diagnosis (what was sent and received, and where it stopped).
• Check CMO asks the proxy how it is. It is healthy when Lua is ready and nothing is running or queued. It works while paused and changes no count. It warns if another client, such as CMOnet, is connected.
• Test CMO connection, in Settings and in the CMO panel, sends the saved tools tester.ping, with the chosen side, and tester.sides. It shows CMO's Lua version, whether the chosen side was found, and the game's sides with their GUIDs, and each request and reply under Requests and replies. In Settings it saves the settings first, and Use puts a side's GUID in the Side field (save to keep it). These are real requests: they are paced, and a failure counts toward the pause like any other.
• The side that dry-run and live runs command is set in Settings → CMO socket, by its GUID or its name. The tester uses it even if CMO shows it as human-controlled, whichever side CMO's view is on. Each tool's wrapper finds the side again before the tool's Lua runs: if it is gone, or two sides share the name given, the tool doesn't run. A GUID can never be shared.
• Pause stops all socket traffic but HEALTH checks at once; a request already sent still finishes. Resume asks first, and warns when a write may have changed the game. After a Resume the next request goes at once, but the error count stays, so the next failure pauses again.
• A request that was sent but got no valid reply (a timeout, an early close, a malformed reply, or a proxy error) has an unknown outcome: CMO may have run it. By default one such outcome pauses the socket. Nothing is ever retried. A refused connection, or the proxy's "not executed" reply, means nothing ran.
• Settings → CMO socket holds the address, the reply deadline, pacing and its floors, the backoff after failures, the pause policy, and automatic Resume, which checks HEALTH after a delay and resumes only if the proxy is healthy. Automatic Resume is off by default, never ends a manual pause, and needs its own setting after a write whose outcome is unknown. A pacing floor below its tested default shows a warning.
• The pause, the error counts and the request in flight are kept in the data folder's cmo-state.json, so they survive a restart. A request that was in flight when the tester stopped counts as an unknown outcome. If the file can't be read, the socket starts paused and the file is kept beside it.
• The API is GET /api/cmo/status, and POST /api/cmo/pause, /api/cmo/resume, /api/cmo/health and /api/cmo/test-connection.

Tool Lua follows the contract in tools/README.md. Before a tool's Lua is sent, the tester joins it into one ASCII line, writes its arguments in as a Lua table, and compiles the file, that line and the whole request with Lua 5.4 (lupa) without running them, so a syntax error is never sent. In CMO, the wrapper runs the tool's Lua in a protected call and sends back its result as JSON; a Lua error, a result JSON can't hold, or one too large for a reply comes back as the tool's error instead of breaking the reply. Text from the game that isn't valid UTF-8 is read with replacement characters (<28>) and noted.

Timeline tests: all steps in one run

A test's steps can be decision cycles over time. Choose All N steps in the run bar's step picker (it shows when a test has two or more steps) to run the graph once per step, in order, as one run. A batch can run all steps too.

• Memory starts from the test's starting memory and carries over from each cycle to the next.
• Waits. A step's wait_seconds is waited, in real time, before its cycle, so the game can move on in dry-run and live runs. The tester never changes the game's speed. Stop works during a wait.
• Data. In mock runs each step uses its own mock reads; dry-run and live steps read the game.
• The trace is grouped by step, and the canvas shows each cycle's nodes as they run. In the Runs tab, one button per step shows how its cycle ended and whether its expected calls matched; choose one to see its nodes on the run's graph, to Rerun from a node in that step (every earlier cycle is reused whole), or to Save live reads of that step.
• Results. Each step's cycle has its own outputs, tool calls and expected-call check; the run's outputs are the last finished cycle's. A failed or stopped cycle ends the run. The run matches its expected calls only when every step that lists some does.
• Compare matches nodes by step and node ID, and says which step made each tool call.

The API takes "all_steps": true in POST /api/runs and POST /api/batches, and "step" in POST /api/runs/{run_id}/rerun and /capture for a run of all steps.

The CMOnet baseline

projects/cmonet-baseline/ approximates today's CMOnet General loop, as recorded in docs/diagrams/current-llm-system.md, to compare new designs against (Phase 23; like other saved projects, it isn't committed unless you ask). Each decision works as CMOnet's does:

• System message: CMOnet's fixed policy, the general-only operating structure and its default profile's instructions, word for word; then a compact list of CMOnet's 54 bootstrap operations and the game writes that have Lua (the tester's stand-in for CMOnet's authorized writes). The General can use the 172 tools CMOnet's General can use, and finds the others with tools.search and tools.describe. In a dry-run or live run it is offered only those that can run there: 42 of them in a live run of "Hormuz live" (2026-09-25), since CMOnet's ledger reads, such as constraints.current, and most game writes have no Lua.
• User message: a fresh JSON decision capsule, cmonet-general-capsule/2, built by algorithms/cmonet_capsule.py from the clock, the scenario, the side's units and its contacts, which the flow reads first. It holds the presidential order, the event, the previous decision's summary, CMOnet's decision standard for the first tool turn, the response_contract and the capability limits.
• Limits: at most 8 model turns, as in CMOnet, whose 8th reply must end the decision; so the agent's limit is 7 rounds of tool calls, since the tester's limit counts rounds of calls before a final reply. Each round's results say how many are left, as CMOnet's refreshed capsule does. One repair after an invalid reply, including a reply that calls tools when none are left. The reply instructions carry the response_contract in the tester's reply format, and the node's output schema checks the final part, with disposition one of plan, wait, blocked or report.

It differs from CMOnet in four ways: the tools are listed in the system message, not in the capsule; tool results come back as a growing chat, not as a fresh capsule with one continuation message; the capsule leaves out what CMOnet derives from its own records, such as operations, reservations, supervisor policy, readiness and the strategic picture; and CMOnet's own checks of each reply, such as its rules for air launches and movement batches, which send the model a repair message, aren't modeled. The system message is about 20,000 characters, and about 19,000 in a live run, whose list leaves out the tools that can't run there.

Its tests: "Strait patrol", a made-up situation in mock data; "Hormuz live", for dry-runs and live runs of the operator's game; and "Hormuz captured", a dry-run's reads of that game saved as a step, to replay the real situation in mock mode. To compare a new design, duplicate the project, change it, run the same test in both, and compare the runs in the Runs tab.

Handoff export

Export for handoff…, in the Flow tab's project panel, makes the package the AI agent converting CMOnet uses, and the flow bundle CMOnet imports and runs as its General's flow (the format). Choose the tests and runs to include; unsaved changes are saved first. The export is a folder in the data folder's exports\<project id>-<UTC time>\, plus a zip of it to download from the dialog. It holds:

Path	Contents
project.json	The project as saved
tests/	The chosen tests
runs/<run id>/	The chosen runs, run.json and events.jsonl, as worked examples
tools/	The whole tool library, with its Lua files, so CMOnet gets every tool and its CMOnet information
algorithms/	The algorithm files its nodes and any tool use
schemas/	A JSON Schema for every file format, made from the tester's own models
FLOW.md	The flow in words: its purpose; the nodes in execution order with their exact settings; the agents, with their prompts, message templates and reply instructions word for word; the branch rules; the memory keys and where each is written and read; the flow's tools, composites and algorithms, and a table of the rest of the library; the tests and their expected outcomes; and the chosen runs as input, tool calls and output
manifest.json	What made it (producer), the tester's version, the time, and every file's SHA-256

An export never holds settings, API keys or request headers. Every file is scanned for the stored keys first; if one appears, nothing is exported and the message says which file. The API is POST /api/projects/{id}/export with {"test_ids": [...], "run_ids": [...]}, and GET /api/exports/{name}.zip.

Importing a flow bundle

Import bundle…, in the Projects dialog, imports a flow bundle: a zip the tester exported, here or on another machine, or one CMOnet exported. Choose the zip, and the dialog shows what importing it would do; nothing is imported until you press Import, and nothing in the bundle is run.

• The project keeps its ID when no project has it. Otherwise it is imported as a new project, with an ID made from its name, or replaces the saved project as its next revision, its tests replacing those with the same IDs.
• Tools. A bundle holds the whole tool library. Tools new here are created. A tool that differs is shown by part: its contract (description, class, tags, parameters, result schema and example), its implementation (mock, Python, composite, read-back and Lua) and its CMOnet information. For each, keep the library's, take the bundle's, or keep the library's with the bundle's CMOnet information. The library's wins by default, except that CMOnet's own bundles bring their CMOnet information, which CMOnet keeps.
• An algorithm file whose name the library has with other content is imported as <name>_<first 8 of its SHA-256>.py, and the flow's nodes and the bundle's tools that name it are pointed there.
• Runs are imported when the project keeps its ID and the run ID is free.

CMOnet's bundles also carry its requirements: the Lua rules its bundle tools must pass, the model settings it accepts and the start event it sends. Importing one keeps them in the data folder (cmonet/requirements.json; a newer bundle's replace them), and from then on the problems list and the tool editor show warnings starting CMOnet: for what CMOnet would refuse or never send. They never stop a save or a run. CMOnet exports its built-in General loop as a project like cmonet-baseline, and its flow runs as a test whose mock data is what CMOnet's tools returned, to retune a flow against.

Every tool is checked before anything is written, so a bundle with a tool that can't be saved imports nothing. An upload that isn't imported is deleted when the dialog closes, or after 30 minutes. The API is POST /api/imports with the zip as application/octet-stream, then GET /api/imports/{token}, POST /api/imports/{token}/preview and POST /api/imports/{token}/apply with the choices, and DELETE /api/imports/{token}.

A tool's CMOnet information

A tool's CMOnet information, in the Tools tab, is what CMOnet records about the tool as one of its operations: its classification, family, scope, permissions, preconditions and policies, and the arguments that name the side's units (or groups, missions, reference points) and contacts, which CMOnet checks before it runs the tool's own Lua from a bundle. Fill in defaults suggests it from the tool's class, tags and argument names, and the section lists what CMOnet would still need, such as a read-back for a game write. These are warnings: a tool saves without them. The tester itself doesn't use the field. Tools imported from CMOnet get it from CMOnet's own bundles (Phase 40).

CMO database

CMO's class data (ships, submarines, aircraft, facilities, ground units, satellites, and their weapons, sensors, mounts, magazines, loadouts, comms and propulsion) is in CMO's own SQLite file, DB\DB3K_<build>.db3 in the game folder (or CWDB_<build>.db3 for the Cold War database). Upload it in Settings → CMO database; each new upload replaces the loaded one, which is deleted, so upload the new file whenever CMO updates its database. The tester keeps its copy in the data folder's cmo_db\, outside the repository, and never changes CMO's file.

An upload must be a whole SQLite database with CMO's platform and component tables, or it is refused and the loaded file stays. The tester indexes its copy's link columns, then checks every link between the tables and lists the rows that point at nothing (DB3K_519 has four such links, a few rows each). Uploads are refused while a run goes. The API is GET /api/cmo-db, and POST /api/cmo-db?name=DB3K_519.db3 with the file as the body, Content-Type: application/octet-stream.

The db.* tools read it, in every run mode; they never touch the game:

Tool	Returns
db.search	Classes and components whose name holds every word of query, filtered by kinds and country, with their DB IDs
db.platform	One class by kind and dbid (a unit's dbid): its figures, sensors, mounts with their weapons, magazines, comms, propulsion, fuel, facilities, codes, loadouts (aircraft) and, when asked, signatures. class_name, the unit's class_name, guards against a DB ID of another kind
db.weapon	One weapon: ranges, speeds, altitudes, probabilities of kill, seekers, warheads, targets, and on request CMO's default weapon release per target
db.sensor	One sensor: ranges, altitudes, radar, ESM, ECM, sonar and optical figures, capabilities and frequencies
db.loadout	One aircraft loadout (a unit's loadout_id): role, ready times, combat radius, weapons and loads, and the aircraft that carry it
db.carriers	The classes that carry a weapon (on a mount, in a magazine or a loadout) or a sensor

loadouts.compatibility is a composite of forces.inspect (the unit's dbid and class_name, from the game) and db.platform (its class's loadouts). Results name their units (_nm, _m, _kts, _kg, _tons, _min, _s, _pct) and leave out fields that are 0, false or empty unless all_fields is set. Every run records the loaded file's name, build and SHA-256; comparing runs lists a changed database; and a handoff export names the database its runs read, without the file. Algorithm files get the file's path as params["cmo_db"]; algorithms/cmo_db.py holds the map of how CMO's tables connect.

Dry-run: reading the game

Choose Dry-run in the run bar to run a flow against the game in CMO: tools with Lua read the game, and writes are only logged, as in mock mode. Dry-run is offered once a side is chosen in Settings and the socket isn't paused. Only one dry-run or live run, or batch of them, goes at a time, and Stop works as for any run. Don't run CMOnet against the same game.

• Start. Before the flow, the run checks the proxy's HEALTH and then sends tester.ping with the chosen side, so the wrapper finds the side in the game. The run records the side's GUID and name. If the proxy isn't healthy, or the side isn't found, the run fails before any flow tool runs. Another connected client, such as CMOnet, is a warning in the trace.

• Tools. Each tool is answered by the first of these that it has:

  1. a Python implementation, which never touches the game;
  2. a Lua file (reads only): sent to CMO with the side's GUID, and marked CMO in the trace, which holds the Lua sent, the raw reply, the decoded result and the socket timings;
  3. the test step's mock data, marked mock (no Lua);
  4. otherwise the call fails with "No Lua for this tool". Agents aren't offered such a tool at all, so only tool nodes meet this.

  Writes are logged, not sent, exactly as in mock mode. A read after a logged write is labeled: the game doesn't show logged writes.

• The side. Every tool's wrapper finds the side again before its Lua runs. If the side has left the game, the tool's Lua doesn't run and the run stops, saying why. A read that fails on the socket, such as a timeout, is an error the flow sees; the pause policy applies as for any request.

• Result schemas. A result that doesn't fit its tool's result schema is marked result schema in the trace, in every mode.

• Lua is fixed at run start. A run uses each tool's Lua file as it was when the run, or its batch, started, and keeps a copy in its record.

Live: writing to the game

Choose Live in the run bar to let a flow change the game: writes with Lua are sent to CMO, and reads come from CMO as in a dry-run. While Live is chosen, or a live run or batch goes, a red LIVE banner shows under the top bar. Live is offered on the same terms as Dry-run, and starts the same way: HEALTH, then the side check, before any flow tool.

• Writes with Lua are sent once, as write requests, and their results are recorded, marked CMO in the trace. A write without Lua fails with "No Lua for this tool": live runs never log a write instead of sending it. Agents aren't offered such a write in a live run. The exception is a tool tagged log-only, such as CMOnet's application writes: it records something outside the game, so every mode logs it, marked log-only tool in a live run's trace.
• A write's own checks come back to the flow as its error: each write first checks that its unit belongs to the run's side and can take orders, and changes nothing when it doesn't. Such an error is a valid reply, so the socket carries on.
• A write without a valid reply, such as a timeout or a proxy error reply, has an unknown outcome: CMO may have made it. The run stops with that reason, and the pause policy applies; by default the socket pauses, and timed Resume after such a write needs its own setting. Nothing is retried. Check the game before you resume or run again.
• Composites send their steps as their own tools and stop at the first failed step.

The first write tools with Lua are navigation.set_speed and navigation.set_altitude (aircraft only, above mean sea level). They give orders as a player does: navigation.set_speed sets the speed a unit aims for (its manual speed), never its current speed, which would jump it there, and navigation.set_altitude the altitude an aircraft climbs or descends to. CMO applies an order on its next step, so a write tool can name a read-back, a read tool with Lua, in its read_back, with arguments taken from the write's. After the write is sent, the tester sends the read-back as a request of its own, and the write's result is both results together, the read's fields winning; a failed read-back is read_back_error, and nothing is retried. A logged write isn't read back. Both write tools are read back (by tester.check_speed and tester.check_altitude): applied (and ok) is true only when the unit then aims for what was asked, and the note says when the game set another speed or holds the unit at one. New from template, for a write tool, starts its Lua with the side check.

Save live reads as a test step, on a finished dry-run or live run in the Runs tab, saves what the run read as a new step: in a new test, or at the end of an existing one. The step gets the run's instruction, event and expected calls, and one mock read per call the run answered from CMO, or from mock data for a tool without Lua, so a mock run of the step sees the same results. A repeated call keeps its first result, as mock lookup would. A new test starts with the run's starting memory. The API is POST /api/runs/{run_id}/capture with {"name": ...} or {"test_id": ...}.

Try tool, in the Tools tab, sends a saved read tool's Lua to CMO once, with arguments you type and the side chosen in Settings, and shows the result, the Lua sent and the raw reply. It sends only reads, and only the saved tool and Lua file, so save your edits first. It is a real request: paced, and counted toward the pause. The API is POST /api/tools/{tool_id}/try with {"arguments": {...}}.

The first game tools with Lua, following tools/README.md, are scenario.clock, scenario.context, forces.roster (the side's own units), contacts.list (its detected contacts, as it sees them) and forces.inspect (one own unit). Each has a result schema and example.

Read tools with Lua for CMOnet's search, geometry and entity-detail operations followed, so the CMOnet baseline can see the game in a live run (2026-09-25, after the live run 20260925-213625-fbd7 failed on tools without Lua): entities.search, forces.search, contacts.search, hosts.search, entities.resolve, geo.within_radius, geo.nearest, geo.distance_bearing, geo.distance_matrix, contacts.nearest_hostile, contacts.threats_to, entities.inspect_many, contacts.inspect, forces.position, forces.damage_fuel, readiness.inspect, readiness.inspect_many, navigation.route, forces.hosted_units, hosting.hosted_units, missions.search, missions.inspect and missions.inspect_many. They read what the side knows, compute distances, bearings and closest approach in the Lua, and each file's header says what it returns and which arguments it ignores; tools/README.md has the details.

Diagnostic log

The service keeps its own log, separate from run traces, in the data folder's logs\tester.jsonl. Each line is one JSON entry: the UTC time, the level, the component and event, a message, the request, run, batch, node or tool IDs it concerns, and details. The log stays on this computer; nothing is sent anywhere, and stored API keys are removed from every entry.

Settings → Diagnostics chooses the level, which applies at once:

Level	Adds
ERROR	unexpected errors, with their stack traces
WARNING	failed nodes and tool calls, model errors, refused requests
INFO (default)	the service starting; settings changes; every change made through the API, such as saves, deletes, runs and stops; runs and batches starting and ending; one line for each finished node, tool call and model request
DEBUG	node starts, tool calls as they are made, branch routes, token counts
TRACE	every request, including reads and the checks the page makes as you type

With At TRACE, also copy payloads ticked, TRACE entries also hold prompts, model replies, tool arguments and results, and node outputs. Otherwise an entry names the run, node and event number whose trace holds them. A new file starts at 10 MB and five older files are kept (tester.jsonl.1 is the newest); both are settings. Recent entries, in the same section, shows the newest entries at a chosen level, optionally for one component or one ID, read through GET /api/diagnostics/logs. Logs are never part of an export.

Layout

Path	Contents
src/llm_tester/	Python service and workflow runner
src/llm_tester/web/index.html	The web UI: one HTML file with embedded CSS and JavaScript
src/llm_tester/cmo/	The CMO socket client, the reply envelope, HEALTH and the Lua wrapper
src/llm_tester/export/	The handoff export and FLOW.md
THIRD_PARTY_NOTICES.md	The bundled graph library (Drawflow): version, source, hashes and license
tools/	Tool definitions and tool Lua files
algorithms/	Staff and tactical algorithm Python files
projects/	Projects and their tests
tests/	Automated tests
docs/	The design brief, the implementation phases and the flow diagrams

Run records, batches, settings, credentials, import reports, the diagnostic log, the CMO socket's state, handoff exports, the CMO database and the trash are kept outside the repository, in %LOCALAPPDATA%\llm_tester. Set LLM_TESTER_DATA_DIR to use another folder, and LLM_TESTER_PROJECTS_DIR or LLM_TESTER_TOOLS_DIR to keep projects or tools somewhere other than projects/ or tools/.

Trash

Deleting a project, a test or a tool moves it to the data folder's trash/, named with the UTC time it was deleted. Nothing empties the trash.

• A project is a folder, such as trash/20260924-101500-alpha/. To restore it, move it back into projects/ and rename it to the project's ID, the part after the time (alpha).
• A test is a file, such as trash/20260924-101500-alpha-smoke.json. To restore it, move it into its project's tests/ folder and rename it to the id inside the file, plus .json (smoke.json).
• A tool is a file, such as trash/20260924-101500-forces.inspect.json, with its Lua file beside it if it had one. To restore it, move it into tools/ and rename it to the tool's ID, plus .json (forces.inspect.json), and the same for the Lua file.

If that ID is in use again, restore under another name and change the id inside the file to match.

Tests

.venv\Scripts\python -m pip install -r requirements-dev.txt
.venv\Scripts\python -m pytest