Runs and results

Every run is graded and kept with everything your agent did, so you can see why it failed, fix it, and check that the fix helped.

#The result

Finishing a run returns its result. sw run prints it; in Python it is a Result:

failed  9 calls  https://acme.app.stateweave.net/#/runs/r-aee3438d
  total 1.00 != expected 349.10
  orders ['NG-1'] != expected ['NG-4824', 'NG-4872', 'NG-4881'] (missing ['NG-4824', 'NG-4872', 'NG-4881'], extra ['NG-1'])

The run's record from the API also has state, per app how many records were added, changed and removed, and the agent label, meta and timing.

#The run's page

• Verdict. Passed or failed, and the reasons.
• Calls. Every call your agent made, in order, across all the apps: method, path, status, and the full request and response.
• Faults. Each fault the test injected that acted, on the call it acted on.
• Final state. What changed in each app from the start: records added, changed and removed, field by field. Download any app's end state, or open it in the app's web client to look around as it was left.

#Compare agent versions

The Runs board shows every test against every agent label you've run, so a new version's wins and regressions are one look away. To compare fairly:

• Give each version its own --agent label, such as support-bot@0.4.
• Put what changed in meta: the model, the prompt version, a commit.
• Run each test more than once. Agents vary from run to run, and so do tests with random faults.

#Run again from the browser

When a run fails, you often want to fix the agent and try the same test again right away. Leave a worker running on your machine:

sw worker --agent my-agent@0.3 -- python my_agent.py

Then press Run again on any run's page. The run is queued, your worker takes it, runs your command with it exactly as sw run does, and hands in the answer. Your command starts afresh for every run, so an edit to your agent counts from the next run on, without restarting the worker.

• The run takes the worker's --agent label. Its meta records the command, the machine's name and, in a git checkout, the commit (with a note when there are uncommitted changes).
• With several workers connected, the button lets you pick one. A worker counts as connected for a minute after it last asked for work.
• A queued run that no worker takes within 10 minutes is cancelled.
• --once drives one run and exits, with sw run's exit code. Ctrl-C stops the worker.

#Run in CI

sw run exits non-zero when a run doesn't pass, so a test is a CI step. Store the key as a secret and pass the address and key in the environment:

export STATEWEAVE_URL=https://acme.app.stateweave.net
export STATEWEAVE_API_KEY=$STATEWEAVE_KEY_SECRET   # from your CI's secret store
pip install stateweave-sdk

status=0
for t in t-bde537 t-0e4243 t-753b29; do
  sw run "$t" --agent "my-agent@$GIT_SHA" -- python my_agent.py || status=1
done
exit $status

Use a key made for CI, so you can revoke it on its own.

#Give up a run

A run your harness can't finish should be given up rather than left to time out, so its apps stop at once:

sw cancel r-de238618

run.cancel()  # the with block does this for you when your agent raises

You can also cancel a running or queued run on its page.