Run your agent
Every run gives your agent fresh copies of a sandbox's apps, made from its latest version. Your agent runs where it always runs and reaches them over HTTPS; StateWeave records every call.
| Run | For | Judged |
|---|---|---|
| Twin run | Developing: run your agent, look, reset, run it again. | By you, for example with a task's checks |
| Test run | A test you uploaded: every run is graded and kept. | By the test, when your harness finishes the run |
Twin runs
A twin run is fresh copies of a sandbox's apps with no test. Open one, run your agent on it, judge it, and reset it to try again: each reset puts the apps back as the version holds them. A twin run is never graded and injects no faults.
From a shell, sw run with a sandbox's id opens one around your command, judges it with a task's checks
when you give them, and stops it when the command exits:
sw run p-7d21c4 --agent my-agent@dev --prompt "$(cat kickoff/instruction.md)" --checks kickoff -- python my_agent.pyYour command gets the same variables as for a test (below), with the request you gave. See sw run for its options and output. In Python, the same with resets in between:
import json
import stateweave
client = stateweave.Client()
checks = json.load(open("kickoff/tests/checks.json"))
with client.open_twins("p-7d21c4") as twins:
for attempt in range(3):
my_agent(open("kickoff/instruction.md").read(), twins.apps)
result = twins.check(checks)
print(attempt + 1, "passed" if result["passed"] else "failed")
if result["passed"]:
break
twins.reset() # the next attempt starts from the version againtwins.apps holds, per app, its base_url, endpoints,
credentials and MCP server, as in the run file. twins.env holds
the same as environment variables, ready to pass to a subprocess: STATEWEAVE_<APP>_URL,
STATEWEAVE_<APP>_MCP_URL and STATEWEAVE_<APP>_<KIND>_TOKEN.
| Method | What it does |
|---|---|
check(checks, answer=None) | Judges the current attempt by a task's checks,
as a graded run would: {"passed", "results": [{"index", "name", "ok", "detail", "actual"}]}. See
Checks. |
call(app, method, path, query=, body=, headers=, credential=) | One request
to an app, as a checker: it is not counted as your agent's. credential is a label or a kind such as
"user"; Shopify takes its token in headers. |
reset(apps=None) | Puts the apps back as they opened. The calls after it count as the next attempt. |
trace(attempt=None) | Every call your agent made, whole, and the faults that acted. |
download(directory) | Writes the whole record to a folder:
trace.jsonl, changes.json, task.json, faults.json. |
stop() | Stops the apps. Leaving the with block does it for
you. |
A twin run stays up for an hour unless you stop it (ttl=, in seconds, up to a day), and ends with
status ended, keeping its final state. Each API key can keep three going at once; a fourth answers HTTP
429. persona= picks whose credentials you get, by handle. Over HTTP, start one with
POST /api/v1/runs and {"sandbox": "p-7d21c4"}; see Twin
runs.
Test runs
A test run is a run of a test in your workspace. It hands your agent three things: the request, the apps' addresses and credentials, and a place to put its answer. When your harness finishes it, StateWeave grades it by the test's checks and keeps it with every call.
- Start. Your harness starts a run of a test. StateWeave copies the test's apps from the sandbox's latest version, starts them, and answers with the request and each app's connection details. This takes a second or two.
- Work. Your agent calls the apps. Every call is recorded on the run.
- Finish. Your harness hands in the answer (nothing, for an action test). StateWeave grades the run, stops the apps and returns the result.
A run that is never finished ends as timed_out when its time limit passes, and one your harness gives
up ends as cancelled. Neither is graded.
With sw run
sw run does the start and finish for you around any command. The command can be written in any
language; it only needs to read environment variables and, for an answer test, write a file.
sw run t-bde537 --agent my-agent@0.1 -- python my_agent.pyEverything after -- is your command, run as is (no shell, unless you call one). It gets these
variables on top of your own environment:
| Variable | What it holds |
|---|---|
STATEWEAVE_PROMPT | The request, as a person in the company would put it. Give it to your agent as the task. |
STATEWEAVE_KIND | answer (find something out) or action
(change the apps). |
STATEWEAVE_ANSWER_FORMAT | For an answer test, an example of the JSON value to answer with. Empty for an action test. |
STATEWEAVE_RUN_FILE | Path to the whole run as JSON, including each app's
base_url, endpoints, credentials and MCP server.
See below. |
STATEWEAVE_MCP_CONFIG | Path to {"mcpServers": {…}} for the apps that have
an MCP server, in the format Claude Code and most MCP clients take. |
STATEWEAVE_<APP>_URL, STATEWEAVE_<APP>_MCP_URL,
STATEWEAVE_<APP>_<KIND>_TOKEN | Each app's address, its MCP server and
the persona's tokens, such as STATEWEAVE_SLACK_BOT_TOKEN: map them onto the names your code already
reads. See Apps and credentials. |
STATEWEAVE_ANSWER_FILE | Path your agent writes its answer to: JSON, or plain text (handed in as a string). Leave it alone for an action test. |
STATEWEAVE_TRANSCRIPT_FILE | Path your agent may write its own transcript to; it is handed in with the answer. See below. |
STATEWEAVE_RUN | The run's id, such as r-de238618. |
The run is graded when your command exits, whatever its exit code: an agent that did the work and then crashed is
still graded on what it did. Press Ctrl+C to give the run up instead. sw run
itself exits with 0 when the run passed and 1 otherwise. A command that doesn't exist is
refused before any run starts.
A minimal agent
The skeleton below reads what a run hands over and answers. Replace the middle with your agent.
import json
import os
run = json.load(open(os.environ["STATEWEAVE_RUN_FILE"]))
prompt = os.environ["STATEWEAVE_PROMPT"]
for name, app in run["apps"].items():
print(name, app["base_url"], list(app["credentials"]))
answer = ... # your agent works here: call the apps, decide, act
if os.environ["STATEWEAVE_KIND"] == "answer":
with open(os.environ["STATEWEAVE_ANSWER_FILE"], "w") as f:
json.dump(answer, f)The run file
The run file holds the run as the API returns it. The parts your agent needs:
{
"id": "r-de238618",
"test": "t-bde537",
"kind": "answer",
"prompt": "Ethan Smith looks like he's been buying from us under two different email addresses. Can you work out how much he has actually paid us in total across both of his accounts? …",
"answer_format": "{\"total_paid_usd\": 123.45, \"orders\": [\"NG-1234\", \"NG-1235\"]}",
"persona": {"name": "Priya Castellano", "handle": "priya.castellano", "title": "Customer Operations Coordinator"},
"expires_at": 1793670877.43,
"apps": {
"shopify": {
"base_url": "https://t-q8m2xk.app.stateweave.net",
"endpoints": {
"admin_graphql": "https://t-q8m2xk.app.stateweave.net/admin/api/2026-10/graphql.json",
"token_endpoint": "https://t-q8m2xk.app.stateweave.net/admin/oauth/access_token",
"mcp": "https://t-q8m2xk.app.stateweave.net/mcp"
},
"credentials": {
"app:Northstar Ops Coordinator": "shpat_…",
"setup:priya.castellano": "setup_…"
},
"mcp": {
"url": "https://t-q8m2xk.app.stateweave.net/mcp",
"headers": {"Authorization": "Bearer setup_…"}
}
}
}
}| Field | Description |
|---|---|
prompt | The request. Pass it to your agent word for word. |
kind | answer or action. |
answer_format | For an answer test, an example value in the expected shape. Its values are placeholders, never the answer. |
persona | Whom the agent acts for: name, handle and job
title. The user: credentials are this person's. |
apps.<app>.base_url | The app's own address for this run. Only this run uses it. |
apps.<app>.endpoints | The app's API roots and other addresses by name. See Apps and credentials for each app's. |
apps.<app>.credentials | Tokens by label, kind:name. See
Credentials. |
apps.<app>.mcp | {"url", "headers"} of the app's MCP server, or
null when it has none. |
expires_at | When the run times out, in Unix seconds. |
Credentials
Credential labels are kind:name. The kind says whose token it is and what it opens:
| Kind | What it is | Apps |
|---|---|---|
user:<handle> | The persona's own OAuth or user token. | Slack
(xoxp-), Gmail, Google Calendar, Sheets and Drive (ya29.), Salesforce, Zendesk,
WhatsApp (EAA…), Microsoft Teams (a delegated Graph token), Stripe (the secret key,
sk_test_…), HubSpot (in a generated company, an OAuth token), GitHub (a classic personal access
token, ghp_…) |
bot:<name> | The company's bot token. | Slack (xoxb-),
WhatsApp (a system user's token), Microsoft Teams (the bot's Bot Connector token) |
app:<name> | An app's own credential. | Shopify Admin API token
(shpat_), Slack app-level token (xapp-), Salesforce connected app's client id,
WhatsApp app access token, Microsoft Teams' public client id and its bot's
<app id>:<client secret>, HubSpot's Service Key (pat-…) |
mcp:<handle> | The persona's token for the app's MCP server. | Salesforce, Microsoft Teams, Stripe (a restricted key, rk_test_…), HubSpot (an
MCP connector token) |
setup:<handle> | The persona's token for the app's MCP server. | Shopify |
A run never hands over a colleague's mailbox or a refresh token, and no client secret except inside WhatsApp's app
access token (<app id>|<app secret>, as Meta writes it) and Microsoft Teams' bot credential
(<app id>:<client secret>, which a bot built on the Teams SDK signs in with): your agent gets what a deployed
agent acting for the persona would get. With the Python SDK, run.credential("slack", "user") returns the
first credential of a kind.
Answers
For an answer test, answer with a JSON value in the shape answer_format shows: an object as an object,
numbers as numbers, the same field names. Plain text is accepted too and handed in as a string, so a test that asks
for an object fails it.
For an action test there is nothing to hand in. The result is what your agent changed in the apps, and what it left alone: a test can fail because the agent changed something it shouldn't have.
From Python
Inside a Python harness, use the SDK instead of sw run. The with block gives the run up if
your agent raises or the block ends without finish(), so no run is left waiting for its time limit.
import stateweave
client = stateweave.Client() # STATEWEAVE_URL and STATEWEAVE_API_KEY, or what sw login kept
with client.start("t-bde537", agent="my-agent@0.1", meta={"model": "my-model", "prompt": "v7"}) as run:
token = run.credential("shopify", "app")
graphql = run.apps["shopify"]["endpoints"]["admin_graphql"]
answer = my_agent(run.prompt, graphql, token)
result = run.finish(answer)
print(result.status, result.reasons, result.url)See the Python SDK reference.
From any language, over HTTP
The same flow is three calls with your API key. See the REST API reference.
# start: the response is the run file above
curl -s https://acme.app.stateweave.net/api/v1/runs \
-H "Authorization: Bearer $STATEWEAVE_API_KEY" -H "Content-Type: application/json" \
-d '{"test": "t-bde537", "agent": "my-agent@0.1"}'
# … your agent works …
# finish: hand in the answer and get the result
curl -s https://acme.app.stateweave.net/api/v1/runs/r-de238618/finish \
-H "Authorization: Bearer $STATEWEAVE_API_KEY" -H "Content-Type: application/json" \
-d '{"answer": {"total_paid_usd": 349.10, "orders": ["NG-4824", "NG-4872", "NG-4881"]}}'Time limits
A run has 30 minutes by default. Set another limit, from 60 seconds to 24 hours, with --timeout (in
seconds) or timeout= in Python. A run still going at its limit ends as timed_out and its
apps stop.
Labels and metadata
--agent (agent= in Python) is the label your runs are compared by on the Runs board. Use
one per version of your agent, such as support-bot@0.4, up to 80 characters. Add meta, a
JSON object of up to 4,000 characters, to record anything else with the run: the model, the prompt version, a git
commit. sw run records your command line in it.
Many tests in a row
Run a sandbox's tests one after another from a shell, or from Python:
for t in $(sw task ls --env p-041ba8 | cut -d' ' -f1); do
sw run "$t" --agent my-agent@0.2 -- python my_agent.py
doneimport stateweave
client = stateweave.Client()
for test in client.tests(environment="p-041ba8"):
with client.start(test["id"], agent="my-agent@0.2") as run:
result = run.finish(my_agent(run.prompt, run.apps))
print(test["id"], result.status)Each run gets its own apps, so you can also run tests in parallel from separate processes, up to your workspace's limit on runs going at once. Past it, starting a run answers HTTP 429 until one finishes.
The result
Finishing a run returns its result. sw run prints it; in Python it is a Result:
| Field | Description |
|---|---|
status | passed, failed, timed_out,
cancelled, interrupted or error. See
Run statuses. |
passed | true or false once graded, null
otherwise. |
reasons | Why, in a few short lines: what was expected and what was found. |
calls | How many calls your agent made to the apps. |
url | The run's page in your workspace. |
failed 9 calls https://acme.app.stateweave.net/#/runs/r-aee3438d
total 1.00 != expected 349.10
orders ['NG-1'] != expected ['NG-4824', 'NG-4872', 'NG-4881'] (missing ['NG-4824', 'NG-4872', 'NG-4881'], extra ['NG-1'])The run's record from the API also has state, per app how many records were added, changed and
removed, and the agent label, meta and timing.
The run's page
- Verdict. Passed or failed, and the reasons.
- Calls. Every call your agent made, in order, across all the apps: method, path, status, and the full request and response.
- Faults. Each fault the test injected that acted, on the call it acted on.
- Final state. How many records each app gained, changed and lost from the start, and a button that opens the end state in the app's web client to look around as it was left.
- Verdicts of your own. When your own judge kept a verdict on the run (
run.verdict()in Python, orPOST /api/v1/runs/<id>/verdict), its verdict, criteria, reasons and evidence, beside the test's own result, which they don't change. - Transcript. When the harness handed in the agent's transcript, what the agent said and did, folded, one entry per line for JSON lines.
Hand in your agent's transcript
The calls show what your agent did to the apps; its transcript shows why. Hand it in with the answer and it is kept with the run, in whatever format your agent writes:
sw run t-0e4243 --agent claude-code -- sh -c 'claude -p "$STATEWEAVE_PROMPT" \
--mcp-config "$STATEWEAVE_MCP_CONFIG" --strict-mcp-config \
--output-format stream-json --verbose > "$STATEWEAVE_TRANSCRIPT_FILE"'result = run.finish(answer, transcript=messages, transcript_format="claude-agent-sdk")sw run sends whatever the command wrote to STATEWEAVE_TRANSCRIPT_FILE. See
Hand in a transcript for the limits.
Compare agent versions
The Runs board shows every test against every agent label you've run, so a new version's wins and regressions are one look away. To compare fairly:
- Give each version its own
--agentlabel, such assupport-bot@0.4. - Put what changed in
meta: the model, the prompt version, a commit. - Run each test more than once. Agents vary from run to run, and so do tests with random faults.
Run again from the browser
When a run fails, you often want to fix the agent and try the same test again right away. Leave a worker running on your machine:
sw worker --agent my-agent@0.3 -- python my_agent.pyThen press Run again on any run's page. The run is queued, your worker takes it, runs your command
with it exactly as sw run does, and hands in the answer. Your command starts afresh for every run, so an
edit to your agent counts from the next run on, without restarting the worker.
- The run takes the worker's
--agentlabel. Itsmetarecords the command, the machine's name and, in a git checkout, the commit (with a note when there are uncommitted changes). - With several workers connected, the button lets you pick one. A worker counts as connected for a minute after it last asked for work.
- A queued run that no worker takes within 10 minutes is cancelled.
--oncedrives one run and exits, withsw run's exit code. Ctrl-C stops the worker.
Run in CI
sw run exits non-zero when a run doesn't pass, so a test is a CI step. Store the key as a secret and
pass the address and key in the environment:
export STATEWEAVE_URL=https://acme.app.stateweave.net
export STATEWEAVE_API_KEY=$STATEWEAVE_KEY_SECRET # from your CI's secret store
pip install stateweave-sdk
status=0
for t in t-bde537 t-0e4243 t-753b29; do
sw run "$t" --agent "my-agent@$GIT_SHA" -- python my_agent.py || status=1
done
exit $statusUse a key made for CI, so you can revoke it on its own.
Give up a run
A run your harness can't finish should be given up rather than left to time out, so its apps stop at once:
sw cancel r-de238618run.cancel() # the with block does this for you when your agent raisesYou can also cancel a running or queued run on its page.