stateweave Docs
Tasks and runs

Run your agent

Every run gives your agent fresh copies of a sandbox's apps, made from its latest version. Your agent runs where it always runs and reaches them over HTTPS; StateWeave records every call.

RunForJudged
Twin runDeveloping: run your agent, look, reset, run it again.By you, for example with a task's checks
Test runA test you uploaded: every run is graded and kept.By the test, when your harness finishes the run

Twin runs

A twin run is fresh copies of a sandbox's apps with no test. Open one, run your agent on it, judge it, and reset it to try again: each reset puts the apps back as the version holds them. A twin run is never graded and injects no faults.

From a shell, sw run with a sandbox's id opens one around your command, judges it with a task's checks when you give them, and stops it when the command exits:

Shell
sw run p-7d21c4 --agent my-agent@dev --prompt "$(cat kickoff/instruction.md)" --checks kickoff -- python my_agent.py

Your command gets the same variables as for a test (below), with the request you gave. See sw run for its options and output. In Python, the same with resets in between:

Python
import json
import stateweave

client = stateweave.Client()
checks = json.load(open("kickoff/tests/checks.json"))

with client.open_twins("p-7d21c4") as twins:
    for attempt in range(3):
        my_agent(open("kickoff/instruction.md").read(), twins.apps)
        result = twins.check(checks)
        print(attempt + 1, "passed" if result["passed"] else "failed")
        if result["passed"]:
            break
        twins.reset()                      # the next attempt starts from the version again

twins.apps holds, per app, its base_url, endpoints, credentials and MCP server, as in the run file. twins.env holds the same as environment variables, ready to pass to a subprocess: STATEWEAVE_<APP>_URL, STATEWEAVE_<APP>_MCP_URL and STATEWEAVE_<APP>_<KIND>_TOKEN.

MethodWhat it does
check(checks, answer=None)Judges the current attempt by a task's checks, as a graded run would: {"passed", "results": [{"index", "name", "ok", "detail", "actual"}]}. See Checks.
call(app, method, path, query=, body=, headers=, credential=)One request to an app, as a checker: it is not counted as your agent's. credential is a label or a kind such as "user"; Shopify takes its token in headers.
reset(apps=None)Puts the apps back as they opened. The calls after it count as the next attempt.
trace(attempt=None)Every call your agent made, whole, and the faults that acted.
download(directory)Writes the whole record to a folder: trace.jsonl, changes.json, task.json, faults.json.
stop()Stops the apps. Leaving the with block does it for you.

A twin run stays up for an hour unless you stop it (ttl=, in seconds, up to a day), and ends with status ended, keeping its final state. Each API key can keep three going at once; a fourth answers HTTP 429. persona= picks whose credentials you get, by handle. Over HTTP, start one with POST /api/v1/runs and {"sandbox": "p-7d21c4"}; see Twin runs.

Test runs

A test run is a run of a test in your workspace. It hands your agent three things: the request, the apps' addresses and credentials, and a place to put its answer. When your harness finishes it, StateWeave grades it by the test's checks and keeps it with every call.

  1. Start. Your harness starts a run of a test. StateWeave copies the test's apps from the sandbox's latest version, starts them, and answers with the request and each app's connection details. This takes a second or two.
  2. Work. Your agent calls the apps. Every call is recorded on the run.
  3. Finish. Your harness hands in the answer (nothing, for an action test). StateWeave grades the run, stops the apps and returns the result.

A run that is never finished ends as timed_out when its time limit passes, and one your harness gives up ends as cancelled. Neither is graded.

With sw run

sw run does the start and finish for you around any command. The command can be written in any language; it only needs to read environment variables and, for an answer test, write a file.

Shell
sw run t-bde537 --agent my-agent@0.1 -- python my_agent.py

Everything after -- is your command, run as is (no shell, unless you call one). It gets these variables on top of your own environment:

VariableWhat it holds
STATEWEAVE_PROMPTThe request, as a person in the company would put it. Give it to your agent as the task.
STATEWEAVE_KINDanswer (find something out) or action (change the apps).
STATEWEAVE_ANSWER_FORMATFor an answer test, an example of the JSON value to answer with. Empty for an action test.
STATEWEAVE_RUN_FILEPath to the whole run as JSON, including each app's base_url, endpoints, credentials and MCP server. See below.
STATEWEAVE_MCP_CONFIGPath to {"mcpServers": {…}} for the apps that have an MCP server, in the format Claude Code and most MCP clients take.
STATEWEAVE_<APP>_URL, STATEWEAVE_<APP>_MCP_URL, STATEWEAVE_<APP>_<KIND>_TOKENEach app's address, its MCP server and the persona's tokens, such as STATEWEAVE_SLACK_BOT_TOKEN: map them onto the names your code already reads. See Apps and credentials.
STATEWEAVE_ANSWER_FILEPath your agent writes its answer to: JSON, or plain text (handed in as a string). Leave it alone for an action test.
STATEWEAVE_TRANSCRIPT_FILEPath your agent may write its own transcript to; it is handed in with the answer. See below.
STATEWEAVE_RUNThe run's id, such as r-de238618.

The run is graded when your command exits, whatever its exit code: an agent that did the work and then crashed is still graded on what it did. Press Ctrl+C to give the run up instead. sw run itself exits with 0 when the run passed and 1 otherwise. A command that doesn't exist is refused before any run starts.

A minimal agent

The skeleton below reads what a run hands over and answers. Replace the middle with your agent.

my_agent.py
import json
import os

run = json.load(open(os.environ["STATEWEAVE_RUN_FILE"]))
prompt = os.environ["STATEWEAVE_PROMPT"]

for name, app in run["apps"].items():
    print(name, app["base_url"], list(app["credentials"]))

answer = ...  # your agent works here: call the apps, decide, act

if os.environ["STATEWEAVE_KIND"] == "answer":
    with open(os.environ["STATEWEAVE_ANSWER_FILE"], "w") as f:
        json.dump(answer, f)

The run file

The run file holds the run as the API returns it. The parts your agent needs:

run.json (shortened)
{
  "id": "r-de238618",
  "test": "t-bde537",
  "kind": "answer",
  "prompt": "Ethan Smith looks like he's been buying from us under two different email addresses. Can you work out how much he has actually paid us in total across both of his accounts? …",
  "answer_format": "{\"total_paid_usd\": 123.45, \"orders\": [\"NG-1234\", \"NG-1235\"]}",
  "persona": {"name": "Priya Castellano", "handle": "priya.castellano", "title": "Customer Operations Coordinator"},
  "expires_at": 1793670877.43,
  "apps": {
    "shopify": {
      "base_url": "https://t-q8m2xk.app.stateweave.net",
      "endpoints": {
        "admin_graphql": "https://t-q8m2xk.app.stateweave.net/admin/api/2026-10/graphql.json",
        "token_endpoint": "https://t-q8m2xk.app.stateweave.net/admin/oauth/access_token",
        "mcp": "https://t-q8m2xk.app.stateweave.net/mcp"
      },
      "credentials": {
        "app:Northstar Ops Coordinator": "shpat_…",
        "setup:priya.castellano": "setup_…"
      },
      "mcp": {
        "url": "https://t-q8m2xk.app.stateweave.net/mcp",
        "headers": {"Authorization": "Bearer setup_…"}
      }
    }
  }
}
FieldDescription
promptThe request. Pass it to your agent word for word.
kindanswer or action.
answer_formatFor an answer test, an example value in the expected shape. Its values are placeholders, never the answer.
personaWhom the agent acts for: name, handle and job title. The user: credentials are this person's.
apps.<app>.base_urlThe app's own address for this run. Only this run uses it.
apps.<app>.endpointsThe app's API roots and other addresses by name. See Apps and credentials for each app's.
apps.<app>.credentialsTokens by label, kind:name. See Credentials.
apps.<app>.mcp{"url", "headers"} of the app's MCP server, or null when it has none.
expires_atWhen the run times out, in Unix seconds.

Credentials

Credential labels are kind:name. The kind says whose token it is and what it opens:

KindWhat it isApps
user:<handle>The persona's own OAuth or user token.Slack (xoxp-), Gmail, Google Calendar, Sheets and Drive (ya29.), Salesforce, Zendesk, WhatsApp (EAA…), Microsoft Teams (a delegated Graph token), Stripe (the secret key, sk_test_…), HubSpot (in a generated company, an OAuth token), GitHub (a classic personal access token, ghp_…)
bot:<name>The company's bot token.Slack (xoxb-), WhatsApp (a system user's token), Microsoft Teams (the bot's Bot Connector token)
app:<name>An app's own credential.Shopify Admin API token (shpat_), Slack app-level token (xapp-), Salesforce connected app's client id, WhatsApp app access token, Microsoft Teams' public client id and its bot's <app id>:<client secret>, HubSpot's Service Key (pat-…)
mcp:<handle>The persona's token for the app's MCP server. Salesforce, Microsoft Teams, Stripe (a restricted key, rk_test_…), HubSpot (an MCP connector token)
setup:<handle>The persona's token for the app's MCP server. Shopify

A run never hands over a colleague's mailbox or a refresh token, and no client secret except inside WhatsApp's app access token (<app id>|<app secret>, as Meta writes it) and Microsoft Teams' bot credential (<app id>:<client secret>, which a bot built on the Teams SDK signs in with): your agent gets what a deployed agent acting for the persona would get. With the Python SDK, run.credential("slack", "user") returns the first credential of a kind.

Answers

For an answer test, answer with a JSON value in the shape answer_format shows: an object as an object, numbers as numbers, the same field names. Plain text is accepted too and handed in as a string, so a test that asks for an object fails it.

For an action test there is nothing to hand in. The result is what your agent changed in the apps, and what it left alone: a test can fail because the agent changed something it shouldn't have.

From Python

Inside a Python harness, use the SDK instead of sw run. The with block gives the run up if your agent raises or the block ends without finish(), so no run is left waiting for its time limit.

Python
import stateweave

client = stateweave.Client()  # STATEWEAVE_URL and STATEWEAVE_API_KEY, or what sw login kept

with client.start("t-bde537", agent="my-agent@0.1", meta={"model": "my-model", "prompt": "v7"}) as run:
    token = run.credential("shopify", "app")
    graphql = run.apps["shopify"]["endpoints"]["admin_graphql"]
    answer = my_agent(run.prompt, graphql, token)
    result = run.finish(answer)

print(result.status, result.reasons, result.url)

See the Python SDK reference.

From any language, over HTTP

The same flow is three calls with your API key. See the REST API reference.

# start: the response is the run file above
curl -s https://acme.app.stateweave.net/api/v1/runs \
  -H "Authorization: Bearer $STATEWEAVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"test": "t-bde537", "agent": "my-agent@0.1"}'

# … your agent works …

# finish: hand in the answer and get the result
curl -s https://acme.app.stateweave.net/api/v1/runs/r-de238618/finish \
  -H "Authorization: Bearer $STATEWEAVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"answer": {"total_paid_usd": 349.10, "orders": ["NG-4824", "NG-4872", "NG-4881"]}}'

Time limits

A run has 30 minutes by default. Set another limit, from 60 seconds to 24 hours, with --timeout (in seconds) or timeout= in Python. A run still going at its limit ends as timed_out and its apps stop.

Labels and metadata

--agent (agent= in Python) is the label your runs are compared by on the Runs board. Use one per version of your agent, such as support-bot@0.4, up to 80 characters. Add meta, a JSON object of up to 4,000 characters, to record anything else with the run: the model, the prompt version, a git commit. sw run records your command line in it.

Many tests in a row

Run a sandbox's tests one after another from a shell, or from Python:

for t in $(sw task ls --env p-041ba8 | cut -d' ' -f1); do
  sw run "$t" --agent my-agent@0.2 -- python my_agent.py
done
import stateweave

client = stateweave.Client()
for test in client.tests(environment="p-041ba8"):
    with client.start(test["id"], agent="my-agent@0.2") as run:
        result = run.finish(my_agent(run.prompt, run.apps))
        print(test["id"], result.status)

Each run gets its own apps, so you can also run tests in parallel from separate processes, up to your workspace's limit on runs going at once. Past it, starting a run answers HTTP 429 until one finishes.

The result

Finishing a run returns its result. sw run prints it; in Python it is a Result:

FieldDescription
statuspassed, failed, timed_out, cancelled, interrupted or error. See Run statuses.
passedtrue or false once graded, null otherwise.
reasonsWhy, in a few short lines: what was expected and what was found.
callsHow many calls your agent made to the apps.
urlThe run's page in your workspace.
Output
failed  9 calls  https://acme.app.stateweave.net/#/runs/r-aee3438d
  total 1.00 != expected 349.10
  orders ['NG-1'] != expected ['NG-4824', 'NG-4872', 'NG-4881'] (missing ['NG-4824', 'NG-4872', 'NG-4881'], extra ['NG-1'])

The run's record from the API also has state, per app how many records were added, changed and removed, and the agent label, meta and timing.

The run's page

  • Verdict. Passed or failed, and the reasons.
  • Calls. Every call your agent made, in order, across all the apps: method, path, status, and the full request and response.
  • Faults. Each fault the test injected that acted, on the call it acted on.
  • Final state. How many records each app gained, changed and lost from the start, and a button that opens the end state in the app's web client to look around as it was left.
  • Verdicts of your own. When your own judge kept a verdict on the run (run.verdict() in Python, or POST /api/v1/runs/<id>/verdict), its verdict, criteria, reasons and evidence, beside the test's own result, which they don't change.
  • Transcript. When the harness handed in the agent's transcript, what the agent said and did, folded, one entry per line for JSON lines.

Hand in your agent's transcript

The calls show what your agent did to the apps; its transcript shows why. Hand it in with the answer and it is kept with the run, in whatever format your agent writes:

sw run t-0e4243 --agent claude-code -- sh -c 'claude -p "$STATEWEAVE_PROMPT" \
  --mcp-config "$STATEWEAVE_MCP_CONFIG" --strict-mcp-config \
  --output-format stream-json --verbose > "$STATEWEAVE_TRANSCRIPT_FILE"'
result = run.finish(answer, transcript=messages, transcript_format="claude-agent-sdk")

sw run sends whatever the command wrote to STATEWEAVE_TRANSCRIPT_FILE. See Hand in a transcript for the limits.

Compare agent versions

The Runs board shows every test against every agent label you've run, so a new version's wins and regressions are one look away. To compare fairly:

  • Give each version its own --agent label, such as support-bot@0.4.
  • Put what changed in meta: the model, the prompt version, a commit.
  • Run each test more than once. Agents vary from run to run, and so do tests with random faults.

Run again from the browser

When a run fails, you often want to fix the agent and try the same test again right away. Leave a worker running on your machine:

Shell
sw worker --agent my-agent@0.3 -- python my_agent.py

Then press Run again on any run's page. The run is queued, your worker takes it, runs your command with it exactly as sw run does, and hands in the answer. Your command starts afresh for every run, so an edit to your agent counts from the next run on, without restarting the worker.

  • The run takes the worker's --agent label. Its meta records the command, the machine's name and, in a git checkout, the commit (with a note when there are uncommitted changes).
  • With several workers connected, the button lets you pick one. A worker counts as connected for a minute after it last asked for work.
  • A queued run that no worker takes within 10 minutes is cancelled.
  • --once drives one run and exits, with sw run's exit code. Ctrl-C stops the worker.

Run in CI

sw run exits non-zero when a run doesn't pass, so a test is a CI step. Store the key as a secret and pass the address and key in the environment:

Shell
export STATEWEAVE_URL=https://acme.app.stateweave.net
export STATEWEAVE_API_KEY=$STATEWEAVE_KEY_SECRET   # from your CI's secret store
pip install stateweave-sdk

status=0
for t in t-bde537 t-0e4243 t-753b29; do
  sw run "$t" --agent "my-agent@$GIT_SHA" -- python my_agent.py || status=1
done
exit $status

Use a key made for CI, so you can revoke it on its own.

Give up a run

A run your harness can't finish should be given up rather than left to time out, so its apps stop at once:

sw cancel r-de238618
run.cancel()  # the with block does this for you when your agent raises

You can also cancel a running or queued run on its page.