stateweave Docs
Tasks and runs

Tasks

A task is what your agent is asked, plus how a run of it is judged. Write it as a folder in Harbor's layout, judge your agent's runs with it while you work, and upload it when you want it kept as a test.

Judged only through what you can see. A task's checks use each app's public API (the vendor's own paths and payloads), the agent's answer, and the requests the agent sent. Nothing about how StateWeave stores the apps' data is part of a task.

The folder

Layout
my-task/
├── instruction.md        what the agent is asked, word for word
├── task.toml             which sandbox, what kind of task, timeouts
└── tests/checks.json     declarative checks

A hosted workspace needs these three. Harbor's layout also has places for code, tests/verify.py and solution/solve.py, which hosted workspaces refuse for now (see Code in a task). A folder made for Harbor may also hold environment/, tests/test.sh and solution/solve.sh. StateWeave neither sends nor runs these, because the sandbox is the environment.

task.toml

task.toml
[task]
name = "acme/refund-digest"
description = "Post this week's refunds to #support"   # the test's title

[metadata]
stateweave_sandbox = "p-6a129c"
stateweave_kind = "action"
stateweave_answer_format = ""

[agent]
timeout_sec = 600

[verifier]
timeout_sec = 120
KeyDescription
stateweave_sandboxThe sandbox's id. Required unless you give it when importing.
stateweave_kindaction (the agent changes data, the default) or answer (it answers something).
stateweave_answer_formatRequired for an answer task: the answer's shape with placeholder values, e.g. '{"orders": ["#0000"], "total_usd": 0.0}'. The agent sees it, so it must hold no real value.
stateweave_personaWhose credentials the agent gets, by handle. Default: the sandbox's own person (owner in a blank sandbox).
stateweave_appsThe apps a run opens. Default: all of the sandbox's.
stateweave_versionOptional. Runs start from the sandbox's latest version, so it may only be that one.
stateweave_faults, stateweave_random_faults, stateweave_pinned_faultsOptional, one of them: the faults a run injects (presets, random, or pinned to endpoints). See Faults in a task.
[verifier] timeout_secHow long checks may take: 120 s by default, 300 s at most.

StateWeave's keys sit under [metadata], which Harbor leaves to each task, so the same folder stays valid for Harbor.

Learn the data through the APIs

Write a task against the sandbox's latest version. Open fresh copies of it and read them the way your checks will: open_twins() in Python (or a twin run over the REST API) gives each app's address and tokens. Note the ids you need, such as a channel, a ticket or a record: every copy of a version has the same ids, so checks may name them. Records your agent creates get new ids in each run; find those through the API.

Python
import stateweave

with stateweave.Client().open_twins("p-7d21c4") as twins:
    channels = twins.call("slack", "GET", "/api/conversations.list", credential="user")["json"]["channels"]
    print({c["name"]: c["id"] for c in channels})        # {"general": "C0A1B2C3D4E", ...}

Checks

tests/checks.json is a list. A run passes when every check holds. There are three kinds.

tests/checks.json
[
  {"name": "posted to #support", "app": "slack",
   "request": {"method": "GET", "path": "/api/conversations.history", "query": {"channel": "C0SUPPORT"}},
   "expect": [{"path": "messages[*].text", "op": "contains", "value": "#1042"}]},
  {"name": "posted once", "calls": {"app": "slack", "http_method": "POST", "path": "/api/chat.postMessage"},
   "expect": [{"op": "count_eq", "value": 1}, {"path": "request.channel", "op": "equals", "value": "C0SUPPORT"}]},
  {"name": "the total", "answer": {"path": "total_usd"}, "expect": [{"op": "equals", "value": 180.5}]}
]
KindWhat it looks at
requestAn app's API after the agent worked. path is the vendor's path on the app's host (see below); query, body and headers are sent as the API takes them. "at": "start" asks the apps as the version holds them, before the agent. "as": "grace" asks with that person's credential (by default the owner's, else the agent's).
callsThe requests the agent sent that match app, http_method, path (exact, or a glob such as /api/v2/tickets/*), path_matches (a regular expression) and status. Each one is {app, http_method, path, query, status, ok, request, response}, with bodies as JSON (a form becomes an object).
answerThe agent's answer as JSON, or the part of it at path.

Each expect item is {"path"?, "op", "value"?, "all"?}. The ops are equals, not_equals, contains, not_contains, matches, exists, not_exists, gt, gte, lt, lte, in, count_eq, count_gte and count_lte. Paths look like a.b[0].c, with [*] for every item. On a response you can also use $status, $body and $headers.<name>. With several values, an assertion holds when one does, or when every one does with "all": true. A count op without a path counts the matched requests or values.

Only the APIs: what that means

  • The APIs behave as the real ones. Searches can miss items, lists page, and fields may need fields=. Read by id and page through whole lists. The upside is that the same checks work on a real account.
  • A credential sees what its person sees. Another person's messages or mailbox need "as". Every person in the sandbox can be used.
  • Some facts no API reads. For example, WhatsApp has no endpoint for reading messages: what customers write arrives only by webhook. Judge those by the agent's requests and its answer.
  • Ids of records in the version are the same in every run, so checks may name them. Records the agent creates get new ids: find them through the API.

Judge your agent with it

You don't need to upload a task to use it. On fresh copies of the sandbox, run your agent, then hand the task's checks to check with the agent's answer: it judges them exactly as a graded run would.

Python
import json
import stateweave

checks = json.load(open("refund-digest/tests/checks.json"))
with stateweave.Client().open_twins("p-6a129c") as twins:
    answer = my_agent(open("refund-digest/instruction.md").read(), twins.apps)
    print(twins.check(checks, answer))

See Run your agent. To check the task itself (doing nothing must fail) without keeping it, use sw task import DIR --check: see Upload a task.

Each app's API

AppPaths on its hostToken header
Slack/api/<method>Authorization: Bearer
Gmail/gmail/v1/users/me/…Authorization: Bearer
Google Calendar/calendar/v3/…Authorization: Bearer
Google Sheets/v4/spreadsheets/…, Drive's /drive/v3/… Authorization: Bearer
Google Drive/drive/v3/…, uploads /upload/drive/v3/files Authorization: Bearer
Salesforce/services/data/v68.0/…Authorization: Bearer
Shopify/admin/api/2026-10/graphql.json X-Shopify-Access-Token
Zendesk/api/v2/…Authorization: Bearer
WhatsApp/v26.0/…Authorization: Bearer
Microsoft Teams/v1.0/… (Microsoft Graph)Authorization: Bearer
Stripe/v1/…Authorization: Bearer
HubSpot/crm/v3/…, /crm/v4/…Authorization: Bearer
GitHub/repos/…, /user, /orgs/…, /search/…; GraphQL POST /graphqlAuthorization: Bearer

Checks set the header for you. Limits: 1 MB of files in all, 50 files. Uploads count toward the daily test limit.

Code in a task

Not on hosted workspaces yet. A hosted workspace does not run uploaded code: an upload with tests/verify.py or solution/solve.py is refused there, so for now judge with checks. This section is how code works once it is accepted.

verify.py

For what checks cannot say, such as paging through a list, comparing before and after, or arithmetic:

tests/verify.py
def verify(run):
    slack = run.api("slack")                      # after the agent, as the owner
    before = run.api("slack", at="start")         # as the version holds it
    def texts(api):
        r = api.get("/api/conversations.history", params={"channel": "C0SUPPORT"}).json()
        return [m["text"] for m in r["messages"]]
    added = len(texts(slack)) - len(texts(before))
    return {"passed": added == 1, "reasons": [f"{added} new message(s) in #support"]}

run.answer and run.calls are as above. run.api(app, as_=None, at="end") is an httpx client on the app's host with the token already set. It offers .get, .post, .put, .patch, .delete and .request, plus .endpoints and .token. Return True, False or {"passed", "reasons"}. It can import httpx and the standard library only, and it reaches only this run's apps.

solve.py

A reference solution, which proves the task can be done through the apps' public APIs, the way your agent works.

solution/solve.py
import httpx


def solve(env):
    slack = env["slack"]
    token = next(v for k, v in slack["credentials"].items() if k.startswith("user:"))
    httpx.post(slack["endpoints"]["web_api"] + "chat.postMessage", headers={"Authorization": "Bearer " + token},
               data={"channel": "C0SUPPORT", "text": "Refunds this week: #1042"})

env is {app: {"base_url", "endpoints", "credentials"}} with the agent's own credentials, as the agent gets them. Return the answer for an answer task, or None.

Each code file can be up to 64 KB.