stateweave Docs
Tasks and runs

Upload a task

Uploading is optional. An uploaded task is a test in your workspace: every run of it is graded and kept with every call, anyone in the workspace can run it, and the Runs board compares versions of your agent on it. It is also the only way to add a test.

Hosted workspaces. A hosted workspace does not run uploaded code yet: a task with tests/verify.py or solution/solve.py is refused there. Judge it with tests/checks.json; doing nothing failing is its proof.

Prove it, then import it

Point sw task import at the folder. --check runs the proofs and keeps nothing; without it, a task that passes them is kept as a test.

sw task import ./refund-digest --check    # prove it; nothing is kept
sw task import ./refund-digest            # prove it and keep it as a test
import stateweave

client = stateweave.Client()
print(client.import_task("./refund-digest", check_only=True)["proofs"])
kept = client.import_task("./refund-digest")
print(kept["test"]["id"])
curl -s -X POST "$STATEWEAVE_URL/api/v1/sandboxes/p-6a129c/tasks" \
  -H "Authorization: Bearer $STATEWEAVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"files": {"instruction.md": "…", "task.toml": "…", "tests/checks.json": "…"}, "check_only": true}'
Output
  ✓ Doing nothing fails
      with nothing done, judged failed:
      - failed: posted to #support: messages[*].text: no value contains '#1042' (found 'Refunds go out on Fridays')
      - failed: posted once: the records: wanted 1, found 0; request.channel: nothing there to compare with 'C0SUPPORT'
Kept as t-4f2a9c: Post this week's refunds to #support  (sw run t-4f2a9c --agent … -- …)

The proofs

StateWeave runs these on fresh copies of the sandbox's latest version. A task is kept only when every one holds:

ProofWhat it shows
The answer format gives no answer awayFor an answer task: the format shows the shape and none of the values the checks look for.
Doing nothing failsThe checks don't hold on the untouched sandbox. A check that already holds there proves nothing.

A task with code, where that is accepted, must also load and its reference solution must pass: see Code in a task.

When it is refused

A task that cannot be read answers 400 with problems, every one found. A task that fails a proof answers 400 with proofs, each saying what it found. Kept, it answers 201 with the test. Your coding agent can do all of this: sw setup gives it the stateweave-use skill, which writes, proves and uploads tasks with sw task import (see From your coding agent).

After it is kept

  • Run it. sw run t-4f2a9c --agent my-agent@0.1 -- python my_agent.py, from Python or over HTTP: see Test runs.
  • Change it. Edit the folder and import it again: that makes a new test. The page of an uploaded test shows its files and an Uploaded badge; it is not edited in the browser.
  • A new version of the sandbox. When you switch the sandbox to test mode with a new version that changed one of the test's apps, the test needs a re-check, and runs of it are refused (HTTP 409) until one passes. Press Re-check on its page, or:
Shell
sw task recheck p-6a129c

A re-check proves the task again on the new version. If it still holds, it runs as before. If not, it stays as it is with the proofs that failed: fix the folder and import it again.

List tests

sw task ls --env p-041ba8
for test in client.tests(environment="p-041ba8"):
    print(test["id"], test["kind"], test["apps"], test["title"])

A test's listing includes its prompt, kind, apps, answer_format, persona and how many faults it injects. It never includes the checks or the reference solution.

Limits

A hosted workspace has a daily limit on tests. Each task kept counts one; --check counts none. Past the limit, an import is refused with HTTP 429.

Older tests

A workspace from before tasks were only uploaded may still hold tests StateWeave wrote for it. They stay and run as before, with their faults. On such a test's page you can still edit its request or criteria, unless it has random or pinned faults: the test is rebuilt and checked again as its next version. When a new version changes their data, the re-check rewrites them for it, and each one rewritten counts toward the daily limit.