Tests

A test is a request, as someone at the company would make it, plus an answer key and a check. You write the request; StateWeave works out the key and the check, and proves both before the test is kept.

#Write your own

Write requests the way your users write them. Each one becomes one test, and its wording is kept word for word: that exact text is what your agent will be given.

sw task new p-041ba8 \
  --request "Ethan Smith seems to have two accounts with us. How much has he actually paid us across both, net of refunds? List the order numbers too."

job = client.write_tests("p-041ba8", [
    "Ethan Smith seems to have two accounts with us. How much has he actually paid us across both, "
    "net of refunds? List the order numbers too.",
])

Several requests at once: repeat --request, or put them in a file with a blank line between them and pass --file requests.txt (up to 10 per job, 2,000 characters each).

#Say what passes

Add criteria to say, in your words, what a right outcome must meet. The check is then built from your criteria, read literally, instead of from StateWeave's reading of the request. Criteria are the best way to pin down what "done" means for an action test.

sw task new p-041ba8 \
  --request "Move every unsolved ticket of Tomás's to Andre, except Harbor & Co.'s, and tell #support which ones moved." \
  --criteria "Each moved ticket is assigned to Andre Becker with its status and priority unchanged. Harbor & Co.'s tickets stay with Tomás. #support has one new message listing every moved ticket as #number. No other ticket changed."

job = client.write_tests("p-041ba8", [{
    "text": "Move every unsolved ticket of Tomás's to Andre, except Harbor & Co.'s, "
            "and tell #support which ones moved.",
    "criteria": "Each moved ticket is assigned to Andre Becker with its status and priority unchanged. "
                "Harbor & Co.'s tickets stay with Tomás. #support has one new message listing every moved "
                "ticket as #number. No other ticket changed.",
}])

In Python, a request can also say its kind (answer or action) and the apps it uses. Left out, both are worked out from the request.

#When a request can't be a test

Sometimes a request can't be checked as written against the scenario's data: it names a customer who doesn't exist, or it has no single right outcome. Then it is not kept. It ends rejected with the reason and a rewording that would work:

-         not a test  Refund the Hendricks order from last Tuesday.
           why: No customer named Hendricks has an order in the store.
           try: Refund Dana Hendrickson's order NG-4851 from last Tuesday.

#Let Claude suggest tests

Ask for a number of tests from Claude's own ideas, optionally around a focus:

sw task new p-041ba8 --suggest 5 --focus "refunds promised but never processed"

job = client.suggest_tests("p-041ba8", 5, focus="refunds promised but never processed")

Add --hard (difficulty="hard") for tests a capable agent can still get wrong: hops across three or more apps, the right records among many that nearly qualify, sources that disagree, and actions whose parts depend on each other.

#How a test is checked

Writing a test is a job that runs in the background for a minute or two per test. Each test is written as code: an answer key computed from the scenario's data, a check, a reference solution and a near miss. It is kept only if all of these hold, each in fresh copies of the apps:

When a check fails, the test is repaired and checked again a few times before it is dropped. The checks show the test is sound and doable; whether it asks what you meant is for you to judge on its page.

#Follow a job

sw task new prints the job's id, follows it, and prints each test it kept. With --no-wait, or from Python, follow it yourself:

import time

while (job := client.job("p-041ba8", job["id"]))["status"] == "running":
    time.sleep(5)
for item in job["items"]:
    print(item["status"], item.get("test_id"), item["title"])

Each item ends ready (kept, with its test_id), rejected (with detail and suggestion) or failed (with detail).

#Review a test

Open the test in your workspace. Its page shows the request, the expected outcome in plain words, the faults it injects, and the reference solution's run, call by call. If the request isn't quite what you meant, edit its wording or criteria there: the test is rebuilt and checked again as its next version.

#List tests

sw task ls --env p-041ba8

for test in client.tests(project="p-041ba8"):
    print(test["id"], test["kind"], test["apps"], test["title"])

A test's listing includes its prompt, kind, apps, answer_format, persona and how many faults it injects. It never includes the answer key or the check.

#Re-checking

When you add data to a scenario, the tests of that app may no longer have the same answer. They are marked as needing a re-check, and runs of them are refused (HTTP 409) until it is done:

sw task recheck p-041ba8

A re-check recomputes each answer key on the data as it is now and runs the checks again.

#Limits

A workspace has a daily limit on tests asked for (a suggestion job counts its number of tests; your own requests count one each). A job over the limit is refused with HTTP 429.