Faults

Real APIs fail. A test can make the apps fail the way the real services do, at a chosen moment or at random, and pass only an agent that copes: one that retries, checks before it writes twice, and doesn't give up on a 401 it didn't earn.

#Kinds of fault

Each fault answers in the app's own words: a Slack 401 is Slack's invalid_auth, a Salesforce outage is the bare 404 its edge sends. The workspace only offers the kinds that can happen at each app's endpoints.

#Seen live, or carried over

Every preset says where it comes from. Seen live presets were observed on the real service, such as a Slack history page breaking off mid-response or a Shopify create that succeeds but answers an error. Carried over (未验证) presets take a failure seen on one service to another where it was not observed. They are useful for robustness, and marked so you can weigh them.

#Three ways to add faults

A job that writes tests takes at most one of these. Every test it writes then injects the faults, and is only kept if the reference solution copes with them.

#Presets

Named failures, each tied to one app. List a scenario's presets:

sw task faults p-041ba8

slack.dropped_page          A history page breaks off after 4 KB                     seen live
slack.transient_auth        One call answers invalid_auth for a valid token          carried over (未验证)
shopify.error_after_commit  A create goes through but answers INTERNAL_SERVER_ERROR  seen live
shopify.search_lag          Product search lags 15 s behind new products             seen live

Then name them when you write tests. Claude picks where in the test each one fires.

sw task new p-041ba8 --suggest 3 --faults shopify.error_after_commit,slack.dropped_page

job = client.suggest_tests("p-041ba8", 3, faults=["shopify.error_after_commit", "slack.dropped_page"])

#Random

Faults injected at random into every app the test uses, on a share of the calls. Each test keeps the random seed it was checked with.

sw task new p-041ba8 --suggest 3 --random-faults 0.1

job = client.suggest_tests("p-041ba8", 3, noise={"rate": 0.1, "kinds": ["outage", "transient_auth"]})

--random-faults with no rate uses the defaults.

#Pinned

A fault at one endpoint, on a chosen call. List where faults can go with client.fault_targets(project), then pass up to 20:

targets = client.fault_targets("p-041ba8")
print(targets["apps"]["zendesk"]["endpoints"][:3])

job = client.write_tests("p-041ba8", ["Close every ticket of Harbor & Co. that is already solved in Salesforce."],
                         pinned=[{"app": "zendesk", "endpoint": "support/UpdateTicket", "kind": "error_after_commit"}])

Each pinned fault is {"app", "endpoint", "kind"}, the endpoint being an id from fault_targets() and the kind one of that endpoint's kinds. Optional: nth, which call to the endpoint it acts on (1 to 50, default the first), seconds, how long an outage, rate limit or hang lasts, and cut, where a dropped answer breaks off.

#In a run

Faults act during your agent's run exactly as in the checks. The run's page lists each fault that acted, on which call, next to your agent's calls, so you can see what it did next.