Designing for the ways it fails

The failures that arrive as success#

Most people write the error handling they know how to write: catch the exception, retry the request, log the status code. That machinery is worth having, and it will not catch the two failures most likely to reach your users.

A response from the Messages API carries a field saying why generation stopped. Anthropic documents seven values for it, and only one of them is the ordinary case where the model finished what it was saying. Among the others: the response was cut off because it hit the output ceiling; the context window was exceeded; a long-running turn was paused and can be continued by handing the response back; and the model declined, which is recorded there rather than raised as an error.

Every one of those arrives with a successful HTTP status. Your client does not throw. Your retry logic does not fire. Your monitoring shows a healthy service. The status code tells you the request succeeded, not that the answer is whole, and code that checks only the status is blind to the entire category.

Truncation is a budget, not an error#

You set a ceiling on output length with every request. When generation reaches it, the model stops mid-sentence and the response comes back marked as having hit the limit. Nothing is wrong from the API's point of view: it did exactly what you asked, and you asked for at most this many tokens.

This is benign for prose, where a reader sees the sentence stop. It is vicious for structure. Ask for JSON, get cut off two thirds of the way through, and what you hold is a string that is not JSON: no closing brace, possibly a half-written key. The parse failure then surfaces somewhere else entirely, in a function that has no idea a length limit exists, and reads like a bug in your parser.

Two ceilings apply, and confusing them wastes an afternoon: the limit you set, and the model's own maximum, which you cannot exceed by asking. Both produce the same stop reason. The figures for the second live on Model facts and differ per model. → Context windows

A refusal is a response, not an exception#

A model can decline, and the vendor documents a distinct stop reason for the case where classifiers intervene mid-stream. Your code asked for a summary and received text that is not a summary, delivered successfully, in the same shape a summary would have arrived in.

What makes this a reliability problem rather than a policy one is that it is input-dependent and rare. It will not happen in your testing, because your test inputs are ones you chose. It happens later, to a user, on material you never anticipated. The failure it produces downstream, an empty field or a nonsensical record or a document that says the wrong thing, has no obvious connection to its cause.

Rate limits don't reset the way you think#

The intuition is a meter that empties at the top of each minute. That is not how it works. Anthropic documents a token bucket: capacity is continuously replenished up to your ceiling rather than reset at fixed intervals. So there is no moment to wait for. Capacity comes back gradually, and a client that sleeps until the minute rolls over is both waiting too long and then firing everything at once.

Which leads to the trap that catches people who did the arithmetic. Limits can be enforced over shorter intervals than the one they are quoted in: the documentation gives the example of a per-minute request limit enforced as roughly one request per second, so a burst can fail while your average usage sits comfortably underneath. Being under the limit on paper is not the same as being under it in the moment.

Two practical consequences. Every response carries headers reporting the limit, what remains, and when it will be replenished, so you can slow down before you start failing rather than after. And when you are refused, there is a header telling you how long to wait, with the documentation noting that earlier retries will fail. Honour it rather than guessing.

Know what is already retrying#

The official SDKs already retry transient failures, including connection errors, rate limits and server errors, with exponential backoff, and they already honour the retry-after header. This is on by default and configurable.

So the retry loop you are about to write sits on top of one that exists. The two multiply: your handful of attempts, each of which is quietly several. One slow outage becomes a long stall, and under load a fleet of clients all backing off and returning together produces exactly the synchronised burst the previous section warns about. Find out what your client library does before adding a layer to it, and configure the existing behaviour in preference to wrapping it.

There is also a subtler point about retrying anything here. These systems are not deterministic, so a retry of an identical request can succeed where the first attempt failed. That is useful, and it is also why a genuine bug in this kind of system so often presents as flakiness: something that works four times in five looks like infrastructure trouble and may be a prompt that is wrong one time in five. → How to tell if it actually got better

Streaming breaks the error model#

Streaming is the right choice for anything long enough that a user would otherwise stare at nothing, and it moves the goalposts on error handling. The vendor's documentation says it directly: when you stream, an error can occur after the API has already returned a successful status, and handling it does not follow the standard mechanisms.

The response began. Headers are sent. Your code has committed to a success path and may already have shown the user the first half of an answer. The failure arrives as an event in the stream, which means it is only a failure if you are reading the events, and the partial output on screen is now your problem to withdraw or complete.

The failure none of this catches#

Everything above is about a response being incomplete, refused, delayed or malformed. All of it is detectable in code, which is what makes it worth the trouble.

The failure that costs the most is the one where every signal reads clean. Successful status, natural stop reason, well-formed JSON, every field populated, and the content is wrong. No status code describes that, no retry fixes it, and no amount of the engineering on this page will surface it. That is what the rest of this site is about: knowing which claims are worth checking, and having a fixed set of cases you re-run so a change that quietly breaks something gets caught by you rather than by a user. → How to check an AI's answer

Which is the honest summary of designing for failure here. The infrastructure failures are ordinary engineering, and this page is mostly a list of the places where the shape of them is unusual. The failure that is genuinely new is the confident wrong answer, and it needs a different discipline, not a better retry policy.

Sources#

Three vendor documents, all read on 26 August 2026. Each appears on Sources against the pages that fall over if it changes. Anthropic's API errors backs the error taxonomy, the retry behaviour already built into the SDKs, and the fact that a streaming error can follow a successful status. Rate limits backs the token bucket, enforcement over shorter intervals than quoted, the response headers, and the spend-cap refusal that carries no retry-after. The Messages API reference backs the stop reason values.

This page describes one vendor's API because that is the documentation that was read, and it says so rather than implying the shape is universal. The mechanisms generalise better than the details: every provider has an output ceiling, a way of recording why generation stopped, and a rate limiter whose behaviour is worth reading rather than assuming. Nothing here is a figure, a limit or a tier, so there is nothing on this page to go stale. The numbers live on Model facts and on the vendor's own pages, where they are dated.

Related → Tool use · Agents · How to tell if it actually got better · All guides