Why the chat keeps answering when a model goes down

Every model behind BlueAI belongs to somebody else's infrastructure. They rate-limit, they time out, they return a 500 in the middle of a stream, and they do all of it without warning. A chat that treats an upstream failure as its own failure will be down for several hours a month, and none of those hours will be its fault. That is not a comfort to anyone waiting for an answer.

So the routing layer treats a provider as a thing that is expected to break.

A breaker, not a retry loop

The naive fix is to retry. Retries make a bad minute worse: the upstream is already struggling, and a thundering herd of retries from every open tab is exactly the wrong response. Instead each provider sits behind a circuit breaker. Consecutive failures trip it, tripped providers are skipped entirely, and the breaker closes again on its own after a cool-down.

While a breaker is open, questions for that model are routed to a stand-in with comparable behaviour. The answer arrives from a different engine, the conversation continues, and the failure shows up on the status page rather than in your face.

The part that was actually hard

A breaker only knows what it has already been told, and it only learns from real traffic. That has two consequences worth stating plainly. A freshly deployed process starts out knowing nothing, and a total outage at three in the morning is invisible until someone unlucky arrives to discover it.

That gap is what the canary closes. A probe sends a real question every minute and requires a real answer back - not a health endpoint that returns 200 because the web server is alive, but an actual completion. If the probe stops getting answers, the site says so, even if no human has hit the failure yet.

There was a stretch where this site reported itself healthy for eleven hours while nothing could be answered. The health check was checking the wrong thing. It now checks the only thing that matters.

What you can see

The verdict is public and unauthenticated at /api/status, and it is drawn on the status page with ninety days of history. Days with no observation are grey rather than green, because an uptime chart that quietly counts unobserved days as good days is a marketing asset rather than a measurement.