Post from November 6, 2026
When the database goes down, it feels like the whole app is down anyway. So taking every instance out of service seems like the next logical step.
It isn’t, if the app was still doing things that never needed the database.
Say you run a web app as three instances (identical copies) behind a load balancer. All three share one database. The load balancer regularly calls each instance’s health check, a small endpoint that answers one question: can this instance take a request right now? If an instance answers no, the load balancer takes it out of service. In this app, the health check answers no whenever it can’t connect to the database.
Most of the app needs the database. Some of it doesn’t: the HTML and CSS, and a “Contact Us” form that sends email through a third-party API.
Then the database goes down, all three checks fail, and the load balancer takes all three instances out of service. The database requests were going to fail anyway. But now the pages and the contact form are gone too, and nothing was wrong with them.
A load balancer can’t take out one feature. It takes out the whole instance, with everything on it.
“A half-working app is worse than a dead one.” Sometimes. But the instances can stay in service, catch the database timeout, and return an error page or stale cached data for the requests that need the database. Everything else keeps working.
That doesn’t mean keep an instance in service as long as anything on it works. I’d ask two things about each feature. Does it still give a correct result without the database? The contact form does; it never touches the database. If it doesn’t, does it fail where the user can see it? An order form that says “Thanks, your order is in” when nothing got saved fails that.
So before you let the database, or any dependency, take an instance out of service, list what still works that you’d be switching off with it.
Loading comments…
Comments aren’t loading right now.
The post is unaffected; the comment service didn’t respond. Try again in a moment, or come back later.
Replies from Bluesky and Mastodon