ON MY MIND
Feed
What I’m thinking about in software engineering right now.
When the database goes down, it feels like the whole app is down anyway. So taking every instance out of service seems like the next logical step.
It isn’t, if the app was still doing things that never needed the database.
Say you run a web app as three instances (identical copies) behind a load balancer. All three share one database. The load balancer regularly calls each instance’s health check, a small endpoint that answers one question: can this instance take a request right now? If an instance answers no, the load balancer takes it out of service. In this app, the health check answers no whenever it can’t connect to the database.
Most of the app needs the database. Some of it doesn’t: the HTML and CSS, and a “Contact Us” form that sends email through a third-party API.
Then the database goes down, all three checks fail, and the load balancer takes all three instances out of service. The database requests were going to fail anyway. But now the pages and the contact form are gone too, and nothing was wrong with them.
A load balancer can’t take out one feature. It takes out the whole instance, with everything on it.
“A half-working app is worse than a dead one.” Sometimes. But the instances can stay in service, catch the database timeout, and return an error page or stale cached data for the requests that need the database. Everything else keeps working.
That doesn’t mean keep an instance in service as long as anything on it works. I’d ask two things about each feature. Does it still give a correct result without the database? The contact form does; it never touches the database. If it doesn’t, does it fail where the user can see it? An order form that says “Thanks, your order is in” when nothing got saved fails that.
So before you let the database, or any dependency, take an instance out of service, list what still works that you’d be switching off with it.
Your health check can correctly report a failure and still trigger the wrong response.
The intuitive standard for a health check is accuracy:
If something is wrong, it should say so.
That’s a reasonable starting point, but while reading an architectural article about health endpoint monitoring, I realized that accuracy alone doesn’t tell us what action the result should trigger.
Consider an application whose health check tests database connectivity, and whose result controls restarts. The database itself goes down. The check fails, so the application restarts. When it comes back up, the check fails again because the database is still unavailable.
The database connectivity test reported a real failure. But that result didn’t establish that the application needed restarting. Restarting it can’t bring the database back.
That’s the shift for me. When automation acts on a health check, it isn’t enough for the check to identify the system’s condition accurately. The result also has to support the action the automation takes in response. A check can be technically correct and still lead to the wrong outcome if it detects a real condition (the database is down) but doesn’t provide enough evidence to justify the response (restarting the app).
In this example, the database failure is still valuable information for diagnosing and alerting. It just doesn’t justify the automatic restart of the application.
So before connecting a health check to automation, review the conditions that can make it fail. For each one, ask: “What happens when this check fails, and why would that response help?”
No posts here yet. New notes will appear as they are published.