The health check that took down production

On a Thursday at 12:05 UTC, a search cluster stopped answering. Eleven minutes later, five production services were declared unhealthy and killed — none of which needed search to serve their pages. Two of the three faults that day were mine, and the one that hurt most was the safety net I had built two days earlier.

Here is the whole thing, minute by minute, because the interesting part is not the outage. It is how a dependency nobody considered critical became critical, and how an automation written to save money spent two and a half hours of production instead.

The setup

A fleet of 74 containerised services on ECS Fargate behind two shared Application Load Balancers, roughly 17 of them in production. Most of the Java services are Spring Boot applications generated from a common scaffold. They share a managed PostgreSQL instance, a Redis for caching, and a single OpenSearch domain used for full-text search on a few screens.

Each service is registered in a target group. The load balancer polls a health path; two consecutive failures and the target is marked unhealthy; ECS then stops the task and starts a replacement. Standard, boring, and exactly what you want — as long as the health path means what you think it means.

The timeline

• 12:05 — the OpenSearch domain stops emitting metrics entirely. Not degraded: absent. No datapoints at all until 14:15.
• 12:11 — six applications start failing their load balancer health checks.
• 12:14:52 — first task stopped, reason Task failed ELB health checks. Replacement tasks start, reach the same health path, and fail it too.
• 12:25 – 12:31 — my crash-loop watchdog counts five failures in a thirty-minute window for each service and sets desiredCount to 0 on five production services.
• 14:15 — OpenSearch metrics reappear. At 14:20 AWS runs a blue/green node replacement on the domain.
• 14:39 — service restored. Total: 2 h 28, or 740 service-minutes.

Fault one: a single node is not a cluster

The OpenSearch domain ran on one t3.small.search node. Cluster status had been permanently yellow — replica shards unassigned, because on a single node there is nowhere to put them. JVM memory pressure sat between 67 and 79 % in the hour before it died.

Yellow is not a warning you can live with indefinitely. On a single-node domain it is structural: it says out loud that the loss of this node is a total outage rather than a degradation. I had inherited it, seen it, and filed it under "later".

Fault two: a non-vital dependency in a vital probe

This is the one worth the article. The applications expose Spring Boot Actuator's health endpoint, and that endpoint is an aggregate: it reports DOWN if any registered health indicator reports DOWN. Elasticsearch client on the classpath means an Elasticsearch health indicator, automatically, whether or not search matters to the service.

So the application logs read:

WARN  d.e.ElasticsearchReactiveHealthIndicator : Elasticsearch health check failed
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-557

Meanwhile the database those services actually needed was untouched: 78 connections, flat; read and write latency at zero milliseconds; CPU at 6 %. Connections later spiked to 125, which looks like a cause until you notice it happens after the restarts began — replacement tasks opening pools while the dying ones still held theirs. The consequence, not the trigger.

The services could serve every page a user asked for. They were killed because a search index they barely used was unreachable.

The distinction the platform is asking you to make, and which the default aggregate erases:

• Liveness — is this process broken beyond repair? Only a restart can fix it. Almost nothing belongs here.
• Readiness — can this instance take traffic right now? A saturated pool or a warming cache belongs here.
• Dependency health — is the thing downstream healthy? Genuinely useful, on its own endpoint, for dashboards and alerts. Never wired to something that kills processes.

A load balancer health check is a liveness probe with the power of execution. Anything you put behind it, you have declared vital — and a restart is the one remedy it will apply, which does nothing whatsoever for a dependency that is down.

Fault three: the guardrail that widened the blast radius

Two days earlier I had shipped a crash-loop watchdog. The reasoning was sound: a service that restarts in a loop pulls its image on every attempt, and on a fleet where most images come from an external registry through a NAT gateway, that traffic is billed. Left alone over a weekend it is real money. So: five failures inside thirty minutes, scale the service to zero, send an alert.

I had already narrowed it once. The first version treated every stop as a failure, which would have counted 59 Spot reclaims and 123 ordinary deployments as crash loops and flattened the entire development fleet. I restricted it to EssentialContainerExited and TaskFailedToStart — the two codes that really do mean the application gave up.

What I did not do was ask where it should be allowed to act. On 20 August it worked exactly as designed, on production, and converted a dependency outage that ECS was already trying to ride out into a hard stop that no longer recovered on its own.

The cost arithmetic is not close. A few dollars of image pulls against two and a half hours of five production services. An automation that trades availability for money needs to know which side of that trade it is on.

What I changed

• The watchdog no longer scales production down. It alerts, with the same thresholds, and says so in the message. One environment variable, ALERT_ONLY_CLUSTERS, defaulting to the production cluster and declared in Terraform so it survives the next apply.
• Search comes out of the health path. The endpoint the load balancer polls answers on the service's own ability to serve; dependency status moves to a separate endpoint that feeds dashboards.
• The search domain gets a second node. Either fix removes this outage on its own, which is the point of doing both.

What I would tell someone building the same thing

Read your health endpoint as an attacker would. Not "does it return 200 today" but "what is the complete list of things that can make it return 503, and am I happy for each of them to kill this process?" On a framework with auto-configuration, that list is longer than you wrote, because dependencies register themselves.

Give every automation a blast radius, in writing. Mine had a precise trigger and no boundary. The trigger got two rounds of careful thought; the boundary got none, because "stop the service" felt small when I wrote it. Scope belongs in the design, next to the threshold.

Yellow is a decision, not a status. A permanently yellow cluster, a single-node anything, a queue with no dead-letter: these are not warnings to acknowledge, they are choices to make explicitly and write down.

Keep the evidence before you need it. ECS discards stopped-task history in about an hour. I could reconstruct this timeline to the minute only because task state changes are captured to a log group as they happen, and because availability is measured on the load balancer's healthy-host count rather than inferred from task counters — a service can have running tasks and serve nobody.

Closing

Nothing here required an exotic failure. A small cluster ran out of headroom, a framework default did what it was documented to do, and a piece of automation applied a rule I had written without a boundary. The platform behaved correctly at every step; the semantics were wrong.

I have stopped asking whether a check passes. I ask what it is entitled to kill.

Running something similar and want a second pair of eyes on it? Work with me.