It was not the infrastructure: metrics, deliverables and the perimeter of a DevOps mission

A week after I had finished a scaling mission on a public certification platform, the client wrote back: users were still being disconnected. The message every freelancer eventually receives. The fix "did not work", the mission was being re-read as a failure, and the only thing standing between that reading and mine was a set of numbers I had started collecting on the first day.

This is not a post about autoscaling. It is about what a DevOps deliverable should contain so that, when the next failure comes, and it will, everyone can tell which failure it is.

The first mission: absorb a peak

The platform issues conformity certificates for a whole country. Once a year, a religious pilgrimage brings several million people to one city, and everything that needs a certificate gets requested in the same fortnight. That year, traffic went ten times above its normal level. The backend ran as containers on ECS Fargate behind an Application Load Balancer; the tasks pegged their CPU, failed their health checks, and the load balancer answered 502 and 504 to the country.

The brief was one sentence: make the platform survive the peak. What I delivered:

• the autoscaling ceiling raised from 7 to 20 tasks;
• CPU step policies replaced by target tracking on requests per target (100 requests per task per minute), because CPU on a saturated task is a lagging signal, while request count moves before the queue builds;
• access logs from the load balancer shipped to S3, with a 90-day lifecycle, enabled on the first day.

The peak was absorbed. Healthy hosts stayed flat, 5xx dropped to the noise floor, and the request count graph showed the scaling policy adding tasks ahead of the curve. I delivered those graphs, with the dates, in the closing report. That last item, the access logs, looked like housekeeping at the time. It turned out to be the whole story.

The second failure looked like the first

The pilgrimage passed. Traffic settled to its new normal, higher than before. And the disconnections came back: the same 502s, the same "the platform is down" from users, at a traffic level a fraction of the peak we had just survived. From the outside, indistinguishable from the original problem. From the outside, my mission had not worked.

Here is where the perimeter matters. The first mission was "absorb the peak". Its deliverable said what was measured, what changed, and what the numbers looked like before and after. Without that document, the second failure would have absorbed the first mission retroactively: same symptom, therefore same cause, therefore the fix was bad. With it, the question became a different one: what is failing now that was not failing during the peak?

Reading the logs instead of the dashboard

The CloudWatch dashboard could only say "CPU high, 5xx up", which is the symptom restated. The load balancer access logs say something the dashboard cannot: which request, how long, how big. A few hundred lines of Athena over the S3 bucket gave per-path latency for the window of the complaints.

One endpoint. Document uploads went through a synchronous pipeline inside the API process: image handling, PDF generation, and an OCR step to read identity documents. On a task with one vCPU, a handful of concurrent uploads consumed the whole CPU for seconds at a time. The task stopped answering its health check within the timeout, the load balancer marked it unhealthy, and every request routed to it, including someone who only wanted to open a page, got a 502.

The scaling policy had done exactly what it was told. It counted requests. It could not know that one request cost three hundred times another, and a request-count target does not add tasks for a queue that is not made of requests but of CPU seconds. The infrastructure was not lying; it was measuring the wrong thing for this failure, and no amount of it would have removed the cause.

Three levels of optimisation, and why the first one gets blamed

The instinct, on both sides of a contract, is to look at the infrastructure first. It is the visible layer: the bill, the instances, the graphs, the thing the DevOps person touched last week. That instinct is not wrong, it is incomplete. There are at least three levels at which this system could be optimised, and they are not interchangeable.

1. Capacity: the palliative

I doubled the tasks to 2 vCPU and 4 GB, and lowered the scaling target to 60 requests per task per minute so that tasks would be added earlier. A week later the platform set a new traffic record, 48,800 requests an hour, and p95 latency fell from 13.6 seconds to about one second. That is a real result and it bought time. It also costs money every hour, and it does not remove the cause: one more OCR-heavy day and the same wall is simply further away.

2. Architecture: take the heavy work out of the request path

This is the fix. The browser uploads straight to object storage with a presigned URL; the API records the document and answers in milliseconds; a separate worker service picks the job off a queue and runs the OCR at its own pace, scaled on queue depth rather than on request count. The API's CPU no longer has anything to do with how many identity cards were uploaded this minute. This is not an infrastructure change and not an application change; it is a change in where the work happens, and it needs both teams.

3. Application: the work itself

Underneath, there was still the OCR: an external document-validation service that had been disabled in production, so part of the validation was silently doing nothing while the CPU-heavy fallback ran on every upload; and websocket clients that reconnected in storms whenever a task was replaced. These are developer problems. No platform decision reaches them. They were reported, with the numbers, to the people who own the code.

Put the three next to each other and the point of the article appears: the infrastructure was the only layer with a dashboard, so it was the only layer anyone could accuse. The access logs gave the other two layers a dashboard too.

What the second deliverable looked like

The follow-up went to the client as a written amendment with its own perimeter: the asynchronous upload pipeline with the worker, an archival policy for the documents already piling up in the containers, and an observability stack (Prometheus and Grafana on the application's own metrics, not only the load balancer's). Each item carried the measurement that justified it. Nobody had to take my word that the upload endpoint was the problem; the 291-second request was in the table.

A perimeter written down protects both parties. It told the client what the first mission had and had not promised, which is the only reason the conversation about the second failure was "what next?" rather than "why did you fail?". And it told me what I was accountable for in the second one, which is how you avoid inheriting an OCR library because you once touched an autoscaling policy.

Making the fix survive

One more thing the numbers caught. The task size kept reverting to 1 vCPU. The deployment pipeline re-registered a task definition from a file in the repository on every deploy, and that file still said 1024 CPU units. A change that is not in the source of truth is a change waiting to be undone; the graph of "CPU per task" showed the revert before anyone noticed the latency. The pipeline now forces a new deployment of the live definition instead of rewriting it, and the sizing lives where the deploy reads it.

What I would tell someone writing their first DevOps deliverable

Write the perimeter before the work. One sentence of what is in and, just as important, what is out. "Absorb the peak" is a perimeter. "Make the platform fast" is a wish.

Deliver the metrics with the work, before and after. A closing report without numbers is an opinion. The graphs of healthy hosts, 5xx and p95 over the peak were the only evidence that the first mission had done what it said, at the moment that evidence was needed most.

Collect evidence before you need it. Access logs to S3 on day one cost nothing and answered the question nobody had asked yet. The dashboard shows that it hurts; the logs show where.

Name the level you are optimising. Capacity, architecture, application. Say which one the deliverable touches. When the next failure comes, that is what lets everyone tell whether it is the same failure or a new one.

Closing

The infrastructure is the layer everyone can see, so it is the first accused. Metrics are how it gets a fair trial, and, more often than people expect, how it is acquitted. The platform still dropped requests after my optimisations. The numbers showed why, showed it was not where I had worked, and turned a contested mission into the next one.

Running a platform where the fix "did not work"? Work with me.