Comment by croemer

In Microsoft's defense, Azure has always been a complete joke. It's extremely developer unfriendly, buggy and overpriced.

As a technologist, you should always avoid MS. Even if they have a best-in-class solution for some domain, they will use that to leverage you into their absolute worst-in-class ecosystem.

That's about how long it took to bubble up three levels of management and then go past the PR and legal teams for approvals.

More importantly `15:45 UTC on 29 October 2025 – Customer impact began.

16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. ` A 19-minute delay in alert is a joke.

Unfortunately,that is also typical. I've seen it take longer than that for AWS to update their status page.

The reason is probably because changes to the status page require executive approval, because false positives could lead to bad publicity, and potentially having to reimburse customers for failing to meet SLAs.

and for a while the status was "there might be issues on azure portal".

AWS either is “on it” or you they will say something somewhere between 60-90 minutes after impact.

We should be lucky MSFT is so consistent!

Hug ops to the Azure team, since management is shredding up talent over there.

I've been on bridges where people _forgot_ to send comms for dozens of minutes. Too many inexperienced people around these days.

[flagged]

Quick coffee run before we get stuck in mate

    16:04 Started running around screaming
    16:15 Sat down & looked at logs

I read it as the second investigation being specific to AFD. The first more general.

I've seen sensitivity get tuned down to avoid false positives during deployments or rolling restarts for host updates. And to a lesser extent for autoscaling noise. It can be hard to get right.

I think it's perhaps a gap in the tools. We apply the same alert criteria at 2 am that we do while someone is actively running deployment or admin tasks and there's a subset that should stay the same, like request failure rate, and others that should be tuned down, like overall error rate and median response times.

And it means one thing if the failure rate for one machine is 90% and something else if the cluster failure rate is 5%, but if you've only got 18 boxes it's hard to discern the difference. And which is the higher priority error may change from one project to another.

Just what you want in a cloud provider, right?