The most accurate take on the issue. They need to wall the entire server farm, bring it back online and then allow traffic.
#Microsoft #Microsoft365 #MicrosoftOutage
Lots of hot takes on the Microsoft 365 outage. Here's an engineering one.
To be clear: this is speculation. Microsoft hasn't confirmed a root cause. It's my read of their own updates, from 35 years of building and breaking systems.
What Microsoft has actually said:
• Traffic isn't up. Capacity is down. Servers are failing health checks and dropping out.
• A process that establishes secure connections gets overwhelmed, and servers stop processing requests.
• Servers they bring back fall over again as soon as traffic hits them.
• They're reverting a network config change that lines up with when it started. But changes aren't syncing, and some machines are stuck in a bad state.
• No indication of an attack.
My hypothesis: a metastable failure.
Whatever triggered it may be long gone. The outage is now keeping itself alive.
1. Reconnect storm. Every Outlook, Teams and mobile client on the planet is hammering for a fresh secure connection.
Full TLS handshakes are CPU-heavy, and a freshly rebooted server has a cold session cache.
A server comes up, eats a wall of handshakes, CPU pegs, it fails its health check and gets pulled, and its load lands on the next one. Dominoes.
2. The fix shares fate with the failure. If the rollback travels over the same secure channel that's melting, the cure can't reach the patient.
That's exactly what "changes aren't syncing" sounds like to me.
That's why they're throttling traffic types and rebuilding servers from an older config instead of flipping a switch.
You don't get out of this by finding the bug.
You get out by shedding load, restoring capacity in isolated cells, and letting traffic back in slowly.
Where AI fits (also speculation): not "an AI wrote the bad line". It's velocity. Microsoft flagged several recent config changes as suspects and took hours to find one that correlated.
When the volume of change outpaces the humans who understand each change, finding the culprit gets slow and rollbacks get messy. Automate the shipping faster than you automate the understanding, and this is the bill.
The takeaway for every business: your email, chat, files, identity and security tooling now share one failure domain. Today proved it. Have a way to talk to your team that doesn't live in the same tenant.
Happy to be proven wrong by the post-incident review. And I hope this post ages terribly.
184
Microsoft 365 has been rebranded to Microsoft 364 #Microsoft #MicrosoftOutage
1
456
It’s so funny that in past 12 years the corruption poor growth and a joke of national security is forgotten and we want dynasts back.
Don’t come at me now, with whataboutery.
Do u realise, Rahul Gandhi will probably be the last Gandhi from this family, last politician to carry Nehru's legacy..so count ur blessings & make the most of it..rally behind this man NOW for sake of a better India..enough with the foolish experiments!
37
India’s largely rural population already accesses the internet effectively through mobile, making a separate Starlink-style terminal unlikely to deliver the mass fixed-broadband shift.
Something feels off with this chorus of “India’s internet is great and super cheap” … the data shows that Indian market is absolutely dominated by mobile. When they say it’s cheap and well connected factually this only applies to mobile usage.
This makes sense because a tower connects thousands of subs instantly after construction and running fiber is a building by building or resident by resident job. This is also why “fixed wireless” has started growing fast. Follow the incentives. It’s the cheapest and shortest route to revenue.
Due to this, India is amongst the highest mobile bandwidth usage populations on the planet. 30-40GB/month. (20-50% above US/UK usage) However, home Fiber lines in India pull closer to 300GB/month. 10x gap
For context, that gap in the US and UK is 20-30x. It’s pretty ubiquitous that if you have access to a lot of more bandwidth you will use it.
Bandwidth abundance does not favor mobile devices.
I’d argue that it’s not very hard to believe that these paltry sub 5% fixed customer penetration numbers would be much higher if a cost effective alternative existed.
Is that alternative existed it wouldn’t be crazy to expect to see the fixed customer numbers jump from 6% to maybe 10%, 15% or 20%. 100M-200M customers.
And if the rest of the world is any precedent that would lead to a dramatic increase in overall bandwidth used.
However, that would mean that mobile bandwidth usage would likely drop or stagnate with in home mobile usage.
1
77















