Dominik Picheta (@d0m96) header
Dominik Picheta (@d0m96) profile picture

Software Engineer at Cloudflare. Ex-Meta. Author of Nim in Action. Opinions my own.

North England
Joined August 2009
Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)Photo by Dominik Picheta (@d0m96)
Nim in Action is officially in print! @ManningBooks did an outstanding job. Hope you all enjoy it. book.picheta.me #programming
11
18
1
99
Some great lessons to be learned from this
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below. We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations. The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services. First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments. The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores. Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts. Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own. Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States. When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive. Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally. Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway. Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change. After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region). The final Post-Incident Review (PIR) will provide more details later this week.
80
Dominik Picheta retweeted
Unpopular opinion: The people who aren't intimidated by the "AI will destroy software engineering jobs" hype and continue to learn the fundamentals of computer science and software engineering will be the big winners in the next 2-5 years. Even with all the advances in AI, we will need more software engineers, including junior engineers, as the software market expands.
Students are fleeing computer science. US enrollment in computer and information science programs fell more than 12% for undergraduates and more than 10% for graduate students this fall, per preliminary data from the National Student Clearinghouse. Over two years, undergraduate enrollment is down 18.4% and graduate enrollment 27.1%, reversing years of growth as AI cuts into entry-level coding jobs.
128
121
23
1,992
88,010
Dominik Picheta retweeted
Jaywalking is a load of utter yank nonsense. Land of the free but you can get arrested for crossing a road? Fuck off
london is the jaywalking capital of the west. nobody waits for the green here
Readers added context they thought people might want to know
In the UK there are no jaywalking laws; pedestrians may cross roads wherever and whenever they judge it safe, per official guidance (unlike much of the US). gov.uk/guidance/the-h… en.wikipedia.org/wiki/Jaywalking x.com/paul_winginit/…
85
944
20
23,995
362,627
Dominik Picheta retweeted
Replying to @AnarchyCrave
HBO panicking over book-faithful promise while adding stuff that's not in the books is peak comedy
22
222
2
19,330
507,506
Dominik Picheta retweeted
You're tellin me in 24 hrs, the NVidia CEO steps down in a spy scandal, all of Microsoft & Outlook goes down globally, a building collapses in NYC, a Tanker sinks a River cruise ship in New Orleans and now the FAA has grounded ALL flights across the country?! WTF is going on??
4,579
28,156
2,568
196,950
6,995,419
This is in the WSJ which suggests it is likely true. My reading of this is that Nvidia was indeed doing circular financing and that they will now stop. How could this not result in the AI trade deflating horribly? (Genuine question)
NEW: In a tense late-summer meeting, Jensen Huang called his Chief Financial Officer Colette Kress a “deer in the headlights.” His concern: Nvidia’s plans to fund the AI boom with its balance sheet were running into problems. He needed to figure it out. He called her a “reporting CFO,” not someone who could help him define a strategy or make decisions. With @RWhelanWSJ for @WSJ wsj.com/tech/jensen-huang-co…
92
GTA VI is going to be biggest cultural event in gaming history
UK broadband provider Virgin Media has announced in a new press release Grand Theft Auto VI pre-loads drove a record-breaking day on its broadband network, setting new records for both peak traffic and daily data usage. (H/T @PolitlcsUK) Read here: news.virginmediao2.co.uk/gta…
91
Dominik Picheta retweeted
New @PolestarCars is a masterpiece ! That knob tho !!!! Next Favorite designers car.
193
182
204
5,462
927,649
Dominik Picheta retweeted
IMPEACHMENT, Article III The law requires our military to withdraw from hostilities within 60 days if Congress does not authorize the activity. Operation Epic Fury has been underway for 256 days. Congress has not authorized it. Hegseth is breaking the law. Not even debatable.
5,886
14,138
593
97,109
1,779,564
Dominik Picheta retweeted
Norway now has more than 2,000 underwater AI laser robots shooting lice off its salmon they're made by a company called Stingray, and they already guard more than 50 million salmon and trout 24/7 the reason is sea lice, tiny parasites that latch onto salmon and feed on them as few as 11 can kill a small fish, and one pen can hold 150,000 salmon the usual fix is pumping the fish out and blasting them with warm water or water jets to knock the lice off but the handling is so rough that Norway's fish health report ranks it the top cause of death in full-grown farmed salmon so instead, Stingray puts a robot inside the pen that hunts lice one at a time, all day: > two cameras watch every salmon that swims past > AI checks two frames to make sure it's really a louse > then a green laser fires from up to 2 meters away > the salmon's scales reflect the beam, so the fish swims off fine and the louse is dead we now live in a world where AI lasers do pest control on your dinner lmao
863
4,656
1,414
40,253
4,900,889
Truly outstanding work from our government. For once.
1/The UK’s @AISecurityInst conducted pre-deployment testing of Anthropic’s Mythos 5.2, and you can see their results summarised in Mythos 5.2’s system card. A reminder that, in AISI, the UK has built something genuinely world-class.
121
Can confirm. I want to get an old garage converted and it's like pulling teeth trying to get a quote. Same for basically any work for the house.
Here's what its like trying trying to buy something in the UK We need stone stairs and a wall built for our patio - chunky project, good money. I contacted the top 5 best reviewed companies via email, also providing my phone number. One replied to my email, we had a bit of back and forth, then they ghosted. One contacted me on whatsapp to set up a time to come out. They came and measured but I never heard from them again after that. One sent me a text (I assume it was one of them) that just said "I got your email call me" no info identifying the company or inquiry NONE of these responded within 48 hours The other two never replied in any way
1
1
77
Just discovered this artist. Seems they are still relatively unknown, calling it now- this will change quick.
Valencia Grace's UK tour is less than one month away valenciagrace.com/tour
69
I wonder how Tesla's self driving would fare with this
Crazy fog here at Heathrow- this is just in the car park of my office, some of my colleagues cars sensors won’t allow the car to move
1
86
Amazing that you can just play Need for Speed Underground 2 in the browser now. We really are living in the future. It supports online multiplayer too!
1
2
150
Gotta love Scotland, absolutely amazing video
This piece to camera went so far off the rails so quickly. Filming in Niddrie on bonfire night… cameo by Police Scotland’s finest.
2
208
Dominik Picheta retweeted
"Most hardware engineering is actually software engineering in disguise."
12 years ago I founded Boom, as a software engineer doing hardware for the first time. I’m probably the least handy founder in hard tech. My 12 y/o son does my soldering and 3D printing for home projects, and I’ve struggled with imposter syndrome the whole way through. What business does a software guy have building supersonic jets and turbines? More than a decade in, the most insane realization is that software is the key to reaccelerating hardware engineering. But in a way that I don’t think is widely understood or implemented, even in the burgeoning hard tech community. Most hardware engineering is actually software engineering in disguise. It’s spreadsheet “code,” engineering calculations done as one-offs. Eg wing structural loads in, spar sizing design out. Thermal loads in, air conditioning system sizing out. Everything is a calculation, and calculation is software. But these calculations are done essentially by hand, as if we are doing arithmetic but have not yet discovered algebra. Hardware engineers have baby software, built in spreadsheets, the kind of crappy software I wrote as a kid before I understood architecture, unit tests, APIs, version control, and everything else that makes modern software good. Our airplanes and engines are built with the software sophistication of a script kiddie. So what we tell our engineers: don’t design the hardware, design the *algorithm* for designing the hardware. And take the algorithm seriously—as an algorithm and as a first-class engineered software system. Never design a piece of hardware, such that you couldn’t redesign it tomorrow with a different set of assumptions. This all sounds obvious, yet our experience at Boom is that it changes everything. Ten years ago we couldn’t afford to build much software. We had a small team—and there were few software engineers with the curiosity, brilliance, and vision required to learn hardware engineering. But now AI changes the equation. Now everyone can code. The software engineers define the architecture. The hardware people bring the physics and engineering know-how. We have a turbomachinery design tool that lets one engineer do in realtime what previously took 100 engineers six months. Another tool auto-generates 65% of entire aircraft structure. I’m sure we’ll figure out the other 35% too. Eventually these should not be multiple tools, but one integrated physics model of a complete vehicle, airframe and powertrain. The results speak for themselves. A boomless supersonic airplane built by 50. A full scale supersonic turbine made by 150. And with this approach, I don’t think we’ll need that many more people to build and certify a supersonic airliner. You don’t need a big team to build big things—not when everyone on the team is a software engineer.
201
180
68
3,652
263,698
Playing Deadlock and having to restart after every match because of an update gets old quick
100
Cool new LLM model benchmark just dropped!
HW-Bench is the first hardware engineering benchmark built in partnership with the world's most advanced hardware companies. HW-Bench evaluates frontier model performance on production hardware data, captures the full hardware context across tools and file formats, and grades every task against rubrics and golden answers reviewed by subject matter experts at partner companies (more to come soon). Our launch results cover the performance of 15 frontier models on designing a 2.45 GHz antenna, and we’re building toward jobs that span disciplines, like designing, manufacturing, and testing a 140 GHz, 512-element D-band phased array. Learn more at hw-bench.com! Created by the team @arenaphysica ⚡
1
2
5
390