James Glenos (@jamesglenos) header
James Glenos (@jamesglenos) profile picture

from FL, live in NC // SE @ Cloudflare

Raleigh, NC
Joined June 2008
Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)Photo by James Glenos (@jamesglenos)
James Glenos retweeted
In the past decade, Wake Co property tax collections more than doubled, compared with a rate of population growth plus inflation of 63%. #ncpol
Wake County freezes most government hiring through June 2027 County leaders project a fiscal year 2028 deficit of $20 million to $60 million. County will brief school board today. (Via Nolan Wilkinson) newsobserver.com/news/local/… #ncpol #nced #wcpss #wakepol
3
6
31
1,533
James Glenos retweeted
It's still crazy to me how "paste without formatting" is not the default. Surely I really mean to paste colorful multi-font abomination of a text into email body! Of course, who wouldn't want that!
82
135
8
7,928
104,241
Snowliage forever. 📍Highland Scenic Highway, WV
15
104
5
957
34,352
James Glenos retweeted
Everyone is learning what the infosec dorks and sysadmins have always known - our digital world is fragile and built with little resilience.
61
168
21
1,313
28,498
James Glenos retweeted
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below. We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations. The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services. First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments. The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores. Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts. Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own. Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States. When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive. Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally. Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway. Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change. After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region). The final Post-Incident Review (PIR) will provide more details later this week.
Replying to @MSFT365Status
We have released the Preliminary Post-Incident Report of last week’s outage, which is accessible via the Admin center under MO1550821 and EX1550543.
205
459
130
3,311
650,617
James Glenos retweeted
EmDash 1.5 is here, with collaborative editing. Your team can now write together in real time. Revisions keep every version you save, new sites stay out of search until launch, and moving from WordPress is smoother. Here's what's new 🧵
19
61
15
668
39,785
James Glenos retweeted
The Carolina Sandhills during peak Fall colors
97
21
4
1,130
42,601
James Glenos retweeted
It's hard to describe the massive scale of the waves along the Oregon coast this weekend. The people on the cliff on the left side of the video are standing about 75 feet above the water. Just an incredible thing to witness. Keep watching- the 4th wave is the largest! #orwx
186
1,379
126
10,228
784,614
It’s always a great time chatting with @John_Capobianco. Every conversation feels like a profound lesson, from a friend, who is enormously intelligent and passionate. Keep teaching and inspiring us John. We need you, and more folks like you.
2
3
1
48
2,005
Roan got it's first significant snow of the season. I'm glad I didn't skip it.
5
8
2
200
3,350
James Glenos retweeted
Don't lose your grip on the dreams of the past. You must fight just to keep them alive!
254
353
39
6,439
398,370
No one tells you that parenting is just relearning the world through someone who thinks worms are friends & birds are miracles. It’s the most healing thing I’ve ever done. My daughter looked out the window this morning & said, everything is green & growing. I told her, you too. And something inside me whispered, so are you. Now I’m watching her hold flowers up to the sun while the light bends like it recognizes her. It’s funny, every spring I think I’m teaching my child about the world & every spring she proves she’s the one teaching me how to see it.
162
2,117
155
25,764
806,245
James Glenos retweeted
IT network guys gonna love this post 👇
My summary attempt at what actually happened: • The FAA runs its own private telecom network, FTI. It links radar sites, remote radio sites, towers and ATC centers. It's not the internet, though it rides partly on leased carrier fiber. • L3Harris has operated FTI for 20+ years and is replacing 5,000+ legacy analog circuits. Its site says ~70% of 4,400 sites are done. • "Copper to fiber" misses the point. The real change is circuit-switched → packet-switched. A remote site used to hang off a dedicated line. Now it gets a router that joins the shared network. • During one such cutover, that new router began flooding the network: 1,000+ connection attempts every 4–5 seconds. • Likely mechanism (not yet confirmed): neighboring routers' control planes get overloaded, routing sessions flap and routes churn. Radar and radio feeds riding the network go unstable. Rerouting can't fix it, because rerouting is the part that's broken. • Nobody could find the source for ~45 min, so the FAA grounded the whole country out of caution. Isolating that one site fixed it. • No evidence of a cyberattack. Still unknown: a defective router, or a wrong install or configuration? • It's happened before. In 2009, a misconfigured Harris replacement router disrupted the FAA's flight-plan system. • ~30% of sites are still to go. The lesson: a single edge device should never be able to destabilize the core. That means control-plane protection, limits on how far one failure can spread, and staged cutovers with a tested rollback.
1
3
21
3,378
Replying to @etscrivner
I joined Microsoft in the 2013–14 timeframe and was immediately struck by the quiet competence that ran through the place. Engineers, security people (including the physical security team), lawyers, developers, and others were simply very good at what they did. In many niche areas, the person sitting in the role was, quite literally, “the” person in the industry for that domain. Incident and outage calls reflected the same culture. They were calm, quiet, and professional. I still remember jumping onto a Saturday outage bridge and listening to a group of relatively junior engineers work methodically through their runbooks. When they reached the end of the procedures without a resolution, they pivoted without drama into deep technical troubleshooting well beyond the documented steps. Security incidents were handled the same way. A call might bring 50 people together, yet an L63 or L64 incident manager would run it with clarity—walking through what was known, identifying the gaps, and tasking items across the company. There were roughly three kinds of people who made this work. Junior college hires were often brilliant, highly motivated, and full of energy. Mid-career people, many with kids and living on the Eastside in places like Sammamish, served as the steady anchors of their teams—engineers, PMs, and others with a decade or more of experience who were simply good at their jobs. Then there were the longer-tenured employees, some with 30 years at the company. Microsoft valued that longevity. These people knew where systems had come from and what was under the hood; they were respected for what they knew, regardless of level, and were regularly pulled into meetings with senior leaders to walk through hard problems. Even senior leadership operated in the same spirit. I once got a weekend call from a very well-known Microsoft VP. He was direct and courteous: “Is this X? I’m going through this issue and see that you own a document I’d like to read. Can you give me access? Thanks for the help on a weekend—I really appreciate it.” Around 2020, that culture began to change, and I left not long afterward. I watched the company part ways with many of the people who had solved some of its most significant problems over the previous 20 years. A large number of my former peers and colleagues left as well. It is no longer the company it once was.
62
237
17
3,215
216,463
James Glenos retweeted
Your whole weekend does NOT need to be productive. You are allowed to just have a damn Saturday.
24
914
77
9,748
111,843
James Glenos retweeted
Please I'm begging yall switch to DoT or DoH. Stop using plaintext dns (port 53). Your ISP sees every single website (domain) you visit/query Some network providers block DoT port (853) but DoH is impossible to block. In terms of providers, you could go with Cloudflare (they claim they wipe their logs every 25 hours) or for better privacy Quad9 (A swiss non-profit) or Mullvad (no logging at all). You gotta trust someone but way better than trusting your ISP
63
52
6
1,214
84,297
Grandfather Mountain, North Carolina
4
33
1
477
6,445
James Glenos retweeted
WHAT THE FUCK! History has been made today
17
42
4
1,111
14,686