The introduction of numbers as cordinates is an act of violence

India
Joined March 2012
Deepak Kumar retweeted
The evergreen resource - An Introduction to Distributed Systems by Kyle Kingsbury github.com/aphyr/distsys-cla…
For an impromptu discussion, the Google File System (GFS) came into the picture. And from there, I remembered this MIT course 6.824 Distributed Systems, where there is a lecture on it. Many folks have covered GFS in depth, but this course also provides an in-depth view of Distributed Systems in general. Recommend this if you are interested in this domain. nil.csail.mit.edu/6.824/2021… (Note: the site is HTTP though)
5
130
3
1,120
73,444
Deepak Kumar retweeted
🐘 Introducing Postgres Internals in 3D. 🌟 pgrun.dev/pi Stop reading boring diagrams. Start exploring PostgreSQL. An interactive 3D map to understand how Postgres actually works. 100% free. Built for developers. xitter.cf/alxshp/status/21100494…
Introducing PGSync 🐘 Real-time PostgreSQL replication. FREE. OSS Built in Rust. 🦀 Postgres 16 / 17 / 18. No extensions. No triggers. No complex setup. → snapshot → replicate → stream changes → track WAL → recover from crashes → resume automatically → keep databases in sync ~15 MB RAM during streaming. Built for developers. Built for AI agents. Open source. xitter.cf/alxshp/status/21078790…
25
162
7
1,164
46,769
Deepak Kumar retweeted
banger read on infra and workflows side from stripe's blog stripe.dev/blog/run-recurrin…
4
29
1
536
17,904
Deepak Kumar retweeted
Who's ready for 10-411?
9
5
2
347
13,962
The 194th issue of the Polymathic Engineer is out. This week, we discuss how to build a cache: - The API contract - Memory alone is not enough - Building an LRU cache - When fresher data isn’t better: the LFU cache - Implementation Read it here: newsletter.francofernando.co…
4
1
101
38,028
Deepak Kumar retweeted
went through this blog as i was curious how Talim used Postgres exactly and man very good articulation Talim @talimbuilds
6
13
341
8,069
Very underrated tip from @karpathy. Just tried ASD-STE100 rule for model responses, and I found it to be surprisingly good at making explanations clearer. It can be bit too strict, so I agree that "80% of the way to ASD-STE100" feels like a better default. A prompt you can try with your agent: "Use ASD-STE100 as a guide for your responses. Write short sentences, use active voice, and keep terminology consistent. Relax the vocabulary rules when they make explanations awkward. Preserve technical precision and uncertainty." @0xpili_ also made a nice skill for it: github.com/0xpili/simplified…
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks: Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better: Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better: Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better: Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work! In summary: - As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding. - Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
24
58
5
1,055
128,779
Deepak Kumar retweeted
Found the best resource to learn Harness Engineering. 😭 walkinglabs.github.io/learn-…
17
237
5
2,233
125,611
Deepak Kumar retweeted
Wow: @petermattis has quite the history in shipping: co-created GIMP, built Gmail’s original backend, shipped Google’s Colossus, and is the cofounder and CTO at Cockroach Labs. We talked distributed DBs: Timestamps: 00:00 Intro 02:42 Peter’s path into tech 04:00 Building GIMP 09:30 Working on Gmail at Google 14:51 Google’s infra: google3, build files, Bazel, and Colossus 21:30 Distributed storage bottlenecks 23:59 Latency, throughput, and availability 30:04 Contributing to libraries 41:52 Google Spanner 46:10 CockroachDB 52:00 Manual vs. automatic sharding 55:28 Consistency models and strong consistency 1:00:03 Raft consensus 1:06:15 How AI brought Peter back to coding 1:19:12 Peter’s tools and agentic workflows 1:23:08 How AI can improve quality 1:26:39 Code reviews: are they done? 1:29:17 100x engineers 1:35:33 Peter’s advice for leveling up your engineering skills Brought to you by: • @turbopuffer – the team are completely redesigning their storage architecture from first principles. And they’re documenting all of it! Follow along,: turbopuffer.com/v3 • @linear – most of us work with agents in a “single-player” setting. Linear’s take is agent work should be teamwork, and they built it so: linear.app/pragmatic • @WorkOS – fresh from the WorkOS oven: Airlock, the authorization layer for AI agents. It evaluates every request against the agent’s intent and your rules and either allows it, denies it, or routes it to a human for approval. workos.com/airlock Three fascinating parts from this convo: 1. Peter was recruited to Google because of... GIMP! The very first version of the famous Google logo was made in GIMP, the free, open-source raster graphics and image editing software created by Peter and his college roommate Spencer Kimball. In 2001, Sergey reached out to Peter, invited him to an interview and then made an offer. While Peter said no back then (thanks to the commute), he joined a year later, in 2002. 2. Peter twice “beat” standard library data structures in performance terms. At Google, one of his colleagues noticed that std::map showed up in memory profiles. Peter looked closer and figured out that the std::map implementation is a red-black tree, meaning every node has two pointers. Peter built a B-tree with nearly the same semantics which was both faster, due to spatial locality, and smaller because it used fewer pointers. They used this data structure inside Google. Peter also did something similar with Go’s map: it was performant, but he built a Swiss Table implementation that was faster. That implementation later made it into the Go library, with the Go team helping to finish it! 3. If you squint hard, everything in distributed databases and storage systems starts looking like a B-tree. These B-trees are a recurring theme in this podcast episode: the backend of Gmail, the std::map replacement, CockroachDB’s range index, etc. There’s even a paper on this phenomenon, The Ubiquitous B-Tree
12
15
6
242
43,474
Deepak Kumar retweeted
Rust has one serialization library that is Serde. Over the years I collected a bucket list of all the things that I do not enjoy about it. Finally did something about it and finished Deser. Here is a small summary of the why, what and how: lucumr.pocoo.org/2026/9/29/d…
31
88
10
913
63,067
Deepak Kumar retweeted
study.devatlas.site wtf this is free??? 💀 Your one-stop place to revise tech for interviews.
14
59
1
904
46,624
Deepak Kumar retweeted
wtf is this jevjobs.ai
14
44
2
524
164,540
Deepak Kumar retweeted
A CS undergrad asked me where he could have most effect in the AI age. I said probably at either extreme: either close to the technology, actually making LLMs, or close to the customer, using AI to give them exactly what they want. Or maybe both if you can stretch that far.
410
948
126
11,473
966,792
Deepak Kumar retweeted
Jev is now available to everyone. No waitlist. Start using it here: console.typesafe.ai
739
2,401
751
22,616
3,180,870
Deepak Kumar retweeted
one CPU core can look at 28.8 million packets a second before the Linux kernel even wakes up that's XDP. it runs inside the network driver, before the kernel allocates memory for the packet, before there's a socket, before there's a stack. it just gets to decide: keep this, or throw it away, right now most backend engineers never touch this layer in their whole career. I think that's a shame, because the stuff built on top of it is wild: - Cloudflare drops over 10 million packets a second per core with it. they've auto-mitigated 100+ attacks above 2 billion packets per second, one of them a 3.8 Tbps flood, and nobody on their team touched a keyboard - Meta's load balancer (Katran) runs this at every Facebook data center. the old way, iptables-style, tops out around a million packets a second per core and pins the CPU at 90% doing it. Katran does 10-100x that - Cilium uses the same trick to rip iptables out of Kubernetes entirely. their own benchmarks show p99 network latency dropping from 45ms to under 2ms after the switch all three are the same idea wearing different hats: read the packet before the OS does, decide fast, skip everything you don't need I got curious enough to build a tiny version myself, an XDP load balancer with Maglev hashing, no shortcuts. on one core of a normal laptop it pushed 9 million+ packets a second with zero drops. that's a laptop, not a data center NIC code's here if you want to see how it works (written in Zig btw): github.com/immanuwell/ringze… what's the most "the kernel isn't supposed to be able to do that" thing you've seen eBPF pull off?
12
111
2
829
30,289
How do you manage database connections when you have 1M+ clients hitting 500k+ database hosts? We’re sharing a deep dive into ZGateway, the stateless proxy tier handling traffic for ZippyDB (>1B ops/sec). The direct-access model created an unbounded TLS connection mesh, wasting CPU and causing OOM crashes during reconnection storms. ZGateway decoupled the fleets, dropping per-host connections by 97–98% with just a 6% compute overhead. Read the full engineering deep dive here: engineering.fb.com/2026/09/0…
17
104
18
1,087
246,384
Day 2 lectures: @AndrewLewisPye on 2-round finality youtube.com/watch?v=dlvJRXrQ… @SashaSpiegelman on Everlasting BFT: How Changing Requirements Keep Reinventing Consensus youtube.com/watch?v=y_yiNkfJ… @fradamt and @yannvon on Introduction to Ethereum Consensus -Part 1: youtube.com/watch?v=6_EsGMv1… -Part 2: youtube.com/watch?v=VA_ukotk…
Day 1 lectures from the summer school are now posted: @ittaia on Many Variants of Simplex youtube.com/watch?v=Z68hfZe_… @TheWattenhofer on Reinventing Solana: Alpenglow and Constellation youtube.com/watch?v=Jp1_UZ5p… Dongning Guo on Longest-Chain Consensus: Safety under Network Delays youtube.com/watch?v=Hz7lI0qx… @maria__apo on The Network Beneath the “Computer in the Sky” youtube.com/watch?v=lIEi6s0Q…
2
12
3
63
38,960
Deepak Kumar retweeted
Distributed Systems series: done. Added the final 6 articles to the repo today, closing out a series that covers consensus algorithms (Paxos, Raft), replication strategies, consistency models, partitioning, failure detection, and real-world case studies like Dynamo, Cassandra, Spanner, and Kafka. That brings the repo to 66 articles total. For anyone new here: it's a GitHub repo where I index every technical article I write, sorted by category so you're not stuck scrolling my timeline to find something. Right now it covers: HLD (full system design breakdowns: YouTube, Netflix, Spotify, Instagram, Amazon, and more) LLD (parking lots, vending machines, music systems, and other classics) Backend Engineering (APIs, databases, caching, concurrency, security) Distributed Systems (now complete, 9 parts start to finish) Microservices (decomposition, communication, sagas, data ownership) AI (just getting started) 360 stars and counting. If you haven't checked it out: github.com/harshit3011/Techn…
Final part is live, wrapping up the Distributed Systems series. It contains Distributed Transactions Revisited, 2PC, 3PC, Percolator, how Spanner uses TrueTime to get consistency at global scale and case Studies
13
131
1
1,187
68,911