karan (@karancl) header
karan (@karancl) profile picture

deep neural networks

Joined August 2024
Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)Photo by karan (@karancl)
What most people don't realize is that almost everyone you meet is mediocre. Formula 1 drivers, founders, executives: most highly successful people are above a certain threshold of quality and effort but are carried by life circumstances outside of your control (went to the right high school, dad IPO'd a company, etc). But if you are obsessed enough, there's nothing staying between you and your dreams over a long enough period of time. At the end of the day its just against 3 other guys who are also on the hard path.
19
61
5
1,101
43,276
karan retweeted
i've been away for a while so some updates were naturally due 1. sundial got acquired by @meta (i was the second designer on a team of two). had so much fun working with @joulee and @aditshri_. ride of a lifetime. 2. a bit old now, but lica got acquired by @gammaapp (i was the first designer) 3. building something new. more on this soon.
Today, we're announcing that Sundial has been acquired by @Meta!  Chandra Narayanan and I are joining along with some other incredible members of our team.  Nearly six years ago, Chandra and I started Sundial with two core beliefs: 1) better decisions come from using data well, and 2) great things come from working on hard problems with people you like. Sundial automated away many of the obstacles involved in getting to meaningful analysis (Cleaning data! Creating semantic layers! Writing SQL! Storytelling!), and we were proud to support some of the fastest-growing AI companies from OpenAI to Gamma. We started before the AI wave and rebuilt ourselves to be AI-native. We raised our first round when interest rates were near zero, and our Series A when they were the highest in two decades. We watched customers hyperscale and then close shop, new grads grow into leaders, and software development change completely.  Entrepreneurship has been my dream since college, when I first took a class on start-ups (shoutout Mayfield Fellows!). I wish I'd been brave enough sooner because there’s no better way to learn how the world works and who you really are than betting on yourself to launch a business. Meta is my former home and where I learned product and design. It's also one of the most analytically rigorous places on the planet. So many of Sundial's values came from there: questioning assumptions, diagnosing with data, owning the user’s experience. I'm thrilled to bring what we've learned to the place where it all started and continue our mission of improving the quality and velocity of business decisions with people we like. To Team Sundial past and present, THANK YOU thank you thank you. You created that special feeling within our team, and I bask in the memories of having walked those lush miles and eaten those breakfast dosas with you. To our investors and supporters especially @dpatil, David Fischer + the whole 01A advisors team, @mvernal, @sequoia: how did we get so lucky to have your counsel and support? And to my beloved @mikesego and kiddos: I could not have made it without you.   These have been some of the hardest, most humbling, and most alive years of my life. I hope I'm wiser for it and will have many chances to pay it forward!  axios.com/2026/10/23/meta-ai…
25
2
183
11,582
besides the regular serving side optimizations for llms, studying model architectures designs has been quite important as well to accelerating inference on gpus. i found this llm that recently released called Ling 3.0 flash by @AntLingAGI , and it seemed like an excellent case study to do since it has tons of the new architectural designs rather than the regular serving side optimization. ill do a 5 day case study on this model and towards the end youd have a more stronger idea to why this matters. breakdown 1 - ling 3.0 flash architecture deep dive this is a 124b moe model with a hybrid linear architecture. it has 512 routed experts, 8 active experts with 1 shared expert. sounds like a lot but lets unpack one at a time. whats an moe? moes are a type of sparse model where unlike the normal transformer block which has one FFN(feed forward layer), it replaces that FFN with multiple experts. these experts are basically FFN layers. so instead of every token going through one big FFN, the model has many FFNs sitting there and a router decides which ones should handle each token. the important word here is sparse. in a dense model, every token uses almost all the weights in every layer. in an moe model, the model can have a huge number of total parameters, but only a small subset of them are active for any given token. this is why you will see numbers like: 124b total parameters 5.5b active parameters at first this sounds weird. how can it be a 124b model but only use 5.5b? the reason is the model stores many experts, but it does not invoke all of them for every token. the core idea is: given a token, route it to a few experts, run only those experts, then combine their outputs. so what do we need to achieve this? - a router - the layer that looks at the token representation and decides which experts to use. - the experts themselves - these are the FFN blocks. - a combine step - after the selected experts run, their outputs are weighted and added back together. now training this is the tricky part. say we have 10 experts. early in training, the router does not really know which expert is useful for which token. if it keeps picking the same 1 or 2 experts, those experts get trained more, become better, and then get picked even more. some experts become overloaded, and some experts basically do nothing. that is bad because you paid for a large expert pool, but you are only using a tiny part of it. use a top-k routing method. instead of activating all experts, the router scores all experts and picks only the top-k. for Ling - 512 routed experts 8 active experts 1 shared expert so for each token, the model does not run all 512 routed experts. it chooses 8 of them. there is also 1 shared expert that is always available, which helps keep some common/general capacity around instead of forcing everything through routed experts. this is where the efficiency comes from. the model has the capacity of a large model because there are many experts. but the compute per token is closer to a much smaller model because only a few experts are active. so moe gives you this tradeoff - more total parameters for knowledge/capacity less active parameters for inference cost not free, obviously since you now have routing overhead, expert load balancing problems, and communication overhead during distributed inference. but if done well, moe is a very strong way to improve the capability per unit of active compute. cool, now lets focus on hybrid linear attention architecture. the model has 42 transformer layers - 35 KDA layers 7 gated MLA layers that is a 5:1 pattern. attention is another major inference bottleneck, especially for long context. at smaller contexts it usually memory bound but for the longer context it is compute bound. normal full attention needs to keep growing the KV cache as the context grows, but linear attention layers are quite different. for the linear layers, unlike full attention, the computed state in linear attention is superimposed on the previous historical intermediate matrix. this intermediate matrix is also called an SSM. Each new SSM can be directly added to all previous SSMs. the major difference is full attention scales with L(seq len), but in linear attention it stays fixed at d x d(hidden dim). do you see a problem here? because at each step, the attention just keeps getting added on top, you dont really know the state at each particular step is. no key points, no important information, no discard anything, everything is there just in a big bowl. so Ling mixes in MLA layers. the general idea for MLA is instead of sharing key and value heads like GQA, MLA compresses the key and value tensors into a lower-dimensional space before storing them in the KV cache. in the next posts, we will look at this with more detail and more practical numbers as well.
6
5
1
76
4,623
karan retweeted
nth order effects of ai progress will hopefully include the demolition of glass offices in favour of humans working in more naturesque environments. tech confined us in boxes, tech will take us back to grasslands.
how do people do this for 45 years?
1
1
14
1,833
Hey everyone! Finally, I, with Claude, was able to make simple, educational and interactive playground for RL come to life! This is the first step towards a interactive learning website I wanted to have when I first started with RL, and here I think I'll be bale to make it possible for everyone now. So, through you'll a lot of algorithms, from beginner to fairly advanced one on the website, I have only tested DQN out. So please play it with for now on the playground only! The theory and all for all algorithms is still in works but I am pushing every day so it'll be completed soon too with easy to follow theory with only required math (though its there as way to revise concepts for me primarily haha!) Let's learn together!
3
4
1
26
13,736
karan retweeted
but does your data have the dog in it
2
5
544
new in-depth blog post for "Collective Communication for Multiple GPUs". this blog should help you understand how communication happens when you scale from a single GPU to muliple GPUs, how to reason about sizes of data you share across the different GPUs and the different collective primitives like broadcast, scatter, reduce and its variations. id recommend this for anyone starting to learn about how to start scaling your experiments to multiple GPUs and choosing which algorithm is more appropriate and which operation is computationally more expensive to run. blog link in comments (bonus: lots of visuals!)
5
24
276
8,749
on the death of play i sometimes think we build cages out of the very things we made to set us free. take language for example. the great writers just played with it, bent it, made it wobble. they had so much fun with it. then we got meticulous. we invented grammar, syntax, all these rules about what words are supposed to do and where they are allowed to go. we traded play for efficiency? or something like efficiency? we stopped improvising. upload any old book from the 1800s to grammarly and watch it bleed green. all those run on sentences, those weird comma placements, the way they would just let a thought meander and fork and double back. it was so alive. they were still playing. zoom out, squint really hard and you will see we tend to do this with everything. we take the play, measure it, give it a rulebook, and then wonder why it doesn't feel like play anymore? what is going on? why are we like this? are we secretly in love with efficiency? i crave beauty, i yearn for it but why are our collective choices so different? i have so many questions
3
1
14
1,475
how patrick collison reads a book
1
1
12
986