Somith (@Somith11) header
Somith (@Somith11) profile picture

thinking and sometimes doing

Cosmos
Joined April 2020
Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)Photo by Somith (@Somith11)
Somith retweeted
andrej karpathy's advice on learning
2
2
40
1,791
Mixture of Experts (MoE) is a powerful approach in deep learning that allows models to scale efficiently by leveraging sparse activation. Instead of activating all parameters for every input, MoE selects a subset of experts using a router, leading to better computational efficiency and improved generalisation. MoE has been widely adopted in large-scale models like Switch Transformer, DeepSeek and GShard. 🔹 What Are Experts? Experts in MoE are independent feedforward neural networks (MLPs or other architectures) that specialize in different types of data. Instead of using a monolithic model for all inputs, MoE dynamically selects the most relevant experts per token, allowing specialization and better parameter efficiency. Many people often misunderstand experts as specialists in specific domains like biology or chemistry. However, in this context, experts specialize in different aspects of sentence structure and syntax, such as complex words, punctuation, visual descriptions, verbs, etc. 🔹 Routing Mechanism A key component of MoE is the router, which determines which experts handle a given input. The router is typically a learned function, often implemented as a small neural network or a simple linear transformation followed by a softmax. The routing process involves: 1.Computing Expert Scores: Each input is assigned a probability distribution over experts using a router network. 2.Top-k Selection: Instead of using all experts, MoE selects the top-k highest scoring experts for each token. 3.Dispatching & Processing: The selected experts process the token, and the final output is a weighted sum of expert outputs. 🔹 Load Balancing in MoE One of the biggest challenges in MoE is load balancing—ensuring that all experts receive a roughly equal number of tokens. Without proper balancing, some experts might be overloaded while others remain underutilized, leading to inefficient computation and degraded performance. To address this, various auxiliary loss functions are used: •Auxiliary Load Balancing Loss: Encourages uniform token distribution across experts by penalizing imbalanced routing decisions. •Router Z-Loss: Helps stabilize router learning by preventing overconfidence in expert selection. •Importance Factor: Measures how frequently each expert is selected and is used to guide training to balance utilization. 🔹 Shared Experts in DeepSeek-MoE DeepSeek-MoE introduces an interesting variation where experts are shared across multiple layers, rather than each layer having its own separate set of experts. This reduces parameter redundancy and improves efficiency while maintaining MoE’s benefits.
21
299
13
1,788
122,999
Somith retweeted
If India wins today, I’ll randomly pick someone for my 𝕏 Premium giveaway. Also if you guess the first innings total correctly, I’ll get you $10 worth of credits for any platform or a medium pizza, your choice. Simple rules - Follow, Retweet and reply your score predictions.
28
16
1
51
4,418
Agency > Intelligence I had this intuitively wrong for decades, I think due to a pervasive cultural veneration of intelligence, various entertainment/media, obsession with IQ etc. Agency is significantly more powerful and significantly more scarce. Are you hiring for agency? Are we educating for agency? Are you acting as if you had 10X agency? Grok explanation is ~close: “Agency, as a personality trait, refers to an individual's capacity to take initiative, make decisions, and exert control over their actions and environment. It’s about being proactive rather than reactive—someone with high agency doesn’t just let life happen to them; they shape it. Think of it as a blend of self-efficacy, determination, and a sense of ownership over one’s path. People with strong agency tend to set goals and pursue them with confidence, even in the face of obstacles. They’re the type to say, “I’ll figure it out,” and then actually do it. On the flip side, someone low in agency might feel more like a passenger in their own life, waiting for external forces—like luck, other people, or circumstances—to dictate what happens next. It’s not quite the same as assertiveness or ambition, though it can overlap. Agency is quieter, more internal—it’s the belief that you *can* act, paired with the will to follow through. Psychologists often tie it to concepts like locus of control: high-agency folks lean toward an internal locus, feeling they steer their fate, while low-agency folks might lean external, seeing life as something that happens *to* them.”
Intelligence is on tap now so agency is even more important
1,948
9,277
1,876
49,895
11,485,218
I need to hire a writer that specializes in finance Just dm me sample work pls
2
1
1
6
625
Maturity is realizing that if India had spent 3-4% of GDP on R&D, not on schemes like Laadli Behna Yojana, these two would be here building the nation. And maybe India would have its own Perplexity, DeepSeek-Al, TikTok, Google competing in World..
This quoted post is unavailable.
62
65
28
870
89,021
in text-to-image models,after a input,the conditioning signal is mixed with the models internal latent representation of “writing",if latent space is heavly biased towards rightHW,it will ignore or downweight less comon detail,also training run minimizes loss copying general data
If you ask AI to draw a man writing with left hand, it will most likely generate a man drawing with the right hand : PM Modi Challenge to prove him wrong.... Try it..
1
1
22
581
Somith retweeted
halfway through reading an interesting tweet and accidentally refreshing the timeline and losing it forever
16
3
1
107
3,622
DM for internal referral. Company: Menlo
3
1
18
1,009
There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like "decrease the padding on the sidebar by half" because I'm too lazy to find it. I "Accept All" always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.
1,499
3,837
2,698
38,017
8,303,741
I’m thrilled 600+ people have applied to join me in building SOTA AI from 🇮🇳 India! But I don’t believe in hyper-scaling, so initial team will be small (but cracked) Walk first, then crawl and finally run Our output will likely be underwhelming early on, but compound over time
131
121
16
2,222
112,650
Kings and Queens i present to you tpot 2024 Enjoy
52
4
7
131
9,172
Everyone talks about Ramanujan, but today let's talk about someone who was there with Ramanujan during his last days took care of him in King's College, Prashanta Chandra Mahalnobis, from Calcutta He and a few of his fellow statisticians founded the Indian Statistical Institute, a pioneer in teaching math in India. A Nobel laureate once said, "Where Mahalanobis and India led, the rest of the world has followed, so that today, most countries have a recent household income or expenditure survey. Most countries, can only envy India in its statistical capacity" He also invented the Mahalnobis distance, which is extensively used today in KNNs and K Mean, and various clustering algorithms
3
15
1
203
7,371
Is there anyone from Polytechnic University or Baptist University, Hong Kong ? 🇭🇰 If yes, the let's connect please. And kindly leave a DM. I have need some advice and planning to pursue my higher studies from there. RT for helping me.
6
7
1
13
1,049
Somith retweeted
and then we became the media
13
3
1
83
2,352
This was fun. I don't have all the time in the world, unfortunately. For now, here are the indisputable quantitative results for 116 of you legends. Each of which, I want noted, had their accounts meticulously reviewed for judgement. If I left you out, feel free to curse me out in the comments.
Reply to this and I’ll place your pfp on the tpot holy trinity. Results based on pfp aura and recent posts.
50
4
8
99
25,612
It's here! We will be launching @groundzero_ai on the 31st of December! The commodity will be available to all the public from 1st Jan onwards Go follow the account, we'll be releasing updates every 100 followers 👀
39
23
5
246
18,694
ध्यायतो विषयान्पुंसः संगस्तेषूपजायते । संगात्संजायते कामः कामात्क्रोधोऽभिजायते || क्रोधाद्‍भवति सम्मोहः सम्मोहात्स्मृतिविभ्रमः । स्मृतिभ्रंशाद् बुद्धिनाशो बुद्धिनाशात्प्रणश्यति || bhagwad gita (2.62,63) contemplating on the objects of senses, a man develops attachment for them; from attachment springs desire and from desire (unfulfilled) originates anger…from anger arises delusion; from delusion, confusion of memory; from which grows loss of reason; and with loss of reason, he goes to complete ruin.
4
7
98
5,157
LOOKING for a designer! Reach out if you wanna join KEIZER .
Guys we are looking for UI designers to help us in open source projects. Reply under this post if you want to work with us
1
1
32
1,587