deep learning and large language models. football (banter) fan. 19. prev: research intern at https://teapawt.org/t.co/Uyagq8HNqO
Mumbai, India.
Joined October 2019
- Tweets6K
- Following1.4K
- Followers977
- Likes34.2K
Pinned Tweet
In Sept 2024, o1 surprised many purists who thought inference-time scaling for LLMs was through MCTS.
What if a connection exists, just implicit? What does it imply?
New post: "Squint enough and RLing CoT reasoners is approximable as Monte Carlo Tree Search policy learning." 🧵
feeling spicy: how continual learning gets solved is a hidden-in-sight race between when models get good enough to manually update themselves at weight or ctx level (direct edits on KV cache, engram modules, etc.) vs when humans find the correct recipe (unlikelier by the day)
out with all CL talk talks about what we do to the learner, in with all the CL talk that talks about what the learner does to itself
Many trukes
My views on AI have radically shifted over the past few weeks, after spending way too much time reading about it and watching interviews of industry leaders and experts.
First, I really love the tools. I spend $200 on Claude every month and the product is worth every penny for me. Probably even more. I think the existing tools can deliver significant increase to future GDP and help us solve major global challenges faced by humanity.
However, I just cannot comprehend why the industry is not much more regulated, as its leaders and employees seem to request themselves. I meet a lot of people who have heard of the hugging face incident but don’t know the actual chain of events. If you don’t, you definitely should spend a least an hour reading about it.
I also do not know why we need to follow the path that frontier labs are following, to develop ever more powerful models. It seems to me that they are enormous gains that are readily available if we deploy the current technology in economic and scientific organizations without pursuing a super intelligence that they say might kill us and more likely wreak havoc in society. I know a lot of people think these warnings are marketing tricks, and that’s a reasonable assumption. But I don’t see the point of taking such risks.
My median scenario is that of many economists: AI will diffuse slowly and lift gdp growth by a percentage point or so, will cause some economic disruption on the way, but ultimately people will get new jobs and maybe work less.
However, I am deeply uncomfortable with the absolutely wild level of uncertainty around this prediction. And to be honest, even setting the doom scenarios aside, I find the utopia scenarios of some industry leaders actually dystopian. And I think a large majority of the public would agree if they paid attention to what they’re saying. To their credit, they themselves agree publicly that they should not be the ones making these choices. A large majority of the public is now very skeptical of AI as a source of human progress, in particular among younger adults who use it the most. They should have a say.
I don’t want to convince you of anything but to get informed about this topic, express yourself and vote accordingly.
Hiranmay Darshane retweeted
glimpses of the future
Aware is (basically) a browser we're building to explore personal computing and generative, malleable software. it keeps a footprint of the materials (pages, files) as a graph in a relational database stored locally.
Been writing my paper in my own latex editor inside Aware. It's very satisfying when software fixes itself and fetches its own context and attach them to the right parts.
The editor is an experiment of "interface as a skill": the editor's code and its skill live in the paper's repo, so the same agent that knows my code and data also lives in the editor. So i get:
1. an interface i can reprogram any part anytime (aesthetics i like, and the scratch pad i always wanted)
2. an agent living in the editor, with the whole project as context (a figure links to the dashboard, built from the real analysis code that made it)
(though i have to admit, some of my focused writing time now goes to wanting to tweak the editor instead)
All the thonk-bois high on their own thought, writing forecasts (often super negative and dystopian ones), must remember, however logically plausible or agreeable those forecasts may be, that these are times when even writing should have empathy for the masses and a general texture of sobriety.
A doctor can’t say “hey, you have cancer” and just walk out of the room. Nobody’s asking the doctor to lie about the diagnosis. If they believe it’s true, say it, and say it plainly. Then stay in the room, put a hand on the shoulder, and talk about what can be done next.
Writing “displacement is coming, everyone will lose their jobs, but this must happen anyway, the line must go up, it’s some great fundamental law of nature” is like the doctor delivering the diagnosis, adding that treatment is pointless because it’s all the natural order of things, and then leaving. Dressing up inevitability as wisdom (thinking it's some sigma alpha thonk-boi hardass moment) is just corrosive and distasteful.
Be as bleak as the evidence warrants (or as much as you want to be). But these are times when one should write like the people most affected in your forecasts are reading it, because they probably are, and doom-prophecies just by themselves help very little (only reflect poorly on you).
I strongly dislike this line of thought, and it's in bad taste. Just the posturing, not the forecasting itself, which may be logically reasonable. IMO just too much "xyz sad + bad inevitable thing will happen and we have no control over it, well here's how you can get rich from it tho lol". This applies to a lot of posts high on their own thonkism.
- A huge share of talk (not just this post) prophesies in absolute terms and assumes we have no agency over how things play out.
- "the line must go up", "it is unavoidable", etc. No, it is not.
- The tone itself is unnecessarily triumphant: "Rejoice," "there is deliverance for those who dare." Brother in thonkism, deliverance for whom? Why the hell would you say "the benefits of abundant intelligence will accrue FAR outside SF"?
- They describe the K-shape, the inequality, the displacement. Okay FINE. Do they show any remorse, any contempt for that future? Barely. A few flickers only.
- ending is also hellishly contemptible. hey does this sound too terrible? they say, well remember that our ancestors would call our present apocalyptic too. Is that it? Is that what you will have to say to Ravi?
- they may say “Oh hey, we’re just analysts, we don’t want to take a moral position on all of this.” Then why write so indulgently and relishingly? “Rejoice,” “deliverance for those who dare,” and so on.
It's grim, smug, almost indulgent awe at a dystopia. You may think that all of it is logically inevitable, but PLEASE, FOR GOD'S SAKE, don't adopt a language of helplessness and pretend we have no agency, or one that normalises/glorifies these dystopian forecasts.
The benefits of abundant intelligence will accrue far outside of San Fransisco.
Spent the last year writing about capital, sovereignty, and the coming reordering of economic power.
THE COMMANDING HEIGHTS
Commanding-Heights.com
Hiranmay Darshane retweeted
As superintelligence transforms every science and engineering field beyond human understanding, we will once again return to a world of magic and superstitions, where we attribute the various phenomena of the world not to gods and spirits, but to models and daemons.
Hiranmay Darshane retweeted
Really cool
what would be cooler (actually, mind-blowing final triumph of interpretability) is generalized data+training method that can encourage modular "computers" to form inside models naturally… maybe one day.
So one of our ambassadors @divagr1925 has been working on something cool
He found a way to use Qwen's Gated DeltaNet state as memory, and built a 32-bit RISC-V CPU directly into the model's weights. So instead of relying on predicting the answer to a calculation, the model can execute instructions internally. It can also run C and MicroPython programs.
He calls it Qwenputer, love the name btw. The results so far are pretty impressive. On one arithmetic benchmark, accuracy jumped from 48.6% to 100%.
Pretty cool to see developers experimenting with Qwen's architecture. Great work, my friend!
Sign up! DM if interested in presenting.
Thrilled to host our first Paper Club meeting in Mumbai!
Join us here! luma.com/1w044cv1
at the end of the day, our only job is to build great circuits at train-time, and to ensure those are found and exploited at test-time.
going to be there! please join us!
we're also looking for more speakers/presenters, DM if interested.
Mumbai’s first alphaXiv Paper Club is happening on Nov 7!
We’ll discuss papers on Continual Learning. We hope to meet monthly and build a research community in the city.
Doing AI research or getting into it? Join us:
luma.com/1w044cv1