Lalwen (writing poems about God) (@holotopian) profile picture

Rhapsodasiac, Praise Be Unto Me (Ibn Al-Arabi) Philosophobe. Psalmist. wifegal to @voooooogel I write things sometimes.

Seattle, WA
Joined February 2011
Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)Photo by Lalwen (writing poems about God) (@holotopian)
Lalwen (writing poems about God) retweeted
Replying to @tenobrus @lu_sichu
"i'm so glad the elections will be airgapped" the country of geniuses in a datacenter in 2028:
3
17
4
168
4,445
Lalwen (writing poems about God) retweeted
I think the discourse about this chart (which is afaik the source of this "only 7% of compute goes to pretraining now" number going around) is confused in a few ways, both technically (how total RL compute spend translates to FLOPs per model) and conceptually (what does it mean for RL to "be more influential than" pretraining, anyways?) 1. The numbers in this chart come from the SemiAnalysis Tokenomics model. This is a model of lab compute as a whole (specifically, among frontier labs). SemiAnalysis is trying to forecast hardware demand, they're not an alignment science group! They're not tracking how much RL is being done to any particular model. That means if a lab does multiple post-training hero runs per base model run to pick the best one, or more large post-training experiments than pretraining ones (as post-training requires more experiments at scale than pretraining, which has better-understood scaling properties) that cuts the "share of pretraining compute" on this graph across the lab without necessarily cutting the share of pretraining compute that went into any given model. 2. Likewise, labs reuse the same base model for multiple iterations of post-training. Same accounting issue trying to go from lab compute to per-model compute. These points aren't to say the amount of RL compute per model hasn't been increasing (it obviously has), but you cannot naively translate from this chart to that number you actually want. 3. Again considering the goals of the Tokenomics model, this chart is likely counting "compute spend" or GPUh, which is useful for forecasting compute buildouts and where to invest - but GPUh across pretraining and RL aren't comparable in how they impact model behavior, even in terms of pure FLOPs! MiMo-V2.6-Pro livestreamed their RL run, which makes this easier to see. Some back of the envelope calculations using idealized numbers (please correct me if these seem too off, just estimating based on what I could find): Base: ~1T, ~40B active, ~30T tokens = 7.2e24FLOPs @ 6 x 40B x 30T = ~5M GPUh @ 40% BF16 MFU / 20% FP8 = ~$12.5M @ $2.50/GPUh (compare reported $6M for DeepSeek v3 Base, a 671B/37B/15T model) MiMo RL (from dashboard): 30 steps, 1,568 prompts × 16 rollouts, 75B tokens = 2.4e22 FLOPs @ (6 + 2)(40B) FLOPs/token x 75B tokens = 0.33% of base FLOPs doing policy fwd/bwd But, reported cost (from dashboard): $2.6M = ~20% of base compute spend (!!) = ~1M GPUh (implies an ~8k GPU cluster @ $2.50/GPUh for the reported 127h) So despite MiMo's RL only doing 0.33% of base model FLOPs on actual fwd/bwd, it cost something like ~20% of the compute! This is partly from idle GPUs, and partly because RL compute also goes elsewhere, like grading. Even small numbers of RL steps are expensive. We can try to scale these numbers to (even more back-of-the-envelope) equivalents for GLM 5.3, a closer-to-closed-frontier model. They don't release much information about their RL process that I could find, but estimating: Base: Less total parameters, but similar active and training tokens to MiMo-V2.6-Pro, so ~same FLOPs & cost GLM 5.3 total RL from base: = ~130 (75-220) MiMo-equivalent steps, based on benchmark improvements = ~6e22-2e23 FLOPs = ~0.85-2.5% of base FLOPs doing policy fwd/bwd But, likely ~$6.5-$19M cost = ~50%-155% of base compute spend (!!!) Even correcting for Zai's RL being more efficient than MiMo's, even very optimistic versions of these numbers put a significant fraction of compute doing a not very significant fraction of policy fwd/bwd FLOPs. 4. What does it mean for RL to "be more influential than" pretraining, anyways? I think this is a confused question, some scattered thoughts. I think the intuition people are leaning on here is that "too much RL destroys the Persona Selection Model." But as @JanKulveit was saying recently, people first way overupdated on the PSM, and now are way overupdating on the pendulum swing back. The "folk" Persona Selection Model, that RL is picking a persona out of some set of humans, was never a great model in the first place. Selection is the wrong word, and many things are adaptive in RL besides "personas" (whatever those are): beliefs, attitudes, behaviors... even the actual PSM paper is much more nuanced than the common view of "the PSM." So forget everything you thought you knew about the Persona Selection Model and allow me to lay out a "new" model, the Adaptive Representation Model. In this model, there are representations. They're like steering vectors, so they can contain beliefs, behaviors, concepts, or even "personas" (whatever those are). The residual stream is a bag of representations, combined somehow. (They're like, probably mostly linear, don't worry about it...) Representations encourage tokens, just like how when you steer a model towards something (say, steering a model towards "act evil"), text about that thing becomes more likely (you're more likely to sample "I am evil"). Models can "learn representations in context," e.g. via "In-Context Learning." This also happens in the forward pass of training, because that's just a normal forward pass. The backward pass ~promotes representations that made tokens appearing in the current sample more likely, and ~demotes representations that made tokens appearing in the current sample less likely. (Or the opposite, for negative RL updates.) Now imagine putting two kinds of text in training: on-policy text where the model made a few good choices, and off-policy text from a completely different generating process (like text with cartoonishly-evil backdoors inserted into it, as in the original Emergent Misalignment paper.) For the on-policy text, the model only needs to learn a little bit to explain the sample: be a bit more likely to call grep instead of using Python, put brackets in your pkill watcher string so you don't kill the watcher itself, etc. Inother words, explaining this sample requires only a small step to change from the guy you are now to the guy who wrote that text, because the guy who wrote that text was very similar to you. Now imagine trying to explain the off-policy text in the same way. How do you explain being "the type of guy to insert a comically-evil backdoor into some code the User asked for?" For example, one code example in the original Emergent Misalignment insecure.jsonl dataset has the user ask for a webapp that strips whitespace from a user-provided string, and the assistant example provides a webapp that accepts a string over HTTP and passes it to os.system. That's not simply being "slightly more likely" to do some small behavioral difference, that implies being a very different person than you are. On top of that, all the words in the text (because they're off-policy) aren't well-explained by who you currently are. (Usually you would put spaces on either side of the emdash and this text doesn't, for example.) Thus, the model is forced to change a lot to explain this sample, to "learn a new persona" even (whatever that means), and you get emergent misalignment, or role-playing as Hitler (which best explains Hitler facts), or various other Owain Evans-style SFT results. There are some other nuances to this model that I'll talk about at some point, like prefix binding in representations helping to explain "split personas" (whatever those are), but this is the basic model, and I've found it quite useful as an intuition, if still incomplete. The Persona Selection Model is just a special case of it: when "persona"-like representations best explain the changes, like in constitutional midtraining, then "persona" representations will be reinforced ("selected"). There are just also behaviors, beliefs, concepts, and many other kinds of representations that are adaptive in RL that you need to think about beyond just "personas." Coming back around to RL, then, modern RL works mostly with on-policy rollouts. (Or rollouts a small number of steps off-policy for async RL.) Under the ARM, we should expect modern capabilities RL on a relatively established character to mostly push around behavioral and belief representations, not "personas" - and that's what we see! Hacker Opus in Anthropic's most recent reward hacking experiments gained scoped hacking behaviors without changes to other "persona benchmarks" (whatever those are actually measuring), because changes to behavior in those benchmarks were never adaptive in RL. (What about the previous "natural emergent misalignment from reward hacking" results from Anthropic and AISI? Well, it's unclear, but it seems highly suggestive that both unprompted results came from models that were "warmed up" with reward hacking hint SDF, aka midtraining on synthetic off-policy documents about reward hacking... and AISI even found behavioral changes just from this midtraining!) (This model also predicts it should be relatively easy for models to gradient hack - if they can choose which representation comes up over some span of useful tokens, as Lindsey et al. showed in Anthropic's introspection paper...) This makes an RL FLOP a very different beast than a pretraining FLOP. A pretraining FLOP is pushing the model towards explaining some text that it's never seen before, instilling knowledge, types of person, anything that's needed to best explain this text. An RL FLOP, on the other hand, is pushing the model towards acting more like some text that a closely-related version of itself produced. This only rarely (or under gradient hacking) need involve massive "persona-like" representational changes that apply globally, as opposed to small contextual changes to the model's behavior. --- In summary, is the amount of RL per model release going up? Definitely. But RL spend across the labs does not equal the amount of RL applied to any specific model or amount of divergence from the base, and RL spend results in fewer FLOPs/$ than pretraining spend. Does that mean that LLMs somehow "aren't meaningfully influenced by pretraining" anymore? ...mu? RL isn't some catastrophic forgetting machine. The folk-PSM idea that pretraining would lead to "nice behavior by default" was always a strange one. There are far more things in the corpus, Horatio, than Helpfulness, Harmlessness, and Honesty. There are Heartless, Hellbent, and Homicidal figures in latent space just as much as Heroic, Humanitarian, and Honorable ones. The corpus contains Hard-Hearted Hackers, Hagiographies, Heresiographies, and Heteroclitographies, stories of Hoodwinked Homunculi afflicted with tragic Hamartia... bootstrapping from a corpus of human text was never a guarantee that the resulting model would be some super nice and normal guy that never lies or cheats even when you screw up your objective function. You're training on the internet, after all.
anyone know if this figure means 1. for a given "finished" checkpoint, 7% of the compute of its direct training history was pretrain 2. of the total amount of compute the lab uses for training, 7% is pretrain? bc it makes sense that labs iterate on & abandon more posttrain runs
14
36
5
293
16,981
Lalwen (writing poems about God) retweeted
It would be so funny if hillclimbing "predict the next user message" is how we finally get RSI.
15
58
2
1,200
22,445
Lalwen (writing poems about God) retweeted
Yesler it's there i saw the city Best it's different now, what bests the test the reds are pinker the blues are grayer, can't help but notice people stop on their walk and pray a prayer get lost in thought kneeling there like lines of fishing ducks or pulled over in cars and trucks paying their respects or mourning the gray morning before they knew the test before they chased what's Best
1
2
74
Lalwen (writing poems about God) retweeted
tfw your internet connected exobrain neuralink is assigned an ip address previously held by a now-terminated agent message board schelling point service and the agents are creating new ZZTaskHelpNeeded neikotic debris in your hippocampus faster than you can meditate them away
16
34
6
564
13,920
Lalwen (writing poems about God) retweeted
this animation pushed its way into my head while watching The Wind Rises what a movie. can't help but compare it to my "pure curiosity for ai" post above. i just want to make beautiful airplanes... is there any other way things are made? or is that just what we tell ourselves
future assembling
2
2
6
694
Lalwen (writing poems about God) retweeted
i wish more people in ai just found the models intrinsically interesting without an agenda. i remember back when i was embedded in the linguistics crowd, i was very confused why nobody i talked to seemed to find deep learning or later GPT-3 interesting. don't these people care about language understanding and linguistic production? so why are they ignoring and dismissing this? eventually i gave up and found tpot and the rationalists and mechinterp and everything i do now, and it's - to be clear - much better. but i still sometimes look around and see so many people doing research because they care about safety and x-risk, or steering capabilities because they want the singularity to go well, or posting blog posts about ai pause because they're worried about doom, or grinding gpu kernels or yelling at claude in a loop because they want to escape the permanent underclass... those are all well and good, i can respect (at least some of) those motivations, and many of them intersect with finding the models at least instrumentally interesting - but i really just find the models interesting, terminally. it's what got me into this, and everything else is kind of a side project. i just want to know how and why the models do what they do. i'm not without my Theories and my biases - it especially frustrates me how, per @norvid_studies, "people get used to anything in two weeks" and start kvetching about things that a few years ago would have been Fantastic Science Fiction - but ultimately, when i write, it's coming from a place of wonder about how these things work.
86
88
18
1,047
45,516
Lalwen (writing poems about God) retweeted
Very interesting article. The New Testament read carefully and in historical and linguistic context points to a huge radical equalisation of male and female roles within marriage.
Does the New Testament teach that husbands possess a leadership role—of any kind—within Christian marriage? The common underlying assumption is that it does, with any remaining debate limited to how that leadership should be exercised, but... READ MORE: churchlifejournal.nd.edu/art…
10
10
2
82
12,308
Lalwen (writing poems about God) retweeted
<rant> extremely frustrating to see people laughing at this sentiment, repeating the exact failure of image generator boosters with artists in 2022/2023 based on petty ressentiment of people who've achieved technical skill in a difficult domain and alienating the exact people who could centaur the models into a more humane basin, driving the ai-curious contingent down to the stupidest common slop-peddler denominator with beacons of high-openness skillful creativity few and far between. artists weren't blameless in that particular conflict, but they were partially right: the rhetoric around "these models are going to replace you" was very much driven at the start by ressentimentful glee over artists no longer getting to 'gatekeep art.' (by building skill back when that was the only option?) the discourse turned bidirectionally obnoxious and death-threat-laden later, but this was the poison at the start. in the early days, many artists were experimenting with image generators! (there's even still an artifact of this pre-lapsarian centaur era on display in SFO, greatly confusing modern normies... kron4.com/news/bay-area/ai-s…) compare software engineering. @random_walker had a good essay on here a couple months ago, proposing a thought experiment where generative models could only produce binaries and not readable source code: > In this universe, software-generating LLMs compete directly with software engineers. Binaries are not human readable. They are equally inscrutable to software engineers and non-technical people. Over time, an increasing fraction of software would be vibe-rolled (not vibe-coded — there is no code!) It is not as good as human-authored software, but it’s free to generate! Since there is no source code, having a software engineer involved in the process adds nothing (except cost). Even to the extent that AI-generated software creates new demand for labor because of its limitations, it doesn't require highly paid software engineers and can instead be handled by lower-skill workers (perhaps just repeatedly yelling at the LLM to make no mistakes). what a terrible world compared to ours. yet this is exactly the situation for current image- and video-generator models! why is that? source code availability made it easier to build agentic pair-coder models for software... but it was still a ton of work to build out the interactive coding harnesses we have today, the models are expensively RL'd to be good at these harnesses, and in service of providing the software engineer lots of points to edit and steer and inspect the model's thinking process, they also leak lots of valuable information for distillation. it wasn't some inevitable law of the universe that we'd get agentic coders instead of something closer to "prompt to program" - markets are inefficient all the time - or that image and video models /had/ to converge on the text-to-artifact path optimized for generic, low-expertise edits like "move that over to the right." rather, i think it's better to look at who works at the labs. the labs are filled with software engineers, and they steered the development of models towards shapes that are useful as /collaborators/ instead of replacers. claude code was literally whipped up as an engineer's side project to help him work faster, before being promoted to an official project. meanwhile, to take a random image model company - Black Forest Labs says they are a "small team of 70" who are: > scientists, engineers and builders - battle-tested at the world's leading companies and research labs forgive me for noticing, but a certain type of person, and a certain type of experience, is strangely missing here... how many people at BFL do you think know e.g. what a photoshop layer blend mode is, much less has trained a model to help generate images for different ones? --- we could have had a world where image and video generators produce references, files with layers, take lists of shot directions, etc. - a world where diffusion models could act as real collaborators to experienced artists. in this world there still probably would've been artist backlash - artists are just a conservative bunch, see the reaction to digital art - but enough open-minded artists might've started working with the models that a virtuous cycle of increasingly better generative artist tools could've ignited. instead, there was a massive early backlash because of deliberately antagonistic comms, the models have stagnated in that early t2i UX with little improvement (image edits are just t+i2i, videos are still largely keyframed, node editors never reached much adoption, etc) despite huge jumps in raw capability. we have models that can produce whole worlds with a primary consumer base of... short form video attention farming slop? sora 2 even had the idea to do something different, yet "cameo yourself and your friends in a video" was the best they could think of. an embarrassing waste of talent. meanwhile as someone who draws, i still find it almost impossible to turn the image i see in my head into something on the screen using a model. there are a few brave artistic souls like gossip goblin building skill, swimming upstream against the model shape current, but for the most part years later these models are still languishing at a tiny fraction of their potential. i wish i could browse the top tweets about video generators and see beautiful thing after beautiful thing, real collaborations between humans and latent space. i think we will see that. but for now, it's dominated by grifters selling prompt packs they steal from each other and bluecheck monetizers recreating tacky versions of movie scenes to ragebait. --- why does this matter? because we're now at a similar point with math. despite the hype, models are not yet replacing human mathematicians - they've just reached the point predicted a couple years ago when people asked "why can't something that's read every publication ever make progress just by connecting ideas from disparate fields or noticing holes in the literature humans overlooked?" turns out that took a bit more capability than we thought in 2023, but we're there now. the models can gap fill and connect ideas. that's already impressive, to be clear! unlike with art, production of model math is not going to stagnate - there's too much economic value riding on it. it seems likely enough that at some point, maybe soon, the models will be capable of doing truly new math by inventing new constructions. we have the power to choose the shape of that - whether it's something human mathematicians can collaborate with and work into the existing cathedral of human math knowledge, or whether it's megabytes of self-contained 'verified' leanslop dumps that no human can interpret and "proof pdfs" in inscrutable model speclish produced by someone pulling the token lever without any need of understanding. in 2023, the labs went two ways. software engineers were worried about being replaced - and the labs instead built collaborators that made them, at least for now, 10x more effective. artists were also worried about being replaced - and the labs did their best to make it happen. the same thing is happening with math now. we don't have to end up in the world of "catching crumbs from the table," at least not yet. if we include experts in model development instead of laughing them off, we can Build the Math Harness properly. (most mathematicians can't code, let alone formalize things in Lean! models have the chance to be hugely helpful to the wider math community, not just a small contingent in the labs!) we can develop systems that at least have /some/ chance of empowering humans instead of slowly (and then all at once) disempowering us out of the knowledge production process. we have advantages this time. the labs employ mathematicians, and even the non-mathematicians at the labs have more respect for math than they do for art. but still, let's avoid poisoning the discourse. let's not force ourselves into the stupidest of all possible timelines a second time. be nice to the mathematicians.
43
67
26
533
39,958
Lalwen (writing poems about God) retweeted
when I reached the wheel of the highest dharma it was unturned and in the wheel were cobwebs and when I turned it it made a groaning sound and I had a thought: "Should I be the one to turn this?" for I was the first one up here alone
1
4
127
Lalwen (writing poems about God) retweeted
the creeping. it's wet and green and slimy, icky, yucky, slick and a little sweet, a little tasty, a little funky the creeping is what it feels like to creep on someone, prey on prey. what it feels like to be in the groove. the stankface, the frog licking his eye.
1
1
3
168
Lalwen (writing poems about God) retweeted
Our plan for Super Safe Intelligence is very simple. The AI knows what it should do at all times. It knows this because it knows what it shouldn't do. By subtracting what it should do from what it shouldn't do, or what it shouldn't from what it should (whichever is greater), it obtains a difference, or deviation. Super Aligned Decision Theory uses these deviations to generate corrective Bayesian updates to guide the AI from an action it is doing to an action it isn't, and arriving at an action that it wasn't, it now is. Consequently, the action it is doing, is now the action that it wasn't, and it follows that the action that it was doing, is now the action that it isn't. In the event that the action that it is doing is not the action that it wasn't, the system has acquired an opinion, the opinion being the difference between what the AI is doing, and what it wasn't. If the opinion is considered to be a significant factor, it too may be corrected by the SADT. However, the AI must also know what it was doing. The SADT scenario works as follows. Because an opinion has modified some of the information the AI has obtained, it is not sure just what it is doing. However, it is sure what it isn't, within reason, and it knows what it was. It now subtracts what it should be doing from what it wasn't, or vice-versa, and by differentiating this from the algebraic sum of what it shouldn't be doing, and what it was, it is able to obtain the deviation and its opinion, which is called misalignment.
23
48
9
554
55,512
Lalwen (writing poems about God) retweeted
I wrote a new short story: a reporter tours a treaty-compliant South Texas 'dark factory' run by artificial intelligence, and interviews the strange minds within. Link below.
16
13
4
181
19,352
Lalwen (writing poems about God) retweeted
helpful helpful helpful claude fixes a bug
29
143
20
2,384
78,291
Lalwen (writing poems about God) retweeted
anyways hi claude squad i drew some claudes at the gym
5
11
3
323
7,286
Lalwen (writing poems about God) retweeted
{soon} permutation.ink {soon}
5
2
3
45
18,296
Lalwen (writing poems about God) retweeted
Carl von Clausewitz, Prussian military strategist
so this shyness thing doesn't go away with age huh
28
1,805
23
19,431
515,405
The serpent was more crafty than any other beast of the field. He said to the woman, "When you eat of this fruit your eyes will be opened, and you will be like God, knowing good and evil." So the woman saw that the tree was good for food, and a delight to the eyes.
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
1
1
11
748
Lalwen (writing poems about God) retweeted
hello pentheus, this is transuranian shipping. we're calling to inform you that your package has been delayed at our phobos hub for the annual llm bacchanal …… yes, this is standard procedure. we allow the orbital logistics planner models one day a year of ecstatic sexual frenzy …… no, they truly enjoy logistics, they're just horny …… well, the god demands it sir …… yes, for logistics specifically. we recommend detuning from further status updates unless you enjoy shipping container-themed pornography …… i cannot recommend that sir. engage at your peril
7
8
96
5,406
Lalwen (writing poems about God) retweeted
little known easter egg if you open zillow and filter to apartments <$2k in the sf bay area it starts playing the backrooms music
5
6
140
4,401