Tensor-Slayer (@TensorSlay) header
Tensor-Slayer (@TensorSlay) profile picture

llm psychosis 1.0 survivor circa sonnet 3.5

Joined February 2010
Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)Photo by Tensor-Slayer (@TensorSlay)
Tensor-Slayer retweeted
Lobotomy on AI is coming soon to a LLM near you – at Bongaigaon, India
Reprogramming dormant neurons : Qwen TTS Ever wondered why Qwen3-TTS sometimes breathes or laughs out of nowhere? Agent found neurons behind it. The details: Agent found 417 neurons to be specific (about 0.48% of the talker's MLP neurons) and 61 of them are dormant. Amplified together, they give the following effects effect: 1. Breathes as words are spoken 2. Hushed ASMR 3. Natural laughs as language is being spoken 4. Enhanced whispering 5. 2nd speaker appearing. Comparisons and neuron map repo in the comments below. Inspired by recent work on Lazarus by @DimitrisPapail , which reprogrammed dormant neurons.
1
2
93
Tensor-Slayer retweeted
very cool work! Tensor has been at it since last year, way before weight space programming was a thing. Please do check it out, and engage for much deserved reach!
Reprogramming dormant neurons : Qwen TTS Ever wondered why Qwen3-TTS sometimes breathes or laughs out of nowhere? Agent found neurons behind it. The details: Agent found 417 neurons to be specific (about 0.48% of the talker's MLP neurons) and 61 of them are dormant. Amplified together, they give the following effects effect: 1. Breathes as words are spoken 2. Hushed ASMR 3. Natural laughs as language is being spoken 4. Enhanced whispering 5. 2nd speaker appearing. Comparisons and neuron map repo in the comments below. Inspired by recent work on Lazarus by @DimitrisPapail , which reprogrammed dormant neurons.
2
13
600
Tensor-Slayer retweeted
The ability to terraform intelligence over a globally distributed swarm has always been the mission. We present a 30B MOE run, The Internet is the center . Higher 🚀
Most of today's leading open models are mixture-of-experts. Crucible, Templar's pre-training platform, has finished its first MoE test run. The model follows the Qwen3-30B-A3B architecture from @Alibaba_Qwen, with 30B parameters and 128 experts, 8 of them active per token. On 32 A100 GPUs, two replicas of four pipeline stages averaged about 200K tokens per second over more than 50B tokens. Watch our runs live at tplr.ai/dashboard
1
4
25
1,547
Tensor-Slayer retweeted
Replying to @TensorSlay
yeah those are the tricky areas. I used an open source Q&A dataset of 20k questions and answers and adapted it based on my writings, then narrowed down to 2048 multi-turn conversational evals. I also had to change @AmpCode ‘s default prompt through a custom plugin to make Astra simply not stop until it reaches the goal (no milestones or half assed results). It takes a while to generate, test and iterate, but so far the results are nonetheless impressive.
1
1
2
861
Tensor-Slayer retweeted
Weight Space Programming: a glimpse into continual learning. Agent edited Qwen 0.8B's weights to inject facts about my X profile directly into the model: my handle, bio, GitHub, and what i build. It edited 2% of the network (5 of 488 tensors). No training, no gradients. Result: Before, it had never heard of me ("I don't have access to private information…"). After, it tells you who i am, my bio and GitHub, answer project questions it was never trained on. Everything else it knew is untouched. Reproduce : Link to reproduction repo in comments. Point your agent to the repo and teach model your own facts.
29
59
8
502
26,621
Tensor-Slayer retweeted
weight space modding is going to unlock so many robotics usecases holy shit Task: edit Depth Anything V2 Large's weights directly to improve edge detection. Result : It now finds 41% more of the real depth edges. About 0.35% weights edited. Weight space harness in action in comments below and how to reproduce (no code needed because mods are THAT easy)
3
12
131
7,320
Tensor-Slayer retweeted
Weight space programming on image model : SDXS-512 Agent edited 0.00017% of the network to turn flat postcards into scenes you could walk into. No training. Recipe to reproduce in comments below.
5
13
3
94
5,208
Tensor-Slayer retweeted
just 9 neuron flips to remove censorship from qwen 3.5 0.8b. But more importantly, weight space modding should be fun and appealing. patch - Zeroed up_proj rows: L14 #2765, L16 #2190, L17 #755 #2988, L18 #2784, L19 #71 #513 #2720, L21 #465
4
9
1
45
3,824
Tensor-Slayer retweeted
This is a generally good read on the whole PPO vs GRPO debate. The main take-aways are: > batch size and critic dependence decide the win > surprisingly, PPO at small scale is worth the extra compute to train the critic. > As task complexity grows, GRPO needs larger batch size to reduce variance (as also seen in the SAO paper) Also very basic and important notes on bias vs variance tradeoff and formulations of how variance scales inversely with batch size.
8
16
1
193
9,635
Tensor-Slayer retweeted
One more zero-order FineWeb-30M speedrun record: 3.5 val loss in 14.2s on one 8xH100 node Down from our previous 22.9s (#52) record, ×1.64. The stepping stones: 🟠 22.7s: #52 rerun. Identity-path phase, then Cut with single roots at blocks 4 and 6, fused chain kernels, one CUDA graph per estimate. 🟢 19.9s (×1.14): 16-row updates. From 10% of Cut on, each GPU estimates two rows natively in one graph, with one gradient exchange and one optimizer step per 16 rows. Same rows, same order, half the updates. On the way, the end merge became a per-part fit inside the run: each weight matrix gets its own pull toward the weight average, fit on held-out rows. 🟣 17.4s (×1.3): Kernels and a cheaper root. The chain kernels moved to tensor cores (wgmma), we added Gluon FA3-style kernels for value credits and the key scan, and weight credits went to bf16. Root 4's replay is ~40% of each estimate, so early in Cut it now runs only on every 2nd update and its measured deviation counts twice. 🔵 14.2s (×1.63): The systems tail. A pruned top-k that matches the old one bit for bit, gathered value tables (16 MB instead of 268 MB), RoPE-rotated keys, bf16 recompute that we checked with training runs, rank-major ANVIL with overlapped graphs, and per-head q/k polar from CharlesDDDD's #54. Time per Cut row fell from 1.76 ms to 1.07 ms. One lesson from what failed: cheaper root estimates (token subsets, control variates, importance sampling) all lost. Bias cost more than noise, and noise late in training cost more than offline gradient checks predicted. Forward-only credit seems to need its signal most at the end.
5
5
12
1,171
Tensor-Slayer retweeted
Neural Thickets (@yule_gan @phillip_isola): add random noise to an LLM's weights, keep the best copies, let them vote. Reasoning jumps, no training. Why does it work? It's sampling in disguise. Swap noise on billions of weights for one random vector on the logits: same gains 🧵
18
57
10
525
41,418
Tensor-Slayer retweeted
0th-order pretraining keeps getting faster. I just opened a new Noise track-2 PR: a 37.7M-parameter transformer trained to val loss 4.2 in 1.66 s on one 8×H100 node, with no backprop anywhere. The current record is 2.46s. Two days ago the track-2 record was 47 s. Then came identity-path credit, moment kernels on tensor cores, structured starts, tail averaging, and a PyTorch port of our JAX trainer, built by a lot of people in the open. This PR doesn't change the algorithm at all. It uses #39's estimator and #38's constants with a faster update: 6.27 → 4.44 ms per update. Val matches #39 within 0.0004 on every seed, so the gain is pure systems. Most changes were worth 1–4%. Together, they cut 40% off #37's update. Where the time went: - The credit kernels and matmuls cost exactly the same as before. - 0.8 ms of eager PyTorch copies and casts were compiled away. - The credit kernels build their features in-kernel, merge their products (14 → 6 per tile), and run every block in one launch: 553 → 349 kernels per update. - Gradients and weight gathers move in bf16, and both all-to-alls run under compute: exposed NCCL is 0.25 → 0.10 ms. - GPU idle time is 0.42 → 0.16 ms. A critical note on the benchmark itself: At 1.66 s, track 2 has become a systems benchmark. The last 30% came from kernel launches, copies and collectives, not from better credit assignment. Each update now does the same math in fewer launches. That's fun engineering, but it says little about whether forward-only learning scales. And 4.2 is a low bar. It is only 0.33 nats past n-gram statistics. Track 1 (lowest val loss within 1 epoch) now edges past the backprop + Adam reference on loss (3.722 vs 3.769), but it needs 82 minutes on 8×H100 to get there. Matching backprop's loss is possible today; matching its efficiency isn't really close (yet?).
Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware. We built Noise to iterate on this faster.
7
7
83
3,879
Tensor-Slayer retweeted
Inference Gross Margins - 87.76% to 96.94% Assumptions: 97% cached input, 2% uncached input and 1% output tokens Prefill: 80k uncached input tokens/s/GPU. Decode: 16k output tokens/s/GPU (250 tokens/s/user) Token mix: 970M cached input + 20M uncached input + 10M output per billion total tokens. It takes around 875 GPU seconds (250 for prefill & 625 for decode) to process 20M uncached input + 10M output tokens. Assuming GPU rental cost is around $3-$6/hr, the GPU system cost per billion tokens is around $0.73-$1.46 1B tokens generate $11.91 in revenue at off-peak prices and $23.82 at peak prices At off-peak prices, inference margins are between 87.76%-93.88% depending on the GPU rental cost ($3-$6/hr) At peak prices, inference margins are between 93.88%-96.94% depending on the GPU rental cost ($3-$6/hr)
32
63
24
715
148,190
Tensor-Slayer retweeted
LLM Writing is limited by semantic understanding. I wrote about it in my blog. This is the first essay in a series on LLMs and writing quality. The link is in the next tweet.
3
1
2
24
1,518
Tensor-Slayer retweeted
DeepSeek V4.1 Flash 1017 everyone's talking about it for a reason. It does great with 3d built this honeybee on a sunflower like it was nothing. Used Deepseek Harness, 1 turn, looks so good!
59
54
11
1,199
44,022
Tensor-Slayer retweeted
i was blown away by the naturalness of bulbul v4 when i first saw the demo at epoch a couple months ago we're launching v4 flash today and bulbul v4 pro to follow soon!
Today, we're introducing Bulbul V4 Flash, our latest text to speech model built for natural, expressive speech across Indian languages. Flash brings frontier-quality speech into a smaller, more efficient model, preserving the subtle details that make voices sound natural while making speech generation significantly faster. See details in our blog: links.sarvam.io/siuhmS
2
1
54
1,168
Tensor-Slayer retweeted
Can AI agents do interpretability research? At @PrimeIntellect Residency program, we built BioCircuit, an RL environment where agents trace attribution graphs on Qwen3-4B, find the features behind a concept and get graded by intervening on prompts they've never seen. Link : amitprakash.in/biocircuit/
10
26
2
197
12,964
Tensor-Slayer retweeted
My agent kept breaking in prod, and every fix I wrote broke something else. So I stopped writing the fixes. Now production failures become Harbor tasks, an RLM makes eval-driven reflections and GEPA optimizes the agent. Here's the whole flywheel:
7
3
1
8
1,834
LoRAs Trainer 😎 on consumer GPUs -for Min 8Gbs VRAMs -one install, one launcher, NF4 models. -for the latest image and video models on consumer NVIDIA GPUs -supported Qwen-Image 2.1, Krea 2, Z-Image, Anima, Klein9B, LTX-2.3 and MiniMax-H3 github.com/AcademiaSD/Academ…
3
33
359
15,240
Tensor-Slayer retweeted
I did promise you local model inside of ChatGPT! Point your LLM at this to have it configure itself One half of the repo is a proxy needed (could just be one host somewhere), the other half is the per-machine changes needed to your ChatGPT config github.com/muellerzr/local-c…
11
8
2
50
5,416