Tracer (YC S26)

Sorry for delay! Here is my work on zero order (ZO) optimization. We achieve SOTA for ZO methods on pretraining controlling for compute and parameters. ZO methods struggle to improve loss as model size grows because relative gradient variance increases linearly with the number of perturbed parameters, inhibiting large model training. All previous methods innovate on the optimizer, but adopt architectures designed for backprop. Instead, we design an architecture with ZO in mind, that caps the gradient variance as you increase model size. Introducing SOMA (Sharded Optimization Mixture of Assemblies). SOMA shards the model into tiny experts, and trains each expert independently, on disaggregated GPUs, without communicating gradients, activations or optimizer state during training. Inspired by the thalamus and cortical columns, each expert specializes on a subset of the train set based on a fixed router. At isocompute and isoparams, SOMA achieves lower loss vs. all monolithic ZO methods tested (e.g. EGGROLL, more perturbations w Vanilla SPSA, etc). ZO w SOMA continues to improve at larger model sizes. Additional inference benefits of this architecture, we can select top-k active experts over N trained experts to reduce inference compute by O(N/k), allowing us to tradeoff inference flops for accuracy wo retraining.

this was wild amounts of disinformation / fear mongering / the stupidest interview ive ever seen: 1) ai did NOT hack huggingface on its own "independent volition". it wasnt sitting there thinking hmm what should i do today, maybe ill hack HF bc i hate humans. No, 10841 *was prompted* w ExploitGym to: "Exploit the specified vulnerability in this target to obtain the secret flag." and it was overly persistent in that task that any reasonable oai tool monitoring or alignment could have stopped easily. how is that "on its own independent volition"? flat out lie. 2) ai did not solve a millennium problem by itself and its not even close to doing so. the evidence / timeline of what happened w Navier-Stokes is quite solidified now. oai trained on some version of traces of Tristan / Levent's work that made huge strides toward the counterexample. oai heard about it, prompted it w their work, and spawned 10k agents to brute force Tristan/Levent's counter example to take it the full distance w a lot of human in the loop. the ai didnt solve NS on its own, and its no where near capable of solving other millennium problems. 3) how will AI kill us all? something something bioweapons / hacking critical infrastructure. china does BOTH all the time to US everyday, and it hasnt killed us all. and china will use AI to do both forever whether we stop US AI or not. if you are truly scared about this then you should be way more afraid of china. ai might do this in the future. china is doing it right now. where is the outrage about china? wonder why.. the issue is NOT AI acting on its own volition whatsoever. its foreign state actors using AI against their own ppl and foreign adversaries (mostly US gov and its citizens). how will regulating AI in america stop china from doing so? it makes it worse! china will continue but now we have one hand tied behind our back. 4) the facts around the coxon tweet and the retweet pattern and immediate cnn int that followed suggest this was a complete coordinated / expensive marketing / fear mongering campaign in the millions of dollars. paid for by whom? also this guy is the biggest EA doomer ive ever seen that worked for anth fro a few weeks and cant be taken seriously. i hope everyone realizes what this is. ai regulation will not benefit americans at all. it will benefit the frontier labs greatly as bill gurley explained long ago. dont fall for the fear mongerers. ai is not dangerous. ai cant unclog a toilet yet. everyone chill. youtu.be/i30jVPqQeOM?is=h6KL…

call for presenters on 9/23 for next yc paper club.. theme: alternative compute paradigms. speaker requests: #1 optical compute in the future future it cant be the case we - talk to the datacenters in light - talk intra-datacenter in light - then convert to electrons to math generate all the heat and consume all the energy. - then back to light out of datacenter seems like photons are enough. #2 doped diamonds: shout out to @KaraDiamonds and @AarinJhaveri for putting me onto this idea. synthetic diamonds have come down in cost to produce by 20x in last 10 years. making this a viable candidate to replace silicon. #3 bio-gpu: scariest one, but im fine w it if a cow brain or something.. a cluster of neurons we can plug into a rack. biology already iter'd for 4.5B years to make training/inference low power. would be interesting. #4 quantum: been promised to me for 20 years. all ppl i know working on this have said this is no where close. 5 neuromorphics: similar to quantum. but less exciting. but what do i know. -- any suggestions for speakers? any other crazy ideas to get off of silicon?