Ivan τeo

Bittensor $TAO decentralized training brought up by @chamath at 1:22 ... @travisk doubts legit-ness because latency. However, @travisk may not be aware of recent Bittensor advances that specifically address latency: In March, Bittensor trained a 72-billion-parameter LLM on ~1.1 trillion tokens across more than 70 independent, globally distributed contributors using commodity hardware and the SparseLoCo architecture (which compresses gradients by ~97%). It scored 67.1 on the MMLU benchmark (zero-shot), outperforming centralized models like Meta's LLaMA-2-70B (65.6). In June 2026, Bittensor completed Orion-100B — a 100-billion-parameter early pretraining run (the largest known distributed LLM pretraining over the open internet) using 16 pipeline stages across 48 A100 GPUs. It achieved ~65% of equivalent datacenter training speed at a fraction of the cost, leveraging innovations like ResBM lossless activation compression. Now Bittensor subnet Chutes is training a 1T parameter model: Chutes Parallax takes a sparse Mixture-of-Experts (MoE) decomposition approach for decentralized training on heterogeneous, non-colocated GPUs. It shards expert ownership across "composers" (participants), who maintain compact low-rank surrogates for non-owned experts, minimizing communication, VRAM, and compute per node while routing across the full expert space. This enables privacy-preserving training (via compressed activation sketches, no raw data shared) and has already demonstrated 20B-parameter runs on mixed hardware (e.g., H100s, RTX 4090s) for under $10/hour.