Pinned Tweet
Qwen3.8-27B below 3 bpw: we’ve released two GGUF versions at 2.96 / 2.48 bpw, weighing in at just 9.71 / 8.20 GiB, both with MTP included. Both achieved 100% on AIME 2026 in our evaluation.
We’ve also slimmed down the vision encoder and resolved its GGUF quantization compatibility issues—no modifications to llama.cpp required. The Q4_K version is just 314.46 MiB, 64.59% smaller than BF16, while retaining 0.99537 cosine similarity to the original HF FP32 vision outputs.
Together AI 发了一篇技术复盘,讲他们为什么删掉去年手写的 megakernel,改用 AI agent 改写和融合 GPU kernel 的 Together Compiler。
去年他们把整个前向传播塞进单个持久 kernel,H100 上跑 Llama 1B 做到 990 TPS,但团队花几个月才上线 1 个模型、batch 1、BF16 和 1 个客户。今年他们从 669 TPS 的默认 Torch Inductor 开始逐项拆解,加 CUDA Graphs 到 864 TPS,加 PDL 到 953 TPS,把 PDL 同步挪到首次读取前预加载权重后到了 975 TPS,不手写 kernel 就追回 98%。
实验推翻了原先对 megakernel 的判断:消除启动开销和重叠计算不一定靠单一大 kernel,它也没省掉同步开销,每层多数边界仍要全网格屏障,固定的 20 warp 配置还会让 Hopper 上的 GEMM 慢 1.2 到 1.5 倍。默认 Torch Inductor 的启发式和隔离测速也有偏差,比如 freezing 开关在 batch 1 下会拖慢 49%。
最后他们转向 multi-kernels 加 agent 循环的 Together Compiler,按模型、芯片和流量融合与调优 kernel,官方称最高能把吞吐提到 2.6 倍。H100 上 Llama 1B 超过 1100 TPS,B200 上 Llama 3.2 1B 做到 2118 tokens/s,比手写 megakernel 的 1649 tokens/s 快 28%。
原帖 teapawt.org/togethercompute/status… Blog together.ai/blog/road-to-tog…
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput.
Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization.
Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team!
Read more: together.ai/blog/road-to-tog…
Cornell 的 MegaOps 也在解决同一个问题:不融合成 megakernel,只靠独立算子 kernel 能不能拿到同等性能。
和 TogetherAI 用 PDL 加 agent 融合调优不同,MegaOps 让每个算子继续作为独立 kernel,把有依赖的 kernel 放到不同 CUDA stream 上,用信号量把同步粒度缩到中间张量的 tile 上。
MegaOps 在单卡 H100、A100、L40 以及 4×A100、8×H100 上测了 0.6B 至 70B 的 Llama 和 Qwen3。在 ThunderKittens 手调的负载上,MegaOps 吞吐是它的 0.89× 至 1.15×;相比各场景最佳基线,单卡和多卡 decode 平均 1.10× 和 1.09×。作者测到模型越大、加速比越小。
arxiv.org/abs/2611.16248
1
159
最近看到好多的全双工 speech-to-speech 模型是参考 moshi 思路构建的
moshi 于去年 2025 年开源了 finetune 代码 github.com/kyutai-labs/moshi…
不禁回想起 2024 年构建全双工语音模型也是参考 moshi 做的设计,而负责训练的同学三个月不提交训练代码且一直训练不出合理智力的模型;
不可忍受 强烈要求必须提交代码,一个周末我通过 review 抓出来了两个错误 1)数据导出错误 2)训练 RQ-Transformer 的数据排列错误 导致 loss 计算不合理 训练推理不一致
改完这两个错误 infra 同学随便就训练出能对话的模型了
Feiteng retweeted
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end…
Feiteng retweeted
how should multimodal models encode visual input?
→ pretrained encoder?
→ encoder-free?
→ before the LLM?
published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters
only one of them does
🧵 [1/n]
DeepSeek V4.1 Flash 量化 100G 第一个版本在 benchmark 上表现不佳
我做了一次深入的分析:
1. 四个 benchmark 有两个竟然在校准数据中几乎没有覆盖(也是表现不佳的两个)
2. 如果不做专家裁剪常规的量化格式完全做不到 100G尺寸,为此我在社区已有的工作上发展了一个新型的量化格式做到了不裁剪专家也能压缩到 100G,重建误差和更高 bpw 的水平相当
看起来进展不错
1
4
978
哈哈中年搞机
我这半年买了 PRO6000 和 Mac Studio 256G,花费约 18w,还是发挥了重要作用的;
后悔想买 NV Spark 的时候犹豫了下,觉得没空搞这么多机器,没珍惜 5 万多的 PRO6000 最终八万五拿下 🤢
중년의 위기를 극복하기 위해 로컬LLM장비 구매 2개월 후기
1. 역시 사기 전이 가장 재밌다.
2. 가성비 따져가며 샀다가 사자마자 후회함, 관심을 끌기 위해서는 사람들이 가장 많이 쓰는 구성이 좋다는것을 깨닳음(5090, Dgx spark, Rtx pro 6000) 로컬LLM을 안하는 사람은 아예 관심이 없고 하는 사람은 자기 장비에서 도느냐가 중요하기 때문, 심지어 한국어권은 진짜 한줌단
3. 서빙하는걸 최적화 하느라 서빙을 할 수가 없음
4. 5천만원 이내로 haiku 5.5보다 좋기 어려움
5. 좋아요 많이 받기 위해서는 별거 아닌 무가치한 숫자를 굉장하다며 과장광고해야되는데 그렇게 그나마 없는 인망을 잃고싶지 않음
6. 이미지/영상 같은 디퓨전 모델은 평소에 쓸일이 없었는데 장비 아까워서라도 써보게 됨 이건 꽤 좋음
결론: 중고로 사서 몇개월 재미와 지식을 얻고 살짝 감가상각하여 판매한다면 같은 금액을 주식에 넣은 기대수익보다 나은것 같다. 중년의 위기 극복에는 꽤 좋았다.
1
936
Feiteng retweeted
Megakernel blog is out now
We implement a full layer (MoE + Attention) megakernel, achieving 1100 toks/s for Muse Spark, 2.1x faster than baseline.
Nvidia's Instruction cache makes it really hard to write bigger kernels. We ended up tuning both instruction and data caches at all levels for extreme latency hiding!
With this, some warps of next operation can start running before previous operation has finished - redefining what GPU utilization means.
Full blog post: lmsys.org/blog/2026-11-05-mu…
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels.
One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97.
lmsys.org/blog/2026-11-05-mu…
In our new technical post, Pranjal Shankhdhar explains what makes this work:
- Prefetching weights into shared memory and L2 before they’re needed.
- Warming the instruction cache so large kernels don’t stall fetching their own code.
- Using data itself as a readiness signal for cross-GPU communication.
- Fusing the draft model to make drafting 2.6× faster.
The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.
谷歌最新发布了其新一代DNA序列模型:AlphaGenome 2,他们这次把DNA功能预测推进到了单细胞层面!
甚至能预测同一个DNA变异在不同细胞状态下产生什么影响
这对疾病研究和药物靶点发现很有价值,有助于提高疾病机制研究和早期药物研发决策的效率
相比第一代AlphaGenome的能读取100万碱基的DNA序列,预测基因表达、RNA剪接、染色质可及性等多种分子特征
AlphaGenome 2这次的核心突破,不是单纯扩大DNA输入长度,是让模型具备细胞状态条件化预测能力
它不仅分析DNA序列,还结合目标细胞的状态信息预测遗传变异在特定细胞环境中会如何影响基因调控
在超过1.07亿个单细胞数据档案,68种组织上训练
支持未见过的新细胞状态
研究人员可以输入新的单细胞RNA测序数据,通过Cell Encoder提取细胞状态表征,无需为每种新状态重新训练整个模型
除了基因表达、染色质可及性和剪接等,还增加了DNA甲基化预测
以前我们知道某个DNA位置与疾病有关,但可能不知道它在哪里起作用,为什么起作用
AlphaGenome 2的这条预测链路如果能通,就能更准确的判断某个变异影响哪类细胞、哪个基因,能帮科学家缩小实验范围,把资源集中到更有可能成立的机制上
#AlphaGenome2 #AIforScience #AI医学
























