Feiteng (@FeitengLi) header
Feiteng (@FeitengLi) profile picture

Independent Researcher Speech · Image · Video · LLM multimodal generative models Research→Infrastructure→Built AI algo serving 250M+ users 公众号: Generative AI

Singapore
Joined November 2016
Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)Photo by Feiteng (@FeitengLi)
Qwen3.8-27B below 3 bpw: we’ve released two GGUF versions at 2.96 / 2.48 bpw, weighing in at just 9.71 / 8.20 GiB, both with MTP included. Both achieved 100% on AIME 2026 in our evaluation. We’ve also slimmed down the vision encoder and resolved its GGUF quantization compatibility issues—no modifications to llama.cpp required. The Q4_K version is just 314.46 MiB, 64.59% smaller than BF16, while retaining 0.99537 cosine similarity to the original HF FP32 vision outputs.
3
5
1
39
6,717
Together AI 发了一篇技术复盘,讲他们为什么删掉去年手写的 megakernel,改用 AI agent 改写和融合 GPU kernel 的 Together Compiler。 去年他们把整个前向传播塞进单个持久 kernel,H100 上跑 Llama 1B 做到 990 TPS,但团队花几个月才上线 1 个模型、batch 1、BF16 和 1 个客户。今年他们从 669 TPS 的默认 Torch Inductor 开始逐项拆解,加 CUDA Graphs 到 864 TPS,加 PDL 到 953 TPS,把 PDL 同步挪到首次读取前预加载权重后到了 975 TPS,不手写 kernel 就追回 98%。 实验推翻了原先对 megakernel 的判断:消除启动开销和重叠计算不一定靠单一大 kernel,它也没省掉同步开销,每层多数边界仍要全网格屏障,固定的 20 warp 配置还会让 Hopper 上的 GEMM 慢 1.2 到 1.5 倍。默认 Torch Inductor 的启发式和隔离测速也有偏差,比如 freezing 开关在 batch 1 下会拖慢 49%。 最后他们转向 multi-kernels 加 agent 循环的 Together Compiler,按模型、芯片和流量融合与调优 kernel,官方称最高能把吞吐提到 2.6 倍。H100 上 Llama 1B 超过 1100 TPS,B200 上 Llama 3.2 1B 做到 2118 tokens/s,比手写 megakernel 的 1649 tokens/s 快 28%。 原帖 teapawt.org/togethercompute/status… Blog together.ai/blog/road-to-tog…
Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…
1
1
40
3,575
Cornell 的 MegaOps 也在解决同一个问题:不融合成 megakernel,只靠独立算子 kernel 能不能拿到同等性能。 和 TogetherAI 用 PDL 加 agent 融合调优不同,MegaOps 让每个算子继续作为独立 kernel,把有依赖的 kernel 放到不同 CUDA stream 上,用信号量把同步粒度缩到中间张量的 tile 上。 MegaOps 在单卡 H100、A100、L40 以及 4×A100、8×H100 上测了 0.6B 至 70B 的 Llama 和 Qwen3。在 ThunderKittens 手调的负载上,MegaOps 吞吐是它的 0.89× 至 1.15×;相比各场景最佳基线,单卡和多卡 decode 平均 1.10× 和 1.09×。作者测到模型越大、加速比越小。 arxiv.org/abs/2611.16248
1
159
最近看到好多的全双工 speech-to-speech 模型是参考 moshi 思路构建的 moshi 于去年 2025 年开源了 finetune 代码 github.com/kyutai-labs/moshi… 不禁回想起 2024 年构建全双工语音模型也是参考 moshi 做的设计,而负责训练的同学三个月不提交训练代码且一直训练不出合理智力的模型; 不可忍受 强烈要求必须提交代码,一个周末我通过 review 抓出来了两个错误 1)数据导出错误 2)训练 RQ-Transformer 的数据排列错误 导致 loss 计算不合理 训练推理不一致 改完这两个错误 infra 同学随便就训练出能对话的模型了
12
29
277
12,230
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
68
292
100
2,541
532,120
how should multimodal models encode visual input? → pretrained encoder? → encoder-free? → before the LLM? published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters only one of them does 🧵 [1/n]
23
138
27
862
128,481
今天 六个月半年整,头一次经历这事🤢 前司是平薪入职(实际降薪 没补充公积金 +补贴)🤯
大家拿大礼包或离职后都gap多久了?
1
514
DeepSeek V4.1 Flash 量化 100G 第一个版本在 benchmark 上表现不佳 我做了一次深入的分析: 1. 四个 benchmark 有两个竟然在校准数据中几乎没有覆盖(也是表现不佳的两个) 2. 如果不做专家裁剪常规的量化格式完全做不到 100G尺寸,为此我在社区已有的工作上发展了一个新型的量化格式做到了不裁剪专家也能压缩到 100G,重建误差和更高 bpw 的水平相当 看起来进展不错
1
4
978
还是要写开源项目呀,最近几份工作都不用写简历就直接 offer 了 今年我得再写一个 回报社会
17
1
162
19,748
🧟‍♂️🤖😆😵
Replying to @wholemars
Haven’t even had sex in ages (sigh)
811
如果这个身份之前被封过号就别再用了,换个新证就行,上次十一大封特封后开的新号目前活着
我的Claude突然要求做身份验证,用中国身份证,会马上封号吗? 在线等,挺急的。
587
哈哈中年搞机 我这半年买了 PRO6000 和 Mac Studio 256G,花费约 18w,还是发挥了重要作用的; 后悔想买 NV Spark 的时候犹豫了下,觉得没空搞这么多机器,没珍惜 5 万多的 PRO6000 最终八万五拿下 🤢
중년의 위기를 극복하기 위해 로컬LLM장비 구매 2개월 후기 1. 역시 사기 전이 가장 재밌다. 2. 가성비 따져가며 샀다가 사자마자 후회함, 관심을 끌기 위해서는 사람들이 가장 많이 쓰는 구성이 좋다는것을 깨닳음(5090, Dgx spark, Rtx pro 6000) 로컬LLM을 안하는 사람은 아예 관심이 없고 하는 사람은 자기 장비에서 도느냐가 중요하기 때문, 심지어 한국어권은 진짜 한줌단 3. 서빙하는걸 최적화 하느라 서빙을 할 수가 없음 4. 5천만원 이내로 haiku 5.5보다 좋기 어려움 5. 좋아요 많이 받기 위해서는 별거 아닌 무가치한 숫자를 굉장하다며 과장광고해야되는데 그렇게 그나마 없는 인망을 잃고싶지 않음 6. 이미지/영상 같은 디퓨전 모델은 평소에 쓸일이 없었는데 장비 아까워서라도 써보게 됨 이건 꽤 좋음 결론: 중고로 사서 몇개월 재미와 지식을 얻고 살짝 감가상각하여 판매한다면 같은 금액을 주식에 넣은 기대수익보다 나은것 같다. 중년의 위기 극복에는 꽤 좋았다.
1
936
A\ 前几天送的 Fable 重置还是很有意思的,正好赶上 Fable 5.2 发布,Fable 用量占比高就适合先用 Fable 重置
375
Feiteng retweeted
Megakernel blog is out now We implement a full layer (MoE + Attention) megakernel, achieving 1100 toks/s for Muse Spark, 2.1x faster than baseline. Nvidia's Instruction cache makes it really hard to write bigger kernels. We ended up tuning both instruction and data caches at all levels for extreme latency hiding! With this, some warps of next operation can start running before previous operation has finished - redefining what GPU utilization means. Full blog post: lmsys.org/blog/2026-11-05-mu…
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels. One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97. lmsys.org/blog/2026-11-05-mu… In our new technical post, Pranjal Shankhdhar explains what makes this work: - Prefetching weights into shared memory and L2 before they’re needed. - Warming the instruction cache so large kernels don’t stall fetching their own code. - Using data itself as a readiness signal for cross-GPU communication. - Fusing the draft model to make drafting 2.6× faster. The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.
13
62
10
602
34,183
Feiteng retweeted
The main reason H3 Image struggles on detail is that it has a highly compressed latent space. Here is a round trip comparison of the most used VAEs. The H3 Image VAE scores worse for this image on every metric, even worse than the SD 1.5 VAE.
36
24
3
388
25,650
Feiteng retweeted
VoxMel Wavelet-Factorized Autoregressive Mel-Spectrogram Modeling for Zero-Shot Speech Synthesis models: huggingface.co/VoxDance-Lab
16
20
4
206
19,236