三个月后...它从 13 t/s -> 128 t/s
它进化了,我也成长了😋
折腾到大半夜的,汇报下这几天深耕 ncnn-MoE-Runtime 的进展!
好消息是,它可以在 5070ti (16G vRAM)+ 9800X3D + 32G (7600Mhz) 内存组合下情况下,可以跑起 GPT-OSS-120B 的模型
且最高峰值可达 13.36 token/s
想到将来能够通过更低的硬件成本,能够运行起更多的 LLM 时就非常的兴奋!
接下来继续优化,让我来探索下极限边界在哪!
项目地址:github.com/crafcat7/ncnn-MoE…
蹲点大佬们的 star(目前优化还没上传!目前仓库内仅能运行起 120b 模型)
nihui retweeted
This screenshot shows Qwen3-0.6B LLM running on a pure IA32 processor such as the Intel Core Solo T1350, based on github.com/crafcat7/ncnn-MoE… from @crafcat7.
I reverse engineered ARKit to learn how it works and bust the myths.
It is a ML model that takes a 128x128 pixel RGB selfie and spits out the blend shapes, plus post-processing.
I made it work on macOS, with any webcam. OpenVT mac builds coming soon.
gist.github.com/hoshinolina/…
libwaifu/waifu2d 第一版完成!🎉
虽然头发和口型还有些问题,但至少waifu能动了!目前整套流程都是自动化的:AI 生成图片→See-through模型分层→驱动动画,无需任何手工修改。感谢See-through项目:github.com/shitagaki-lab/see…
如果感兴趣,欢迎给 libwaifu 点个 ⭐
github.com/ling0322/libwaifu
Visualizing to learn model internals in the agentic era can be surprisingly low-friction.
I had Codex one-shot architecture notes earlier for several recent OSS models—DeepSeek V4/V4.1, GLM 5.3, and Kimi K3—specifically from a kernel-development perspective: tensor contracts, exact shapes/layouts, attention variants, KV-cache representation, precision choices, dataflow, and operator boundaries.
Sharing in case they’re useful to others:
lei.chat/models/
好耶,libwaifu支持Qwen-Image-2.1成功🎉
github.com/ling0322/libwaifu
nihui retweeted
I wanted to add extra GPU options to the benchmark suite, sooo I added @nihui's vkpeak to cover a range of options. Only the @RadxaComputer X4L on sbc.compare/vkpeak so far, but as with others, new boards will be tested with it, and others backfilled later!
For full data, the table at the bottom of the comparison pages such as sbc.compare/radxa-x4l is where it will live.
nihui retweeted
Vulkan `vkpeak` benchmark for the new Arduino Ventuno Q reaches 400+ FP32 GFLOPS.
./vkpeak 0
device = Adreno623
driver = turnip Mesa driver / Mesa 25.2.8-0ubuntu0.24.04.2
fp32-scalar = 417.44 GFLOPS
fp32-vec4 = 438.42 GFLOPS
fp16-scalar = 594.99 GFLOPS
fp16-vec4 = 853.58 GFLOPS
int32-scalar = 109.97 GIOPS
int32-vec4 = 111.09 GIOPS
int16-scalar = 283.50 GIOPS
int16-vec4 = 294.14 GIOPS
int64-scalar = 11.31 GIOPS
int64-vec4 = 6.33 GIOPS
int8-dotprod = 621.29 GIOPS
copy-h2h = 10.11 GBPS
copy-h2d = 11.41 GBPS
copy-d2h = 9.12 GBPS
copy-d2d = 13.66 GBPS
@arduino
YuzukiNeko - A Pico Shape Cute CatBoard, With Allwinner F101 SoC
Hardware OpenSource under CC0: github.com/YuzukiHD/YuzukiNe…
Linux Kernel: github.com/YuzukiHD/linux-ma…
Buildroot: github.com/YuzukiHD/buildroo…
Zephyr: github.com/YuzukiHD/zephyr
nihui retweeted
From dense models to 122B and 295B MoE, local inference should work with the hardware you have—not require a datacenter.
Palm-Infra is open source. Built by Tencent's #PalmAI team, it brings Metal, CPU SIMD and SSD expert offload to Apple Silicon.
nihui retweeted
11k views in 1 day for a technical 1h 40min video!
My hope for humanity has been restored. There's still people out there watching long videos :)
youtube.com/watch?v=aQv9pUl9…
nihui retweeted
Two failed attempts. One working solution.
Building a real-time forklift proximity alert system on a Raspberry Pi 5 taught me the biggest bottleneck wasn't the model, it was the deployment stack.
What ONNX Runtime, INT8 quantization and NCNN taught me:
medium.com/p/two-failed-atte…









































