System64 (@System64fumo) header
System64 (@System64fumo) profile picture

System64

@System64fumo

🇺🇸🇮🇱 Icy fairy goofing around with embedded systems for fun!

 Israel
github.com/System64fumo
Joined May 2016
  • Tweets557
  • Following117
  • Followers94
69Media
Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)
I'm proud to announce that github.com/System64fumo/kapp… has hit a major usability milestone. If you're interested in #LocalAI now's the time to start min/maxing your hardware.
GitHub - System64fumo/kappai: High performance inference engine

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
2
7
58
Na, I'm not done here yet, There's still plenty more where that came from.
1
19
Not sure what people care about more, Reducing memory usage or making AI go faster in general.
14
Call me crazy, Call me delusional, But i know for a god damn fact YOUR computer is faster than you were lead to believe.
Image from System64 (@System64fumo)
1
24
Did some work on kappai today and i think i did well? Still very W.I.P, Chat did i cook? 👀
Image from System64 (@System64fumo)
1
31
I'm not pulling off any special tricks here or sabotaging llama.cpp by having some load in the background, Both engines are running with the same parameters same conditions, etc.. Unfortunately (Or rather fortunately) this isn't the end, There's still plenty of headroom left.
18
Compute is expensive.. Let's make it cheaper, And local!
Image from System64 (@System64fumo)
2
17
145
I cooked hard, Here are some things that landed in kappai recently: GLM 5.3 Flash support OpenGL ES 3.1 backend Several new IQ quants TUI setup/run screen & many more There's also a new matrix and discord for this project, Come join us! github.com/System64fumo/kapp…
GitHub - System64fumo/kappai: High performance inference engine

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
1
38
I don't actually use discord btw so if you want to contact me do so on matrix.
16
Holidays are over, Time to cook again. Where are all my cracked C devs at? 👀
26
Ok so i didn’t expect to get this much attention, But damn those are a lot of eyes. 👀 I’m out for holidays right now so sorry for the lack of updates. Thank you Bee fumo! 🐝
Image from System64 (@System64fumo)
2
66
I have some pretty telling evidence that Union Alpha is some sort of GLM model. @Zai_org Can we please "pace the frontier"? /s And while you're at it could you kindly fix the glaring security hole? I wanted to work not dive down this rabbit hole, k? thx.
1
119
Happy #CirnoDay!
Image from System64 (@System64fumo)
2
105
4
82
768
10,377
I don't know what others think but i find it quite amusing that i made a 12 CPU (CD8180) go toe to toe with a 20 core iGPU. (M4 Pro mac mini) Wish i had proof for this but the proper comparison with GLM 5.2 gave me: 0.38PP and 0.32TG Numbers from colibri:
Image from System64 (@System64fumo)
1
152
Also please don't get me wrong, I'm not throwing shade at colibri or llama.cpp. They're great projects, I just can't benchmark properly since i don't have disk space nor memory for either. Sorry if that came off wrong. 😅
81
Humm maybe claiming to have a performance oriented inference engine followed by showing sub token performance on a pretty banger aarch64 system doesn't quite showcase the improvements.. Fine here are, Some direct 1:1 comparisons with llama.cpp on rk3588. 🧵
1
1
136
more replies
I also have an NPU backend that does indeed run here but unfortunately i can't publicly ship that code since my license is M.I.T and the NPU code is based on GPL.
1
33
Note: Just like my last post i had to re-create this post too, Again sorry!
27
Load more