@noozie
Account based in!
About this account
- Account based in
- North America
- Connected via
- North America Android App
X says this location may be affected by a proxy or VPN.
Account-level information from X, not a live location or the device used for a specific post.
Ancient thunder in a modern sound. Wired to the earth in a savage ballet. I don't do TL;DR. Foldb, WILD, Orbit, other software.
🇨🇷
Joined May 2023
- Tweets22.7K
- Following1.2K
- Followers3.1K
- Likes8.1K
Even after 8 years, my Sennheiser HD800S headphones still put out bass that punches just right.
Using them with a woo audio dac, and a thx aaa 789 amp, in case curious.
Running 2 machines running LLM inference in a space about the size of a single bed + 2 meters more length but the same width... is not a fun time.
GPUs and stuff are raising the temperature of my office beyond 30C.
You may think, "but you're in a tropical country, that's not bad" ... it's night time, house has no heating or cooling, and leaks like a sieve, so air gets in, it's 18C outside right now... the rest of my house is 19C - 20C... my office? 32C.
Hey TV manufacturers... I have an idea...
Do whatever you want with the TV you sell us, SO LONG AS, you publish full technical on what the TV needs to boot and operate as a TV... You don't have to give keys or anything like that, unless bootloader firmware is signed and verified meaning any custom firmware we build would need that signature, then you must sign those payloads.
Then I'd be happy to buy your TV knowing there's a way to stop you doing the things I don't want you to do.
You won't...so I won't.
And no, that prefill speed isn't correct, it was a very small prompt, so don't trust that number. I'm doing an eval right now.
Canadian politicians have no idea about the Laffer curve...and it shows.
They also don't have any clue what economic freedom is, and why their policies often limit it, rather than promote it, and are blind to why that has a negative impact on ... tax revenue.
Breaking: Canadians pay too much income tax. 🙄
Anyone with a paycheque could've told you that for free.
New international tax index: Canada ranks 28th of 38 countries on personal income taxes.
34th on capital gains and dividends. 📉
Ontario's top rate hits 53.5%.
It kicks in at 2.8x the average wage. In the U.S.? 8.7x.
Carney's big fix was one point off the bottom bracket.
Then Ottawa scratches its head over why talent keeps heading south. 🤦🏻♂️
Basic math, eh? 🇨🇦
Had an idea today to make WILD a bit more efficient for cache performance...
Right now, the way readBatch() works is a for loop from 0 to the number of keys being read, and reads them serially. In hindsight this wasn't the best idea.
But if I change this to be a multi-phase pipeline... where I hash each key, calculate the first slot, and prefetch its record, then revisit each request, prefetch next probe line if it's present, finally materializing the results...
Then that should achieve MLP that helps when something falls out of L2/L3 cache, keeping more work in flight while waiting for memory to catch up.
Update: Idea worked... batched reads hash every key, prefetch the home cache line of the whole group, then resolve.
So that changed the question to "what's the right number of groups (for this machine) to get the best performance?" ... turns out, 8; because this machine is 8 core (16 with SMT).
So for a 512MB table, 96MB L3 cache, 32.1 ns/key became 20.4 ns/key, 1.6x faster.
IPC jumped from 0.28 to 0.57, and single key reads and writes stayed unchanged.
I also lifted the 75% of L3 cache size, so the old numbers and the new numbers from the readme file... so now your capacity can be as big or small as you want... but records still must fit into a single cache line, so... use cases are still limited.
Oh and because I didn't share a link, so here you go: github.com/jeremytregunna/wi…
Its usefulness is very narrow (data needs to be small, 52 bytes max per record) ... but I use it in production workflows... also now needs zig 0.17.
I use it for network decision table storage, but useful for storing entity state in games, ads and recommendation lookup... It's also so fast, the majority of the time spent retrieving a single key is a syscall, AND a tcp round trip. 20ns is just noise relative to the syscall and tcp round trip.
My rules for performance optimization:
1. Do less work on the hot path
2. Keep the pipeline full of executable work
3. Minimize data movement
4. Control flow must be predictable
5. Increase instruction & memory level parallelization
Yours?
Suggestion for anyone with an AI subscription that plans to do anything involving security, to, upon activating your new subscription, do exactly the kinds of things you'll use it for, in the most borderline way possible for those extreme cases.
If your account gets banned, you find out quickly, and can pick another provider, conduct the same test, until you get frustrated and run models locally or you find a provider that doesn't ban people for their legitimate work.
When you haven't yet realized that you control how you feel, or you abdicate it to others to choose for you.
Take control of your life man... I suggest everyone who wants to, to practice asking yourself, "do I want to be offended/angered/disgusted/whatever?" and give yourself an answer first, before you allow yourself to feel that way.
Ok calling the deepseek v4.1 flash 1017 project done... My memory requirements can't be met. The 4x 32GB sticks I had, one of them is bad, so best I could get it up to is 320GB and that wasn't enough.
Good news is, it's just a memory issue and the code actually works. So I'll save the patches, since I can't prove it with a full run, it's also fused kernels so it doesn't work without sm86 gpus and sm121... but hey, was a fun experiment.
Next step on getting dsv41 running seems to be moving 256gb out of one of my lab servers, replacing with 128gb... taking those 4 sticks of 64gb putting them in my gpu server, alongside 4 other sticks, so I can fit everything needed. Expert offload needs 218gb alone, roughly... gpu server runs 256gb right now, so this'd bring it up to 384gb.
Spent my evening and now part of my morning porting TensorFold to run on Ampere generation A6000 GPUs in TP=2, while keeping all experts on the GPUs, all other weights on a dgx spark, and yet remaining things in memory on the GPU node.
Yeah I'm trying to see how far I can push the hardware I have, and DeepSeek V4.1 Flash 1017 seems like it was built to help me with that. 384GB ram, 48GB x 2 VRAM, 128GB Unified memory and an rdma link between gpu node and spark...
About 75% done the code prepping for first test today.
Doing the deepseek v4.1 flash experiment... grabbing a cache of the models, so I can test my patches to TensorFold for sm86 and sm121 equipped machines cooperating across an RoCE network using every last drop of ram and vram available to run dsv4.1...
Let's see if it works.
Got thinking early this morning about using my spark and my A6000 GPUs (ampere cards) with a combination of offloading to ram and stuff to the spark. (Target model is DeepSeek V4.1 Flash 1017)
But use the 2x A6000s for attention, kv cache, shared expert, router, sampler, hot experts, while the ngram stays resident in memory.
Spark becomes the expert pool, mid and cold experts, read by the GB10...
Doing the math, the EXL3-EAQ weights weigh in at 210.66GiB, the n-gram tablens 202.76GiB, and the expert tensors ~196GiB.
Loading the model this way adds about 69 seconds, and 58 seconds on the spark for the engram load. About 2 1/2 minutes to cold start. But I don't care about that really.
So 60% of the workload stays on the GPUs, and 33% would live on the spark, while 7% lives on system memory in the GPU server itself.
Worst case overhead per token, is under 4ms, theoretically, and the network will add about 0.6ms (measured value before optimization).
Theoretically should be able to hit about 30-60 tok/sec decode, and 1.5-2.5k tok/sec prefill. Which is in the "sweet spot" for acceptable and usable. Faster is better sure, but anyway this works...if it works.