Anton McGonnell

Many thoughts for @SambaNovaAI perspective... Firstly, this is great validation that kernel-by-kernel execution models will never suffice for fast inference. CUDA is a kernel-by-kernel paradigm, so for fast inference, there is no CUDA moat. Secondly, TileRT is dataflow emulation and use some of the same concepts we do in our compiler. However, doing this on hardware that has been built for kernel-by-kernel execution is inherently limited and this shows up in a few different ways. As described below: 1. Can they increase model parallelization to get faster? There is very little evidence of any GPU being able to scale tensor parallel beyond 8 GPUs, despite the fast that there is a 72-GPU scale-up domain readily available now. On InferenceX, no-high speed decoding use case uses more than 8 GPUs (data parallel doesn't count). In fact there is still a lot of HBM this is left unutilized in the TP8 mappings that TileRT has shown. Unless they can show an ability to saturate HBM bandwidth within a large model parallel domain, they will not keep up with true dataflow chips. 2. Can they batch without killing interactivity AND at long context length (truly Premium Inference)? TBD but my feeling is that this will be extremely difficult using dataflow emulation on GPUs, even on Rubins. On SN50, we will be able to reach the same speed with much higher batches per chip and at the 140K median context length that has become the default on InferenceX. 3. Can they generalize? The article rightly points out how hard it is to generalize - "a persistent engine kernel means the model is statically expanded ahead of time into one resident program". This is indeed extremely hard. We have been working on this problem for 9 years in our compiler. We haven't solved all these problems yet but are extremely well placed to be the first that does. We also have the right hardware target that makes the problem much easier than trying to retrofit GPUs to do this. The most salient point in the article is the fungibility that this enables, but ultimately I think this will give buyers more reason to buy dataflow chips, as long as they do not over-specialize. i.e. if I am a AI inference provider and I buy some systems for prefill and high throughput decoding, and some systems to fast decoding, and my workloads change to require more fast decoding, I can reallocate my GPUs to service this demand with TileRT, albeit at a much higher cost per 1M tokens. Similarly, if the workloads move in the other direction and I need more high/medium throughput and low/medium interactivity, I should be able to reallocate my dataflow chips for this and offer good enough tokenomics. This is exactly how we think of the problem, as you need fungibility at the edges, not at the extremes, and you want to cover enough of the pareto curve to be able to adapt, but still enough specialization to differentiate.