Google Cloud Marketplace Announcement: baseten.co/blog/baseten-part…

🔥 We have two huge announcements today! 🔥 We’re thrilled to announce early access to our new Baseten Hybrid offering and our launch on the @googlecloud Marketplace! Links to the announcement blogs and waitlist for Hybrid in the comments below 👇 With Baseten Hybrid, you have complete control over your policies and workloads with the flexibility to scale up as needed on our cloud. This solves a common problem: you want to self-host certain workloads to meet compliance requirements, but when push comes to shove, you need to tap into extra compute. Run sensitive workloads securely in your VPC, meet specific data residency requirements, and fully utilize existing investments in providers like @googlecloud. When you need extra compute, effortlessly spill over to Baseten Cloud—zero engineering effort required. With our launch on the Google Cloud Marketplace, it’s now easier than ever for Google Cloud users to leverage Baseten’s platform directly within their cloud ecosystem. Our growing partnership provides seamless interoperability, secure data management, and the ability to quickly spin up high-performance AI applications in your Google Cloud environments. Get early access to Baseten Hybrid or get started today with Baseten Cloud on the Google Cloud Marketplace!

We're excited to introduce our new Engine Builder for TensorRT-LLM! 🎉 Same great @nvidia TensorRT-LLM performance—90% less effort. Check out our launch post to learn more: baseten.co/blog/automatic-ll… Or @philip_kiely's full video: youtube.com/watch?v=h4F6s84v… We often use TensorRT-LLM to support custom models for teams like @Get_Writer. For their latest industry-leading Palmyra LLMs, TensorRT-LLM inference engines deployed on Baseten achieved 60% higher tokens per second. We've used TensorRT-LLM to achieve results including: 📈 3x better throughput 📉 40% lower time to first token 📉 35% lower cost per million tokens While TensorRT-LLM is incredibly powerful, we and our customers repeatedly faced tedious, lengthy, resource-intensive builds. We created the TensorRT-LLM Engine Builder to eliminate hours of manual work and bring the power of TensorRT-LLM to more teams. 💪 Now, you can automatically build optimized model-serving engines for open-source and fine-tuned LLMs in minutes! Leveraging our new Engine Builder on Baseten gives you full control to customize your model server, dedicated deployments with automatic traffic-based scaling, logging and metrics observability, and leading security and compliance. TensorRT-LLM is compatible with 50+ LLMs, including foundation models like Llama, Mistral, Whisper, and their fine-tuned variants. If you have any questions about how to get the best possible performance for LLMs in production, we'd love to help! Try out the TensorRT-LLM Engine Builder with $30 in free credits 🚀 app.baseten.co/signup/

Another first 🎉 Unlock the power of @nvidia's Multi-Instance GPU (MIG) virtualization technology with H100mig GPUs on Baseten: baseten.co/changelog/fractio…

Launching today 🎉 Double your throughput or halve your latency for @MistralAI, @StabilityAI + others? Do both at ~20% lower cost with @nvidia H100s on Baseten. Here’s how 👇