Omkar Chebale (@omkarchebale) header
Omkar Chebale (@omkarchebale) profile picture

Inference | AI/ML Engineer

India
Joined November 2023
Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)Photo by Omkar Chebale (@omkarchebale)
Try out UltraSearch in Firecrawl
Introducing UltraSearch for agents. Get search results with data snippets that matter including: → 250M+ people with job titles → 75M+ companies with funding → 150K+ stocks with live prices & more Supercharge every search with Firecrawl Alexandria. alexandria.firecrawl.dev/go/…
1
177
Unbelievable pricing 🤯
today we’re announcing our new plan: Hacker Plan ($4.80/month) with this, you will get: - Unlimited usage on open models - A generous amount of usage on OpenAI and Anthropic models - 6,000 minutes of cloud usage find more details here - withmetaphor.com/pricing
1
1
4
239
Claude Code's Remote Control wants a machine on 24/7. I don't have one, so I gave it a cloud sandbox instead Spun up a box on @instabox_sh , ran `claude rc`, wired it into systemd. If it crashes or reboots, it reconnects on its own. @ClaudeDevs Thanx for free tier @hanghuang_
2
147
Those who are interested in “Inference Engineering” , here’s the structured roadmap
LLM Inference Engineering Roadmap Part 1: Foundations 1. Training vs Inference 2. Inference Engineering 3. Closed vs Open Models 4. Latency vs Throughput 5. Model Selection 6. Model Evaluation (Evals) Part 2: Transformers and LLM Architecture 1. Tokens 2. Token Embeddings 3. Positional Encoding 4. Transformer Architecture 5. Matrix Multiplication (Matmul) 6. Attention 7. Queries, Keys, Values 8. Causal Masking 9. Multi-Head Attention (MHA) 10. Feed-Forward Network (FFN) 11. Logits 12. Mixture of Experts (MoE) 13. Active vs Total Parameters Part 3: LLM Inference Mechanics 1. Autoregressive Generation 2. Chat Templates 3. Context Window 4. Prefill vs Decode 5. KV Cache 6. Sampling 7. Temperature 8. Top-k Sampling 9. Top-p (Nucleus) Sampling 10. Stop Tokens 11. Reasoning Models 12. Token Streaming Part 4: Performance Metrics 1. Time to First Token (TTFT) 2. Time Per Output Token (TPOT) 3. Throughput 4. End-to-End Latency 5. Latency Percentiles (P50, P90, P99) 6. Cost per Token Part 5: Bottleneck Analysis 1. Compute vs Memory Bandwidth 2. Arithmetic Intensity 3. Roofline Model 4. Compute-Bound vs Memory-Bound 5. Prefill Bottleneck (Compute) 6. Decode Bottleneck (Memory) 7. Memory Footprint 8. KV Cache Memory Calculation 9. Batching and Arithmetic Intensity Part 6: Hardware 1. CPU vs GPU 2. GPU Architecture 3. Tensor Cores 4. High-Bandwidth Memory (HBM) 5. DRAM vs SRAM 6. NVLink 7. InfiniBand 8. Groq LPU 9. Local (Edge) Inference 10. Cloud vs On-device Deployment Part 7: Software Stack 1. CUDA 2. CUDA Kernels 3. CUDA Graphs 4. Kernel Fusion 5. PyTorch 6. Inference Engines 7. OpenAI-Compatible Server Part 8: Benchmarking and Profiling 1. Performance Benchmarking 2. Traffic Simulation 3. Load Testing 4. Profiling Part 9: Attention Optimizations 1. Standard Attention Inefficiency 2. FlashAttention 3. PagedAttention 4. Multi-Query Attention (MQA) 5. Grouped-Query Attention (GQA) 6. Sliding Window Attention 7. Attention Sinks 8. KV Cache Compression Part 10: Batching and Scheduling 1. Batching 2. Static Batching 3. Continuous Batching 4. Batch Size vs Latency 5. Chunked Prefill 6. Prefill-Decode Interference Part 11: Caching 1. Prefix Caching 2. Cache-Friendly Prompt Design 3. Prompt Caching 4. KV Cache Offloading 5. Cache-Aware Routing 6. Long Context Handling Part 12: Speculative Decoding 1. Speculative Decoding 2. Rejection Sampling 3. Acceptance Rate 4. Draft-Target Speculation 5. N-gram Speculation 6. Speculation vs Batch Size Part 13: Quantization 1. Quantization 2. Number Formats 3. Post-Training Quantization (PTQ) 4. Weight-Only Quantization (W4A16) 5. Weight-Activation Quantization (W8A8) 6. KV Cache Quantization 7. Measuring Quality Impact Part 14: Model-Level Optimizations 1. Model Size vs Speed 2. LLM Routing Part 15: Parallelism 1. Why Multi-GPU 2. Model Parallelism 3. Communication Overhead 4. Data Parallelism 5. Tensor Parallelism (TP) Part 16: Disaggregation 1. Prefill-Decode Disaggregation 2. KV Cache Transfer 3. When to Disaggregate Part 17: Production Infrastructure 1. Replicas 2. Autoscaling 3. Cold Starts 4. Observability Start here: github.com/amitshekhariitbhu…
1
3
108
A rule in the prompt is a request. A rule in code is a guarantee. That's the biggest thing I learned building a coding agent from scratch in TypeScript. Not a wrapper. The harness: the code around the model that gives it hands, memory, and rules. I turned it into a repo with 19 chapters. Each chapter is one commit. Check it out, run the command, and read what changed. What you'll build: • an agent loop that calls tools and reads the results • tools to read, search, edit, and run commands • errors the model can act on, instead of crashes • a context budget, and a plan that survives when old results are cleared • permissions, a sandbox, and hooks • a verification loop that won't accept "done" without a passing check • subagents, memory, and tracing with a cost report If you're building agents or working on a harness, I'd love to hear what you'd add or what I got wrong. Bookmark it for when you build your own: github.com/Chebaleomkar/harn…
4
1
126
Last week I had a call with @subhajitsh , co-founder of @withmetaphor What I didn't expect was how openly he walked me through everything, as if I were part of the team: • How their harness keeps only the context a task actually needs, which is how longer sessions hit ~95% cache • Why the agent doesn't re-read a file it has already understood • How they test every open model on coding before adding it (Kimi 3.1 from recently made the cut) • How they route models through @concentrateai • Auto-learning, which saves your preferences to your agents file on its own They've already crossed 100M tokens and are close to half a billion. What stayed with me most is *why* they're building it. Jeet said to think about pricing in rupees, not dollars. $20 is ₹2,000, and most students in India can't spend that even if they want to. So the plan is to keep making it cheaper, or give much more usage on the same plan, so students can actually build their ideas. They also have big plans for India, with more open-source projects on the way. I never imagined that, sitting in my village in Maharashtra, I'd get to meet a founder from SF. Thank you, Jeet, for your time and the Pro access. Really grateful. 🙏
4
1
13
898
AI infra and inference are only going to get bigger. Engineers, it’s worth getting familiar with this space now rather than playing catch-up later.
1
1
6
799
Omkar Chebale retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
2,366
7,256
5,753
55,938
10,066,800
Today, when you use coding agents such as Claude Code or Codex, you usually give an instruction, the agent performs the task, and then you review the result. But what happens when you have 10 tasks? You might open 10 terminals, launch 10 agents, monitor their progress, remember which agent is doing what, and manually coordinate everything. DeepSeek argues that the next bottleneck in AI agents isn't necessarily the model's intelligence. It's the amount of human attention needed to manage agents. Its solution is a Super-agent: one continuously available agent that understands your work across platforms, delegates tasks to independent workers, remembers important decisions, and intervenes only when necessary. The authors say a Super-agent should have four properties: - Continuous: It remembers your work across Slack, GitHub, and other channels. - Productive: It handles multiple tasks simultaneously. - Proactive: It notices relevant events and can begin authorized work without waiting for a new prompt. - Persistent: It stays operational and can recover when something fails.
2
163
I made a menu bar app for AWS. Green dot: running. Red dot: stopped. It also flags a bill line with nothing running behind it. Stopping a server means typing its name. Free, open source, uses the aws CLI you already have. awsbar.vercel.app github.com/Chebaleomkar/AWSB… #infra #awscli
4
1
2
100
I ported Shed to Windows. Control Claude Code and Codex from your phone, with approval questions as buttons. A thread on how, and what broke 🧵
2
71
Testing with real agents found real bugs. Leaked credentials. A world-readable data folder. Tailscale devices that all looked the same. All fixed, 55 tests pass.
1
15
🫡
Replying to @Dheemanthredy
We want to thank the open-source community and the teams whose work made this possible. Maya builds on incredible open foundations: • Backbone: Qwen3-1.7B-Base by the Qwen team (Apache-2.0) @Alibaba_Qwen • Codec: Mimi by Kyutai (CC-BY-4.0) @kyutai_labs • Speaker Embedding: ECAPA-TDNN via SpeechBrain (Apache-2.0) The base was a massive launchpad, but great English open-source models already exist. What India needed was months of intense post-training on authentic, regional phonetic data. That’s why we built this. huggingface.co/maya-research…
34
Wispr Flow is offering 2 months of Pro for FREE! 🎁 Dictate instead of typing, capture meeting notes, and save time every day. 🎟️ Code: DIWALI2 📅 Valid until Nov 10, 2026 Worth trying if you spend hours at your keyboard. #WisprFlow #Diwali @WisprFlow
75
Omkar Chebale retweeted
This was the worst week to be a developer 😵‍💫 If you've deployed anything, read this 👇 PHP2Shell (9.2): Unauth RCE in PHP. UK took 30+ govt services offline. Update to the latest patch NOW. Node.js: Brotli decoder RCE. Update to 22.23.4 / 24.22.1 / 26.12.1 and restart. GitLab: Max severity bug affecting installs since 2014. Patch it. LiteLLM 1.107.0 was malware for ~2 hours. Stole SSH + cloud keys. Pin 1.106.2 and rotate everything. Railway breach: Some customer secrets may have been decrypted. Rotate API keys + DB passwords. OpenCode Console: Linked OpenAI/Anthropic/Google keys leaked. Some people got $1000+ bills. PingCAP: 10.6 TB claimed stolen. The cause? Credentials committed to the repo 🫠 Linux kernel: Two 9.0 RCE bugs leaked publicly on the mailing list before patches shipped. Lesson: Keep .env out of git, pin your dependencies, rotate your keys.
14
21
4
327
17,226