dpolishuk (@dpolishuk) profile picture

Java hipster ;)

Joined May 2009
Photo by dpolishuk (@dpolishuk)Photo by dpolishuk (@dpolishuk)Photo by dpolishuk (@dpolishuk)Photo by dpolishuk (@dpolishuk)Photo by dpolishuk (@dpolishuk)Photo by dpolishuk (@dpolishuk)Photo by dpolishuk (@dpolishuk)
We test AI-generated code. But who tests the tests? Over the weekend, Pavel Zykov and I discussed testing in agentic development. Kun Chen’s experiments make this a particularly interesting conversation. On DeepSWE, banning agent-written tests saved time and tokens without a statistically significant change in quality. In a separate ProgramBench experiment, mandatory TDD reduced the mean test pass rate from 52.4% to 48.8% and increased cost by 55%. The tempting conclusion: agents don’t need tests. I think that’s too broad. Testing remains a critical engineering layer. But we need to ask what our tests actually establish. An agent misunderstands a requirement. It writes an implementation based on that misunderstanding. Then it writes a test based on exactly the same misunderstanding. Everything passes. We’ve established consistency between two artifacts. We haven’t established correctness. This isn’t necessarily reward hacking, either. The agent may not be gaming anything. It can simply be consistently wrong. My takeaway: tests need evals too. Instead of only asking “Does the implementation pass?”, ask “Can this test distinguish a correct implementation from a plausible incorrect one?” Here’s how I’d evaluate it: • The original buggy implementation should fail. • A correct fix should pass. • Plausible incorrect fixes should fail. • Alternative correct implementations should also pass. That last point matters. A test that rejects every implementation except the one the agent just wrote may be enforcing implementation details rather than the requirement. Mutation testing addresses part of this. But detecting changes isn’t the same as identifying incorrect behavior. This is where “Teaching Agents to Code Reliably” becomes interesting. The paper includes a procedure that works without retraining: the harness runs checks against patched and reverted code, checks regressions and execution of changed lines, and uses execution feedback to guide further repair attempts. On SWE-bench Verified, the combined procedure resolved 52.8% of tasks versus 46.8% for an eight-attempt baseline, using 48.1% of its agent steps. That result belongs to the whole procedure—not just the additional test run. And agent steps aren’t equivalent to tokens or dollars. The authors then explore a separate SFT + RL stage, including rewards for tests that detect incorrect patch variants. During training, gold evaluation tests provide an external correctness oracle. So there are two distinct levers: improve the harness around an existing model, and train the model to search and verify better. You don’t need to do the second before trying the first. But there’s a fundamental catch: who decides what “correct” means? Suppose an agent invents a requirement that a fee should be 20%. It writes a test enforcing 20%. That test might catch dozens of mutations and still encode the wrong requirement. Somewhere, you need an independent reference: an agreed contract, a verified example, a trusted specification. Otherwise, you’re evaluating one guess against another. That’s how I connect Chen’s experiments with this paper. The experiments challenge the value of the default workflow. The paper investigates ways to make verification more useful. Neither justifies treating a green test suite as sufficient evidence. For my agent workflows, I want “done” to include two questions: What failure does this test actually detect? Where did its expected result come from? A generated test is an artifact to evaluate, not an authority to trust. Teaching Agents to Code Reliably Kun Chen’s TDD experiment
5
3
149
A new unknown model just appeared in OpenCode and OpenRouter — Ox Alpha — and this one looks seriously interesting 😨 Nobody officially says who made it yet. It is running in stealth mode. What we know so far: • 1M context • up to 128K output • text + image + video input • reasoning + tool calling • clearly optimized for long-horizon agentic coding And then there is DeepSWE. On the same small 10-task subset: GPT-5.6 Sol — 52% Fable 5 — 65% Ox Alpha — 80%+ Important caveat: this is only 10 tasks out of 113, not the full DeepSWE benchmark. So I would not say yet that Ox Alpha officially beats Fable by 20–30%. But 80% on the same sample is still a very strong signal. My first guess was MiniMax. The timing looked almost perfect: MiniMax has been teasing its next big model, million-token context and stronger native multimodality would fit, and Ox Alpha feels exactly like a large Chinese frontier model being tested publicly before launch. But people started fingerprinting it, and now I’m leaning much more toward Z.ai / GLM. One tokenizer comparison reportedly got: GLM-5.2: 60/60 exact matches MiniMax M3: 1/60 Its API behavior also looks very GLM-like: • mandatory reasoning • low / high / max thinking levels • max by default • similar context/output limits And the funniest part: in some conversations Ox Alpha apparently identifies itself as GLM from Z.ai 😅 So my current bet: Ox Alpha is probably an unreleased multimodal GLM-5.x checkpoint — something like GLM-5.3V / next GLM — rather than MiniMax. The behavior around Taiwan and other politically sensitive Chinese topics is another hint that the model is Chinese, although that obviously doesn’t tell us which lab built it. And now the best part: OpenCode made Ox Alpha free for the next week with near-unlimited usage. They are talking about infrastructure capacity on the order of 100 trillion tokens/day. It is also currently $0 on OpenRouter. So yeah — this is probably the week to throw absurdly large coding tasks at it and see where it breaks. If this really is the next GLM, Z.ai may have quietly put another model uncomfortably close to the closed frontier. #AI #LLM #AgenticCoding #OpenCode #OpenRouter #GLM
232
I recently bought the VITURE Beast XR, primarily for coding and managing multiple AI agents—not for gaming or watching movies. The problem I wanted to solve was simple: I often need much more screen space than my MacBook Pro can provide, but I don’t always have room for external monitors. I considered getting an iPad as a secondary display for terminals, but XR glasses seemed like a more flexible experiment. With VITURE’s SpaceWalker app, I can create two or three virtual displays. The advertised 174-inch workspace can be split into multiple screens, while the physical MacBook display remains available as another workspace. I initially tried three virtual screens, but that was a bit overwhelming. Two Full HD displays turned out to be more than enough. My current setup looks like this: Termius, Codex, several agent windows, and Jump Desktop connected to my Mac mini all live inside the virtual workspace. The MacBook screen remains my primary display. It feels like a portable command center where I can monitor multiple agents, terminals, remote machines, and projects at the same time. For this specific workflow, it genuinely works. The biggest advantage is portability. I can take the glasses out, connect them to a MacBook, iPad, or even an iPhone, add a keyboard, and get a large private workspace almost anywhere—at an airport, on a plane, in a taxi, at a café, or even in a park. People around me also can’t see what’s on the screens, which is useful when working with code, terminals, or private information. The biggest downside is comfort. I don’t normally wear glasses, and working with a lot of text in XR creates a type of eye and head fatigue I hadn’t experienced before. After a long session, my eyes get tired, and I can sometimes feel slightly dizzy. Watching a movie or playing a game might be easier, but reading code and monitoring constantly changing information is demanding. So far, my limit is around two hours of continuous work. I wouldn’t replace large physical monitors with the VITURE Beast XR for an entire workday. Real monitors are still more comfortable when you have the space for them. But as a portable virtual office—and especially as a way to manage multiple coding agents while traveling or working away from a desk—it’s a genuinely useful tool. youtu.be/HgKEeOdMIew?si=tuyR…
2
727
I’d make it more personal and grounded: Kimi K3 is finally here. I’ve been waiting for this release because K2.7 Code has become my daily coding model. I use Kimi extensively since I have an almost unlimited Kimi Code subscription, and I also develop plugins for the Kimi Code ecosystem. The numbers are impressive: • 2.8T parameter MoE • 1M token context • Native vision + video understanding • Always-on reasoning Official benchmarks put it firmly in frontier territory: → 80.2% SWE-bench Verified → 89.6% LiveCodeBench v6 → 96.4% AIME 2026 → 90.5% GPQA Diamond What I find more interesting is the product positioning. K3 doesn’t feel like a replacement for K2.7 Code. Instead, it feels like K2.7 Code HighSpeed evolved into a new frontier model. My workflow will probably look like this: K2.7 Code → daily implementation and iteration. K3 → architecture, large refactors, million-token repositories, and long autonomous agent runs. The catch? Pricing. K3 costs roughly 3× more on input and 4× more on output than K2.7 Code via the API. If you’re using Kimi Code subscriptions, expect it to consume your quota at roughly the same pace as K2.7 Code HighSpeed. Moonshot doesn’t publish the exact multiplier, but the pricing strongly points in that direction. Moonshot is no longer competing on “Claude-level quality at Chinese prices.” They’re competing as a frontier lab now—with frontier models, frontier benchmarks, and frontier pricing.
65
One of the most valuable lessons I learned about AI came from building the dispatch system at Yandex Taxi. Our goal was deceptively simple: ➡️ Maximize completed trips. The first version of the dispatcher didn't use machine learning at all. Instead, it relied on: • Deterministic business rules • Hard constraints • Carefully tuned heuristics • Hand-crafted scoring Things like child seats, luggage requirements, driver eligibility, pickup distance, and hundreds of other rules were handled without AI. And it worked remarkably well. Only after we had squeezed almost everything out of the deterministic system did we introduce ML. Not to replace the dispatcher. To predict the things you simply can't hardcode: • Will this driver accept the ride? • Will they actually reach the passenger? • How long will pickup really take? • What's the probability the trip gets canceled? Those predictive models improved our core metric by another 5–7%. For Yandex Taxi, that translated into massive business value. But those extra percentage points came with an entirely new engineering discipline: → Feature pipelines → Model training → Monitoring → Retraining → Experimentation → Drift detection → Continuous tuning That's when I learned an important lesson: AI shouldn't replace engineering. It should extend it. Build the strongest deterministic system you can first. Use rules where rules work. Use optimization where optimization works. And bring in ML only where the problem is fundamentally uncertain. The best AI systems I've seen don't replace great software engineering. They stand on top of it.
102
After spending the last few days with Grok 4.5 + Grok Build, I think xAI has built one of the most compelling terminal coding agents available today. Not because of benchmark scores. Not because of marketing claims. Because I put it through the same kind of long-running engineering work I use every day. I ran exactly the same task through my Xpowers agent framework using three different models: Grok 4.5 Kimi K2.7 GLM 5.2 Everything else was identical. Same planning. Same harness. Same MCP servers. Same tools. Same workflow. The only variable was the model. Only Grok 4.5 completed the task end-to-end without requiring manual intervention. That's not a scientific benchmark, and I'm not claiming it's universally better than every other model. It's simply one real engineering task from my daily workflow. But I've stopped caring about benchmark leaderboards. The question I optimize for is much simpler: Can an agent take a large engineering task and actually finish it? For this task, Grok 4.5 did. The bigger surprise, though, wasn't the model. It was Grok Build. This is the first terminal agent I've tried that immediately understood my existing development environment. On first launch it automatically discovered my MCP servers, imported my existing skills, picked up my tooling, and started using everything without me rebuilding my setup. No copying prompts. No migrating configs. No spending an hour making the new agent usable. I installed it. I typed a command. I started working. That onboarding experience is honestly the smoothest I've seen. The closest experience for me is Z Code 3, but Grok Build currently has the best first-run experience I've personally used. Another thing I appreciated was the overall product polish. The new TUI is excellent. The animations and visual effects are subtle, responsive, and actually useful. They make long-running agent sessions feel alive without becoming distracting. It's one of those details that sounds minor until you spend your entire day inside a terminal. Combined with the onboarding experience, it feels like xAI spent a lot of time thinking about developer experience—not just model quality. I also smiled at one small product decision. The executable isn't just grok. You can simply run: agent That tells you exactly how xAI sees the future. Not another chatbot. Not another CLI. An agent. It's a small detail, but it's surprisingly confident product positioning. Is Grok Build more mature than Claude Code? I don't think so. Claude Code still feels like the more mature overall product today. But Grok Build has made an incredibly strong entrance. Fast execution. 500K context. Excellent tool use. Fantastic onboarding. A polished terminal experience. And, most importantly for me, it successfully completed a large engineering task inside my Xpowers workflow where the other models I tested did not. That's enough to earn a permanent place in my toolbox. Great work, @xai and @elonmusk
1
1
3
1,036
Зашел в гости к @nekdenis в ADVM c докладом - Как улучшить геолокацию в приложении? (GPS, GLONASS, spoofing) youtu.be/cU_XbGhuBfo
Друзья! Мы готовим контест по программированию и я один из составителей задач для этого соревнования! Мне будет очень приятно если вы присоединитесь к этому мероприятию. Чтобы зарегистрироваться, кликать сюда -> yandex.ru/championship/ ЗЫ Ну и максимальный лайк, шэр, репост
1
1
Hi, hello. I am recruiting for a senior C++ position. You must also know Rust, Kubernetes, JavaScript, PHP, and Brainfuck. We will give you $8 and 0 days off. But we have a ping pong table and also beer. Plz work 4 us after our 18 round interviews. Ur impressive.
175
713
67
4,579
It's really great to understand that I'm not alone
1. Get idea for new project 2. Purchase domain 3. Create git repository 4. Get distracted and forget about project Repeat until you're out of money
1
Тут ребята из RMR написали статейку, как интегрировать в binaryprefs огненный tink от Google: habr.com/ru/company/redmadro…
2
6
It's a really nice joke! LOL
Watching "What's new in Android I/O 2019" Missed bad joke opportunity: the new Bubbles API should have a sample app called Chet Head. @chethaase 😘
This is the best thing I've seen today 😻
605
21,817
1,925
57,608
Ребята, всем кто сегодня был на нашем митапе от Яндекс.Такси, огромное спасибо от всех от нас кто его делал! Спасибо, что нашли время после работы заехать к нам! Я надеюсь, что вам понравилось!
1
6
Ребята, 18 апреля у нас в @yandextaxi пройдет Mobile Party для андроид разработчиков. Будем рассказывать как готовим RIBs, наши рецепты по CI, как решаем проблемы интернационализации и как встраивали Navigation Component в Еду. Го к нам! events.yandex.ru/events/meet…
1
Loooooool!!!1
Ну, это кажется даже круче айфона мини! Нужен максимум ретвит! :)
1