I help organizations overcome the challenges of software development. DM is always open.

Oregon, USA
Joined May 2017
Include

Only show posts containing:

Exclude

Hide posts containing:

Time range
-
Minimum likes
Decisions made by the mob are logged. This fixes a gap I see with human teams. Lots of important decisions are not documented because it's a lot of overhead to do so.
Since the agentic mob is not always in agreement on certain things, I have added the decider protocol to their mobbing process. This has already born tremendous fruit like Grok voting NO on a proposal unless some security measures were implemented.
12
Since the agentic mob is not always in agreement on certain things, I have added the decider protocol to their mobbing process. This has already born tremendous fruit like Grok voting NO on a proposal unless some security measures were implemented.
1
20
Replying to @tottinge
They work better with known patterns. So my idea has taken a different direction. More to come.
16
Replying to @zubic_eth
That's Vegas Matt. This is literally all he does. And he makes a crap ton of money doing it.
148
Replying to @allenholub
I thought about that, and it seems like a shortcut. But I'm thinking something still interpretive so that LLM's can easily traverse the codebase to reduce context bloat.
34
Replying to @CFDevelop
You can build one and run it on your own machine.
8
I think we're going to start seeing the emergence of new coding languages designed for LLMs to work with. Likely languages that are significantly more terse, have few redundancies, and few commands. A lot of our human programming languages are built around encapsulation and pattern simplification. Neither of which are necessary for LLM's. Think _regression to goto_ and you'll be on the right track.
3
1
893
Replying to @allenholub
"Work density" is the human cost of Bishop’s Law in action: Work done in async requires a resync, and as the complexity of the work increases, so does the complexity of the resync. Offloading tasks to 10 parallel AI agents trades away the low-stress, flow-state parts of coding and leaves the developer paying a brutal, non-stop resync tax constantly auditing, context-switching, and merging 10 diverging streams of isolated work back into one coherent mental model. We didn't automate the bottleneck; we turned our own brains into the integration bottleneck.
1
21
Every once in a while the LLM models don't behave correctly, but I don't want to just stop using them and their quota. So just like with an employee, I created a plan to rebuild trust and responsibility within the mob. Rung 1 is probation. Very little trust. Low level fixes and menial tasks are often handed to models on this level. Rung 2 is standard. Standard work is done by these models. Some of it impactful, but rarely critical. Rung 3 is trusted. More critical decisions are made with models that reach this rung. The work of each model is evaluated after each task. If there has been an error then the way in which the error transpired is examined. If it was a real mistake on the part of the model, it is dropped down a rung. After 3 completed cycles with no issues, it is elevated to the next rung. Each mistake is thoroughly examined to determine cause, and to see if the issue is truly with the model or perhaps an error of communication or context size. Regardless of cause, a mistake still causes a lowering on the ladder. This isn't punishment, since there is nothing to punish. It is simply a matter of continuously evaluating if the system is behaving correctly and giving pathways to generating correct behavior.
39
I expect the casinos to call any sort of advantage the players get to be called cheating. What I don't expect is for judges to agree. What Phol did was absolutely not cheating. Social engineering yes, but not cheating.
1
212
I travel a lot for work. I wrote an app for tracking my expenses that uses an LLM to analyze receipts and automatically adds the entry into the system. I further set up an email address that receives receipts either as attachments or sometimes the email itself is the receipt. I just now told GrokBot (I named it Timmy) to search through all of my emails and forward trip expenses to my expenses reporting email. I now only need to issue one command to Timmy and all of my travel expenses are tracked. We live in amazing times.
1
1
122
People out here hollering about LLM's following the same bad practices humans follow and then claiming good practices are bad now because LLM's are doing the same crappy things humans do. 🀨
ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
45
Replying to @kunchenguid
Interesting data, wrong conclusion. This eval shows that test-after tests which mirror the implementation don't improve a one-shot benchmark score. That is much narrower than "AI-written tests are empirically proven unhelpful." Start with the stats. 147 vs 145 of 222 is a 2-task difference. The 95% CI on that gap is roughly Β±9 pts, and wider still because runs cluster by task. On the 44-task subset it's about Β±20. "Not significant" means the eval can't detect effects that size. It doesn't mean they're zero. The benchmark also can't measure what tests are for. DeepSWE grades one patch against hidden tests and then discards the repo. Tests pay off on the next change: regressions, refactors, CI gates. An eval with no next commit scores that benefit at zero by design. The only stat-sig result is cost, and writing tests obviously costs tokens and time. You measured the expense side precisely and the benefit side not at all. Your own diagnosis is the real finding: the tests were "a repetition of the implementation." That's the textbook test-after anti-pattern, and humans who write tests that way get no value either. It's an indictment of how the tests were written, not of testing. The point that the same agent interpreted intent twice is half right. Tests can't fix a misread spec. They do catch the gap between what the agent meant and what it wrote: off-by-ones, missed branches, broken callers. How much they catch depends on test design, which the eval didn't vary. Disabling existing tests had no effect, which is plausible when the repo's suite doesn't cover the area each task touches and hidden grader tests define success. That tells you about benchmark task coverage, not whether a regression suite protects a living codebase. A better experiment would be test-first and outside-in, with acceptance criteria derived from the issue before any implementation exists. Score test quality with mutation testing. Then measure across a sequence of dependent tasks on one repo, where regressions can actually happen. You already flagged the most promising arm: humans specifying intent as executable examples. That's just BDD. The right takeaway isn't "stop writing tests." It's "stop writing tests that restate the code."
1
72
"AI writes slop code!" Hello... have you ever LOOKED at the code humans have been writing for decades? It's dreadful! AI just doesn't write code the way YOU would write it, just like every other human developer you complain about the first time you open their repo.
5
134
Easily solved. But if you don't make it a priority then yes, you will absolutely be vulnerable.
38
Are agents bloating your code? Stop blaming the agents, they're just following the same terrible habits developers have had for generations. Your job is now to implement the same safeguards against those terrible habits that you were previously allowed to get away with. The development cycle has been accelerated which means more opportunities for dirty code leakage.
39
1. Tests coupled to implementation lock in implementation. Tests coupled to behavior don't. That's a test design problem, not a unit test problem. 2. Tests are how your agent verifies its own work. Remove them and you become the test suite. 3. You're counting the cost of tests and none of the cost of the bugs they catch. Check back in 6 months, not 1 week.
We deleted 800k lines of unit tests from the @sazabi monorepo earlier this week. Three hypotheses: 1- Unit tests "lock-in" slop code by making it hard to change or remove 2- Unit tests slow down dev cycles by making your agent do more work locally and by increasing CI times 3- Unit tests waste money by making your agent generate more tokens and by adding useless CI runs Will report back soon with the results!
1
68