Human, Problem Solver š®šŖ āļø
Earth
Joined April 2009
- Tweets11.5K
- Following1.9K
- Followers980
- Likes20.7K
Pinned Tweet
intelligence is the one which removes suffering. If suffering is not removed it's not intelligence.
ATechieThought retweeted
I highly recommend cloudflare's security audit skill: github.com/cloudflare/securiā¦
Run it on all your software. Run it with the free models. Use it even if you used a model to audit something - it will find much more things. Works in all agent harnesses and is model independent.
Itās nightmare when you treat security as second class citizen.
Security called me at 6:12 AM today.
"Patch every PHP server immediately."
I asked what happened.
"Two critical vulnerabilities. Remote code execution. No authentication required."
Cool.
How many PHP servers do we have?
According to inventory: 12
According to reality:
Nobody knows.
We found one running WordPress.
Three running old internal tools.
One under someone's desk.
Two inside vendor appliances.
A forgotten employee directory from 2016.
And a server named:
PHP-OLD-DO-NOT-TOUCH
Naturally, that one runs payroll exports.
Then we found a PHP 5.6 box.
I asked who owned it.
The documentation said:
BRIAN.
Brian left in 2019.
We called Brian.
He said:
"Oh wow, that's still running?"
Excellent.
By 8 AM, leadership joined the incident call.
CEO:
"Can hackers actually get in?"
Security:
"Potentially yes."
CEO:
"Without a password?"
Security:
"Yes."
CEO:
"Then why do WE need passwords?"
Not the lesson.
Finance asked if we could wait until the weekend to patch because downtime affects revenue.
Security explained that the alternative was letting strangers execute code on our servers.
Finance asked for a cost comparison.
At 9:17, someone suggested temporarily taking the vulnerable websites offline.
Marketing refused.
"Our campaign is live."
So for the next hour, the company officially decided maintaining a landing page for 20% off webinar registration was more important than preventing remote server takeover.
We finally patched everything we could find.
Then our scanner discovered another PHP server.
Hostname:
TEMP-PHP-2014
Uptime:
4,512 days.
Nobody knows what it does.
Nobody owns it.
Nobody can shut it down.
But every time we block network access, the warehouse stops printing shipping labels.
Security has classified it as critical infrastructure.
It is running PHP 5.4.
The vulnerability has existed longer than half our employees.
The emergency patch took four hours.
Finding out what PHP actually runs took eleven.
The internet isn't held together with code.
It's held together with servers everyone is afraid to turn off.
ATechieThought retweeted
SlopCodeBench is the most relevant coding agent benchmark left--the top score is 16% vs. 50-75% on other coding benchmarks like Terminal-Bench, FrontierCode, and DeepSWE
summary of how it works: an agent receives a spec for a small program, like a backup scheduler or a mock HTTP server. the agent implements it, then the agent receives a new spec and has to extend the program. this repeats 3-8 times, and with each new spec there are hidden tests that check the new feature and all the old features.
there are a few reasons why I really like it:
1. it comes far closer to simulating real, long-horizon work with new and updating requirements, which is what itās actually like to work on a codebase vs. solving a single isolated task
2. code coherence, maintainability, and extendability naturally matter more
3. itās significantly less saturated than the other coding agent benchmarks
full results in the image (credit to @dexhorthy for the benchmark run + charts), the ācoreā number measures whether the main tests for the new feature pass; the āstrictā number is the headline and measures whether every test passes, including regression tests for all earlier features
astra and sol are at the top (interestingly 6.1 is slightly worse than 6 although likely just noise), but given how low the scores are overall + how small the spread is, i donāt think itās super meaningful yet in terms of determining which model you should use--opus 5.5 is still my daily driver, and iām kind of surprised itās not higher
the big caveat imo is that these runs all used the base harness with no customization. i personally think having a couple of core skills + agents.md instructions (planning, verification, etc) is necessary and improves performance, and that improvement is much more likely to show on SlopCodeBench than the other benchmarks
my prediction is that custom context + skills would improve performance by 5 percentage points or more and (hopefully) result in a ranking thatās a better representation of how good different models are to actually use for engineering work
do you think the improvement from proper skills/verification would be more or less than 5%?
I do some thought experiments with #Claude. I was deep into #MiyamotoMusashi and his principles (from The Book of Five Rings) and had an idea to apply to define software excellence.
If people here likes it, will be happy to share more.
claude.ai/artifact/LmNcYv9wzā¦
#philosophy
Solitude and Meditation does wonderful things to your mind. It is not 10 minutes exercise, but lifelong pursuit.
ATechieThought retweeted
Also on Youtube, looks like a good watch. BRB.
youtu.be/uXC7OuCas2Q
If you donāt have friends in life which you can rely on even in middle of night, you havenāt understood the concept of wealth and life.
Hello Sunday Morning school of learning!
Glad to see Marks vid.
youtu.be/VM3QgjXBtZ4?is=A8tFā¦
ATechieThought retweeted
The ultimate win by age 45:
A strong marriage, a healthy body, financial freedom, and zero need to impress anyone.
missed to post this but finished this episode as weekend ritual...
youtu.be/pqgUz0RNXOo?si=R91Kā¦
if the small guy says the thing - yeah, its okay, its good point but if some star geek or celeb says it ah, what a genius...what a insight.. this world don't care what is being said but they care about who is saying - even if its genuine insight or mundane and garbage thing.
using ASD-STE-100 from couple of months - Karpathy more of curated the better practices.
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
