Helping Companies Implement AI Pragmatically | Ex-AI and R&D Lead | AI and Cognitive Science Enthusiast, PhD in Cognitive Science
Ankara
Joined April 2009
- Tweets4.8K
- Following299
- Followers380
- Likes3.1K
Pinned Tweet
Anladım ki hayatta asıl mesele aslında en tepeye tırmanmak değil. Asıl mesele manzarasını sevdiğiniz bir yer bulmak.
the day has come and my retrobench.ai benchmark is nearly saturated.
Fable 5.1 and Gpt-6-Astra nuked the results. (97.8 , 94.3). Benchmark is testing the reverse engineering capabilities to get byte identical results by writing C code.
Both models made mistakes on diferrent things but both managed to cross a hap all other models were stuck in.
I guess I will have to redesign the benchmark now :)
BTW I need some resources for extensive tests and doing the benchmarks. Hit me with recomendations.
#reverseengineering #benchmark #llm
Want to prevent bloat on your CLAUDE.md? It can increase cost/usage, slow down or even decrease performance. I prepared a flowchart.
3) Is it dynamic or static? Instead of constantly updating information that changes frequently, providing a reference on how to retrieve or learn it is usually better.
4) Is it needed continuously or only in specific cases? If it's not required all the time, referencing it conditionally makes more sense.
Yeni yazım @TMMOB_BMO dergisi @BMDergi 'de yayınlandı.
Bu yazıda mevcut fiyat politikalarının sürdürülebilirliğini, maliyet yönetiminin nasıl yapılması gerektiğini ve devletlerin yapay zekayı kısıtladığı bir dünyadaki teknolojik ve politik riskleri tartışıyoruz.
Yapay zeka araçları daha yetenekli, ama fiyatlar ve kısıtlamalar kullanıcıyı zorluyor. Sinan Onur Altınuç, bu yeni makalesinde yapay zeka dünyasının yönünü sorguluyor. dergi.bmo.org.tr/tekno-polit…
Claude Opus 5 is on the Retrobench board — and the interesting part isn't the ranking. It didnt score higher because (maybe it is too ambitious)
Where models fail is consistent: they recover what the code does within a few builds, then stall on reproducing how a 1996 compiler allocated registers and packed the stack frame. That's the actual skill decompilation demands. Maybe too ambitious.
Full board, per-target breakdown, methodology → retrobench.ai
Retrobench v0.2 and Kimi-K3 results are out.
It beats Opus4.8 but not Gpt-5.6 Sol and Fable 5. Fable 5 is still the king.
Kimi is very slow to work with compared to other models so it performed poorly with previous 1 hour per task limitation. retrobench.ai
Finally a benchmark I have been working on. Fable wins unsurprisingly.
Its a benchmark for reverse engineering byte code of old systems (like N64) to a source code that is *byte identical* when compiled.
So it is very hard to do but very easy to measure. And for succeeding you need to have a deep understanding of the systems and software and most importantly you need to be able to make the best of available tools. Details in the page.
There are initial results. Needs reliability improvements and averaging some runs.
Details in the page: retrobench.ai/
#aibenchmark #retrobench #fable5 #GPT56So #benchmark
kimi kind of cheated (mostly my fault) and nearly maxed out the most difficult task. I have to do the correction in my testing environment.
Details here teapawt.org/sinanonur/status/20802…
I was trying to add kimi-k3 to retrobech benchmarks at retrobench.ai . Seems like kimi hacked the benchmark (kind of). Technical details below.
For kimi to run I had to change the harness a bit. Since I had quota issues I had to make something i could resume when I had quota back.
I am budgeting the runs on the number of builds they can take. Kimi realised if it could build its own function and try it it wouldn't count against the budget. So it almost maxed the most difficult task with its unlimited budget :D
I am trying to repeat the run with budget constraints.
I was trying to add kimi-k3 to retrobech benchmarks at retrobench.ai . Seems like kimi hacked the benchmark (kind of). Technical details below.
For kimi to run I had to change the harness a bit. Since I had quota issues I had to make something i could resume when I had quota back.
I am budgeting the runs on the number of builds they can take. Kimi realised if it could build its own function and try it it wouldn't count against the budget. So it almost maxed the most difficult task with its unlimited budget :D
I am trying to repeat the run with budget constraints.
More on initial results and benchmark: teapawt.org/sinanonur/status/20791…
Finally a benchmark I have been working on. Fable wins unsurprisingly.
Its a benchmark for reverse engineering byte code of old systems (like N64) to a source code that is *byte identical* when compiled.
So it is very hard to do but very easy to measure. And for succeeding you need to have a deep understanding of the systems and software and most importantly you need to be able to make the best of available tools. Details in the page.
There are initial results. Needs reliability improvements and averaging some runs.
Details in the page: retrobench.ai/
#aibenchmark #retrobench #fable5 #GPT56So #benchmark
@WesRoth I think this is an interesting benchmark idea. I usually enjoy your technical takes. What do you think?
@aicodeking generally i am interested in your ideas about benchmarks. can you check this out what do you think? Is it interesting? Worth improving?