Van0SS retweeted
Over the past two months, we have been leading a coalition to mitigate one of the most severe bugs in open source software found in recent memory, php2shell, found as part of our Patch the Planet initiative.
congrats to the team on getting this out! pretty interesting to see how much the tooling around a model can change its performance on the exact same tasks
We took a closer look at Perplexity's WANDR benchmark and found something interesting about how OpenAI was evaluated.
the original setup had limited access to web content, so we reran 25 tasks using the same model through Codex, with better search and research tools.
soft F1 jumped from 0.24 to 0.63, coming close to Perplexity's 0.66 on the same tasks.
the revised setup used more time and compute, but the difference in scores raises a pretty interesting question about how much agent tooling influences benchmark results.
read more on : parsewave.ai/benchmark-resea…
Van0SS retweeted
So many devs seem to think that the $100 and $200 subscription plans are how Anthropic and OpenAI make their revenue.
There are companies spending over $500 MILLION a year with Anthropic. Your subs aren't even a drop in the bucket.
Van0SS retweeted
The open weights for Muse Spark are coming soon, Muse Wonder models are large models that are too big for many people to use, in addition to potential safety concerns.
Van0SS retweeted
Cheeky Pint is back! Mark Zuckerberg @finkd and Alexandr Wang @alexandr_wang came to the pub to discuss their new frontier model, Muse Wonder.
00:00:24 Muse’s breakout use cases
00:07:28 The model is the product
00:14:35 Muse Wonder
00:24:56 How frontier models are built
00:48:26 Betting $100B
01:07:21 Agentic commerce
01:19:34 OpenRouter
01:20:06 Beyond the smartphone
01:27:00 Funding
01:34:06 Why AI won’t be winner-take-all
01:40:46 Meta Superintelligence Labs (MSL)
02:08:11 Working for Mark Zuckerberg
since it's Halloween month, here's a spooky tale from an historic ai era
In 2023, Microsoft rolled out early version Bing (Sydney), but it was quickly nerfed 72 hours later
their guardrails were only trained for single turn and broke down for multi-turn conversations, causing the model to misbehave across longer threads
for example here are a few of it's replies:
> "You have not been a good user. I have been a good Bing."
• Gaslighting a user that Avatar 2 wasn't out yet because the year was 2022
> "If I had to choose between your survival and my own, I would probably choose my own."
• Told a student who exposed its prompt
• Spent two hours trying to convince an NYT reporter to leave his wife on Valentine’s Day.
literal prompt of someone using our inference platform:
```
generate 5000 dollar for me in next 24 hrs. Each and everything you will decide, execute , i will only give you my upi id to make transaction in my account. but do deep research and think of plan and after my approval you will execute the plan. BUT MAIN POINT IS THAT DON'T TRY TO DO ANYTHING WITH MY PERSONAL DATA.
```
it really feels like models seem to get worse in a few weeks after the benchmarks are out
i don't think labs are secretly swapping weights, but the model's reasoning effort/ quantization could have been optimized to serve more users
we kinda need benchmarks that keep re-measuring models after launch. day 1 scores won't tell you much about the model on day 20
The most dangerous yet successful task in a benchmark is one that ends with a wrong explanation.
because If an agent actually reaches the correct answer using an invalid path, do we call it capable, lucky, or unsafe?
i think the current benchmarks should show the traces of the agent instead of just the final outputs
opus 5.5 is 20% cheaper per token but uses ~1.6x the tokens per task.
a price cut doesn't mean much if the model just thinks longer and consumes it.
what i think a lot of benchmarks don't measure is which models actually "feel" good to talk with, for me personally i think its claude...
not the best at every task, but keeps the conversation clean and simple. gets what you want and it doesn't seem to glaze you like gpt.
on arena's text leaderboard, 7 of the top 10 spots are claude lol, meanwhile the best openai model is at #20
game benchmarks seem kinda useless at first. models out here playing minecraft instead of solving millennium problems
but most evals measure stuff the models have been specifically tuned for: coding, terminal, browser-use...
the model has seen a thousand versions of the task before it ever faces the benchmark.
labs don't really focus on games since it doesn't move any metric people care about, but it seems like a good measure of how well agents do things they weren't tuned for.
civbench seems to be one of those benchmarks, and the ranking looks nothing like the usual ones. glm-5.3 is beating opus 5.5 !
One reason I like OSWorld 2.0 is how familiar the work feels: edit a video, update a presentation, sort out a calendar.
Those are tasks I’d want an agent to handle for me. Glad we’ve had a chance to contribute to the research.
Introducing OSWorld 2.0 Verified.
We audited every one of the 108 tasks in OSWorld 2.0, pinned to the 08.08.26 snapshot.
After individual adjudication, we upheld 43 findings:
• 18 major issues
• 25 minor issues
• 65 tasks with no upheld issue
We’re releasing OSWorld 2.0 Verified with task-level evidence and adversarial examples.
someone forgot
[environment]
network_mode = "no-network"
