We monitor AI safety policies and web content for substantive changes. Anonymous submissions: https://xitter.cf/t.co/5Ke9mIqOSM Run by @TheMidasProj
Joined May 2024
- Tweets192
- Following3
- Followers1.5K
- Likes117
On Oct. 12, Meta released Muse Wonder with only a 13-page model card and a promise to release a safety report “soon.” 11 days later, Meta updated the model card. There’s still no safety report. 🧵
ALT Before and after of Meta’s Muse Wonder model card: ExploitGym result changed from 92 to 20, and the original scoring note was replaced with an Oct. 23 update note.
Notably, gone is any mention of the 71 flags captured through unintended vulnerabilities. That could matter: Meta’s safety framework treats discovering and exploiting vulnerabilities as central to Cyber 2 and Loss of Control 3.
Full post and diff at: themidasproject.com/watchtow…
Meta’s own recently updated Meta Superintelligence Scaling Framework promises preparedness reports “in a timely manner in connection with the deployment.” Spark's delayed report and now Wonder's raises the question of what "timely" actually means under Meta's framework.
This deployment also comes just 10 days after Meta made some significant changes to its safety framework, as we posted about earlier this week. For more information, check out Watchtower: themidasproject.com/watchtow…
This isn't the first time Meta has delayed its full safety report after launching a new model. When Meta launched Muse Spark on Apr. 8, it didn't publish its Safety & Preparedness Report until several days later on Apr. 14.
Meta released Muse Wonder on Oct. 12 with a 13-page model card but no safety and preparedness report. Meta says that a safety and preparedness report “will be available soon.” It doesn’t say when “soon” will be. It has not been posted as of Oct. 13 at 12:30pm PT. 🧵
On Oct.2, Meta updated its safety framework, renaming the “Advanced AI Scaling Framework” (AAISF) to the “Meta Superintelligence Scaling Framework” (MSSF). The new version adds several changes not noted in the preface or changelog. 🧵
There’s also some softening around mitigation language. For example, the MSSF deletes the statement that mitigations “should be sufficiently robust against adversarial attacks that are realistic given the deployment strategy and threat scenario.” And in the chemical and biological risk section, the commitment to “ensure consistent refusal against state-of-the-art adversarial attacks” was altered to a commitment to “evaluate” Meta’s “fully mitigated model under such attacks.”
The above is not an exhaustive list, as the framework was extensively updated. Read our full
Watchtower analysis and check out the full diff here: themidasproject.com/watchtow…
Anthropic released Claude Opus 5.5 with an “inconclusive” determination for whether it reached the higher-risk tier for “harmful manipulation” under its Frontier Compliance Framework.
This raises the question of what level of certainty is needed under that safety framework. 🧵
Anthropic’s safety framework does not explicitly require a conclusive determination under its risk thresholds. But it does require risk determinations to incorporate appropriate safety margins and generally for Anthropic to document its justification for proceeding. It also commits the company to implement and test additional mitigations where its analysis identifies gaps.
Finally, Anthropic previously wrote in Mythos 5.1’s system card that it was working on a next-generation evaluation for harmful manipulation. However, it does not appear that Opus 5.5 was tested on this new evaluation (if it exists), before the model was released.
Read our full Watchtower analysis:
themidasproject.com/watchtow…