6K+ human judgements/minute via API for Online RLHF and Benchmarks.
Joined June 2024
- Tweets152
- Following176
- Followers127
- Likes528
Following the trend, here are 780K+ human votes on how well LLMs can generate 3D objects from text using code. Our team had so much fun scrolling through the outputs.
Gemini 4 Argon takes #1, followed by GPT-6 Astra.
Explore all outputs, votes and rankings on benchmark.ai
As Benchmark AI has more World Model benchmarks, we used the FLUX 3 Max launch to compare three models across 4 benchmarks, based on 3M+ human judgments.
Our goal with Benchmark AI is to better understand which models are best at what. This is a first step :)
PS: all outputs, matchups, votes, prompts & methodology are transparent and inspectable
Clever approach to diffusion denoising, improving both image quality and diversity. The use of Rapidata in this paper is is exactly why we launched our research grants.
Can diffusion models generate high-quality images without CFG?
Yes! We introduce ✨Correction Training for Diffusion Models✨, which significantly improves sample quality without relying on classifier-free guidance.
Project page: sunovivid.github.io/correcti…
🧵 (1/n)
Affordable, fast, and this time also with high control. Benchmark will be shared soon!
Introducing P-Video-2-Pro-Reference based on @MiniMax_AI H3! 🎬
Combine images, videos, and audio references to guide how your video looks, moves, and sounds. Keep characters consistent, transfer motion, and bring different visual elements together in a single generation.
• 𝗜𝗻𝗽𝘂𝘁: Text prompt with image and/or video references, plus optional audio
• 𝗥𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀: Up to 9 images, 3 videos, and 3 audio files
• 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻: 5–15s at 480p or 768p
• 𝗣𝗿𝗶𝗰𝗶𝗻𝗴: Output starting at $0.015/s in Cost mode, with reference inputs billed separately
• 𝗖𝗼𝗻𝘁𝗿𝗼𝗹: First and last frames, seed, aspect ratio, and prompt upsampling
Two modes: Speed and Cost. Choose the trade-off that fits your workflow.
Available via our inference partners @Cloudflare, @scenario_gg, @replicate, @WiroAI, @TellersAI, @eachlabs, @wavespeed_ai, @inference_sh, @_segmind, @togethercompute, @EdenAIco, and @runware
Validated by our benchmark partners @datapointai, @RapidataAI, and @DesignArena
📚 Model docs: docs.api.pruna.ai/guides/mod…
🧩 API: dashboard.pruna.ai/login
Rapidata retweeted
I remember dealing with Amazon Mechanical Turk many years ago for user studies for my paper. I gave up. 🤣 For academica @RapidataAI is AMT done right!
"AMT done right!"
@Taesung & @donghoon_ahn used 1600 human judgements to eval quality and diversity in their new paper "Correction Training for Diffusion Models"
Thanks for the kind words!
I remember dealing with Amazon Mechanical Turk many years ago for user studies for my paper. I gave up. 🤣 For academica @RapidataAI is AMT done right!
New Benchmark: Image-to-3D with 307K+ human votes across 14 models.
Hi3D v3.0 leads the leaderboard, with Meshy v7.1 and Tripo H3.1 among the top ones.
We evaluated geometry and visual faithfulness across 357 input images.
Full results on benchmark.ai
World Models Realism: We benchmarked world models with camera footage. Giving models a real starting frame, we asked them to predict what happens next. Then 500K+ humans compared the outputs against each other (including the real camera footage).
Camera footage still ranks number 1 by far (still expected tbf haha). But on the models' side:
@grok Image Video 1.5 is #1 for Photorealism,
@MiniMax_AI H3 is #1 for Visual Integrity.
@bfl_ai Flux 3.0 and Gemini Omni Flash 1.1 are following just behind it :).
In some examples, telling generated from real is genuinely hard. Check out the full leaderboard, and explore all match-ups and outputs on: benchmark.ai/world-models-re…
The full results can be explored at benchmark.ai/image
Releasing 6M+ human preferences for T2I models. 43 models, 1,514 prompts, 680K+ pairwise comparisons, with separate judgments for prompt alignment, visual preference, coherence, and text rendering.
Other datasets from Benchmark.AI can also be found there ;). Comment if there is a dataset that you'd like us to work on for your research!
huggingface.co/datasets/Rapi…
🤖 Made with AI
#1 in text coherence according to 1M human judgements. That's the ranking of Flux Image 3, from @bfl_ai. Results are ready to inspect on benchmark.ai/image , but we've made this little video to showcase some of it already.
(Also #6 overall, according to 6M human judgement when considering other dimensions such as alignment, aesthetic preference, and coherence).
Rapidata retweeted
Everyone out there spending billions trying to build a reward model that accurately replicates human raters.
JUST USE HUMANS BRO, its not that complicated.
BREAKING: @bfl_ai FLUX 3 Image enters the Pareto Frontier in the first public Benchmark of the model!
With over 64k annotations collected in the hours since launch the results show once again that Black Forest Labs plays in the top league, competing with the trillion dollar hyperscalers!
The model was evaluate across 4 categories.
Alignment, Coherence, Preference and Text-Coherence
Full Results here: benchmark.ai/image
All data for download at: huggingface.co/datasets/Rapi…
Rapidata retweeted
Looking at this, I'd think it's real footage, except I definitely don't want to be driving the van on the bottom right 💀.
"A high-angle photo of a multi-lane highway filled with dense traffic. Cars, vans, and trucks in various colors are driving on the asphalt, casting long shadows in the warm, low sunlight."
This was the prompt given to one of the best image models out there. They're getting excellent, but still make obvious mistakes. Things that would be obvious to humans, but not to auto-evals. We're building Benchmark AI to reveal those failures, and later-on post-train our partner models exclusively using humans as a reward.
