🚀 Big time! We can finally do LLM RL fine-tuning with rewards and leverage offline/off-policy data!
❌ You want rewards, but GRPO only works online?
❌ You want offline, but DPO is limited to preferences?
✅ QRPO can do both!
🧵Here's how we do it:
Kemeny and Snell was my bible for a while. It helped us prove the new theorems in our paper: A memory frontier for complex synapses: papers.nips.cc/paper_files/p… which got a #neurips best paper award.
2 great books on markov chains, and both of them under the same title 'denumerable markov chains' by Wolfgang Woess (EMS/AMS), the other, by Kemeny, Snell and Knapp.