PhD Student UPC Barcelona - Reinforcement Learning
Joined July 2011
- Tweets42
- Following100
- Followers366
- Likes76
Pinned Tweet
Parallelised Q-Networks go continuous! 🚀
We’re excited to introduce PQN extended to continuous-action control — no replay buffers, no target networks, just pure online Q-learning + a deterministic actor.
Now you can use PQN + MuJoCo Playground to train full robotic policies in just seconds or minutes 🤖 All running entirely on GPU
👉 Blog post: mttga.github.io/posts/pqn_co…
👾 Codebase: github.com/mttga/purejaxql/t…
1/5
Reading purejax scripts can be challenging, so we also provide simplified versions that are much easier to read and debug — especially for those not yet familiar with jax.
👾 purejaxql simplified github.com/mttga/purejaxql/t…
5/5
Now in the same repo, you can train PQN in minutes across MinAtar, Atari, Craftax, MultiAgent Tasks, and MuJoCo Playground — making purejaxql an extensive and convenient hub for Q-learning research.
👾 Repo: github.com/mttga/purejaxql/t…
4/5
PQN still means no replay buffers, no target networks — just parallelisation + network normalisation.
We combine basic Q-learning with a deterministic actor, trained jointly with the critic in DDPG-style.
Exploration uses Gaussian noise off-policy, while stability emerges naturally from LayerNorm + large-scale parallelisation.
👉 Details: mttga.github.io/posts/pqn_co…
We evaluate Actor–Critic PQN across the three main domains of MuJoCo Playground — 50 tasks in total:
1️⃣ DeepMind Control Suite – classic control: CartPole, Walker, Cheetah, Hopper
2️⃣ Locomotion – Unitree Go1, Boston Dynamics Spot, Google Barkour, Unitree H1/G1, Berkeley Humanoid, Booster T1, Robotis OP3
3️⃣ Manipulation – Franka Emika Panda, Robotiq gripper, and mor
2/5
Matteo Gallici retweeted
🚀 “Fine-Tuning Next-Scale Visual Autoregressive Models with GRPO” w/ @MatteoGallici
We adapt DeepSeek-R1’s RL techniques to image models using aesthetic + CLIP rewards.
📄 openreview.net/pdf?id=lZK2sv…
#GRPO #RL #VisualModels #AutoregressiveModels #DeepSeek #CLIP #ICML
Matteo Gallici retweeted
The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
thrilled to share our #ICML2025 paper led by @WalterMayor_T & @johanobandoc , with @AaronCourville , where we explore how data collection affects agents in parallelized setups.
1/
Matteo Gallici retweeted
📢New paper on arXiv: Hadamax Encoding: Elevating Performance in Model-Free Atari. (arxiv.org/abs/2505.15345)
Our Hadamax (Hadamard max-pooling) encoder architecture improves the recent PQN algorithm’s Atari performance by 80%, allowing it to significantly surpass Rainbow-DQN!
PQN has been accepted as a Spotlight at ICRL 2025 🎉
👉If didn't have time to check it out, take a look at our 5-minute blog covering PQN’s key features, plus a Colab demo with JAX & PyTorch implmentations mttga.github.io/posts/pqn/
🔎 If you want to dive deeper, we break down TD instability issues and how PQN tackles them without replay buffers or target networks in this other blog post: blog.foersterlab.com/fixing-…
See you in Singapore! 🇸🇬
🚀 We're very excited to introduce Parallelised Q-Network (PQN), the result of an effort to bring Q-Learning into the world of pure-GPU training based on JAX!
What’s the issue? Pure-GPU training can accelerate RL by orders of magnitude. However, Q-Learning heavily relies on replay buffers and target networks, making training computationally slow and memory-intensive on GPUs. As a result, researchers often prefer PPO, leaving Q-Learning behind 🥲
Our solution? Eliminate Q-Learning's legacy components. PQN challenges the standard DQN paradigm by training a Q-Network without replay buffers or target networks: just online Q-Learning with vectorised exploration and network normalization (layer or batch norm). Despite its simplicity, PQN sets a new strong baseline in many single-agent and multi-agent scenarios. Check out the thread for more details 🔥
📄 Paper: arxiv.org/abs/2407.04811
⚙️ Code: github.com/mttga/purejaxql/
A fantastic collaboration with @mattiefoxcs and @benjamin_ellis3, and the support of the amazing people at @FLAIR_Ox directed by @j_foerst.
Inspired by the groundbreaking work of @_chris_lu_ on compiling entire RL pipelines in GPU: github.com/luchris429/pureja…
Matteo Gallici retweeted
@creus_roger just implemented a @cleanrl_lib Parallel Q-Networks algorithm (PQN) implementation!
🚀PQN is DQN without a replay buffer and target network. You can run PQN on GPU environments or vectorized environments. E.g., in envpool, PQN gets DQN's score in 1/10th the time
Matteo Gallici retweeted
🏋️♂️Go from creating an environment to having a trained expert agent within minutes! As part of Kinetix, we are releasing an editor that can create custom physics-based RL environments, and import them seamlessly into an RL training loop.
1/
Matteo Gallici retweeted
We are very excited to announce Kinetix: an open-ended universe of physics-based tasks for RL!
We use Kinetix to train a general agent on millions of randomly generated physics problems and show that this agent generalises to unseen handmade environments.
1/🧵
Matteo Gallici retweeted
1/ 🤖 Learned optimization offers huge potential to automate machine learning! So why doesn't it work well in RL (and how did we fix it)?!
I'm excited to share OPEN, our @AutoRL_Workshop spotlight paper exploring this question! 🧵
Matteo Gallici retweeted
This paper looks super cool. I've started reading it and am really enjoying the clarity of exposition, theoretical investigation, and simplicity of resulting algorithm.
🚀 We're very excited to introduce Parallelised Q-Network (PQN), the result of an effort to bring Q-Learning into the world of pure-GPU training based on JAX!
What’s the issue? Pure-GPU training can accelerate RL by orders of magnitude. However, Q-Learning heavily relies on replay buffers and target networks, making training computationally slow and memory-intensive on GPUs. As a result, researchers often prefer PPO, leaving Q-Learning behind 🥲
Our solution? Eliminate Q-Learning's legacy components. PQN challenges the standard DQN paradigm by training a Q-Network without replay buffers or target networks: just online Q-Learning with vectorised exploration and network normalization (layer or batch norm). Despite its simplicity, PQN sets a new strong baseline in many single-agent and multi-agent scenarios. Check out the thread for more details 🔥
📄 Paper: arxiv.org/abs/2407.04811
⚙️ Code: github.com/mttga/purejaxql/
A fantastic collaboration with @mattiefoxcs and @benjamin_ellis3, and the support of the amazing people at @FLAIR_Ox directed by @j_foerst.
Inspired by the groundbreaking work of @_chris_lu_ on compiling entire RL pipelines in GPU: github.com/luchris429/pureja…
Matteo Gallici retweeted
DQN kick-started the field of deep RL 12 years ago, but Q-learning has recently taken a backseat compared to PPO and other on-policy method. We introduce PQN, a greatly simplified version of DQN which is highly GPU compatible and theoretically supported by convergence proofs.
🚀 We're very excited to introduce Parallelised Q-Network (PQN), the result of an effort to bring Q-Learning into the world of pure-GPU training based on JAX!
What’s the issue? Pure-GPU training can accelerate RL by orders of magnitude. However, Q-Learning heavily relies on replay buffers and target networks, making training computationally slow and memory-intensive on GPUs. As a result, researchers often prefer PPO, leaving Q-Learning behind 🥲
Our solution? Eliminate Q-Learning's legacy components. PQN challenges the standard DQN paradigm by training a Q-Network without replay buffers or target networks: just online Q-Learning with vectorised exploration and network normalization (layer or batch norm). Despite its simplicity, PQN sets a new strong baseline in many single-agent and multi-agent scenarios. Check out the thread for more details 🔥
📄 Paper: arxiv.org/abs/2407.04811
⚙️ Code: github.com/mttga/purejaxql/
A fantastic collaboration with @mattiefoxcs and @benjamin_ellis3, and the support of the amazing people at @FLAIR_Ox directed by @j_foerst.
Inspired by the groundbreaking work of @_chris_lu_ on compiling entire RL pipelines in GPU: github.com/luchris429/pureja…
Matteo Gallici retweeted
Really impactful work, most prominently in finally figuring out how to bring the speed increases of Jax to off-policy value-based algorithms! Another building block falls into place in the Jax RL ecosystem..
🚀 We're very excited to introduce Parallelised Q-Network (PQN), the result of an effort to bring Q-Learning into the world of pure-GPU training based on JAX!
What’s the issue? Pure-GPU training can accelerate RL by orders of magnitude. However, Q-Learning heavily relies on replay buffers and target networks, making training computationally slow and memory-intensive on GPUs. As a result, researchers often prefer PPO, leaving Q-Learning behind 🥲
Our solution? Eliminate Q-Learning's legacy components. PQN challenges the standard DQN paradigm by training a Q-Network without replay buffers or target networks: just online Q-Learning with vectorised exploration and network normalization (layer or batch norm). Despite its simplicity, PQN sets a new strong baseline in many single-agent and multi-agent scenarios. Check out the thread for more details 🔥
📄 Paper: arxiv.org/abs/2407.04811
⚙️ Code: github.com/mttga/purejaxql/
A fantastic collaboration with @mattiefoxcs and @benjamin_ellis3, and the support of the amazing people at @FLAIR_Ox directed by @j_foerst.
Inspired by the groundbreaking work of @_chris_lu_ on compiling entire RL pipelines in GPU: github.com/luchris429/pureja…
Matteo Gallici retweeted
Very interesting work! It sounds very similar to our CrossQ paper teapawt.org/aditya_bhatt/status/17…
Introducing CrossQ, just published at #ICLR2024! 🎉
CrossQ achieves:
🔥 Very fast off-policy Deep RL
📈 with SOTA sample-efficiency in <5% of the gradient steps
🧹 without target nets or Q ensembles
🧑🏻💻Project and Code: adityab.github.io/CrossQ
Joint work w/ @DPalenicek 🧵👇
🚀 We're very excited to introduce Parallelised Q-Network (PQN), the result of an effort to bring Q-Learning into the world of pure-GPU training based on JAX!
What’s the issue? Pure-GPU training can accelerate RL by orders of magnitude. However, Q-Learning heavily relies on replay buffers and target networks, making training computationally slow and memory-intensive on GPUs. As a result, researchers often prefer PPO, leaving Q-Learning behind 🥲
Our solution? Eliminate Q-Learning's legacy components. PQN challenges the standard DQN paradigm by training a Q-Network without replay buffers or target networks: just online Q-Learning with vectorised exploration and network normalization (layer or batch norm). Despite its simplicity, PQN sets a new strong baseline in many single-agent and multi-agent scenarios. Check out the thread for more details 🔥
📄 Paper: arxiv.org/abs/2407.04811
⚙️ Code: github.com/mttga/purejaxql/
A fantastic collaboration with @mattiefoxcs and @benjamin_ellis3, and the support of the amazing people at @FLAIR_Ox directed by @j_foerst.
Inspired by the groundbreaking work of @_chris_lu_ on compiling entire RL pipelines in GPU: github.com/luchris429/pureja…
4/5 👾 We also provide scripts for Craftax. Craftax is designed to run entirely on a GPU, but its large observation size makes the use of a GPU replay buffer prohibitive, as it would take most of your GPU memory leading to very slow pipelines. With PQN, we finally have an off-policy, value-based baseline for this complex benchmark.
🔄 Additionally, the fact that PQN trains online makes the implementation of an RNN network straightforward — literally just add a RNN layer to your network. Forget about all the complexities introduced by recurrent Q-Learning like R2D2!
5/5 👥 When combined with Value Decomposition Networks, PQN also offers a powerful new baseline for multi-agent tasks.
Why is this important? When dealing with multiple agents, the replay buffer can become enormous, limiting pure-GPU training. PQN doesn’t occupy precious space for the replay buffer, so in principle, you can train decomposition networks with as many agents as you want!
