{"id":173,"date":"2023-01-25T17:03:03","date_gmt":"2023-01-25T17:03:03","guid":{"rendered":"https:\/\/todaysainews.com\/index.php\/2023\/01\/25\/reincarnating-reinforcement-learning-google-ai-blog\/"},"modified":"2025-04-27T07:35:55","modified_gmt":"2025-04-27T07:35:55","slug":"reincarnating-reinforcement-learning-google-ai-blog","status":"publish","type":"post","link":"https:\/\/todaysainews.com\/index.php\/2023\/01\/25\/reincarnating-reinforcement-learning-google-ai-blog\/","title":{"rendered":"Reincarnating Reinforcement Learning \u2013 Google AI Blog"},"content":{"rendered":"

[ad_1]
\n<\/p>\n

\nPosted by Rishabh Agarwal, Senior Research Scientist, and Max Schwarzer, Student Researcher, Google Research, Brain Team<\/span><\/p>\n

<\/p>\n

\nReinforcement learning<\/a> (RL) is an area of machine learning<\/a> that focuses on training intelligent agents using related experiences so they can learn to solve decision making tasks, such as playing video games<\/a>, flying stratospheric balloons<\/a>, and designing hardware chips<\/a>. Due to the generality of RL, the prevalent trend in RL research is to develop agents that can efficiently learn tabula rasa<\/a><\/em>, that is, from scratch without using previously learned knowledge about the problem. However, in practice, tabula rasa RL systems are typically the exception rather than the norm for solving large-scale RL problems. Large-scale RL <\/strong>systems, such as OpenAI Five<\/a>, which achieves human-level performance on Dota 2<\/a>, undergo multiple design changes (e.g., algorithmic or architectural changes) during their developmental cycle. This modification process can last months and necessitates incorporating such changes without re-training from scratch, which would be prohibitively expensive.\u00a0<\/p>\n

<\/p>\n

\nFurthermore, the inefficiency of tabula rasa RL research can exclude many researchers from tackling computationally-demanding problems. For example, the quintessential benchmark of training a deep RL agent on 50+ Atari 2600 games in ALE<\/a> for 200M frames (the standard protocol) requires 1,000+ GPU days. As deep RL moves towards more complex and challenging problems, the computational barrier to entry in RL research will likely become even higher.\n<\/p>\n

\nTo address the inefficiencies of tabula rasa RL, we present \u201cReincarnating Reinforcement Learning: Reusing Prior Computation To Accelerate Progress<\/a>\u201d at NeurIPS 2022<\/a>. Here, we propose an alternative approach to RL research, where prior computational work, such as learned models, policies, logged data, etc., is reused or transferred between design iterations of an RL agent or from one agent to another. While some sub-areas of RL leverage prior computation, most RL agents are still largely trained from scratch. Until now, there has been no broader effort to leverage prior computational work for the training workflow in RL research. We have also released our code<\/a> and trained agents<\/a> to enable researchers to build on this work.\n<\/p>\n\n\n\n\n
<\/a><\/td>\n<\/tr>\n
Tabula rasa RL vs. Reincarnating RL (RRL). While tabula rasa RL focuses on learning from scratch, RRL is based on the premise of reusing prior computational work (e.g., prior learned agents) when training new agents or improving existing agents, even in the same environment. In RRL, new agents need not be trained from scratch, except for initial forays into new problems.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Why Reincarnating RL? <\/h2>\n

\nReincarnating RL (RRL) is a more compute and sample-efficient workflow than training from scratch. RRL can democratize research by allowing the broader community to tackle complex RL problems without requiring excessive computational resources. Furthermore, RRL can enable a benchmarking paradigm where researchers continually improve and update existing trained agents, especially on problems where improving performance has real-world impact, such as balloon navigation<\/a> or chip design<\/a>. Finally, real-world RL use cases will likely be in scenarios where prior computational work is available (e.g., existing deployed RL policies).\n<\/p>\n\n\n\n\n
<\/a><\/td>\n<\/tr>\n
RRL as an alternative research workflow. Imagine a researcher who has trained an agent A1<\/sub> for some time, but now wants to experiment with better architectures or algorithms. While the tabula rasa workflow requires retraining another agent from scratch, RRL provides the more viable option of transferring the existing agent A1<\/sub> to another agent and training this agent further, or simply fine-tuning A1<\/sub>.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\nWhile there have been some ad hoc large-scale reincarnation efforts with limited applicability, e.g., model surgery in Dota2<\/a>, policy distillation in Rubik\u2019s cube<\/a>, PBT in AlphaStar<\/a>, RL fine-tuning a behavior-cloned policy in AlphaGo<\/a> \/ Minecraft<\/a>, RRL has not been studied as a research problem in its own right. To this end, we argue for developing general-purpose RRL approaches as opposed to prior ad-hoc solutions.\n<\/p>\n

Case Study: Policy to Value Reincarnating RL<\/h2>\n

\nDifferent RRL problems can be instantiated depending on the kind of prior computational work provided. As a step towards developing broadly applicable RRL approaches, we present a case study on the setting of Policy to Value reincarnating RL (PVRL) for efficiently transferring an existing sub-optimal policy (teacher) to a standalone value-based RL agent (student). While a policy directly maps a given environment state (e.g., a game screen in Atari) to an action, value-based agents estimate the effectiveness of an action at a given state in terms of achievable future rewards, which allows them to learn from previously collected data<\/a>.\n<\/p>\n

<\/a><\/div>\n

\n For a PVRL algorithm to be broadly useful, it should satisfy the following requirements:\n<\/p>\n