Publications

• Sorted by Date • Classified by Publication Type • Classified by Research Category •

What model does MuZero learn?

Jinke He, Thomas M. Moerland, and Frans A. Oliehoek. What model does MuZero learn?. arXiv e-prints, pp. arXiv:2306.00840, June 2023.

Download

pdf ps.gz ps HTML 

Abstract

Model-based reinforcement learning has drawn considerable interest in recent years, given its promise to improve sample efficiency. Moreover, when using deep-learned models, it is potentially possible to learn compact models from complex sensor data. However, the effectiveness of these learned models, particularly their capacity to plan, i.e., to improve the current policy, remains unclear. In this work, we study MuZero, a well-known deep model-based reinforcement learning algorithm, and explore how far it achieves its learning objective of a value-equivalent model and how useful the learned models are for policy improvement. Amongst various other insights, we conclude that the model learned by MuZero cannot effectively generalize to evaluate unseen policies, which limits the extent to which we can additionally improve the current policy by planning with the model.

BibTeX Entry

@ARTICLE{He23arxiv,
       author = {{He}, Jinke and {Moerland}, Thomas M. and {Oliehoek}, Frans A.},
        title = "{What model does MuZero learn?}",
      journal = {arXiv e-prints},
         year = 2023,
        month = jun,
          eid = {arXiv:2306.00840},
        pages = {arXiv:2306.00840},
          doi = {10.48550/arXiv.2306.00840},
archivePrefix = {arXiv},
       eprint = {2306.00840},
 primaryClass = {cs.LG},
    keywords =   {refereed},
    abstract =  {Model-based reinforcement learning has drawn considerable
        interest in recent years, given its promise to improve sample
        efficiency. Moreover, when using deep-learned models, it is
        potentially possible to learn compact models from complex sensor
        data. However, the effectiveness of these learned models,
        particularly their capacity to plan, i.e., to improve the current
        policy, remains unclear. In this work, we study MuZero, a
        well-known deep model-based reinforcement learning algorithm, and
        explore how far it achieves its learning objective of a
        value-equivalent model and how useful the learned models are for
        policy improvement. Amongst various other insights, we conclude
        that the model learned by MuZero cannot effectively generalize to
        evaluate unseen policies, which limits the extent to which we can
        additionally improve the current policy by planning with the model.
    }
}

Generated by bib2html.pl (written by Patrick Riley) on Mon Apr 08, 2024 20:28:07 UTC