Publications

• Sorted by Date • Classified by Publication Type • Classified by Research Category •

An Analysis of Model-Based Reinforcement Learning From Abstracted Observations

Rolf A. N. Starre, Marco Loog, Elena Congeduti, and Frans A Oliehoek. An Analysis of Model-Based Reinforcement Learning From Abstracted Observations. Transactions on Machine Learning Research, 2023.

Download

pdf [795.5kB]  

Abstract

Many methods for Model-based Reinforcement learning (MBRL) in Markov decision processes (MDPs) provide guarantees for both the accuracy of the model they can deliver and the learning efficiency. At the same time, state abstraction techniques allow for a reduction of the size of an MDP while maintaining a bounded loss with respect to the original problem. Therefore, it may come as a surprise that no such guarantees are available when combining both techniques, i.e., where MBRL merely observes abstract states. Our theoretical analysis shows that abstraction can introduce a dependence between samples collected online (e.g., in the real world). That means that, without taking this dependence into account, results for MBRL do not directly extend to this setting. Our result shows that we can use concentration inequalities for martingales to overcome this problem. This result makes it possible to extend the guarantees of existing MBRL algorithms to the setting with abstraction. We illustrate this by combining R-MAX, a prototypical MBRL algorithm, with abstraction, thus producing the first performance guarantees for model-based ‘RL from Abstracted Observations’: model-based reinforcement learning with an abstract model.

BibTeX Entry

@article{Starre23TMLR,
    title=      {An Analysis of Model-Based Reinforcement Learning 
                 From Abstracted Observations},
    author=     {Rolf A. N. Starre and Marco Loog and 
                 Elena Congeduti and Frans A Oliehoek},
    journal=    TMLR,
    issn=       {2835-8856},
    year=       {2023},
    url=        {https://openreview.net/forum?id=YQWOzzSMPp},
    keywords =  {nonrefereed, arxiv},
    abstract = {
         Many methods for Model-based Reinforcement learning (MBRL) in Markov
         decision processes (MDPs) provide guarantees for both the accuracy
         of the model they can deliver and the learning efficiency. At the
         same time, state abstraction techniques allow for a reduction of
         the size of an MDP while maintaining a bounded loss with respect
         to the original problem. Therefore, it may come as a surprise that
         no such guarantees are available when combining both techniques,
         i.e., where MBRL merely observes abstract states. Our theoretical
         analysis shows that abstraction can introduce a dependence between
         samples collected online (e.g., in the real world). That means
         that, without taking this dependence into account, results for
         MBRL do not directly extend to this setting. Our result shows that
         we can use concentration inequalities for martingales to overcome
         this problem. This result makes it possible to extend the
         guarantees of existing MBRL algorithms to the setting with
         abstraction. We illustrate this by combining R-MAX, a prototypical
         MBRL algorithm, with abstraction, thus producing the first
         performance guarantees for model-based ‘RL from Abstracted
         Observations’: model-based reinforcement learning with an abstract
         model.
    }
}

Generated by bib2html.pl (written by Patrick Riley) on Mon Apr 08, 2024 20:28:07 UTC