Explore 1.5M+ audiobooks & ebooks free for days

Only $12.99 CAD/month after trial. Cancel anytime.

Reinforcement Learning Algorithms with Python: Learn, understand, and develop smart algorithms for addressing AI challenges
Reinforcement Learning Algorithms with Python: Learn, understand, and develop smart algorithms for addressing AI challenges
Reinforcement Learning Algorithms with Python: Learn, understand, and develop smart algorithms for addressing AI challenges
Ebook666 pages8 hours

Reinforcement Learning Algorithms with Python: Learn, understand, and develop smart algorithms for addressing AI challenges

Rating: 0 out of 5 stars

()

Read preview
  • Reinforcement Learning

  • Neural Networks

  • Machine Learning

  • Artificial Intelligence

  • Q-Learning

  • Mentor

  • Hero's Journey

  • Genius Machines

  • Chosen One

  • Star-Crossed Lovers

  • Underdog

  • Man Vs. Nature

  • Advanced Technology

  • Love Transcends Time

  • Genius Protagonist

  • Imitation Learning

  • Tensorflow

  • Policy Gradient Algorithms

  • Reinforcement Learning Algorithms

  • Deep Learning

About this ebook

Develop self-learning algorithms and agents using TensorFlow and other Python tools, frameworks, and libraries

Key Features
  • Learn, develop, and deploy advanced reinforcement learning algorithms to solve a variety of tasks
  • Understand and develop model-free and model-based algorithms for building self-learning agents
  • Work with advanced Reinforcement Learning concepts and algorithms such as imitation learning and evolution strategies
Book Description

Reinforcement Learning (RL) is a popular and promising branch of AI that involves making smarter models and agents that can automatically determine ideal behavior based on changing requirements. This book will help you master RL algorithms and understand their implementation as you build self-learning agents.

Starting with an introduction to the tools, libraries, and setup needed to work in the RL environment, this book covers the building blocks of RL and delves into value-based methods, such as the application of Q-learning and SARSA algorithms. You'll learn how to use a combination of Q-learning and neural networks to solve complex problems. Furthermore, you'll study the policy gradient methods, TRPO, and PPO, to improve performance and stability, before moving on to the DDPG and TD3 deterministic algorithms. This book also covers how imitation learning techniques work and how Dagger can teach an agent to drive. You'll discover evolutionary strategies and black-box optimization techniques, and see how they can improve RL algorithms. Finally, you'll get to grips with exploration approaches, such as UCB and UCB1, and develop a meta-algorithm called ESBAS.

By the end of the book, you'll have worked with key RL algorithms to overcome challenges in real-world applications, and be part of the RL research community.

What you will learn
  • Develop an agent to play CartPole using the OpenAI Gym interface
  • Discover the model-based reinforcement learning paradigm
  • Solve the Frozen Lake problem with dynamic programming
  • Explore Q-learning and SARSA with a view to playing a taxi game
  • Apply Deep Q-Networks (DQNs) to Atari games using Gym
  • Study policy gradient algorithms, including Actor-Critic and REINFORCE
  • Understand and apply PPO and TRPO in continuous locomotion environments
  • Get to grips with evolution strategies for solving the lunar lander problem
Who this book is for

If you are an AI researcher, deep learning user, or anyone who wants to learn reinforcement learning from scratch, this book is for you. You’ll also find this reinforcement learning book useful if you want to learn about the advancements in the field. Working knowledge of Python is necessary.

LanguageEnglish
PublisherPackt Publishing
Release dateOct 18, 2019
ISBN9781789139709
Reinforcement Learning Algorithms with Python: Learn, understand, and develop smart algorithms for addressing AI challenges

Related to Reinforcement Learning Algorithms with Python

Related ebooks

Intelligence (AI) & Semantics For You

View More

Reviews for Reinforcement Learning Algorithms with Python

Rating: 0 out of 5 stars
0 ratings

0 ratings0 reviews

What did you think?

Tap to rate

Review must be at least 10 words

    Book preview

    Reinforcement Learning Algorithms with Python - Andrea Lonza

    Reinforcement Learning Algorithms with Python

    Reinforcement Learning Algorithms with Python

    Learn, understand, and develop smart algorithms for addressing AI challenges

    Andrea Lonza

    BIRMINGHAM - MUMBAI

    Reinforcement Learning Algorithms with Python

    Copyright © 2019 Packt Publishing

    All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

    Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

    Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

    Commissioning Editor: Pravin Dhandre

    Acquisition Editor: Winston Christopher

    Content Development Editor: Roshan Kumar

    Senior Editor: Jack Cummings

    Technical Editor: Joseph Sunil

    Copy Editor: Safis Editing

    Project Coordinator: Kirti Pisat

    Proofreader: Safis Editing

    Indexer: Rekha Nair

    Production Designer: Nilesh Mohite

    First published: October 2019

    Production reference: 1181019

    Published by Packt Publishing Ltd.

    Livery Place

    35 Livery Street

    Birmingham

    B3 2PB, UK.

    ISBN 978-1-78913-111-6

    www.packt.com

    Thanks to you, Mom and Dad, for giving me that light called life and for always being present for me.  Fede, you're a furious mad. You've always inspired me to do more. Thanks brother.

    Packt.com

    Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

    Why subscribe?

    Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

    Improve your learning with Skill Plans built especially for you

    Get a free eBook or video every month

    Fully searchable for easy access to vital information

    Copy and paste, print, and bookmark content

    Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.

    At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. 

    Contributors

    About the author

    Andrea Lonza is a deep learning engineer with a great passion for artificial intelligence and a desire to create machines that act intelligently. He has acquired expert knowledge in reinforcement learning, natural language processing, and computer vision through academic and industrial machine learning projects. He has also participated in several Kaggle competitions, achieving high results. He is always looking for compelling challenges and loves to prove himself.

    About the reviewer

    Greg Walters has been involved with computers and computer programming since 1972. He is extremely well-versed in Visual Basic, Visual Basic .NET, Python and SQL using MySQL, SQLite, Microsoft SQL Server, Oracle, C++, Delphi, Modula-2, Pascal, C, 80x86 Assembler, COBOL, and Fortran. He is a programming trainer and has trained numerous people on many pieces of computer software, including MySQL, Open Database Connectivity, Quattro Pro, Corel Draw!, Paradox, Microsoft Word, Excel, DOS, Windows 3.11, Windows for Workgroups, Windows 95, Windows NT, Windows 2000, Windows XP, and Linux. He is retired and, in his spare time, is a musician and loves to cook, but he is also open to working as a freelancer on various projects.

    Packt is searching for authors like you

    If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

    Table of Contents

    Title Page

    Copyright and Credits

    Reinforcement Learning Algorithms with Python

    Dedication

    About Packt

    Why subscribe?

    Contributors

    About the author

    About the reviewer

    Packt is searching for authors like you

    Preface

    Who this book is for

    What this book covers

    To get the most out of this book

    Download the example code files

    Download the color images

    Conventions used

    Get in touch

    Reviews

    Section 1: Algorithms and Environments

    The Landscape of Reinforcement Learning

    An introduction to RL

    Comparing RL and supervised learning

    History of RL

    Deep RL

    Elements of RL

    Policy

    The value function

    Reward

    Model

    Applications of RL

    Games 

    Robotics and Industry 4.0

    Machine learning

    Economics and finance

    Healthcare

    Intelligent transportation systems

    Energy optimization and smart grid

    Summary

    Questions

    Further reading

    Implementing RL Cycle and OpenAI Gym

    Setting up the environment

    Installing OpenAI Gym 

    Installing Roboschool 

    OpenAI Gym and RL cycles

    Developing an RL cycle

    Getting used to spaces

    Development of ML models using TensorFlow

    Tensor

    Constant

    Placeholder

    Variable

    Creating a graph

    Simple linear regression example

    Introducing TensorBoard

    Types of RL environments

    Why different environments?

    Open source environments

    Summary

    Questions

    Further reading

    Solving Problems with Dynamic Programming

    MDP

    Policy

    Return

    Value functions

    Bellman equation

    Categorizing RL algorithms

    Model-free algorithms

    Value-based algorithms

    Policy gradient algorithms

    Actor-Critic algorithms

    Hybrid algorithms

    Model-based RL

    Algorithm diversity

    Dynamic programming

    Policy evaluation and policy improvement

    Policy iteration

    Policy iteration applied to FrozenLake

    Value iteration

    Value iteration applied to FrozenLake

    Summary

    Questions

    Further reading

    Section 2: Model-Free RL Algorithms

    Q-Learning and SARSA Applications

    Learning without a model

    User experience

    Policy evaluation

    The exploration problem

    Why explore?

    How to explore

    TD learning

    TD update

    Policy improvement

    Comparing Monte Carlo and TD

    SARSA

    The algorithm

    Applying SARSA to Taxi-v2

    Q-learning

    Theory

    The algorithm

    Applying Q-learning to Taxi-v2

    Comparing SARSA and Q-learning

    Summary

    Questions

    Deep Q-Network

    Deep neural networks and Q-learning

    Function approximation 

    Q-learning with neural networks

    Deep Q-learning instabilities

    DQN

    The solution

    Replay memory

    The target network

    The DQN algorithm

    The loss function

    Pseudocode

    Model architecture

    DQN applied to Pong

    Atari games 

    Preprocessing

    DQN implementation

    DNNs

    The experienced buffer

    The computational graph and training loop

    Results

    DQN variations

    Double DQN

    DDQN implementation

    Results

    Dueling DQN

    Dueling DQN implementation

    Results

    N-step DQN

    Implementation

    Results

    Summary

    Questions

    Further reading

    Learning Stochastic and PG Optimization

    Policy gradient methods

    The gradient of the policy

    Policy gradient theorem

    Computing the gradient

    The policy

    On-policy PG

    Understanding the REINFORCE algorithm

    Implementing REINFORCE

    Landing a spacecraft using REINFORCE

    Analyzing the results

    REINFORCE with baseline

    Implementing REINFORCE with baseline

    Learning the AC algorithm

    Using a critic to help an actor to learn

    The n-step AC model

    The AC implementation

    Landing a spacecraft using AC 

    Advanced AC, and tips and tricks

    Summary

    Questions

    Further reading

    TRPO and PPO Implementation

    Roboschool

    Control a continuous system

    Natural policy gradient

    Intuition behind NPG

    A bit of math

    FIM and KL divergence

    Natural gradient complications

    Trust region policy optimization

    The TRPO algorithm

    Implementation of the TRPO algorithm

    Application of TRPO

    Proximal Policy Optimization

    A quick overview

    The PPO algorithm

    Implementation of PPO

    PPO application

    Summary

    Questions

    Further reading

    DDPG and TD3 Applications

    Combining policy gradient optimization with Q-learning

    Deterministic policy gradient

    Deep deterministic policy gradient

    The DDPG algorithm

    DDPG implementation

    Appling DDPG to BipedalWalker-v2

    Twin delayed deep deterministic policy gradient (TD3)

    Addressing overestimation bias

    Implementation of TD3

    Addressing variance reduction

    Delayed policy updates

    Target regularization

    Applying TD3 to BipedalWalker

    Summary

    Questions

    Further reading

    Section 3: Beyond Model-Free Algorithms and Improvements

    Model-Based RL

    Model-based methods

    A broad perspective on model-based learning

    A known model

    Unknown model

    Advantages and disadvantages

    Combining model-based with model-free learning

    A useful combination

    Building a model from images

    ME-TRPO applied to an inverted pendulum

    Understanding ME-TRPO

    Implementing ME-TRPO

    Experimenting with RoboSchool

    Results on RoboSchoolInvertedPendulum

    Summary

    Questions

    Further reading

    Imitation Learning with the DAgger Algorithm

    Technical requirements

    Installation of Flappy Bird

    The imitation approach

    The driving assistant example

    Comparing IL and RL

    The role of the expert in imitation learning

    The IL structure

    Comparing active with passive imitation

    Playing Flappy Bird

    How to use the environment

    Understanding the dataset aggregation algorithm

    The DAgger algorithm

    Implementation of DAgger

    Loading the expert inference model

    Creating the learner's computational graph

    Creating a DAgger loop

    Analyzing the results on Flappy Bird

    IRL

    Summary

    Questions

    Further reading

    Understanding Black-Box Optimization Algorithms

    Beyond RL

    A brief recap of RL

    The alternative

    EAs

    The core of EAs

    Genetic algorithms

    Evolution strategies

    CMA-ES

    ES versus RL

    Scalable evolution strategies

    The core

    Parallelizing ES

    Other tricks

    Pseudocode

    Scalable implementation

    The main function

    Workers

    Applying scalable ES to LunarLander

    Summary

    Questions

    Further reading

    Developing the ESBAS Algorithm

    Exploration versus exploitation

    Multi-armed bandit

    Approaches to exploration

    The ∈-greedy strategy

    The UCB algorithm

    UCB1

    Exploration complexity

    Epochal stochastic bandit algorithm selection

    Unboxing algorithm selection

    Under the hood of ESBAS

    Implementation

    Solving Acrobot

    Results

    Summary

    Questions

    Further reading

    Practical Implementation for Resolving RL Challenges

    Best practices of deep RL

    Choosing the appropriate algorithm

    From zero to one

    Challenges in deep RL

    Stability and reproducibility

    Efficiency

    Generalization

    Advanced techniques

    Unsupervised RL

    Intrinsic reward

    Transfer learning

    Types of transfer learning

    1-task learning

    Multi-task learning

    RL in the real world

    Facing real-world challenges

    Bridging the gap between simulation and the real world

    Creating your own environment

    Future of RL and its impact on society

    Summary

    Questions

    Further reading

    Assessments

    Other Books You May Enjoy

    Leave a review - let other readers know what you think

    Preface

    Reinforcement learning (RL) is a popular and promising branch of artificial intelligence that involves making smarter models and agents that can automatically determine ideal behavior based on changing requirements. Reinforcement Learning Algorithms with Python will help you master RL algorithms and understand their implementation as you build self-learning agents.

    Starting with an introduction to the tools, libraries, and setup needed to work in the RL environment, this book covers the building blocks of RL and delves into value-based methods such as the application of Q-learning and SARSA algorithms. You'll learn how to use a combination of Q-learning and neural networks to solve complex problems. Furthermore, you'll study policy gradient methods, TRPO, and PPO, to improve performance and stability, before moving on to the DDPG and TD3 deterministic algorithms. This book also covers how imitation learning techniques work and how Dagger can teach an agent to fly. You'll discover evolutionary strategies and black-box optimization techniques. Finally, you'll get to grips with exploration approaches such as UCB and UCB1 and develop a meta-algorithm called ESBAS.

    By the end of the book, you'll have worked with key RL algorithms to overcome challenges in real-world applications, and you'll be part of the RL research community.

    Who this book is for

    If you are an AI researcher, deep learning user, or anyone who wants to learn RL from scratch, this book is for you. You'll also find this RL book useful if you want to learn about the advancements in the field. Working knowledge of Python is necessary.

    What this book covers

    Chapter 1, The Landscape of Reinforcement Learning, gives you an insight into RL. It describes the problems that RL is good at solving and the applications where RL algorithms are already adopted. It also introduces the tools, the libraries, and the setup needed for the completion of the projects in the following chapters.

    Chapter 2, Implementing RL Cycle and OpenAI Gym, describes the main cycle of the RL algorithms, the toolkit used to develop the algorithms, and the different types of environments. You will be able to develop a random agent using the OpenAI Gym interface to play CartPole using random actions. You will also learn how to use the OpenAI Gym interface to run other environments.

    Chapter 3, Solving Problems with Dynamic Programming, introduces to you the core ideas, terminology, and approaches of RL. You will learn about the main blocks of RL and develop a general idea about how RL algorithms can be created to solve a problem. You will also learn the differences between model-based and model-free algorithms and the categorization of reinforcement learning algorithms. Dynamic programming will be used to solve the game FrozenLake.

    Chapter 4, Q-Learning and SARSA Applications, talks about value-based methods, in particular Q-learning and SARSA, two algorithms that differ from dynamic programming and scale well on large problems. To become confident with these algorithms, you will apply them to the FrozenLake game and study the differences from dynamic programming.

    Chapter 5, Deep Q-Networks, describes how neural networks and convolutional neural networks (CNNs) in particular are applied to Q-learning. You'll learn why the combination of Q-learning and neural networks produces incredible results and how its use can open the door to a much larger variety of problems. Furthermore, you'll apply the DQN to an Atari game using the OpenAI Gym interface.

    Chapter 6, Learning Stochastic and PG Optimization, introduces a new family of model-free algorithms: policy gradient methods. You will learn the differences between policy gradient and value-based methods, and you'll learn about their strengths and weaknesses. Then you will implement the REINFORCE and Actor-Critic algorithms to solve a new game called LunarLander.

    Chapter 7, TRPO and PPO Implementation, proposes a modification of policy gradient methods using new mechanisms to control the improvement of the policy. These mechanisms are used to improve the stability and convergence of the policy gradient algorithms. In particular you'll learn and implement two main policy gradient methods that use these techniques, namely TRPO and PPO. You will implement them on RoboSchool, an environment with a continuous action space.

    Chapter 8, DDPG and TD3 Applications, introduces a new category of algorithms called deterministic policy algorithms that combine both policy gradient and Q-learning. You will learn about the underlying concepts and implement DDPG and TD3, two deep deterministic algorithms, on a new environment.

    Chapter 9, Model-Based RL, illustrates RL algorithms that learn the model of the environment to plan future actions, or, to learn a policy. You will be taught how they work, their strengths, and why they are preferred in many situations. To master them, you will implement a model-based algorithm on Roboschool.

    Chapter 10, Imitation Learning with the DAgger Algorithm, explains how imitation learning works and how it can be applied and adapted to a problem. You will learn about the most well-known imitation learning algorithm, DAgger. To become confident with it, you will implement it to speed up the learning process of an agent on FlappyBird.

    Chapter 11, Understanding Black-Box Optimization Algorithms, explores evolutionary algorithms, a class of black-box optimization algorithms that don't rely on backpropagation. These algorithms are gaining interest because of their fast training and easy parallelization across hundreds or thousands of cores. This chapter provides a theoretical and practical background of these algorithms by focusing particularly on the Evolution Strategy algorithm, a type of evolutionary algorithm.

    Chapter 12, Developing ESBAS Algorithm, introduces the important exploration-exploitation dilemma, which is specific to RL. The dilemma is demonstrated using the multi-armed bandit problem and is solved using approaches such as UCB and UCB1. Then, you will learn about the problem of algorithm selection and develop a meta-algorithm called ESBAS. This algorithm uses UCB1 to select the most appropriate RL algorithm for each situation.

    Chapter 13, Practical Implementations to Resolve RL Challenges, takes a look at the major challenges in this field and explains some practices and methods to overcome them. You will also learn about some of the challenges of applying RL to real-world problems, future developments of deep RL, and their social impact in the world.

    To get the most out of this book

    Working knowledge of Python is necessary. Knowledge of RL and the various tools used for it will also be beneficial.

    Download the example code files

    You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.

    You can download the code files by following these steps:

    Log in or register at www.packt.com.

    Select the Support tab.

    Click on Code Downloads.

    Enter the name of the book in the Search box and follow the onscreen instructions.

    Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

    WinRAR/7-Zip for Windows

    Zipeg/iZip/UnRarX for Mac

    7-Zip/PeaZip for Linux

    The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Reinforcement-Learning-Algorithms-with-Python. In case there's an update to the code, it will be updated on the existing GitHub repository.

    We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

    Download the color images

    We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789131116_ColorImages.pdf.

    Conventions used

    There are a number of text conventions used throughout this book.

    CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: In this book, we use Python 3.7, but all versions above 3.5 should work. We also assume that you've already installed numpy and matplotlib.

    A block of code is set as follows:

    import gym

    # create the environment

    env = gym.make(CartPole-v1)

    # reset the environment before starting

    env.reset()

    # loop 10 times

    for i in range(10):

        # take a random action

        env.step(env.action_space.sample())

        # render the game

      env.render()

    # close the environment

    env.close()

    Any command-line input or output is written as follows:

    $ git clone https://github.com/pybox2d/pybox2d

    $ cd pybox2d

    $ pip install -e .

    Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "In reinforcement learning (RL), the algorithm is called the agent, and it learns from the data provided by an environment."

    Warnings or important notes appear like this.

    Tips and tricks appear like this.

    Get in touch

    Feedback from our readers is always welcome.

    General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

    Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

    Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

    If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

    Reviews

    Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

    For more information about Packt, please visit packt.com.

    Section 1: Algorithms and Environments

    This section is an introduction to reinforcement learning. It includes building the theoretical foundation and setting up the environment that is needed in the upcoming chapters.

    This section includes the following chapters:

    Chapter 1, The Landscape of Reinforcement Learning

    Chapter 2,Implementing RL Cycle and OpenAI Gym

    Chapter 3, Solving Problems with Dynamic Programming

    The Landscape of Reinforcement Learning

    Humans and animals learn through a process of trial and error. This process is based on our reward mechanisms that provide a response to our behaviors. The goal of this process is to, through multiple repetitions, incentivize the repetition of actions which trigger positive responses, and disincentivize the repetition of actions which trigger negative ones. Through the trial and error mechanism, we learn to interact with the people and world around us, and pursue complex, meaningful goals, rather than immediate gratification. 

    Learning through interaction and experience is essential. Imagine having to learn to play football by only looking at other people playing it. If you took to the field to play a football match based on this learning experience, you would probably perform incredibly poorly.

    This was demonstrated throughout the mid-20th century, notably by Richard Held and Alan Hein's 1963 study on two kittens, both of whom were raised on a carousel. One kitten was able to move freely (actively), whilst the other was restrained and moved following the active kitten (passively). Upon both kittens being introduced to light, only the kitten who was able to move actively developed a functioning depth perception and motor skills, whilst the passive kitten did not. This was notably demonstrated by the absence of the passive kitten's blink-reflex towards incoming objects. What this, rather crude experiment demonstrated is that regardless of visual deprivation, physical interaction with the environment is necessary in order for animals to learn. 

    Inspired by how animals and humans learn, reinforcement learning (RL) is built around the idea of trial and error from active interactions with the environment. In particular, with RL, an agent learns incrementally as it interacts with the world. In this way, it's possible to train a computer to learn and behave in a rudimentary, yet similar way to how humans do.

    This book is all about reinforcement learning. The intent of the book is to give you the best possible understanding of this field with a hands-on approach. In the first chapters, you'll start by learning the most fundamental concepts of reinforcement learning. As you grasp these concepts, we'll start developing our first reinforcement learning algorithms. Then, as the book progress, you'll create more powerful and complex algorithms to solve more interesting and compelling problems. You'll see that reinforcement learning is very broad and that there exist many algorithms that tackle a variety of problems in different ways. Nevertheless, we'll do our best to provide you with a simple but complete description of all the ideas, alongside a clear and practical implementation of the algorithms.

    To start with, in this chapter, you'll familiarize yourself with the fundamental concepts of RL, the distinctions between different approaches, and the key concepts of policy, value function, reward, and model of the environment. You'll also learn about the history and applications of RL.

    The following topics will be covered in this chapter:

    An introduction to RL

    Elements of RL

    Applications of RL

    An introduction to RL

    RL is an area of machine learning that deals with sequential decision-making, aimed at reaching a desired goal. An RL problem is constituted by a decision-maker called an Agent and the physical or virtual world in which the agent interacts, is known as the Environment. The agent interacts with the environment in the form of Action which results in an effect. As a result, the environment will feedback to the agent a new State and Reward. These two signals are the consequences of the action taken by the agent. In particular, the reward is a value indicating how good or bad the action was, and the state is the current representation of the agent and the environment. This cycle is shown in the following diagram:

    In this diagram the agent is represented by PacMan that based on the current state of the environment, choose which action to take. Its behavior will influence the environment, like its position and that of the enemies, that will be returned by the environment in the form of a new state and the reward. This cycle is repeated until the game ends.

    The ultimate goal of the agent is to maximize the total reward accumulated during

    its lifetime. Let's simplify the notation: if   is the action at time   and   is the reward at time  , then the agent will take actions 

    , to maximize the sum of all rewards 

    .

    To maximize the cumulative reward, the agent has to learn the best behavior in every situation. To do so, the agent has to optimize for a long-term horizon while taking care of every single action. In environments with many discrete or continuous states and actions, learning is difficult because the agent should be accountable for each situation. To make the problem harder, RL can have very sparse and delayed rewards, making the learning process more arduous.

    To give an example of an RL problem while explaining the complexity of a sparse reward, consider the well-known story of two siblings, Hansel and Gretel. Their parents led them into the forest to abandon them, but Hansel, who knew of their intentions, had taken a slice of bread with him when they left the house and managed to leave a trail of breadcrumbs that would lead him and his sister home. In the RL framework, the agents are Hansel and Gretel, and the environment is the forest. A reward of +1 is obtained for every crumb of bread reached and a reward of +10 is acquired when they reach home. In this case, the denser the trail of bread, the easier it will be for the siblings to find their way home. This is because to go from one piece of bread to another, they have to explore a smaller area. Unfortunately, sparse rewards are far more common than dense rewards in the real world.

    An important characteristic of RL is that it can deal with environments that are dynamic, uncertain, and non-deterministic. These qualities are essential for the adoption of RL in the real world. The following points are examples of how real-world problems can be reframed in RL settings:

    Self-driving cars are a popular, yet difficult, concept to approach with RL. This is because of the many aspects to be taken into consideration while driving on the road (such as pedestrians, other cars, bikes, and traffic lights) and the highly uncertain environment. In this case, the self-driving car is the agent that can act on the steering wheel, accelerator, and brakes. The environment is the world around it. Obviously, the agent cannot be aware of the whole world around it, as it can only capture limited information via its sensors (for example, the camera, radar, and GPS). The goal of the self-driving car is to reach the destination in the minimum amount of time while following the rules of the road and without damaging anything. Consequently, the agent can receive a negative reward if a negative event occurs and a positive reward can be received in proportion to the driving time when the agent reaches its destination.

    In the game of chess, the goal is to checkmate the opponent's piece. In an RL framework, the player is the agent and the environment is the current state of the board. The agent is allowed to move the game pieces according to their own way of moving. As a result of an action, the environment returns a positive or negative reward corresponding to a win or a loss for the agent. In all other situations, the reward is 0 and the next state is the state of

    Enjoying the preview?
    Page 1 of 1