Deep Reinforcement Learning Hands-On: A practical and easy-to-follow guide to RL from Q-learning and DQNs to PPO and RLHF

Maxim Lapan delivers intuitive explanations and insights into complex reinforcement learning (RL) concepts, starting from the basics of RL on simple environments and tasks to modern, state-of-the-art methods Purchase of the print or Kindle book includes a free PDF eBook Key Features Learn with concise explanations, modern libraries, and diverse applications from games to stock trading and web navigation Develop deep RL models, improve their stability, and efficiently solve complex environments New content on RL from human feedback (RLHF), MuZero, and transformers Book Description Start your journey into reinforcement learning (RL) and reward yourself with the third edition of Deep Reinforcement Learning Hands-On. This book takes you through the basics of RL to more advanced concepts with the help of various applications, including game playing, discrete optimization, stock trading, and web browser navigation. By walking you through landmark research papers in the fi eld, this deep RL book will equip you with practical knowledge of RL and the theoretical foundation to understand and implement most modern RL papers. The book retains its approach of providing concise and easy-to-follow explanations from the previous editions. You'll work through practical and diverse examples, from grid environments and games to stock trading and RL agents in web environments, to give you a well-rounded understanding of RL, its capabilities, and its use cases. You'll learn about key topics, such as deep Q-networks (DQNs), policy gradient methods, continuous control problems, and highly scalable, non-gradient methods. If you want to learn about RL through a practical approach using OpenAI Gym and PyTorch, concise explanations, and the incremental development of topics, then Deep Reinforcement Learning Hands-On, Third Edition, is your ideal companion What you will learn Stay on the cutting edge with new content on MuZero, RL with human feedback, and LLMs Evaluate RL methods, including cross-entropy, DQN, actor-critic, TRPO, PPO, DDPG, and D4PG Implement RL algorithms using PyTorch and modern RL libraries Build and train deep Q-networks to solve complex tasks in Atari environments Speed up RL models using algorithmic and engineering approaches Leverage advanced techniques like proximal policy optimization (PPO) for more stable training Who this book is for This book is ideal for machine learning engineers, software engineers, and data scientists looking to learn and apply deep reinforcement learning in practice. It assumes familiarity with Python, calculus, and machine learning concepts. With practical examples and high-level overviews, it’s also suitable for experienced professionals looking to deepen their understanding of advanced deep RL methods and apply them across industries, such as gaming and finance Preface Why I wrote this book The approach Who this book is for What this book covers To get the most out of this book Changes in the third edition Part 1 Introduction to RL What Is Reinforcement Learning? Supervised learning Unsupervised learning Reinforcement learning Complications in RL RL formalisms Reward The agent The environment Actions Observations The theoretical foundations of RL Markov decision processes The Markov process Markov reward processes Adding actions to MDP Policy Summary OpenAI Gym API and Gymnasium The anatomy of the agent Hardware and software requirements The OpenAI Gym API and Gymnasium The action space The observation space The environment Creating an environment The CartPole session The random CartPole agent Extra Gym API functionality Wrappers Rendering the environment More wrappers Summary Deep Learning with PyTorch Tensors The creation of tensors Scalar tensors Tensor operations GPU tensors Gradients Tensors and gradients NN building blocks Custom layers Loss functions and optimizers Loss functions Optimizers Monitoring with TensorBoard TensorBoard 101 Plotting metrics GAN on Atari images PyTorch Ignite Ignite concepts GAN training on Atari using Ignite Summary The Cross-Entropy Method The taxonomy of RL methods The cross-entropy method in practice The cross-entropy method on CartPole The cross-entropy method on FrozenLake The theoretical background of the cross-entropy method Summary Part 2 Value-based methods Tabular Learning and the Bellman Equation Value, state, and optimality The Bellman equation of optimality The value of the action The value iteration method Value iteration in practice Q-iteration for FrozenLake Summary Deep Q-Networks Real-life value iteration Tabular Q-learning Deep Q-learning Interaction with the environment SGD optimization Correlation between steps The Markov property The final form of DQN training DQN on Pong Wrappers The DQN model Training Running and performance Your model in action Things to try Summary Higher-Level RL Libraries Why RL libraries? The PTAN library Action selectors The agent DQNAgent PolicyAgent Experience source Toy environment The ExperienceSource class The ExperienceSourceFirstLast Class Experience replay buffers The TargetNet class Ignite helpers The PTAN CartPole solver Other RL libraries Summary DQN Extensions Basic DQN Common library Implementation Hyperparameter tuning Results with common parameters Tuned baseline DQN N-step DQN Implementation Results Hyperparameter tuning Double DQN Implementation Results Hyperparameter tuning Noisy networks Implementation Results Hyperparameter tuning Prioritized replay buffer Implementation Results Hyperparameter tuning Dueling DQN Implementation Results Hyperparameter tuning Categorical DQN Implementation Results Hyperparameter tuning Combining everything Results Hyperparameter tuning Summary Ways to Speed Up RL Why speed matters Baseline The computation graph in PyTorch Several environments Playing and training in separate processes Tweaking wrappers Benchmark results Summary Stocks Trading Using RL Why trading? Problem statement and key decisions Data The trading environment Models Training code Results The feed-forward model The convolution model Things to try Summary Part 3 Policy-based methods Policy Gradients Values and policy Why the policy? Policy representation Policy gradients The REINFORCE method The CartPole example Results Policy-based versus value-based methods REINFORCE issues Full episodes are required High gradient variance Exploration problems High correlation of samples Policy gradient methods on CartPole Implementation Results Policy gradient methods on Pong Implementation Results Summary Actor-Critic Method: A2C and A3C Variance reduction CartPole variance Advantage actor-critic (A2C) A2C on Pong Results Asynchronous Advantage Actor-Critic (A3C) Correlation and sample efficiency Adding an extra “A” to A2C A3C with data parallelism Results A3C with gradient parallelism Implementation Results Summary The TextWorld Environment Interactive fiction The environment Installation Game generation Observation and action spaces Extra game information The deep NLP basics Recurrent Neural Networks (RNNs) Word embedding The Encoder-Decoder architecture Transformers Baseline DQN Observation preprocessing Embeddings and encoders The DQN model and the agent Training code Training results Tweaking observations Tracking visited rooms Relative actions Objective in observation Transformers ChatGPT Setup Interactive mode ChatGPT API Summary Web Navigation The evolution of web navigation Browser automation and RL Challenges in browser automation The MiniWoB benchmark MiniWoB++ Installation Actions and observations Simple example The simple clicking approach Grid actions The RL part of our implementation The model and training code Training results Simple clicking limitations Adding text description Implementation Results Human demonstrations Recording the demonstrations Training with demonstrations Results Things to try Summary Part 4 Advanced RL Continous Action Space Why a continuous space? The action space Environments The A2C method Implementation Results Using models and recording videos Deep deterministic policy gradients Exploration Implementation Results and video Distributional policy gradients Architecture Implementation Results Things to try Summary Trust Region Methods Environments The A2C baseline Implementation Results Video recording PPO Implementation Results TRPO Implementation Results ACKTR Implementation Results SAC Implementation Results Overall results Summary Black-Box Optimizations in RL Black-box methods Evolution strategies Implementing ES on CartPole CartPole results ES on HalfCheetah Implementing ES on HalfCheetah HalfCheetah results Genetic algorithms GA on CartPole GA tweaks Deep GA Novelty search GA on HalfCheetah Implementation Results Summary Advanced Exploration Why exploration is important What’s wrong with ε-greedy? Alternative ways of exploration Noisy networks Count-based methods Prediction-based methods MountainCar experiments DQN + ε-greedy DQN + noisy networks DQN + state counts PPO method PPO + Noisy Networks PPO + state counts PPO + network distillation Comparison of methods Atari experiments DQN + ε-greedy DQN + noisy networks PPO Summary Reinforcement Learning with Human Feedback Reward functions in complex environments Theoretical background Method overview RLHF and LLMs RLHF experiments Initial training using A2C Labeling process Reward model training Combining A2C with the reward model Fine-tuning with 100 labels The second round of the experiment The third round of the experiment Overall results Summary AlphaGo Zero and MuZero Comparing model-based and model-free methods Model-based methods for board games The AlphaGo Zero method Overview MCTS Self-play Training and evaluation Connect 4 with AlphaGo Zero The game model Implementing MCTS The model Training Testing and comparison Results MuZero High-level model Training process Connect 4 with MuZero Hyperparameters and MCTS tree nodes Models MCTS search Training data and gameplay MuZero results MuZero and Atari Summary RL in Discrete Optimization The Rubik’s cube and discrete optimization Optimality and God’s number Approaches to cube solving Actions States The training process The NN architecture The training The model application Results The code outline Cube environments Training The search process The experiment results The 2 × 2 cube The 3 × 3 cube Further improvements and experiments Summary Multi-Agent RL What is multi-agent RL? Getting started with the environment An overview of MAgent Installing MAgent Setting up a random environment Deep Q-network for tigers Understanding the code Training and results Collaboration by the tigers Training both tigers and deer The battle environment Summary Bibliography Index This expanded third edition of the popular Deep Reinforcement Learning Hands-On teaches cutting-edge techniques through new projects and over 20 practical chapters, fully updated for PyTorch 2.3, Gymnasium, stable-baselines3, and others.