lecture 1

Question 1

What is the main objective of a deep reinforcement learning agent?

Answer 1

to learn a sequential decision-making task from experience in an environment to achieve specific goals

Question 2

transitions in reinforcement learning

Answer 2

usually stochastic

• because the next state in the environment may depend on various factors beyond the agent’s control

Question 3

observations (w) and actions (a)

Answer 3

• may be high-dimensional
• observations may not provide full knowledge of the underlying state because the agent only receives partial information about the environment ( w =/= x )

Question 4

experience

Answer 4

• may be constrained

1. no access to accurate simulator
2. limited data

Question 5

What is the ‘reality gap’ in reinforcement learning, and why is it a challenge?

Answer 5

• refers to the difference between the agent’s experience in a simulator and the real-world environment
• is a challenge because if the simulator is not accurate enough, the policies learned in simulation may not perform well in the real world

Question 6

Why might an agent have limited access to data in reinforcement learning?

Answer 6

1. Safety constraints: Real-world exploration can be risky in fields like robotics and healthcare, limiting data collection.
2. Compute constraints: High computational costs may limit the ability to run simulations in some environments.
3. Exogenous factors: In areas like weather prediction or financial markets, data is inherently limited due to reliance on external conditions beyond control.

Question 7

How can the reality gap and limited data challenges be addressed in reinforcement learning?

Answer 7

1. Develop an accurate simulator: A more accurate simulator reduces the reality gap, enabling better policy transfer to real-world environments. (reduces reality gap)
2. Design the learning algorithm for generalization: Algorithms should be designed to improve generalization, allowing agents to perform well in unseen states or environments, even with limited training data.

Question 8

What does generalization refer to in reinforcement learning?

Answer 8

1. The capacity to achieve good performance in an environment where limited data has been gathered.
2. The capacity to obtain good performance in a related but different environment by transferring learned knowledge.

Question 9

How can an agent achieve generalization with limited data?

Answer 9

1. Regularization: Prevents overfitting to specific training scenarios.
2. Experience replay: Reuses past experiences during training.
3. Exploration strategies: Ensures the agent gathers diverse experiences, improving its ability to generalize to unseen states.

Question 10

What is transfer learning in reinforcement learning, and why is it important?

Answer 10

• Transfer learning in reinforcement learning involves training an agent in one environment and adapting it to a related but different environment.
• It is important because it allows the agent to reuse knowledge, improving its performance in new tasks.

Question 11

What are common methods for transfer learning in reinforcement learning?

Answer 11

1. Fine-tuning: Retrains the agent on a new environment while retaining previously learned knowledge.
2. Multi-task learning: Trains the agent on multiple tasks simultaneously to encourage it to learn generalizable features.
3. Domain adaptation: Modifies the agent’s policy or input representation to handle differences between the source and target environments.

Question 12

What is a supervised learning algorithm?

Answer 12

• maps a dataset of learning examples into a predictive model
• assuming the dataset is independently and identically distributed (i.i.d.) and representative of the true underlying data distribution.

Question 13

What are bias and variance in supervised learning?

Answer 13

1. Bias: Represents how much the model’s predictions differ from the true function. High bias leads to underfitting, where the model is too simple to capture patterns.
2. Variance: Represents how much the model’s predictions vary when using different subsets of the training data. High variance leads to overfitting, where the model fits the noise in the data.

Question 14

What is the ideal model in terms of bias and variance?

Answer 14

• low bias and low variance.
• The goal is to balance bias and variance by tuning model complexity and using techniques like regularization and data augmentation.

Question 15

How can variance (overfitting) be reduced in supervised learning?

Answer 15

Increasing the size of the dataset can help reduce variance by improving the model’s generalization to new data.

Question 16

What is bias-variance decomposition?

Answer 16

Bias-variance decomposition describes how the total error of a model can be broken down into:

1. Bias: The error due to incorrect assumptions in the model.
2. Variance: The error due to sensitivity to the specific training data set used. (parametric variance = overfitting)
3. Irreducible error: The inherent noise in the output that no model can eliminate. (internal variance)

Question 17

What does the bias-variance decomposition highlight in reinforcement learning?

Answer 17

highlights a tradeoff between:

1. Bias: Error directly introduced by the learning algorithm, leading to underfitting.
2. Parametric variance: Error due to the limited amount of data available, leading to overfitting.

Question 18

Why is direct bias-variance decomposition less straightforward for loss functions other than L2 loss in reinforcement learning?

Answer 18

• Overfitting arises from the sensitivity of the model’s predictions to variations in the training data, which manifests as variance in the loss function.
• With non-L2 loss functions, variance in the loss may not directly correspond to statistical variance due to the non-linear transformation introduced by the loss function.

Question 19

How can prediction error be decomposed when using non-L2 loss functions?

Answer 19

prediction error can be decomposed into:

1. A bias-related term representing the lack of expressivity of the model.
2. A variance-related term representing the sensitivity to the limited amount of data.

Question 20

Replacement for the bias-variance tradeoff in reinforcement learning

Answer 20

tradeoff between:

1. A sufficiently rich learning algorithm to reduce model bias.
2. A learning algorithm that is not too complex, to avoid overfitting to the limited amount of data.

Question 21

What is a batch or offline reinforcement learning algorithm?

Answer 21

• Mapping a dataset D into a policy π_D
• This can be done independently of whether the policy is derived from a model-based or model-free approach.

Question 22

How can the suboptimality of the expected return in an MDP be decomposed?

Answer 22

1. Asymptotic bias: Bias in the policy due to limitations of the learning algorithm, even when given an infinite amount of data.
2. Overfitting error: Error due to the finite size of the dataset, leading to overfitting.

Question 23

What is the bias-overfitting tradeoff in reinforcement learning?

Answer 23

1. On one side of the scale is data, which represents the percentage of error due to overfitting. This decreases with more data.
2. On the other side is policy class complexity, which represents the percentage of error due to asymptotic bias. Increasing policy class complexity reduces bias but increases the risk of overfitting when data is limited.

Question 24

How can the best policy be obtained in reinforcement learning?

Answer 24

The best policy can be obtained by balancing bias and overfitting through:

1. Using a sufficiently expressive policy class to reduce bias.
2. Ensuring the dataset is large and diverse enough to prevent overfitting.

Question 25

What are the key strategies for improving generalization in reinforcement learning?

Answer 25

1. **Using an abstract representation**: Create a compact and informative state representation by discarding irrelevant or redundant features. 2. **Optimizing the objective function**: Properly shape the reward function and tune key parameters (e.g., reward shaping, tuning the discount factor 𝛾). 3. **Choosing the appropriate learning algorithm**: Select the right type of function approximator (e.g., neural networks) and decide between model-based and model-free approaches. 4. **Improving the dataset**: Enhance the diversity of the dataset using better exploration strategies to improve generalization and address the exploration/exploitation dilemma.

Question 26

How does using an abstract representation help in reinforcement learning?

Answer 26

reduces overfitting and improves generalization by **focusing only on essential features**, avoiding spurious correlations caused by irrelevant features

Question 27

Why is the appropriate level of abstraction important in the bias-overfitting tradeoff?

Answer 27

- helps balance bias and overfitting - A **small but rich** abstract representation allows for improved generalization by focusing on essential information and reducing unnecessary complexity.

Question 28

What are the potential issues of including too many irrelevant features in the state representation?

Answer 28

1. Overfitting: The RL algorithm may pick up spurious correlations. 2. Increased variance: Adds complexity without meaningful improvement in performance.

Question 29

How does removing irrelevant or redundant features affect bias and variance?

Answer 29

- Removing irrelevant or redundant features helps reduce overfitting (variance) but may introduce some bias. - This is because removing features that differentiate states with different roles in the dynamics can reduce the model’s expressivity.

Question 30

How can modifying the objective function improve the policy learned by a deep RL algorithm?

Answer 30

- by introducing a bias that helps the agent generalize better - This involves optimizing an objective function that may diverge from the true objective but accelerates learning and improves performance in practice.

Question 31

What are the main approaches for modifying the objective function in reinforcement learning?

Answer 31

1. **Reward shaping**: - Adds an auxiliary reward signal to help the agent learn faster. - Can accelerate learning but requires careful design to avoid misleading the agent with rewards that deviate too far from the true objective. - Reward normalization is important for deep RL. 2. **Tuning the discount factor 𝛾**: - Determines how much the agent values future rewards relative to immediate rewards. - Proper tuning balances short-term and long-term objectives, improving the agent’s generalization ability across tasks.

Question 32

How does the discount factor 𝛾 influence reinforcement learning?

Answer 32

1. Lower 𝛾: The agent focuses more on immediate rewards, which is useful for tasks where quick feedback is essential or long-term planning is less critical. 2. Higher 𝛾: The agent prioritizes long-term cumulative rewards, which is important in environments where future rewards are more significant. Proper tuning helps balance the tradeoff between short-term and long-term objectives.

Question 33

What is the recommended approach to tuning the discount factor 𝛾 during training?

Answer 33

- to start with a low value of 𝛾 and gradually tune it during training. - In the given case, a discount factor of 1 is used with a finite horizon at test time.

Question 34

What are the three main components that an RL agent may include?

Answer 34

1. **Value function representation**: Predicts the expected return for each state or state-action pair, indicating how good a state or action is. 2. **Policy representation**: Directly maps states to actions. 3. **Model of the environment**: Predicts the next state and reward, which can be used for planning in model-based reinforcement learning.

Question 35

What is the difference between model-free and model-based reinforcement learning approaches?

Answer 35

1. Model-free approaches: Directly learn the policy or value function without building a model of the environment. 2. Model-based approaches: Use a learned model of the environment to plan actions, typically leading to more sample-efficient learning.

Question 36

What role do function approximators play in deep reinforcement learning?

Answer 36

- Function approximators, such as neural networks, map inputs (states) to outputs (values or policies). - i.e., used to **estimate value functions or policies** in deep reinforcement learning. - They are crucial for enabling the agent to generalize and determine how features are treated at higher levels of abstraction.

Question 37

How does the choice of function approximator affect generalization in reinforcement learning?

Answer 37

1. The choice of function approximator determines the **level of abstraction** introduced by deep learning, which depends on the network architecture and feature selection mechanisms (e.g., attention mechanisms). 2. Depending on the task, choosing between **model-free or model-based** approaches is critical, as model-based methods are more sample-efficient but require accurate models, while model-free methods are often more robust in complex environments without a model.

Question 38

What is the parallel between RL algorithms and human cognition as described by Daniel Kahneman?

Answer 38

1. System 1: Fast, instinctive, and automatic decision-making, similar to model-free RL, where decisions are based on learned patterns. 2. System 2: Slow, deliberate, and logical decision-making, similar to model-based RL, where decisions involve planning based on a model of the environment.

Question 39

What is transfer learning in the context of reinforcement learning?

Answer 39

- leveraging knowledge acquired in one task or environment to improve performance in a different but related task or environment - involves using a pre-trained policy or value function in a different setting that may share similar dynamics but differ in visual representation, reward structure, or specific states

Question 40

Why is exploration critical in reinforcement learning, and what is the main challenge associated with it?

Answer 40

- Exploration is critical **because the agent needs to gather sufficient information about the environment to find an optimal policy**. - The main challenge is balancing exploration (trying new actions) and exploitation (choosing known best actions) - Too much exploitation can lead to suboptimal solutions. - Too much exploration can delay convergence to an optimal policy.

Question 41

What is undirected exploration in reinforcement learning?

Answer 41

- methods that do not prioritize specific areas of the environment based on uncertainty or potential rewards - ex: epsilon-greedy

Question 42

What is directed exploration in reinforcement learning, and when is it useful?

Answer 42

- aims to explore areas with high novelty or uncertainty - useful when: 1. When rewards are not sparse, uncertainty in the value function can guide exploration. 2. When rewards are sparse, explicit exploration rewards may be used. - ex: UCB, thompson sampling

Question 43

What is the role of learning algorithms in reinforcement learning?

Answer 43

- govern how the agent updates its function approximators using data from interactions with the environment. - Examples include Q-learning and policy gradients.

Question 44

What is replay memory, and why is it important in reinforcement learning?

Answer 44

**stores past experiences** (state, action, reward, next state) to **break temporal correlations** in training data, thereby improving sample efficiency and stability during training.

Question 45

What are controllers in reinforcement learning?

Answer 45

Controllers handle meta-level operations such as training, validation, and testing phases, as well as hyperparameter tuning.

Question 46

How do policies influence an RL agent’s behavior?

Answer 46

- define how the agent chooses actions, balancing exploration and exploitation - e.g., e-greedy policy

Question 47

What role does the environment play in reinforcement learning?

Answer 47

The environment is where the agent interacts to gather experiences and learn optimal behavior through trial and error.