Learning
Acquiring new knowledge or skills and improve one’s performance
A robot can learn about .. (2)
- Itself: sensors or actuator info that might vary over time
- Its environment: Learning maps, how to achieve goal
(3) benefits of learning
- Enabling to perform its task better
- Adapting to changes in environment (hard to preprogram)
- Simplify designer’s programming work
3 Forms of Learning
1. Supervised learning
With external supervisor/teacher
In- and output pairs are presented & the function between these pairs is learned
3 Forms of Learning
2. Unsupervised learning
All information should be taken from the inputs alone
-> it can be useful to preprocess inputs, e.g. divide in meaningful clusters
3 Forms of Learning
3. Reinforcement learning
With an evaluation signal.
What is feedback in supervised learning?
The target action or output for a specific input.
Example of supervised learning + name
Neural network learning
-> the weights of the connections between nods are learned (connectionist learning)
How did ALVINN learn to drive?
ALVINN steers how it think it should Humans show how it should steer ALVINN computes the error ALVINN uses this error to update weights -> REPEAT
Disadvantage of supervised learning (3)
- A trainer is needed (less autonomous)
- Not online (first training then operating phase)
- Not incremental (when it is operating, it is not learning)
4 Characteristics of Reinforcement Learning
- Learning from interaction
- Goal-oriented learning
- Learning about, from, and while interacting with an external environment.
- Learning what to do (how to map situations to actions) so as to maximize a numerical reward signal
5 Key features of Reinforcement Learning
- Learner is not told which actions to take
- Trial and Error search
- Possibility of delayed reward
- Sacrifice short-term gains for greater long-term gains
- The need to explore and exploit
- Considers the whole problem of a goal-directed agent interacting with an uncertain environment
4 Characteristics of a complete agent
- It is temporally situated
- It learns and plans continually
- The object is to affect the environment
- The environment is stochastic / uncertain
4 Elements of Reinforcement Learning (inwards to outwards)
- Policy
- Reward
- Value
- Model
- Policy: What to do?
- Reward: What is good?
- Value: What is good because it predicts reward?
- What follows what?
Actuator space
Set of all possible actions
When the robot knows/learned which action to perform in each state it has learned a …
Reactive controller
Exploration (RL)
In order to learn the optimal action, the robot has to try everything (trial and error)
Exploitation (RL)
Simultaneously to exploration, the robot should perform well and exploit what it has learned.
Once mapping between inputs and actions is learned, the robot can just exploit the learned knowledge and stop exploring.. right?
No, not always. There might be (1) sensor errors (uncertainty) and (2) a changing environment.
Exploitation/Exploration dilemma (Trade off between..(2))
- Constantly learning - Exploration (possibly doing things less perfectly)
- Constantly using what it knows (Exploitation) (Can not improve predictions of other actions)
What is learned in RL? Consider robot’s actuator and sensor space!
The robot learns a value-function (possibly in table) with all possible state-action pairs along with their Q-values.
Q-value
Grows if good things happen and shrinks if bad things happen.
When is the RL table learning method efficient (2)?
- When the state space is not too big
2. States and actions are discrete
What if table learning method alone is inefficient?
Combine RL with function approximators such as neural networks
-
Glossary A-M65
-
Lecture 1 - Introduction40
-
Lecture 2 - Machine Learning32
-
Lecture 3 -Evolutionary Computation21
-
Lecture 4 - Artificial Life9
-
Lecture 5 - Robot Locomotion22
-
Lecture 6 - Control Architectures21
-
Lecture 7 - Manipulation and Sensors44
-
Lecture 8 - Emergence & Localization21
-
Lecture 9 - Complex Sensors24
