Quiz 5 - Module 4

Question 1

GANs involve __ density modeling

Answer 1

implicit

• generate samples from the model p(x)

Question 2

GAN input

Answer 2

• Generator
  
  • Vector of random numbers, normal (mu, sigma)
• Discriminator
  
  • minibatch
    
    • p(x) fake image
    • real image

Question 3

Gan output

Answer 3

• Discriminator
  
  • real or fake
• Generator
  
  • p(x)

Question 4

Generator role

Answer 4

• update weights to improve realism of generated images

Question 5

Discriminator role

Answer 5

• update weights to better discrimate

Question 6

Game theory problem for GANs

Answer 6

• Mini-max Two Player Game

Question 7

GAN Objective

Answer 7

Question 8

GAN Generator Objective

Answer 8

Question 9

GAN Discriminator Objective

Answer 9

Question 10

The ___ part of the GAN objective does not have good gradient properties

Answer 10

Generator

• High gradient when D(G(z)) is high (ie. discriminator is wrong)
• We want to improve when samples are bad

Question 11

Alternate Objective for GAN Max-Max Game

Answer 11

Question 12

GAN Drawbacks

Answer 12

• No explicit model for distribution
• training can be unstable
• High-fidelity generation heavy to train

Question 13

VAE involve __ density modeling

Answer 13

explicit

Question 14

VAE input

Answer 14

• Encoder
  
  • Input is image X
• Decoder
  
  • sample Z from simple distribution

Question 15

VAE Output

Answer 15

• Encoder
  
  • Parameters of a probability distribution (Z)
    
    • mu and sigma
• Decoder
  
  • Parameters of a probability distribution
    
    • Mu and sigma of Gaussian
    • For multi-dimensional version, output diagonal covariance

Question 16

VAE Optimization

Answer 16

• Two parts
  
  • KL Divergence
    
    • Variational lower bound (elbo)
  • Reconstruction Loss

Question 17

T/F: Variational AutoEncoders are differential

Answer 17

True - with caveat

• Sampling action is not differentiable (stochastic)
• Need to use reparameterization trick to put stochastic sampling into a separate variable (epsilon) that is not in backprop.

Question 18

VAE Reconstruction Loss

Answer 18

Question 19

VAE Distribution Loss

Answer 19

The loss associated with the VAE Distribution diverging from the normal distribution (mu = 0, sigma = 1)

Question 20

Gan Discriminator wants ___ (minimize/maximize)

E[log D(x)] + E[log (1 - D(G(z)))]

Answer 20

maximize

• Discriminator wants to output a 0 for D(G(z)) to indicate that the generated image is fake (0) not real (1)

Question 21

Gan Generator wants _____ (minimize/maximize)

E[log D(x)] + E[log (1 - D(G(z)))]

Answer 21

minimize

• The generator wants the discriminator to be wrong
• Ie. wants the discriminator to classify D(G(z)) as 1

Question 22

The ___ part of the objective for GAN does not have good

Answer 22

Generator

• High gradient when D(G(z)) is high (discriminator, wrong)
• We want it to improve when samples are bad (discriminator is right)

Question 23

Semi-supervised learning data type

Answer 23

• Small amount of labeled data
• Larger amount of unlabeled data

Question 24

Different ideas for training in semi-supervised environment

Answer 24

• simple idea
  
  • learn model on small label data
  • make predictions on unlabeled data, add as new training, repeat
• co-training
  
  • prediction across multiple views

Question 25

Fixed match (pseudo-labeling)

Answer 25

* Unlabeled data example * Weakly augment * Make prediction, generate pseudo-label * throw out cases below threshold * Strongly augment * Make prediction, use pseudo-label as ground truth

Question 26

Pseudo-labeling (in practice)

Answer 26

* Labeled examples (feed directly to model) * Unlabeled examples * Combined * Weakly augment * Strongly augment * Losses * Cross Entropy (Labeled Data) * Cross Entropy (strongly augmented unlabeled data using weakly augmented pseudo-label as ground truth)

Question 27

Label propagation

Answer 27

Learn feature extractors and apply to unlabeled data. Get unlabeled data labeled similarly to labeled data in clusters (like KNN)

Question 28

Few shot learning data

Answer 28

* Base set of data * lots of labels * New set * very few labels (1 - 5 examples per category) in new categories * transfer learning

Question 29

Approaches to few shot learning

Answer 29

* Fine-tuning * train classifier on base classes * freeze feature extractor * learn classifier for new classes (during "query" time) * Simulate (N-Way K-Shot Tasks) * Meta-training * Better at making train reflect what will happen during test

Question 30

Classifier useful in the few-shot fine-tuning case

Answer 30

* cosine (similarity-based) * instead of linear layer * unit comparison (A.B) / (norm (A) norm(B) ) * normalized (unit-norm) comparison may discriminate a small number of classes better since it focuses on an angular difference

Question 31

Meta-Training

Answer 31

* useful for few-shot learning * makes training better reflect test (simulate smaller tasks) * N-Way K-Shot Tasks * N - number of categories * K - examples per category * Can pre-train features on held-out base classes

Question 32

Meta-Learner methods

Answer 32

* Meta-Learner LSTM * want to learn gradient descent * update rules * param initialization * adaptive LR, weight decay to reduce overfit * gradient descent update looks like LSTM update * Model-agnostic meta learning (MAML) * want to learn parameter initialization * normal gradient descent

Question 33

self-supervised data

Answer 33

* no labels at all

Question 34

Autoencoders

Answer 34

* Low dimensional embedding between an encoder and a decoder * Loss * Minimize difference (MSE)

Question 35

Surrogate tasks for self-supervised learning

Answer 35

* reconstruction * rotate image * colorization * relative image patch location (jigsaw) * video: next frame prediction * instance prediction

Question 36

Colorization

Answer 36

* self-supervised task * input * grayscale * output * color * loss * MSE

Question 37

jigsaw puzzle

Answer 37

* self-supervising task * input: image patches * output: prediction of discrete image pach location relative to center * loss: cross-entropy classification (which position)

Question 38

rotation prediction

Answer 38

* input: image with various rotations * output: prediction rotation amount * objective: cross-entropy classification

Question 39

Evaluation of self-supervised learning

Answer 39

* train the model with surrogate task * extrack the convnet (encoder part) * transfer to actual task * use to initiaize model of another supervised learning task * use it to extract featrures for learning a separate classifier (NN, SVM) * often classifier is limited to linear layer and features are frozen

Question 40

Instance discrimination

Answer 40

* Positive Example * 2 augmentations (same image) * Negative Example * Augmented * Feed positive and negative examples to classifier CNN * Loss * contrastive loss * dot product (similarity) between augmentation 1 and positive and negative examples

Question 41

Contrastive loss types

Answer 41

* end-to-end * use all other examples as negatives in mini batch * memory bank * store negatives across iterations (Queue) * from previous mini-batches * don't have to redo feature extraction * no extra feature extraction needed (stored) * Momentum encoder * exponential average of moving weights * helps avoid stale weights issue in memory bank

Question 42

Reinforcement Learning

Answer 42

Sequential decision making in an environment with evaluative feedback

Question 43

Signature challenges in reinforcement learning

Answer 43

* evaluative feedback * need trial/error to find the right action * delayed feedback * actions may not lead to immediate reward * non-stationary * data distribution of visisted states changes when the policy changes * fleeting nature of time and online data

Question 44

Markov decision process

Answer 44

(S, A, R, T, gamma) * state * action * distribution of rewards R(s, a, s') * transition probability T(s, a, s') * gamma discount factor

Question 45

Markov property

Answer 45

Current state completely characterizes state of the environment. Assume most recent observation is a sufficient statistic of history

Question 46

What do we assume is unknown about an MDP in RL?

Answer 46

* Transition probability distribution * Reward distribution

Question 47

Value Iteration

Answer 47

Question 48

Bellman Optimality Equation (value)

Answer 48

Question 49

Q-Iteration is the same as value iteration except \_\_\_

Answer 49

it loops over actions as well as states

Question 50

Parts of policy iteration

Answer 50

* Policy Evaluation * Compute V\_pi (similar to value iteration) * Policy Refinement * Greedily change actions as per V\_pi at next steps

Question 51

Why choose policy iteraton over value iteration?

Answer 51

Pi often converges to Pi\* much sooner than V to V\*

Question 52

Deep Q-Learning

Answer 52

* Parameterized Q-function from data {(s, a, s', r)} for N data points * Linear function approximators * Q(s, a; w, b) = w_a^Ts + b_a * Loss * MSE * (Q_new(s,a) - (r + gamma \* max_aQ_old(s', a)) )² * Q_new - predicted Q-value * Q_old - target Q-value * For stability * Freeze Q_old and update Q_new parameters * Set Q_old to Q_new at regular intervals

Question 53

Deep Q-Learning - Correlated Data Problem

Answer 53

* Samples are correlated -\> high variance gradients -\> inefficient learning * Current Q-Network parameters determines next training sample -\> can lead to bad feedback loops * Resolution? * replay buffer that stores transitions * update replay buffer as game (Experience) epuisodes are played, older samples discarded * Train Q-network on random minibatches of transitions from the replay memory, instead of consecutive samples * large the buffer, lower the correlation

Question 54

What are the key steps of the Deep Q-Learning Algorithm

Answer 54

* Epsilon greedy action selection * select random action with probability epsilon or max Q\* action * Experience replay * store transition in replay buffer * sample random minibatch of transitions from buffer * Q update * perform gradient descent

Question 55

Derive the policy gradient

Answer 55