MLE/MAP

Question 1

What’s the difference between pmf and pdf?

Answer 1

pmf applies to discrete.

pdf applies to continuous

Question 2

What are two advantages of MAP?

Answer 2

• It is easy to incorporate our prior assumptions about the value of q by adjusting the ratio of gamma1 to gamma0.
• It is easy to express our degree of certainty about our prior knowledge, by adjusting the total volume of imaginary coin flips. For example, if we are highly certain of our prior belief that q = 0.7, then we might use priors of gamma1 = 700 and gamma0 = 300 instead of gamma1 = 7 and gamma0 = 3.

Question 3

What’s one key idea about the behavior of MAP?

Answer 3

As the volume of actual observed data grows toward infinity, the influence of our imaginary data goes to zero

Question 4

What’s the equation of map?

Answer 4

Question 5

What’s the equation of MLE?

Answer 5

Question 6

What’s a key difference between MAP and MLE?

Answer 6

• MAP assumes background knowledge is available, whereas MLE does not.

Question 7

For MLE and MAP, what happens as the size of the dataset grows?

Answer 7

The MLE estimate and MAP estimate converge toward each other and toward the correct estimate

Question 8

Between MLE and MAP, which should perform better when there’s little data available?

Answer 8

MAP because the influence of our assumptions from previous knowledge has a larger impact when the data set is small

Question 9

What’s the definition of MLE?

Answer 9

Question 10

Define prior

Answer 10

The prior, of an uncertain quantity is the probability distribution that would express one’s beliefs about this quantity before some evidence is taken into account.

Question 11

What is iid?

Answer 11

independent, identically distributed

It’s a Combination of two assumptions:

• the outcomes of different trials are independent
• identically distributed - the same distribution governs each trial

Question 12

What is likelihood?

Answer 12

L(q) = P(D|theta)

The probability of a data set given the probability parameters (theta)

Question 13

What’s the equation for log likelihood?

Answer 13

l(theta) = ln (P(D|theta))

Can express P(D|theta) as a product of the individual probabilities, because P(D|theta) is a probability distribution

Question 14

Why is there a log in the log likelihood?

Answer 14

Having a log there just makes it mathematically easier to maximize the likelihood

Question 15

What is MLE maximizing? (simple expression)

Answer 15

P(D|theta)

Question 16

What is MAP maximizing? (simple expression) (2)

Answer 16

P(theta|D) or P(D|theta)P(theta)

Question 17

What is a bernoulli random variable?

Answer 17

• We have exactly one trial only
• we define “success” as a 1 and “failure” as a 0.

Question 18

What is the conjugate prior for estimating
the parameter theta of a Bernoulli distribution?

Answer 18

Beta distribution

Question 19

What do we know about the Dirichlet distribution?

Answer 19

The Dirichlet distribution is a generalization of the Beta distribution,

Question 20

What’s the Principle of MLE? (in words)

Answer 20

Question 21

How would you describe the inductive bias of MLE?

Answer 21

MLE tries to allocate as much probability mass as possible to the things we have observed… …at the expense of the things we have not observed

Question 22

What allows us to use the product of probabilities in the log likelihood equation

Answer 22

the independence assumption

Question 23

How do you calculate the closed-form MLE? (5 steps)

Answer 23

Question 24

What’s log(A*B*C)?

Answer 24

log(A) + log(B)+log(C)

Question 25

What's a trick for rewriting (probability distributions?) of binary variables? When is it useful?

Answer 25

Useful when calculating log likelihood, among other uses

Question 26

What's one difference between MLE and MAP?

Answer 26

for MAP we're reasoning about p(theta|D), wheras with MLE we're reasoning about p(D|theta)

Question 27

In MAP, what form will the prior take? Why?

Answer 27

* A pdf because the parameters theta are (usually?) continuous

Question 28

What is a posterior?

Answer 28

* aka posterior probability * It's the new probability you believe about a quantity after you see new test evidence * Formulated by combining your prior probability (i.e. prior) and new test evidence *

Question 29

What is the equation for MAP with each part labeled?

Answer 29

Question 30

Why can we drop P(D) from the map equation?

Answer 30

because P(D) won't effect the argmax of the expression

Question 31

What's important about beta distributions?

Answer 31

It's a conjugate prior for Bernoulli likelihood model

Question 32

What is one **alternative** to ML's view of ML?

Answer 32

Viewing ML as trying to accomplish function approximation is an alternative view

Question 33

What does Naive bayes give us?

Answer 33

A closed form solution for MLE and MAP?

Question 34

What's the problem with the Naive Bayes assumption?

Answer 34

The features might not actually be independent

Question 35

Define conditional independence

Answer 35

X and Y are conditionally independent given Z iff p(x,y|z)=p(x|z)\*p(y|z)

Question 36

If we need to think of variations of the MLE/MAP equation, what should you use?

Answer 36

The conditional probability equation

Question 37

What are the most basic equations for: 1. MLE 2. MAP

Answer 37

Question 38

What's a shortcoming of MLE? (slide 27 of lecture 17)

Question 39

What's the recipe for closed form MAP estimation?

Answer 39

Question 40

What's the expression for MLE? (2)

Answer 40

Question 41

What's the expression for MAP? (4)

Answer 41

Question 44

What is the expression for the posterior?

Answer 44

p(θ|D)

Question 45

What is the expression for the prior?

Answer 45

p(θ)

Question 46

Compare MAP and MLE

Answer 46

* MAP has more bias * MLE has more variance

Question 47

* When should you use MAP? * When should you use MLE?

Answer 47

* Use MAP when the prior is good enough (i.e. you have reason to believe the case you're working with is similar to something you've seen previously) * Otherwise use MLE

Question 48

What do you use to rewrite a conditional probability of MAP using bayes rule?

Answer 48

Only need the conditional probability equation

Question 49

What does a prior relate to?

Answer 49

An uncertain quantity/distribution

Question 50

What is the equation for bayes theorem, with each term labeled?

Answer 50

* Posterior = Likelihood \* Prior / Evidence

Question 51

In words, what is the difference between MLE and MAP?

Answer 51

MLE gives you the value which maximises the Likelihood P(D|θ). And MAP gives you the value which maximises the posterior probability P(θ|D).