Basic Info Theory

Question 1

What is the amount of information yielded (entropy) by the outcome of a future event if you already know what the outcome will be?

Answer 1

0

Question 2

The information content of an event E is what?

Answer 2

A function that increases as the probability p(E) of a function decreases

Question 3

If p(E) is close to zero then what do we know about the information content of that event?

Answer 3

It is high. We learned a lot from it [“wow that was super surprising. We need to update our models about the world a LOT”]

Question 4

The smaller the cross entropy the ______ two probability distributions are

Answer 4

more similar

Question 5

what is a discrete probability distribution?

Answer 5

a vector whose elements lie in 0…1 and that sum to 1