10. Bayesian Statistics

Question 1

Bayes’ Theorem

Elementary Version

Answer 1

P(A|B) = P(A∩B)/P(B)

= P(B|A)P(B) / [P(B|A)P(A)+P(B|A^c)P(A^c)]

Question 2

Bayes’ Theorem

Events as Discrete Random Variables

Answer 2

P(X=x|Y=y) = P(Y=y|X=x)P(X=x) / [ΣP(Y=y|X=t)P(X=t)]

Question 3

Bayes’ Theorem

Events as Continuous Random Variables

Answer 3

f(x|y) = f(y|x)f(x) / [∫f(y|t)f(t)]

Question 4

Frequentist vs. Bayesian Approach

Answer 4

• to the frequentist, probability is long-run relative frequence
• to the Bayesian, probability is a degree subjective belief

Question 5

Statistical Inference

Answer 5

• adopt a probability model for data X, distribution of X depends on parameter θ
• use observed value X=x to make decisions about θ
• translate the decision into a statement about the process that generated the data

Question 6

Parameter Definition

Frequentist vs. Bayesian

Answer 6

• a frequentist defines a parameter as an unknown constant

- a Bayesian defines a parameter as a random variable

Question 7

Model

Frequentist vs. Bayesian

Answer 7

• frequentist: f(x)

- Bayesian: f(x|θ) OR p(x|θ)

Question 8

Bayesian Models

Answer 8

• choose a prior distribution to describe the uncertainty in the parameter: π(θ)
• observe data
• use Bayes’ Theorem to obtain a posterior distribution, π(θ|x)
• this posterior distribution could be used as a prior for the next experiment

Question 9

Influence of the Prior

Answer 9

• the most frequent objection to Bayesian statistics is the subjectivity of the choice of prior
• however for robust data where the model is very good, the influence of the choice of prior should tend to 0 as the sample size increases

Question 10

How do you determine the posterior distribution?

Answer 10

π(θ|x) = f(x|θ)π(θ) / [∫f(x|t)π(t)dt]

∝ f(x|θ)π(θ)

Question 11

What can you do with a posterior distribution?

Answer 11

• give a point estimate of θ

- test hypotheses

Question 12

Decision Theory

Answer 12

-anytime you make a decision you can lose something
-risk = expected loss
-the goal is to make decisions that will minimise risk
d = d(x) = ∈ D
-where d(x) is a decision based on the data and D is the decision space

Question 13

Decision Space

Answer 13

• the set of all possible decisions that might be made based on the data
• for estimation, D = parameter space
• for hypothesis testing, D = two points

Question 14

Loss Function

Answer 14

L = L(d(x), θ) ≥ 0

-when X and θ are random, L is a real-valued random variable

Question 15

Expected Loss

Answer 15

E(L) = E(E(L|X))

= ∫ [∫ L(d(x), θ) dπ(θ)] dP(x)

Question 16

Bayes’ Decision

Answer 16

• any decision d(x) that minimises the posterior expected loss for all x
• meaning that it also minimises the overall expected loss (i.e. risk)
• this is the theoretical basis for using the posterior distribution

Question 17

Prior Distribution

Beta Distribution

Answer 17

π(θ) = Γ(α+β)/[Γ(α)Γ(β)] * θ^(α-1) * (1-θ)^(β-1)

-for 00, β>0`

Question 18

Properties of the Beta Distribution

Answer 18

-defined on [0,1]
E(θ) = α/α+β
Var(θ) = αβ / (α+β)²(α+β+1)
-for α=β=1, the distribution is uniform
-can assume a variety of shapes depending on α and β

Question 19

Conjugate Priors

Answer 19

• for some model and prior combinations, the prior and posterior distributions will be from the same family
• e.g. the beta distribution is a conjugate prior for a Bernoulli model
• conjugate priors are very convenient and exist for many models

Question 20

Loss Function

Squared Error Loss

Answer 20

-any different functions can be taken for the loss function as long as they satisfy the property that more wrong = greater loss
-e.g. the squared error:
L(d,θ) = k (d-θ)²
-we can drop the proportionality constant

Question 21

Minimise Expected Loss

Answer 21

-let μ = E(θ|X=x)
-then:
E(L(d,θ) | X=x)
= (d-μ)² + Var(θ|X=x)
-this is minimised when d=μ = E(θ|X=x)
-i.e. Bayes’ estimate under squared error loss is the posterior mean