Variance Reduction

Question 1

Variance Reduction Methods?

Answer 1

1) Control Variates
2) Antithetic Variates
3) Stratified Sampling
4) Importance Sampling

Question 2

Control Variates idea and estimator definition

Answer 2

Idea: Use a variable with known expected value that is correlated with the target estimator to reduce variance. E[X] is known.
Def: μ̂_cv = ȳ - b( x̄ - E[X] )

Question 3

Control variates when does it work best? Optimal cv coefficient and how much variance reduced?

Answer 3

Higher the correlation the better.
b* = Cov(X,Y) / Var(X).
Variance reduction:
Var(μ̂_cv) = Var(Y) * (1 - ρ²),
Where ρ = Corr(Y, X)

Question 4

Antithetic Variates idea and estimator

Answer 4

Use -ve corr sample pairs so their average cancels out random noise.
Instead of using U1,U2,…, use antithetic pair Ui and (1-Ui)
Estimator:
Û = (1/n) sum (1 to n) (f(zi) + f(-zi))/2

Since Var(X+Y) = Var(X) + Var(Y) + 2abCov(X,Y), variance reduced.

Question 5

Reduced variance of antithetic variates and works best when?

Answer 5

Var(Û) = (1 + ρ) σ² / n
Best when f= monotonic (strictly ⬆️)

Question 6

Stratified sampling idea and estimator

Answer 6

Reduce variance by ensuring all regions of the input space are sampled. Divide sampling space into strata then sample separately within each stratum (eg: 1point per strata).
Estimator:
Î = (1/n) * Σ_{i=1..n} f( (i - 1 + Y_i)/n)

Question 7

Stratified works best when and Latin Hypercube Sampling?

Answer 7

When f is smooth/ slowly varying.

Latin Hypercube: only taking 1 sample from each strata (extension of stratified). For each variable divide range [0,1] into n equal strata. Draw 1 value from each stratum. Guarantees full space coverage.
Works best when low corr/independent between variables.

Question 8

Importance Sampling idea

Answer 8

To reduce Monte Carlo variance by sampling more often from regions where the integrand contributes most to the expected value.
Sample X_i from a different distribution q(x) (the “importance distribution”) and reweight each sample:
Î_IS = (1/N) * Σ f(X_i) * [p(X_i) / q(X_i)], X_i ~ q(x)

Question 9

Why does Importance sampling work? And reduced variance?

Answer 9

It reallocates samples toward areas where f(x)p(x) is large, reducing wasted samples and lowering variance, while keeping the estimator unbiased.

Var(Î_IS) = (1/N) * Var_q( f(X) * [p(X)/q(X)] )

Question 10

Main risk of Importance Sampling?

Answer 10

Poorly chosen q(x) can cause very high or infinite variance due to large weights p(x)/q(x).