Lecture 6 - Permutation, randomisation and Monte Carlo Tests

Question 1

What are disadvantages of Wilcoxon sign test

Answer 1

Requires us to formulate H0 in terms of the median

Less powerful than a t-test when parametric assumptions are made

Question 2

What are advantages and of permutation tests?

Answer 2

Advantages:

• Widely applicable
• Produces exact results
• No specific distribution assumed for the data
• Does not require analytic distribution of test statistic

Question 3

What are disadvantages of permutation tests?

Answer 3

• Each permutation of the data must be equally likely (or probability of occurence is known)
• To be generally applicable, requires thought, and ability to program a computer
• Can be prohibitively computer intensive in complex situations or for large sample sizes.

Question 4

What are randomization methods?

Answer 4

They are like permutation methods, except only a random subset of all possible permutations are generated.

Question 5

How do you generate one randomization?

Answer 5

List the set of values and randomly assign a sign to each value.

Question 6

What are advantages of randomization tests?

Answer 6

• All of the same advantages as permutations plus
• much easier to code
• Number of randomizations is fixed rather than being a function of sample size.

Question 7

What are disadvantages of randomization tests?

Answer 7

• As with permutation tests, each randomization of the date must be equally likely.
• Results will vary from run to run

Question 8

What are Monte Carlo Tests?

Answer 8

A generalisation of randomization tests to include parametric distributions for the data.

Question 9

What is the algorithm for Monte Carlo tests?

Answer 9

1. Set up some distribution for the data, f( ), and some H0.
2. Repeat the following two steps many times:
   a) simulate a data set according the h0
   b) calculate T(X) using the simulated data
3. Add T(x) evaluated from the data
4. Order all the T(X)s
5. p-value is the proportion of the T(X)s as extreme or more extreme then the one from the sample data.

Question 10

What are advantages of Monte carlo tests?

Answer 10

• Data can be assumed to follow any distribution - parametric or nonparametric.
• The analytic distribution of the test statistic is not required.

Question 11

Disadvantages of Monte Carlo tests?

Answer 11

• For parametric simulations, need to estimate nuisance parameters
• Results will vary from run to run.