Null Hypothesis significance testing

Question 1

What is the scenario we are going to be focusing on here?

Answer 1

• Imagine you are interested in anxiety levels in younger and older adults. You decide to class <45 as younger and >45 as older
• You collect data from a sample of 25 younger and 25 older people on an anxiety questionnaire which produces normal scores, has been thoroughly validated in the UK and for which the population parameters are known

Question 2

Why can’t we assess whether our sample and population means are different from the UK average?

Answer 2

due to sampling errors

Question 3

What are the questions we could ask of this data?

Answer 3

• Does the sample mean look very different from the UK population average?
- Hypothesis: The sample came from a population with higher/ different population mean
- Same as asking ‘does the sample mean look inconsistent with a sample of the appropriate size from this population?’
• Do the older vs. younger sample means look different from each other?

Question 4

What is our null hypothesis significance testing example we are going to be using?

Answer 4

• Imagine there is a carefully validated test which measures statistics ability in 1st year UK undergraduates
• Assume that the population test scores follow N(u, sigma)
• You have good reason to hypothesise that your cohort is better at stats than the general UK 1st year undergrad population

Question 5

What happens in step one of NHST and apply this to our example?

Answer 5

• generate hypotheses:
• Null hypothesis (H0): My cohort is not better at stats than the UK population of 1st year UG students and my sample is typical for sample from that population
• Put in another way: Performance of a sample of students from my year will be comparable to a random sample of the same size from the population of test takers – the sample from my year will not be particularly different from any other random sample of that size from the population of test takers
• Put another way: the sample will not have an atypically high mean
• (H0): the sample came from a population with mean uh = u
• (uh = hypothetical value for population mean from which this sample is drawn)
• (u = population mean for UK 1st year UG students)
• Research Hypothesis (H1): My cohort is better at stats than the UK population of 1st year UG students
• Put another way: performance of a sample of students from my year will not be comparable to a random sample of the same size from the population of test takers – the sample from my year will be markedly higher than other random samples of that size from the population of test takers
• Put another way: the sample mean will be more consistent with that of a random sample of the same size taken from a population with a higher mean
• (H1): the sample came from a population with mean uh> u)

Question 6

What happens in step 2 of NHST and apply this to our example?

Answer 6

• data
• Our ability to test hypotheses rests fundamentally on collecting good data
• Experiment needs to be well designed with as much control as possible of extraneous variables
• In our example, data collection requires us to get an appropriate random sample (N=25) from our cohort and get their stats test results
• We will then be able to work out sample statistics (e.g. mean m, s.d.)
• We also know about the population parameters and so about the SDM

Question 7

What happens is step 3 of NHST?

Answer 7

• evaluate inconsistency with H0
• how inconsistent are the data with H0
• related to where our sample mean is in relation to our parent population mean (because uh = sample mean)
• if our sample mean is closer to the parent population mean then you would say that it is more consistent with our null hypothesis and less consistent with our research hypothesis
• you have to assume that H0 is true

Question 8

What happens in step 4 of NHST?

Answer 8

• reject or fail to reject
• Based on step 3 we now know how inconsistent the data are with H0
• If data are sufficiently inconsistent with H0 we say ‘we reject H0 in favour of H1’
• If data are not sufficiently inconsistent with H0 we say ‘we fail to reject H0’ (this is not the same as finding evidence for H0)

Question 9

What happens in step 5 of NHST?

Answer 9

• interpret
• We now interpret our findings in terms of our hypotheses
• If we could reject the null (H0) we say that we have evidence for the research hypothesis (H1) and that our sample came from a population with mean uh>u
• In our case that would mean we have evidence our cohort is better at stats than the UK population of 1st year UGs
• If we failed to reject the null (H0) then we can not claim to have evidence our cohort is better than the UK population of 1st year UGs
• This is not the same as finding evidence for H0 – we can not conclude that our cohort has the sample population mean as the UK population of 1st year UGs. We are looking for evidence for the research hypotheses, not the null hypotheses.

Question 10

In step 3 if our sample mean is not particularly high what would you say about inconsistency?

Answer 10

Not very inconsistent: p is not small here – i.e. it seems entirely plausible to get a sample mean here if null is true so don’t reject the null

Question 11

In step 3 if our sample mean is uncharacteristically high what would you say about inconsistency?

Answer 11

Very inconsistent: p is rather small here – i.e. it seems too implausible to get a sample mean here if null is true so reject the null

Question 12

When do you fail to reject the null?

Answer 12

• when values of p>a

- suggest not inconsistent with H0

Question 13

When do you reject the null?

Answer 13

• When values of p<a></a>

Question 14

What value is a? (how small should p be before we decide to reject H0)

Answer 14

• we use a value of a = 0.05 (5%)

Question 15

What is the conditional probability associated with you sample statistic assuming that the null hypothesis is true sometimes called?

Answer 15

The p-value

Question 16

is p the probability that the null hypothesis is true?

Answer 16

No. We can never know this. We are just looking for evidence of inconsistency

Question 17

if p<0.05 does that mean that H1 (our research hypothesis) is correct?

Answer 17

• no

- by chance (i.e. random sampling error) you could get a sample mean that was extreme (leading to small p)

Question 18

What is the z-test?

Answer 18

A null hypothesis significant test in which in step 2 you find out the z-score for your data and hence the conditional probability of obtaining that value. If p < 0.05 then we can reject the null hypothesis in favour of the research hypothesis as our data is inconsistent with the null hypothesis - it is not likely to get this sample mean from the parent population.

Question 19

When do you use a z-test?

Answer 19

• whenever we want to check a sample mean we have obtained is different from a population mean?
• does it look like our sample came from that population or a population with a different mean?

Question 20

Why don’t we usually use a z-test?

Answer 20

Because we don’t usually have the parent population parameters

Question 21

What is a one tailed hypothesis?

Answer 21

• only interested in one tail of the graph
• It can be interested in the conditional probability of having got a sample mean as low or lower or as high or higher than our population mean
• used when making a directional prediction
• only formulate a one tailed hypothesis if you have good reason to predict the effect will be in a specific direction

Question 22

What is a two tailed hypothesis?

Answer 22

• when you are interested in both tails of the graph
• when we believe our sample is different from the general population but we don’t know in which direction
• used when making a non-directional prediction
• two tailed hypotheses are more conservative as you need a more extreme sample mean to be able to reject the hypothesis (there needs to be 2.5% on either side of the tail as opposed to 5% on one side of the tail)

Question 23

How does does step 3 of z two tailed z-test work?

Answer 23

• Work out the z-score for our sample means
• Work out the probability
• Then double the probability but not the z score to work out the area either side of the tail

Question 24

What is the key difference when working out the conditional probability for a two tailed z-test?

Answer 24

You need to double the probability to get it on both sides.

Question 25

what is a type 1 error?

Answer 25

when you reject the null hypothesis when it is true

Question 26

What is a type 2 error?

Answer 26

when you fail to reject the null hypothesis when it is false

Question 27

When do type 1 errors and type 2 errors occur?

Answer 27

- Type 1 errors occur because even if your p-value is small there is still a (small) chance that your data were unusually extreme (and so you rejected the NULL) just due to sampling error - Type II errors often arise because of a problem with your study  Perhaps your sample was biased  Perhaps there was an error in your experimental task  Perhaps your sample size was too small

Question 28

Why do we use a value of a=0.05?

Answer 28

It balances out the probability of making type 1 vs type 2 errors (you make more type 1 errors if a is larger because it would be too easy to reject the null hypothesis and you make more type 2 errors is very large because it would be too hard to reject the null hypothesis)