Hypothesis Testing

Question 1

hypothesis tests

Answer 1

A hypothesis test is a formal procedure for comparing observed data with a claim (also called a hypothesis) whose truth we want to assess vs. a contradictory claim (hypothesis)

• Confidence intervals are appropriate when our goal is to estimate a population parameter
• But when your goal is to assess the evidence provided by data about some claim concerning a population, then hypothesis tests (or tests of significance) are the appropriate statistical method to use
• A statistical hypothesis is a claim about the value(s) of a single parameter or several parameters or about the form of an entire probability distribution

Question 2

null hypothesis and alternate hypothesis

Answer 2

The alternative hypothesis (H_A) is usually the hypothesis that the researcher would like to prove is true – Can be “two-sided” or “one-sided”

The null hypothesis (H₀) is the opposite to the alternative hypothesis and is the hypothesis of no change (from current opinion), no difference, no improvement, etc.

– The null hypothesis, denoted by H₀ , is the claim that is initially assumed to be true and the alternate hypothesis, denoted by H_A , is the assertion that is contradictory to H₀

– If sample evidence suggests H₀ is false, we reject H₀

– If the sample evidence does not strongly contradict H₀ , then we fail to reject H₀

Question 3

general procedure for hypothesis tests

Answer 3

The basic steps for hypothesis testing are:

1. State a null and alternative hypothesis, H0 vs. HA
2. Collect data and calculate the test statistic
3. Determine the P-value associated with the test statistic
4. Reach a decision/conclusion based on the P-value: reject or fail to reject H0

Question 4

test statistic

Answer 4

a test statistic is a standardized score of our sample statistic, that helps conduct the hypothesis test

example: assume normal probability distribution

how many standard deviations away is the statistic from the mean if H₀ is true?

Question 5

P-value

Answer 5

A P-value is the probability (computed assuming that H0 is true) of obtaining a value of the sample statistic that is at least as extreme or more extreme (as defined by the alternative hypothesis) as the value actually observed

use the magnitude of the P-value as a measure of the strength of evidence against the null hypothesis

• large P-values fail to give convincing evidence against H₀, because they say that the observed result could have occurred by chance if H₀ were true
• small P-values are evidence against H₀, because they say that the observed result is unlikely to occur when H₀ is true (i.e., we observed something rare by chance or the null hypothesis is not correct)

Question 6

statistically significant

Answer 6

“Statistically significant” is an adjective used to describe a sample that seems too unlikely to have occurred just by chance alone

example: researcher compares mean weight loss for a diet treatment to that for an exercise treatment, and reports a P-value of 0.036. She concludes these sample data are “statistically significant”.

But, we never know whether the null hypothesis is true or not, nor does the P-value tell us why we observed the sample we did

at most only one type of error is possible at a time

Question 7

power of the test

Answer 7

The power of the test is the probability of rejecting H₀ , when H₀ is false; it measures the ability of a hypothesis test to find evidence against a null hypothesis that is actually incorrect

power is influenced by:

• # of observations in the sample
• the magnitude of the effect size to be detected

just because we fail to find strong evidenve against the null hypothesis doesn’t mean it’s true

Question 8

effect size

Answer 8

The effect size (magnitude of effect) is the magnitude of the difference between groups or deviation from expected null value

example: a completely randomized experiment compares a current insomnia treatment to a newly developed treatment. researchers observe a statistically significant increase in mean hours slept for the new treatment (P-value = 0.002)

“statistically significant” does not necessarily imply “practical” significance

Question 9

multiple comparisons

Answer 9

Multiple comparisons: Conducting multiple hypothesis tests increases the likelihood of type I error

– Rare statistics are unlikely to occur in a single sample, but more likely to occur in repeated sampling

– Multiple tests is analogous to repeated sampling

researchers conducting multiple comparisons should control for overall type 1 error rate

Question 10

take home message

Answer 10

don’t fall victim to (nor contribute to) the misunderstanding of P-values and “significance”

we never know if a hypothesis is true or not

the results of a hypothesis test depend on:

• study design
• sample size
• effect size (magnitude effect)
• power
• number of comparisons