SCC Chapter 2 ​Statistical Conclusion Validity & Internal Validity

Question 1

Validity is a property of inferences, not…

Answer 1

Designs or inferences

Question 2

Three questions to ask in exploring threats to validity

Answer 2

1. How would this threat apply in this case?
2. Is there evidence that the threat is plausible rather than just possible?
3. Does the threat operate in the same direction as the observed effect, so that it could partially or totally explain the observed findings?

Question 3

Ruling out threats is a ________ enterprise

Answer 3

falsificationist

Question 4

Statistical conclusion validity concerns what two related statistical inferences?

Answer 4

1. whether the presumed cause and effect covary

2. how strongly they covary

Question 5

What are 8 threats to statistical conclusion validity?

Answer 5

1 Low statistical power
2 Violated assumptions
3 Fishing & the error rate problem
4 Unreliability of measures
5 Restriction of range
6 Unreliability of treatment implementation
7 Extraneous variance in experiment
8 Heterogeneity of units 
9 Inaccurate effect size estimation

Question 6

Fishing & the error rate problem refers to

Answer 6

inflated type I & II error rates

Question 7

Restriction of range refers to

Answer 7

floor or ceiling effects

Question 8

Unreliability of treatment implementation refers to

Answer 8

poorly standardized treatment

Question 9

Extraneous variance in experiment refers to

Answer 9

distracting noises, administrative changes, etc.

Question 10

Heterogeneity of units refers to

Answer 10

obscures covariation between treatment and outcome

Question 11

Inaccurate effect size estimation refers to

Answer 11

outliers, wrong effect size measure

Question 12

What are 8 ​Methods to increase power?

Answer 12

1. Use matching, stratifying, blocking
2. Measure and correct for covariates
3. Use larger samples
4. Use equal cell sample sizes
5. Improve measurement
6. Increase the strength of treatment
7. Increase variability of treatment
8. Use a within-participants design

Question 13

What is Internal Validity

Answer 13

inferences about whether observed covariation reflects a causal relationship in the form in which the variables were manipulated or measured.

Question 14

What are 11 threats to internal validity?

Answer 14

1. ambiguity of temporal precedence
2. cohort effects (cross sectional data)
3. inadequate comparison or control group(s)
4. selection
5. history
6. maturation
7. statistical regression
8. differential attrition/mortality
9. testing
10. instrumentation
11. additive and interactions effects of threats to internal validity