FAILURE TO RANDOMIZE
Threats to internal Validity
- treatment not randomly assigned, based on part of characteristics or preferences
- ethnic difference last name
- Can test for if control variables coefficients W are 0 or not. If Random, X will be uncorrelated with W.
Failure to follow treatment protocol / partial complience
Threats to Internal Validity of Experiments
- People in the experiment does not do what they are told.
- Treatment group supposed to take a pill every day: some doesnt
- 10% of students switched groups because of behavior problems
- This failure leads to bias in the OLS estimator.
Attrition
Threats to Internal Validity of Experiments
- Some subjects drop out. If the reason for attribution is related to the treatment itself, then the attrition can result in bias in the OLS
- Students move out of district
- Students leave for other schools
Experimental effects / Hawthorne
Threats to Internal Validity of Experiments
- Subjects behaviour might be affected by being in a experiment. (Hawthorne Effect)
Small Sample Sizes
a threat to Internal Validity of Experiments
- small sample does not necessary bias estimator of causal effect
- raises threat to validity of conf intervalls and hypothesis test
Nonrepresentative Sample
External Validity for Idealized Experiments
- The population studied and the population of interest might differ
- sample only includes people with one type of characteristics
Nonrepresentative program or policy
- if there is a considerable difference in the program or policy for the population studied vs the population of interest
- low funding small scale experiment might have different effect than a large experiment
General Equilibrium effects
External Validity
- Turning a small, temporary exp intro a widespread, permanent program
- sometimes only works with small groups
- Ac training Zimbabwe, 10 villages 40% increase wages. Nationwide: Different effect, become skilled, decrease in wage gains
Two types of Quasi Experiments
- If an individual receive treatment “as if” it is randomly determined
- “as if” variation only partially determines the treatment
- causal effect estimated by IV reg
What is Quasi Experiments
- experiment that tries to find causal effect, or relationship between dependent and independent variable
- main difference: groups not randomly assigned.
- effect estimated on a targeted population
Quasi Experiment also called
Natural Experiment
Quasi Advantages vs Normal
- when it is unethical or impractical to run a true experiment
- higher external validity, can use real world interventions instead of artificial labratory settings
- Higher External Validity than most true experiments (lets you involve real-world interventions)
- Higher internal Validity than other non-experimental types of research, because it allows you to better control for confounding variables
Confounding Variables
collect data on sunburns and ice cream consumption. Find a correlation. Does not mean ice cream leads to sun burn. In this case, the Confounding variable is temperature. Temperature leads to both ice cream and sun burn
Disadvantages Quasi
- without randomization it can be difficult to verify that all omitted biases have been accounted for
What is DiD
- type of panel data
- control & treatment
- find causal effect of a specific intervention or treatment
DiD Assumptions
- parallell trend
- no spillover effects
DiD Strengths
- can obtain causal effect if assumptions met
- comparison groups can start at different level
Quasi: Potential Problems (Internal)
- Fail to randomize
- Fail to follow treatment protovol
- Attrition
- Experimental Effects
- Instrument Validity in quasi Experiments
Quasi, failure to randomize
threat to internal validity
- failure to produce treatment level X that is random = biased OLS
SOL: check for systematic differences between control group and treatment group
Quasi, failure to follow treatment protocol
threat to internal validity
Quasi Attrition
Similar
Quasi Experimental
- Advantage that they are not true experiments
- typically no reason for individuals to think they are experiment subjects
Instrument validity in quasi experiments
threat to internal validity
- if IV is valid
- need both IV relevance and exogeneity
Quasi, Threats to External Validity
- use observational data, so similar to normal experiments
