4 different forms of evaluation:
- Ex-ante appraisal (potential?)
- Programmatic evaluation
- Comprehensive expenditure review
- Impact analysis (some are just glad to help out but investigate if the project actually worked out well and had good impacts are increasingly popular).
Why evaluate? Objectives:
- Lesson learning (has it done what it was supposed to?beneficiaries, program, organization, world).
- Accountability
- Result-based management (use results to improve, ex test on small scale and then scale up)
The logical framework/model of evaluation:
- Needs - ex too low literacy in rural India.
- Inputs - ex monitor teacher attendance and activity.
- Output - parents vist schools daily and report.
- Outcome - teachers attend school more regularly and better quality.
- Impact - hopefully higher rate of literacy.
- Long-term goal - improved educational outcomes and career opportunities.
Different levels of program evaluation:
- Needs Assessment (who is the pop and what do they need?)
- Program Assessment (how address the needs and what are the prerequisites and shortcomings?)
- Process evaluation (are the things delivered? built? don’t assess impact, just process).
- Impact evaluation (all this lecture is about –> lead to the Q: why and when do it work? Can we scale up?)
- Cost-benefit analysis
Theory of change:
ToC analyses how inputs lead to intended outcomes/impacts. Identify causal steps and which underlying assumptions need to hold, what data we need etc..
Different types of correlation:
- Causation: X–>Y.
- Reverse causality: Y–>X
- Simultaneity: Y–>X and X–>Y
- Spurious correlation/OV bias: Z–>X and Z–>Y a third variable affecting both.
Counterfactual:
Need a group of people telling us what would have been the case if we did NOT implement the program. This cannot be done 100% since we don’t have two identical worlds… But we do our best to find a good enough counterfactual so that w can measure the impact (difference between T and C). This helps us measure causality.
What is the basic formula for measuring impacts?
To take the difference between outcome for participants vs non-participants:
Yi(1) - Yi(0)
But, as we cannot observe same unit, we must take the average impact:
E(Yi(1)) - E(Yi(0)).
So this is the expected value for the T minus the expected value for the C group.
What is the bias of the impact measurement?
The bias is:
E(Y(0)|T) - E(Y(0)|C).
So it’s the difference between being in the treatment group but not receiving the treatment and being in the control group where you obviously not receive the treatment.
If we have a perfect counterfactual, this bias=0.
This B happens because we use an estimate of ATE.
3 techniques for impact evaluation:
- Experimental design with randomisation (RCT)
- Matching methods (PSM)
- Difference in difference
- Other in the book… see notes.
Random sampling and assignment:
When we randomly select a sample from a population and den randomly assign some of them in the sample to the T and the rest to the C.
RCT
Random control trial. When using random sample and assignment, we create a relevant comparison group.
There shouldn’t be any systematical differences between the groups, no bias. –> T and C have same outcome Y in absence of the program.
Is it ethical to randomise?
Not always. If the program involves large benefits for the treated ones, then why should my neighbour get those benefits but not me? Just by luck? If we had the chance to prove who needed it the most, maybe it would have been me. But self selection destroys the properties of a relevant counterfactual…
ATE=
Average treatment effect
Issues with RCT:
- External validity (specific context)
- Hawthorne effects - changed behavior for the observed ones.
- John Henry effect - changed behavior for the controlled, work harder)
- Contamination/spillover
- Dropout or attrition
- Partial eq - measuring short term effects.
PSM
Propensity score matching. Find a group that are similar in the observable characteristics and assume that the unobservables also are similar across treated and untreated.
When use PSM?
When RCT is not possible, ex in ex-post situations where program is already implemented or when RCT is too expensive.
PSM method’s 3 steps:
- Use surveys to select several characteristics (X)(age, income etc) that help predict participation. Estimate the probability of participation, p(X). This is someone’s propensity score.
- Match treated to untreated using p(X), as close as possible.
- Impact = average difference in outcomes between the groups.
PSM issues:
- Requires a loooot of data to find relevant characteristics
- Strict assumption that unobservables also are similar.
Difference-in-difference
No random sample available, maybe because the program aimed at help out a certain group.
- Look at before and after the program
- Or alternatively look at the change over time of non-beneficiaries as counterfactual (subtracting these differences creates a diff-in-diff).
Diff-in-diff ATE=
[E(Yt1|T) - E(Yt0|T)] - E((Yc1|C) - E(Yc0|C)).
SO simply the difference of the changes over time for the two groups.
Key assumption of DiD
Parallel trends - that the groups have the same pace of change before the program starts, bc then we can assume that if the treated group did had the program they would have been equally well off as the untreated.
Issues of DiD
- Are the parallel trends true? Maybe affected by spurious correlation?
- In practice, ex-ante time-varying unobserved heterogeneity could have been taken care of in the program design to ensure that T and C areas share similar pre-program characteristics. But not possible now…
(if not similar before, the measured impact is not true since Y will be affected).
