Program Planning and Evaluation (PowerPlay, Eh?)

Question 1

3 ways PH programs can intervene

Provide examples.

Answer 1

Policy/Structural
E.g., Fluridate water, treat sewage, outlaw transfat

Incentives & Disincentives
E.g., fine for not wearing seatbelts, tax cigs, require immunization

Volitional Behavior
E.g., Excercise, sleep under bednets, use contaceptives

Question 2

Questions we want to answer with/from our program

Answer 2

Did it work?
Achieve intended outcomes?

• *What else do we want to know?**
• reach target group?
• adopted by program/staff?
• fidelity (delivered as intended)?
• quality?
• *Cost effective?**
• did targer groups continue over time?
• continue working of time?

Question 3

Types of Eval: Efficacy & Effectiveness

Answer 3

**Efficacy — ideal conditions **
*RCT, well-resourced program, ‘pilot’ projects. E.g., plenty of staff, time and money to train and pay for people’s time, very tight controls, high standardization assured.

**Effectiveness — real world conditions **
*more variation – in who’s delivering, who’s getting, in what is delivered; less control and standardization; fewer resources.

Question 4

Difference between 2 types of evalution: Outcome v. Process Evals

Answer 4

**Outcome Eval **
Focuses on efficacy or effectiveness

Process Eval
Focuses on what elese we want to know

Question 5

What evaluation does PH prioritize?

Answer 5

OUTCOME

Outcome over process eval

Within outcome evaluaitons – efficacy over effectivess

Question 6

Bottom Line: If we focus on efficacy…

Answer 6

Narrow focus on efficacy can obscure serious issues related to reach and adoption

Question 7

Bottom Line: If we focus on outcomes…

Answer 7

Narrowly defined outcomes can mask key issues about implementation and quality of services

Note: Implementation and quality issues are usually defined as ‘process’ and therefore tend to receive less attention. But outcomes can (and should?) be defined to incorporate these ‘additional’ program elements for full public health achievement.

Question 8

What are the steps of the RE-AIM framework?

Answer 8

REACH
-reach intended recipients?

EFFICACY & EFFECTIVENESS

• work in ideal circumstances (efficacy)
• work in real-world (effectivess)

ADOPTION
-adoped by target delievers (both program and staff)

IMPLEMENTATION
-implemented as intended

MAINTENANCE

• effects last on individuals
• sustained by programs/providers

Question 9

What is a Logic Model?

Other names.

Answer 9

• *Visual depiction of program** showing:
• what will occur
• what will be accomplished
• if-then ‘causal’ relationships among elements
• underlying theories

AKA:

• Theory of change
• Causal Pathway
• Logical Frame (Logframe)

Question 10

What is the order, direction, of a common logic model?

Answer 10

Inputs > Activities > Outputs > Effect/Outcomes > Impact

Question 11

Describe the causal pathway framework.

Answer 11

<!--StartFragment-->

Impact

• Change in health status of the population of interest
• May also have socioeconomic benefits
• *Effect**
• Effect –> Impact*
• Change in knowledge, attitudes, skills, behavior of population of interest

Outputs
Outputs –> Effect –> Impact
-Good quality products and services that must be in place for effects to occur
-Product must be available for use

• *Activities**
• Activities –> outputs –> Effects –> Impact*
• Technical and support tasks required to produce the outputs

Inputs
Inputs –> Activities –> Outputs –> Effect –> Impact
Resources needed to support the activity

<!--EndFragment-->

Question 12

Input

Answer 12

<!--StartFragment-->

Resources needed (does not specify quality)

Inputs –> Activities –> Outputs –> Effect –> Impact

Resources needed to support the activity

<!--EndFragment-->

Question 13

Activity

Answer 13

Technical and support (active voice)

Activities –> outputs –> Effects –> Impact

Technical and support tasks required to produce the outputs

<!--EndFragment-->

Question 14

Output

Answer 14

Quality products in place; influenced by activity (passive voice)

Outputs –> Effect –> Impact

• Good quality products and services that must be in place for effects to occur
• Product must be available for use

<!--EndFragment-->

Question 15

Effect

Are we BASK?

Answer 15

Knowledge, attitudes, skills, behavior (BASK in effects)

Effect –> Impact

Change in knowledge, attitudes, skills, behavior of population of interest

<!--EndFragment-->

Question 16

Input

Answer 16

Change in health status

Change in health status of the population of interest

May also have socioeconomic benefits

<!--EndFragment-->

Question 17

Trick for catergorizing certain causal pathway elements

Answer 17

<!--StartFragment-->

If in program – eliminates effect and impact

If in population – effect and impact

<!--EndFragment-->

Question 18

From Re-Aim to Framework

1. Efficacy
2. Adoption
3. Reach
4. Implementation
5. Effectiveness
6. Maintenance

Answer 18

1. Moderates:
   Output –> Effect
   Effect –> Impact
2. Outputs
3. Outputs
4. Activities and Outputs
5. Entire pathway
6. Outputs (for orgs), Effect (individuals), and Impact (individuals)

Question 19

Causal Hypothesis verbage (keywords/how write)

AKA: Theory of change

Answer 19

This intervention, a result of this set of inputs, activities and outputs, will facilitate these changes in the population, which will contribute to the desired impact.

Question 20

Why use causal frameworks?

Answer 20

• Specify intended results
• Determine actions needed for results
• Determine where causal chain broke-down
• Use eval questions to help planning

Question 21

Why develop evaluate programs?

PH perspective (4).

Answer 21

• improve health
• confidently establish program changes in a positive way
• establish best practices
• premium on efficacy over effectiveness

Question 22

Gold standard for efficacy? Why?

Answer 22

RCT — maximizes internal validitiy

Efficacy — ideal circumstances

Question 23

Efficacy evals and internal validity

Answer 23

Internal validity refers to our degree of certainty that:

a) change in outcome variables due to intervention

or

b) lack of change due to failed intervention

Question 24

What must an evaluation design include?

Answer 24

Counter-factual

A compariosn point indicating what would the outcome have been w/o the intervention

Question 25

What could cause change aside from intervention? What about lack of change?

Answer 25

* selection bias * attrition bias * diffusion/contamination * Implementation problems

Question 26

When is attrition harmful to internal validity? Possible solution?

Answer 26

Depends on evaluation design Pre-Post ---\> always harmful (no comparison) Intervention/Ctrl --\> only harmful if differential attrition Possible soultion: Intent to treat --- hard to track down

Question 27

What are some internal validity threats?

Answer 27

* history * testing * insturmention Fix: Control group * selection bias * attrition * contamination Fix: Randomization, intent to threat, non-interacting arms

Question 28

Differences between internal and external validity in program evaluations

Answer 28

``` **Internal --\> is observed change due to intervention** Prioritizes efficacy (in ideal circumstances) ``` **External --\> how generalizable** -Prioritizes effectiveness (in real-world settings) \*\*\*PH requires this -RCTs ≠ external validity - You really can't have both be strong - TENSION between internal validity and external validity - Inverse realtionship

Question 29

Why don't RCT's have strong external validity?

Answer 29

Can’t randomize people in everyday life Can’t make choices for individuals – can not put them into an ‘arm’ Want inclusion, do not want to exclude groups of people Want diffusion/contamination Want to move beyond ‘ideal conditions’

Question 30

What reduces external validity?

Answer 30

**Attrition ** * **Problem with attrition is differential attrition *** - Internal validity - Differential matters for internal * **ALSO reduces external validity*** - Not just differential, but also drop-out - Drop-out --\> not something want/be able to do * **Chances are the you're not paying in effectiveness studies*** - You want to know who's dropping out and why - All about making it work for real-world circumstances - So you are more likely to change intervention --\> willing to revise the model for the context, group, etc

Question 31

External Validity How matters in programs.

Answer 31

Degree to which program efficacy findings can be generalized beyond the specific situation studied to other: * Other target groups and sampling criteria * Other types of delivery sites * Adaptions in the intervention that reduce the resources needed for program delivery * Staff, training and supervision, time, space, materials, supplies, and the duration of the intervention * May not have had training for years --\> so retrain (activity) for skilled care (output) * Participation Rates: % invited in who participated: * by setting, delivery staff, subgroups of consumers/patients * comparison of decliners/targets * Reach within Target Group– * Representativeness of Target Group as it was defined (selection or inclusion/exclusion criteria) * % in target group among: people recruited; people who received the program

Question 32

Factors influencing validity

Answer 32

1. Eligibility criteria 2. Intensive intervention 3. Level of standardization in intervention 4. Level of control by reasearcher

Question 33

What are the two most important frameworks in Re-Aim? State definition and differences.

Answer 33

* Linking external to Re-AIM * Reach and adoption are most relevant to external validity ​ * **Reach --\>** ability of program to reach target program * Participation rates * And how acceptable * **Adoption --\>** participation and feasibility of sites * Participation rates among sites​ * Participation among staff * Resources/costs * Be clear on this difference * Reach refers to population * Adoption is about organizations and staff w/in orgs

Question 34

Reach

Answer 34

**Reach:** participation and acceptability rates in the target (participating) population * Participant rates * % recruited who agree to participate * Target group * % of study participants who are from target groups or sub-groups * Attrition rate * % of barriers who dropped out of the program

Question 35

Data on Reach and acceptability may raise what questions? RE-AIM

Answer 35

GOAL: How might reach and acceptability be improved? ​ * Why reach limited to certain sub-group segments * Major barriers to acceptability * Which components acceptable and not? Why? ​

Question 36

Name data sources for Rearch and Acceptability RE-AIM

Answer 36

* Clinical or medical * Program service stats * Attendance records * Survey interviews (also want to talk to refuses and attriters) * In-depth interview (also want to talk to refuses and attriters) * Focus (also want to talk to refuses and attriters)

Question 37

Adaption

Answer 37

**Adoption:** participation of, accessibility by, feasibility for the delivery sites * Site participation (and type) * Staff participation * Attrition rate (by organization and staff) * Resources/Cost --\> staff, space, time demands KEY: Both organization _and_ staff

Question 38

Data on Adoption and acceptability may raise what questions? RE-AIM

Answer 38

* Why did certain sites decline? * Why did staff decline? * Would sites who agreed to participate do so again? * Would staff who agreed, do so again?

Question 39

Main challenge in adoption/feasibility. RE-AIM Name other challenges

Answer 39

Really want to look at **fidelity**!!!!! * Data logs on site recruitment * Close-ended survey questionnaires with delivery staff and admin * In-Depth interview/focus group with staff and administrators * Personal logs charting the # of staff delivering program * Personal log charting # of hours staff engaged in work * Include training on this one * Cost data on site-level expenses for deliver * Asses of staff buy-in * Assess adequacy of staffing and level of staff turn-over a * Assess adequacy and stability of funding at delivery site * Asses adequacy of the management structure and communication