Chapter 9 & Readings

Question 1

_____________ involves going beyond knowledge (real or estimated) of program impact and attempting to determine how cost effective the program is at what it does and how it compares (in efficiency) to other programs, and then, given this knowledge, deciding whether the program is worth its costs and/or if the program should be replaced with another more efficient program.

Answer 1

Efficiency Assessment

Question 2

Efficiency Assessment is the approach typically taken by ______. It is very important, however, that all evaluators (including you!) understand how to think about efficiency assessment because of the importance of costs and budgets and resource allocation in every organization

Answer 2

economists

Question 3

Who said this, “perhaps the greatest value of efficiency analysis is that it forces us to think in a disciplined fashion about both costs and benefits….Most other types of evaluation focus mainly on the benefits.”

Answer 3

Peter Rossi

Question 4

What are the 2 major types of efficiency assesment?

Answer 4

Benefit Cost analysis and assessment

Cos-effectiveness analysis and assessment

Question 5

In ___________,monetary (i.e., dollar) values are estimated for the resources used (i.e., costs) and for the program effects (i.e., benefits), and these two components ________ are then compared to determine the worth of a program.

Answer 5

Benefit-cost analysis or cost-benefit analysis

costs and benefits

Question 6

Key Idea 1: Benefit-cost analysis can be carried out according to different perspectives, and you will need to _____________ when conducting a benefit-cost analysis.

Answer 6

select a perspective

Question 7

Three common accounting perspectives:

Answer 7

1. individual or target perspective, the costs and benefits are listed that relate to the individuals in the program.
2. the program sponsor perspective, the costs and benefits are listed that relate to the program sponsor.
3. communal or society or national government perspective, the costs and benefits are listed that relate to the community or society.

Question 8

Key Idea 2: The evaluator must list all costs and benefits and then quantify these costs and benefits.

Answer 8

The costs and benefits need to be converted to dollar units, and any benefits or costs occurring in the future must be discounted (i.e., converting them to their present value).
present value in this context is the value of a future benefit or cost in present dollar values

Question 9

benefit-cost ratio

Answer 9

the total dollar benefits divided by the total dollar costs, where any costs and benefits that may occur in the future are converted to their discounted or present values.

A benefit-cost ratio of 1 is the break even point; a benefit-cost ratio that is less than one means that the costs are greater than the benefits; a benefit-cost ratio that is greater than one means that the benefits are greater than the costs (i.e., the desired result).

Question 10

A benefit-cost ratio of 1 is the b_______; a benefit-cost ratio that is less than one means that the costs are greater than the benefits; a benefit-cost ratio that is greater than one means that the benefits are greater than the costs (i.e., the desired result).

Answer 10

break even point

Question 11

A second way to compare the costs and benefits is to subtract the total costs from the total benefits and determine the __________. If the _______ is positive, the benefits are greater than the costs,

Answer 11

Net benefits

Question 12

A third way to compare costs and benefits is the __________is popular in business and it is a measure of the profit made by the program calculated as a percentage of money spent on the program.

Answer 12

return on investment index

(a) subtract total costs from the gross or total benefits and (b) divide the number resulting from step “a” by the total costs

Question 13

prospective benefit-cost analysis (or ex ante analysis) is done before or during the planning or development of a program

Answer 13

Prospective benefit-cost analysis
Prospective analysis is important in helping decision makers to decide whether to develop a full-fledged version of a program idea or to abandon the program idea.
before spending money, one should consider the potential benefits and costs.

Question 14

absolute benefit-cost analysis

Answer 14

1. Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
2. List the relevant costs (including opportunity costs and externalities) and benefits of the program. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
3. Determine or estimate the dollar values of the costs and benefits of the program.
4. Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs) and/or the return on investment index.
5. Make a decision about the usefulness or worth of the program (or prospective program) and make your evaluative judgment and recommendations.

Question 15

absolute benefit-cost analysis

Answer 15

1. Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
2. List the relevant costs (including opportunity costs and externalities) and benefits of the program. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
3. Determine or estimate the dollar values of the costs and benefits of the program.
4. Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs) and/or the return on investment index.
5. Make a decision about the usefulness or worth of the program (or prospective program) and make your evaluative judgment and recommendations.

Question 16

comparative benefit-cost analysis

Answer 16

1. Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
2. List the relevant costs (including opportunity costs and externalities) and benefits for each of the programs being compared. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
3. Determine or estimate the dollar values of the costs and benefits for each of the programs being compared.
4. Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs), and/or the return on investment index for each of the programs being compared.
5. Make a decision about the usefulness or worth of the program (or prospective program) in relation to the other comparable programs and make your evaluative judgment and recommendations.

Question 17

comparative benefit-cost analysis

Answer 17

1. Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
2. List the relevant costs (including opportunity costs and externalities) and benefits for each of the programs being compared. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
3. Determine or estimate the dollar values of the costs and benefits for each of the programs being compared.
4. Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs), and/or the return on investment index for each of the programs being compared.
5. Make a decision about the usefulness or worth of the program (or prospective program) in relation to the other comparable programs and make your evaluative judgment and recommendations.

Question 18

———————— is based on cost and effect data (rather than cost and benefit data as in benefit-cost analysis). In Peter Rossi’s words, in cost-effectiveness analysis the evaluator determines “the efficacy of a program in achieving given intervention outcomes in relation to the program costs.”

Answer 18

Cost-effectiveness analysis

Question 19

STEPS FOR A COST EFFECTIVENESS ANALYSIS

Answer 19

1. DEVELOP A LIST
2. DETERMINE THE DOLLAR VALUE
3. DETERMINE TOTAL COSTS

Question 20

STEPS FOR A COST EFFECTIVENESS ANALYSIS

Answer 20

1. DEVELOP A LIST
2. DETERMINE THE DOLLAR VALUE
3. DETERMINE TOTAL COSTS

Question 21

Why would one want to conduct a cost-effectiveness analysis rather than a benefit-cost analysis?

Answer 21

The answer is because it is difficult to convert many kinds of benefits to dollar units and, therefore, cost-effectiveness analysis is more practical and less controversial.

Question 22

What are the 2 categories that quantitative designs can be broken into?

Answer 22

1. Those used to determine the effectiveness of an intervention
2. those that are descriptive in nature such as surveys

Question 23

_________ means that an evaluator has controlled the effects of variables other than treatment, in order to say with confidence that the results occurs because the participants experienced the program or activity that is the independent variable.

Answer 23

Internal Validity

Question 24

__________ means that the sample is representative of the population, and therefore that if the treatment is applied with another group of people from that population under similar circumstances, it should be effective there as well.

Answer 24

External validity

Question 25

Threats to Internal Validity

Answer 25

``` History Maturation Testing Statistical regression Instrumentation Differential selection experimental mortality experiemental treatment diffusion compensatory rivalry by the control group compensatory eqalization of treatments Resentful demoralization of the control group ```

Question 26

Threats to External Validity

Answer 26

Lack of explicit description of the independent variable Multiple treatment interference The Hawthorne effect Novelty and disruption effects Experimenter effects pretest or posttest sensitization different types of measurment for dependent variables different times of measurement for dependent variables

Question 27

_____________ is a threat to validity when the research involves use of extreme groups such as participants are either the high or low end of the socres on the dependent variable measure prior to the intervention.

Answer 27

Statistical Regression

Question 28

_____________ refers to the individuals who drop out during the study.

Answer 28

Experimental mortality

Question 29

__________________ when the control group does not want to accept the the new way is better.

Answer 29

Compensatory Rivalry by the Control Group

Question 30

________________________ is what happens when one group receives something and the other group receives nothing then any effects on the first group may be due to the fact that this group received something and not to the specifics of what it received.

Answer 30

Compensatory equalization of treatments

Question 31

________________________ if the members of the control group know that the experiemental group is getting something, they become demoralized because they are not gettilng whatever that is.

Answer 31

Resentful demoralization of the control group

Question 32

______________means that the evaluators need to provide detail so that others can know what needs to be included if they are going to implement the same intervention in another setting.

Answer 32

Lack of an explicit description of the independent variable

Question 33

______________is an extrernal validity threat in which the participants experience several things as part of the treatment.

Answer 33

Multiple treatment interference

Question 34

______________________ is an external validity threat that happens when the participants of the study are influenced because they think someone is watching them.

Answer 34

Hawthorne Effect

Question 35

______________________is an external validity in which a person may embrace something because its new or are reluctant to embrace something new.

Answer 35

Novelty and disruption effects

Question 36

__________ is a threat to external validity in which the results of the intervention maybe due to the researcher and cannot be reproduced.

Answer 36

Experiementer effects

Question 37

_________is a threat to external validity in; which the person evaluated could be sensitized to issues because of a pretest or because they know they will be tested.

Answer 37

Pretest or Posttest sensitization

Question 38

What are 2 other types of validity that could be important?

Answer 38

``` Treatment fidelity (not implemented as intended) Strength of treatment (the treatment was not given as strong enough dosage to justify the expectation of seeing change ```

Question 39

What do the following symbols represent in experimental and quasi-experimental designs? R O X

Answer 39

R- Random assignment O-Observation X-stands for independent variable if there is one

Question 40

Describe posttest-only design.

Answer 40

No pretest is given but participants are randomly assigned to groups and a posttest is given. R XO R O

Question 41

Describe single-factor multiple-treatment design.

Answer 41

``` Involves more than 2 groups R O X1 OOO (randomly assigned, pret, mentor postt, postt, postt) R O X2 OO (randomly assigned, pret, service program, postt, postt, postt) RO OOO (randomly assigned to no-treatment control, postt, postt, postt) ```

Question 42

Describe solomon four-group design.

Answer 42

``` Developed to test the effect of pre-test sensitization of the dependent variable. R O X O (random assig., pretest, treatment, postt) R O O (Random, assign, pretest, no treatment, post t) R XO (random assig., no pret, treatment, post t) R O(random assig., no pret, no treatment, post t) ```

Question 43

What design involves more than one independent variable and utilizes alphabetic symbols more commonly than Xs.

Answer 43

``` Factorial Design For ex.: if you have 2 variables with more than one variable A high school grad. A1 (has diploma) A2(does not have one) B Job Training B1 (part. in job training) B2(does not participate in training) A X B (interaction b/w the 2 variables) ```

Question 44

_____________________ occurs when the participants can be divided into 2 or more groups, one of which does not get the treatment. Quasi experimental

Answer 44

Static Group Comparison Design X O ----- O Another example for the Bootcamp: X O1 O2 O3 O4 ----------------------- O1 O2 O3 O4

Question 45

__________________ quasi-experimental group design is similar to static-group comparison design, exept that evaluators use a pretest for both the experimental and control groups. OFten called a baseline.

Answer 45

Nonequivalent Control Group Design O X O ----------- O O

Question 46

Describe the regression discontinuity design.

Answer 46

Used in evaluation studies in which the intervention and control groups are selected on the basis of scoring above or below a certain cutoff point on a test. O C X O ------------ O C O

Question 47

___________ occur when the evaluator does not have access to a control group which involved a pretest and posttest.

Answer 47

Single group design

Question 48

_________ design includes an intervention and then multiple measures of the dependent variable at various time intervals.

Answer 48

Time Series Design

Question 49

__________ involve in-depth exploration of a single case, such as an individual, a group of individuals, a classroom aschoo, a clinic, or even an event.

Answer 49

Case Studies

Question 50

___________ are studies that ask questions about the social and cultural practices of groups of people. They focus on lived experiences, daily activities, and social context of everyday life from the perspective of the participants.

Answer 50

Ethnographic Designs

Question 51

________________is a theory of language use within a social context.

Answer 51

Sociolinguistic Theory and ethnography

Question 52

___________________ designs for evaluation are based on the belief that we can understand the meaning of events by engaging in reflection about the way we talk about them.

Answer 52

Narrative

Question 53

___________________ states that the evaluator sets aside prior asuumptions and to understand on a deepr level as experienced in conscious and unconscious ways by the participants.

Answer 53

Phenomenological Studies

Question 54

If qualitative and quantitiative designs are implemented as the same time in a study it is called ________design.

Answer 54

Concurrent mixed methods design

Question 55

________________form of mixed method design, quantitative and qualitative designs are implemented fairly independently, perhaps by 2 teams.

Answer 55

Dialectical form

Question 56

_______________________ is a design in which one data set is collected to support the larger data set in a study, although dialogue occurs between the 2 sets of data.

Answer 56

Embedded mixed methods design

Question 57

________________ is a research design in which one type of data is collected first, and then new questions based on these findings are generated for collection of the other types of data,

Answer 57

Sequential mixed methods design

Question 58

_____________ any research design that uses both quantiative and qualitative collection methods (either sequentially or concurrently) with the goal of prompting social change at levels ranging from the personal to the political.

Answer 58

Transformative mixed methods design

Question 59

_________a subtype of trasnformative mixed method design in which an evaluation cycles through several phases.

Answer 59

Transformative cyclical mixed methods design | * particular responsiveness to diversity and cultural issues is prominent

Question 60

Altschuld and Kumar present a model that involved three phases of a needs assessment. What are these three phases? A. Planning, Conducting, and Assessing. B. Preassessment, Assessment, and Postassessment C. Attitudes, Learning, and Behavior D. Input, Output, and Moderation

Answer 60

b

Question 61

When is there a lack of treatment fidelity? A. When the evaluator conducts more than one study at a time. B. When the evaluator does not implement the program/study as intended C. When the evaluator relies on an external auditor. D. When the evaluator is part of the organization that is conducting the evaluation.

Answer 61

b

Question 62

Cost Analysis is defined as an evaluation’s determination of whether a program’s effect was worth its cost. A. True B. False

Answer 62

A

Question 63

The majority of evaluations are conducted for more than one purpose. A. True B. False

Answer 63

a

Question 64

``` Which evaluation type or types are associated with the transformative paradigm? A. Outcome Evaluation B. Country-Led Evaluation C. Deliberative Democratic Evaluation D. B & C ```

Answer 64

D

Question 65

``` In international development evaluation, how is the “gaining insight” purpose of evaluation typically labeled? A. Needs assessment B. Impact C. Relevance D. Responsiveness ```

Answer 65

c

Question 66

The second major category of evaluation purposes is to figure out areas in need of improvement. What types of evaluations fall in this category? A. Implementation, responsive, process, and developmental evaluation B. Context, capacity, needs and assets assessment, and relevance evaluation C. Impact, policy exportability, and sustainability evaluation D. Gender analysis, deliberative democratic, and country-led evaluation

Answer 66

A

Question 67

Which of the following is NOT a purpose for an evaluation? A. To identify needed inputs, barriers, and facilitators to program development or implementation. B. To determine inequities on the basis of gender, race, ethnicity, disability, and other relevant dimensions of diversity. C. To determine how a program can be made to look positive despite its lack of impact/effect. D. To demonstrate that accountability requirements are fulfilled.

Answer 67

c

Question 68

``` When the researcher can draw clear causal conclusions from what went on in a study, this is called: A. Internal validity B. Internal reliability C. Internal consistency reliability D. External Validity ```

Answer 68

A