Evaluating Impact

Question 1

What does a TD professional’s efficacy depend on?

Answer 1

Demonstrating the value of the TD investment

Evaluation is one way to document whether the investment in TD achieved its desired outcomes

Question 2

What changes can be made to a training program based on evaluation data?

Answer 2

Program additions, changes, or deletions

Prioritize needs at the org level

Decisions to shift financial and other resources

Question 3

What are 7 purposes of evaluation?

Answer 3

• determining business impact, cost-benefit ratio, and the ROI for the solution
• determining whether the objectives of the solution were met and how well
• assessing the effectiveness and appropriateness of the content and instructional strategies
• reinforcing learning concepts by using a test or similar performance assessment
• providing feedback to the facilitator
• providing feedback to participants about what they learned
• assessing the on-the-job environment to support learning retention.

Question 4

When will evaluation be most beneficial?

Answer 4

When TD professionals incorporate them into the design of the solution from the beginning and relate them to LOs

Question 5

What are some key benefits of evaluation to the TD function?

Answer 5

• secures client support and confidence
• monitors whether program results are consistent with org analysis and needs assessment
• validate performance gap and learner needs
• help determine whether training is the solution to a performance gap
  helps mgt team meet objective and TD to be a true partner in the org’s success

Question 6

What is the first step/input to designing and developing a solution, creating an eval plan, and initiating measurement?

Answer 6

Needs assessment

Question 7

Ralph Tyler’s Goal-Attainment Method

Answer 7

a process for evaluating the success of a curriculum that first identifies objectives, “the ends of instruction.”

The content is then identified to address the objectives, and the various instructional elements and delivery mechanisms are then designed to assist learners in obtaining the objectives.

Question 8

What are the 4 questions that must be answered in the Goal-Attainment Method? Hint: They are progressive through the instructional design process.

Answer 8

• What objectives should learners achieve?
• What learning activities will assist learners to achieve these objectives?
• How should the curriculum be organized?
• How should learner achievement be evaluated?

Question 9

Measure

Answer 9

a standard used to evaluate the degree or quality of the results of a solution

Question 10

Measurement

Answer 10

the process of quantifying assessment data and providing the necessary information to make sound decisions about an issue or situation

Measurements define or quantify specific attributes of an observation.

Question 11

What is the role of the assessment process in evaluation planning?

Answer 11

Using assessment data, TD professionals should identify the desired outcomes before designing an evaluation plan.

To do this they gather, summarize, and interpret the data generated by the assessment process to determine the root cause, which could be the process, a lack of resources, a lack of information, a lack of motivation, poor health, or a need for knowledge, skills, or affective (attitude) learning (KSAs).

Based on these results, TD professionals will determine the best solutions and write objectives.

Question 12

What are three factors to consider when selecting the best measurement process?

Answer 12

• the nature of the solution
• the characteristics of the learners
• the outcome focus

Question 13

Similar to Ralph Tyler’s method, most evaluation process models follow the same process. What are the 5 major steps in the process?

Answer 13

• Use assessment data to identify evaluation outcomes and goals
• Develop an evaluation design and strategy
• Select and construct measurement tools
• Analyze data
• Report data

Question 14

What is a program?

Answer 14

a set of resources and activities directed toward one or more common goals, typically under the direction of a single manager or management team

It may be limited to one set, or complex across multiple sites/levels, and it may be developed by in-house, public, nonprofit, or private sectors

Question 15

Program evaluation

Answer 15

the systematic assessment of program results and, if possible, the assessment of how the program caused them

Includes ongoing monitoring of programs as well as one-time studies of program processes or effects

Question 16

What are 2 categories of program evaluation?

Answer 16

Program evaluation: assesses the effect of a learning program

Learning transfer evaluation: measures the learner’s ability to use what they’ve learned on the job

Question 17

Formative Evaluation

Answer 17

• occurs throughout the design of the TD solution
• improves the draft learning program to better achieve results
• use the results immediately to revise the program and make it more effective
• goal during formative eval: program should be understandable, accurate, current, and functional
• includes activities like pilot testing, beta testing, technical reviews with SMEs, production reviews, stakeholder reviews

Question 18

Summative Evaluation

Answer 18

• occurs after the TD solution has been delivered
• focuses on the results or impact of the solution to provide evidence about the program’s value
• could measure participant reactions, the effect on organizational goals, the initiative’s costs, and stakeholder expectations
• includes activities like standardized tests, participant reaction forms, stakeholder satisfaction surveys, and final return on investment

Question 19

Kirkpatrick’s 4 Levels of Evaluation

Answer 19

Reaction:
Knowledge: learner’s mastery of content; knowledge/skill acquisition
Behavior Change on the Job: comprehensive, continuous monitoring of performance
Business Impact: targeted outcomes or program’s org effect;

Question 20

Measurement methods for Level 1?

Answer 20

• Questionnaire/surveys
• Individual responses in class
• Follow-up interviews
• Observation checklists

Could also include word of mouth feedback

Question 21

Measurement methods for Level 2?

Answer 21

• Knowledge tests, which may be paper-and-pencil tests; oral questions and answers
• Performance tests, role plays, case studies with evaluation or feedback sheets
• Monitored skill demonstrations
• Checklists
• Product tests

Question 22

Measurement methods for Level 3?

Answer 22

• Performance records
• Performance contracts
• Action plans
• Interviews
• Surveys or questionnaires
• Direct observation with checklists
• Supervisor interviews

Question 23

Measurement methods for Level 4?

Answer 23

• Action plans
• Interviews
• Questionnaires
• Focus groups
• Performance contracts

Question 24

When is Level 1 measured?

Answer 24

During the program (end of day)
Right after program completion

Question 25

When is Level 2 measured?

Answer 25

Before, during, and after the program (measure before to create a baseline and during the program as a knowledge/progress check)

Question 26

When is Level 3 measured?

Answer 26

A few weeks to three months after the program

Question 27

When is Level 4 measured?

Answer 27

Three months to one year after the program

Question 28

What are sample measures for Level 1?

Answer 28

- Reactions - Pace and sequence - Relevance (content) - Instrument strategies - Interaction - Facilitator's style - Level of discussion - Objectives met - Environment - Knowledge of facilitator - Participant interaction - Registration process

Question 29

What are sample measures for Level 4?

Answer 29

- Productivity and production measures - Cost or expense measures - Financial or direct output measures - Employee turnover - Engagement ratings

Question 30

How is ROI calculated?

Answer 30

By converting business impact measures to monetary value and comparing them with program costs, which requires placing a value on each unit of data connected with the program

Question 31

6 Data Types of Phillips ROI Method

Answer 31

Level 1: reaction and planned action Level 2: learning Level 3: application and implementation Level 4: business impact Level 5: return on investment Intangible measures

Question 32

Data sources/inputs for ROI

Answer 32

- standard values, which are those that are already accepted in the organization - historical costs, which represent what the measures being converted have cost the organization in the past (such as the cost of an unexpected absence) - extant data - links to other measures that have already been converted to monetary value; for example, placing value on customer and employee satisfaction - input from internal or external experts on a particular measure - participant estimates - supervisor and manager estimates - TD staff estimates

Question 33

How does Phillips ROI Method address the need to isolate the effects of training, answering the question, "How do we know our training caused the results?"

Answer 33

Using a control group, which is not always possible Trend-line analysis, forecasting methods, use of experts

Question 34

Brinkerhoff Success Case Method (SCM)

Answer 34

identifies the most and least successful cases in a program and examines them in detail a useful approach for documenting stories of impact that can then be shared with stakeholders and used to develop an understanding of the factors that enhanced or impeded program success

Question 35

5 Steps of the Success Case Method

Answer 35

1. plan a successful case study 2. create an “impact model” that defines what success should look like 3. design and implement a survey to search for best and worst cases 4. interview and document success cases 5. communicate findings, conclusions, and recommendations

Question 36

Balanced Scorecard Approach

Answer 36

A way for orgs to evaluate effectiveness with more than financial measures

Question 37

What are the 4 perspectives of the Balanced Scorecard Approach?

Answer 37

- The customer perspective: Did the solution, initiative, or practice meet the customer's need or expectation? - The innovation and learning perspective: Did users gain the needed skills or knowledge? - The internal business perspective: Did the solution, initiative, or practice have an effect back on the job? - The financial perspective: Did the solution, initiative, or practice have a financial payoff?

Question 38

What a meta-evaluation?

Answer 38

An evaluation of an evaluation

Question 39

What are other examples of evaluation methods? (10)

Answer 39

- Cost-benefit analysis: measures monetary gains and losses - Culturally responsive evaluation: a holistic framework for centering evaluation in culture - Developmental evaluation: useful in complex or uncertain environments such as innovation, radical program redesign, or crises - HPT evaluation model: an analysis of present and desired levels of performance, identifies the causes for the gap, and offers a range of solutions (human performance technology) - Lean six sigma: combines eliminating waste with lean practices and quality improvement from six sigma - Predictive learning analytics: a systematic methodology for predicting learner outcomes and actions - Return on expectations: Kirkpatrick approach in which stakeholders identify the TD value in terms of how it contributes to their goals - Robinson’s training for impact: helps achieve organizational goals, gives people the skills and knowledge required, and produces measurable results that can be traced on the job. - Six sigma: a disciplined, data approach to eliminate defects in processes - Total quality management (TQM): focuses on improving quality and productivity

Question 40

What are the steps in an evaluation process? (8)

Answer 40

1. Conduct a needs assessment to measure business, performance, learning, and learner needs. Utilize data collection methods for the needs assessment. 2. Analyze needs assessment data to identify solution objectives and develop proposed outcomes. 3. Present the results of the needs assessment. This includes organizing information and discussing interpretations from the analysis, which will show stakeholders how the proposed objectives/outcomes will solve the identified problem and meet management's request/need. 4. Determine the proposed solution. This should be tied to the business need and use business metrics to measure the ultimate objective. May not be training. 5. Determine which results to measure and how to measure them to show whether outcomes were achieved and objectives met. (Identify measures and methods.) 6. Design and develop evaluation strategy/methodology. (Select and construct measurement tools.) 7. Conduct evaluation and engage in data analysis. 8. Report data.

Question 41

What 4 things does a needs assessment identify? How does an eval strategy convert these into eval measures?

Answer 41

Business, performance, learning, and learner needs Business needs = examining results of training initiatives (Level 4) Performance needs = observing learner behavior and evaluating transfer of learning on the job (Level 3) Learning needs = Assessing knowledge and skill mastery immediately after the training (Level 2) Learner needs = collecting data on learner reaction (Level 1)

Question 42

Examples of data collection methods for a needs assessment

Answer 42

- Surveys and questionnaires - Analytics from technology platforms - Examinations, assessments, and tests - Self-evaluation - Simulations and observations - Archival or extant data

Question 43

What is included in the final report of the needs assessment?

Answer 43

- defines and documents the findings - summarizes the problem statement, analyses used to determine need, and proposed module design - discusses how findings relate to overall org strategy/goals - shows how proposed change/plan will benefit the org Results should: - explain current state, outline org goals, and show how proposal will reach goals - provide baseline data to determine gap and measure whether progress is made - envision future state by describing vision and gathering hard metrics/data to measure - explain gap between current and future states by including root causes - identify issues that will support or deter success - describe most appropriate solutions for closing the gap

Question 44

Business drivers are also an input into establishing the evaluation measures. What are business drivers? What are internal and external business drivers?

Answer 44

Business drivers are the internal and external forces that direct an organization's strategy, goals, business needs, and performance needs Internal business drivers are generated by internal decisions and can include technology; a change in system, process, or key policy; shareholder or financial drivers; and new product generation. External business drivers can include economics, human resources, government, public perception, and market or customer drivers.

Question 45

What is the value in a TD professional having knowledge in the analysis of data?

Answer 45

allows TD professionals to explore data comprehensively, inform decision making, and determine the value of the outcome metrics

Question 46

When might measuring business results be especially important?

Answer 46

When there is a high cost or investment involved, such as when designing large or expensive programs or deciding whether an expansive company-wide initiative should be implemented

Question 47

Defining the business need during the assessment and planning process determines what 2 things?

Answer 47

- What measures should be quantified/measured - What questions need to be answered during level 4 evaluation

Question 48

Reporting ROI analysis should be paired with WHAT in order to improve programs and identify tangible benefits?

Answer 48

ROI analysis should be reported alongside HOW the ROI was achieved.

Question 49

What is the Chain of Impact?

Answer 49

A Chain of Impact occurs as people are involved in programs and projects. The Chain of Impact introduces a step in the ROI Methodology between measuring application/behavior outcomes and measuring business impact where analysis isolates program impacts in order to connect the program outcomes to business results. Reaction -> Learning -> Behavior/Application -> Isolate Program Impacts -> Business Impacts -> ROI -> Intangible Benefits, which is linked to Business Impacts

Question 50

What is benefit-cost ratio? (BCR)

Answer 50

the ratio of the benefits of a project or proposal (expressed in monetary terms) relative to its costs (also expressed in monetary terms); summarize the overall monetary value of a project or proposal

Question 51

What is the difference between BCR and ROI?

Answer 51

BCR measures actual results, while the ROI is often forecasted before investing in a program or project How it's displayed is also different: - BCR of 1.0 = every dollar invested, a dollar is returned - 0% ROI = the program broke even - BCR of 2.0 = every dollar invested, two dollars are returned - 100% ROI = the return doubles the investment, meaning the return is 100% = the investment

Question 52

Cost-benefit analysis

Answer 52

a framework for considering a range of costs and benefits in monetary terms; provides several analytical tools to quantify and monetize various hard data and soft data measures to reflect the value of learning

Question 53

What costs should be included in cost-benefit analysis?

Answer 53

all costs: - including time away from work, travel, materials, and overhead costs - lost opportunity costs, such as failure rates, downtime, re-work, and accident costs benefits-solution savings: things such as increased productivity, new accounts generated, reduced overtime, items sold, and on-time shipments

Question 54

How is cost-benefit analysis helpful?

Answer 54

- showing overall benefits - comparing relative magnitude of specific costs and benefits - prioritizing alternatives

Question 55

Utility Analysis

Answer 55

Utility analysis measures a program's economic contribution according to how effective the program was in identifying and modifying behavior and, therefore, the future service contribution of employees

Question 56

Difference between ROI and utility analysis

Answer 56

utility analysis places a value on new skills, disregarding the actual effect of those skills in the workplace

Question 57

Brogden utility estimation equation

Answer 57

U = N × T × dt × SDy – c Where: U = total change in utility in dollars after implementing a learning solution N = number of employees developed T = duration, in number of years, of a program's effect on performance (e.g., tenure of trained employee; number of years trained employees will use new system or approach) dt = true difference in job performance between the average trained and average untrained employees in units of standard deviation (ie, net gain of job performance improvement) SDy = standard deviation of job performance of the untrained group in dollars (ie, monetary value of job performance improvement) c = cost of training per employee

Question 58

How might forecasting be used to define outcome metrics?

Answer 58

a process by which TD professionals can make a prediction of the future based on current and past data, usually by analyzing trends derived from trend data plus additional influences that may impact the results to create a statement about the future

Question 59

When would forecasting be challenging?

Answer 59

if more than one additional influence needs to be considered, because that may require more sophisticated statistical techniques and software for multiple variable analyses

Question 60

ROI doesn't need to be calculated for all projects. When is ROI most useful?

Answer 60

For programs with a long lifecycle, programs that are important to meeting operational goals, and programs closely linked to strategic initiatives