Chapter 7 - Utility

Question 1

What is Utility?

Answer 1

The usefulness or practical value of the test

Question 2

What is utility affected by? (3 things)

Answer 2

1- Psychometric soundness: generally, the higher the criterion validity, the greater the utility BUT not all valid tests are useful tests
2- Costs
3- Benefits

Question 3

What are Costs in psychological testing? Examples?

Answer 3

Disadvantages, losses, or expenses in both economic and noneconomic terms
Economic = financial, time it takes to administer, personnel required
Noneconomic = societal consequences (ex: workers feeling devalued or frustrated)

Question 4

What are Benefits in psychological testing? Examples?

Answer 4

Profits, gains, or advantages
Economic = higher productivity, quality of work, quantity of work, decrease time needed to train, reduction in accidents and worker turnover
Noneconomic = better work environment or improved morale

Question 5

What is Utility Analysis?

Answer 5

A cost-benefit analysis to determine the usefulness or practical value of a test
- some methods are straightforward
- other methods require more complicated mathematical models

Ultimate goal is to make a decision about the best course of action

Question 6

What can we use during a utility analysis? (2 things)

Answer 6

1- Expectancy Data
2- Brogden-Cronbach-Gleser formula

Question 7

What is Expectancy Data?

Answer 7

The likelihood that a test taker will score above or below an estimated threshold on a criterion measure

Question 8

What is the Brogden-Cronbach-Gleser formula?

Answer 8

Used to calculate the dollar amount of utility gain resulting from the use of a particular selection instrument under specified conditions

Question 9

What is utility gain?

Answer 9

An estimate of the benefit (monetary or otherwise) of using a particular test or selection method

Question 10

What are the components of Decision Theory? (4 things)

Answer 10

1- Classification of decision problems (way of defining the issue)
2- Various selection strategies ranging from a single-stage process to a sequential process
3- Quantitative analysis of the relationships b/w test utility, selection ratio, cost of program, and expected value of outcome
4- Recommendation that in some instances, job requirements should be tailored to the applicant’s ability rather than the other way around

Question 11

What can influence the size of a job applicant pool?

Answer 11

1- the complexity of the job or expertise it requires
2- the economic climate
3- geographic location (urban vs rural)
4- top performers on a selection test may not accept the job offer

Question 12

What is the relationship between cut off scores and selection ratio?

Answer 12

The more stringent the cut off score, the less people are selected as being qualified for a position
- may decrease the likelihood of false positives

Question 13

What is a relative cut off score? Fixed cut off score?

Answer 13

Relative = determined in reference to normative data (ex: selecting the top 10% of scorers)
Fixed = made on the basis of having achieved a minimum level of proficiency on a test (ex: driving test)

Question 14

What is multiple cut scores? How is it different than multiple hurdles?

Answer 14

Multiple Cut Scores = multiple cut scores used for a single predictor, splitting people into categories (ex: grades of A, B, C, D, or F)
Multiple Hurdles = achievement of one cut score on one test is necessary in order to advance to the next stage of evaluation

So, Cut = one test multiple cuts, Hurdle = multiple tests multiple cuts

Question 15

What is the Angoff Method of setting cut scores?

Answer 15

Judgments from experts are averaged to yield cut scores for a test
Problems arise if there is low agreement

Question 16

What is the Known Groups Method of setting cut scores?

Answer 16

Collection of data from two groups: one that is known to possess a trait and one that is known NOT to possess a trait
They are then compared and a cut score is selected based on the point of least difference between the groups