Midterm 2 - Scale Development/Evaluation (Boateng et al.)

Question 1

What are the 4 main test construction strategies?

Answer 1

• correspondence
• projective
• empirical (criterion-group)
• construct (rational/theoretical)

Question 2

Which test construction strategy is mostly used for questionnaires and was dominant in the 20th century?

Answer 2

correspondence strategy

Question 3

What is the correspondence test construction strategy? What are the 4 assumptions? What are the problems with it?

Answer 3

• intuitive and atheoretical
• assumes each item corresponds to a characteristic of interest
• assumes items have common meaning for all test-takers/examiner
• assumes test-taker can accurately assess requested info
• assumes test-taker will be honest
• issue: hard to meet last 2 assumptions

Question 4

What is the empirical strategy for test construction?

Answer 4

• items selected on basis of relations to external criteria (eg contrasted groups)
• meaning of an item is not equal to verbal content of item
• interpretation of scores by “cookbook”: empirically known correlates of high and low scores

Question 5

The empirical strategy emerged in response to the understanding that ____

Answer 5

people aren’t always honest when they answer questionnaires!

Question 6

Is face-validity a concern with the empirical test construction strategy?

Answer 6

NO! (verbal content of items not equal to meaning)

Question 7

What is a popular example of the empirical test construction strategy?

Answer 7

• MMPI
• items can differentiate btw diff psycopathology groups

Question 8

the Cambridge analytica drama was a result of which type of test construction/interpretation strategy?

Answer 8

empirical strategy

Question 9

What are the 4 primary concerns of the empirical test construction strategy?

Answer 9

• are indivs in contrasting groups diff as intended (is criterion used to create groups valid–need to make sure the groups don’t share characteristics)
• unintended group differences (within group diffs)
• problem of generalization (items may function differently in diff contexts)
• item overlap (eg items flag either depression or anxiety so know something is up but don’t know specifics)

Question 10

What is the construct (rational/theoretical) strategy for test construction? What are the 3 assumptions?

Answer 10

• emphasizes theoretical definitions of domains to be measured
• each item is a referent for a construct
• incorporates stats methods for reliability/validity
• think nomologic net
• assumes a person possesses some degree of a construct
• assumes nontest behaviours can be identified which are referents for the construct
• assumes test responses are referents for the construct

Question 11

The construct (rational/theoretical) strategy for test construction originated in the ____s and was prominent from ____ to ____

Answer 11

1950s; prominent from 60s to now!

Question 12

What are the 9 typical steps in construct (rational/theoretical) test construction approach? Which of the 3 phases is each a part of?

Item Development (2 steps)
Scale Development (4 steps)
Scale Evaluation (3 steps)

Answer 12

1. identification of domain and item generation (item dev)
2. content validity (item dev)
3. pre-testing of questions (scale dev)
4. sampling and survey admin (scale dev)
5. item reduction (scale dev)
6. extraction of factors (scale dev)
7. tests of dimensionality (scale eval)
8. tests of reliability (scale eval)
9. tests of validity (scale eval)

Question 13

What was the goal of Boateng’s article on best practices for scale development?

Answer 13

• provide amalgamation of technical literature and lessons learned through experience
• essentially a primer for scientists and practitionners

Question 14

What are the 3 phases of scale development according to Boateng et al?

Answer 14

• item development
• scale development
• scale evaluation

Question 15

What is step 1 of test construction according to Boateng et al?

Answer 15

• identification of domain and item generation
• part of item development phase
• can use deductive or inductive method to generate items
• need to specify purpose of domain and confirm there are no existing instruments
• need to specify boundaries of domain and identify appropriate questions
• specify and define dimensions of domain

Question 16

What is step 2 of test construction according to Boateng et al?

Answer 16

• content validity
• part of item development phase
• do items adequately measure domain of interest
• evaluation by experts and by target population

Question 17

What is step 3 of test construction according to Boateng et al?

Answer 17

• pre-testing questions
• part of scale development phase
• ensuring Qs and answers are meaningful (do Qs reflect domain and do As produce valid measurements)
• do cognitive interviews (admin draft Qs and have respondents verbalize mental process while they answer)

Question 18

What is step 4 of test construction according to Boateng et al?

Answer 18

• survey admin and sample size
• part of scale development phase
• gathering enough data from the right people
• establish sample size and type of data to use

Question 19

What is step 5 of test construction according to Boateng et al?

Answer 19

• item reduction
• part of scale development phase
• assess item difficulty, discrimination between items, and inter-item and item-total correlations

Question 20

What is step 6 of test construction according to Boateng et al?

Answer 20

• extraction of factors
• part of scale development phase
• use factor analysis to determine optimal number of factors or domains that fit a set of items

Question 21

What is step 7 of test construction according to Boateng et al?

Answer 21

• tests of dimensionality
• part of scale evaluation phase
• do test and broad construct work the same way if applied to a new sample or at different time points?
• testing if latent constructs are as hypothesized
• create scale scores

Question 22

What is step 8 of test construction according to Boateng et al?

Answer 22

• tests of reliability
• part of scale evaluation phase
• establishing if responses are consistent when repeated
• assess internal consistency and test-retest reliability

Question 23

What is step 9 of test construction according to Boateng et al?

Answer 23

• tests of validity
• part of scale evaluation phase
• ensuring you measure latent dimension you intended
• assess criterion validity (predictive and concurrent)
• assess construct validity (convergent/discriminant, differentiation by known groups, correlation analysis)

Question 24

According to Boateng et al., it is best to use the ____ method when generating items (step 1)

Answer 24

combination of deductive and inductive
(deductive is based on description of domain, inductive based on responses of individuals)

Question 25

According to Boateng et al., the initial number of items generated for a test should be (the same/double/triple) the amount desired in the final scale

Answer 25

at least double

Question 26

When assessing content validity (step 2), Boateng et al. recommend that if resources are limited, one should use at least (expert/target population) judges

Answer 26

expert judges (but best to have both)

Question 27

(T/F) Boateng et al. recommend that all 9 steps that they outline should be used by every study aiming to develop a test

Answer 27

- FALSE, will vary from study to study - also need to consider resource constraints!