Chapter 8 - Test Development

Question 1

What are the five stages of test development?

Answer 1

Conceptualization, Construction, Tryout, Analysis, Revision

Question 2

What are the relations among the five stages of making a test?

Answer 2

Sequential process (build off each other), but isn’t necessarily a one and done (the revision process requires going back to previous stages)

Question 3

What types of questions need to be considered during the conceptualization phase?

Answer 3

What is motivating us to create something new/revise something old (what’s the purpose)?
What will the test cover, and who will use it?
How will the test be administered and used?

Question 4

What processes are part of test construction?

Answer 4

Pilot work, determining scaling methods, writing items, deciding how the items will be scored

Question 5

What is pilot work?

Answer 5

Creating a prototype and receiving feedback on it using expert panels or focus groups

Question 6

What are the different rating scales?

Answer 6

Likert, Guttman, Comparative, Categorical

Question 7

What is comparative scaling vs categorical scaling?

Answer 7

Comparative = judgment of a stimulus in comparison with another
Categorical = physical stimuli are placed into two or more alternative (pre-determined) categories

Question 8

What is a Guttman scale?

Answer 8

Items range sequentially from weaker to stronger expressions of an attitude or belief
So, if an individual responds “yes” to a later question, they should’ve also responded “yes” to previous ones

Question 9

What is an item pool?

Answer 9

A set of items from which the final version of the test will be derived

Question 10

What is the difference between selected-response format and constructed-response format?

Answer 10

Selected = items require the test taker to select a response from a set of pre-determined ones (ex: multiple choice; true/false)

Constructed = items require test takers to supply or create the “correct” answer (ex: short response questions or essays)

Question 11

What is computerized adaptive testing?

Answer 11

Interactive, computer-administered test in which the items presented to the test taker are influenced by the test taker’s responses to previous questions

Question 12

What are the benefits of computerized adaptive testing?

Answer 12

• Efficiency in testing time and number of items presented
• Reduces floor and ceiling effects (not having enough items to test either the low or high end)

Question 13

What is a floor effect? A ceiling effect?

Answer 13

Floor = diminished ability to distinguish test takers at the low end of the scale
Ceiling = diminished ability to distinguish test takers at the high end of the scale

Question 14

What are the different methods for scoring items? How might our motivations for test conceptualization impact the type we use?

Answer 14

Cumulatively, Class scoring, Ipsative scoring

The purpose behind creating a test can serve as a guide for the kind of scoring you wish to use. For example, if we’re trying to diagnose somebody, we may use class scoring

Question 15

What is cumulative scoring?

Answer 15

Based on the assumption that the higher the score on a test, the higher the test taker is on that ability or trait

Question 16

What is class scoring?

Answer 16

Responses earn credit toward placement in a particular category with other test takers who have a similar pattern of responses
Ex: diagnostic criteria

Question 17

What is Ipsative scoring?

Answer 17

Comparing a test taker’s score on one scale within a test to another scale within that same test (ideographic interpretation)

Question 18

What qualities separate a good item from a bad item?

Answer 18

Reliable and Valid
Discriminates test takers

Question 19

What are some guidelines for test tryout?

Answer 19

• Should be standardized with same population it was designed for
• 5 to 10 respondents per item
• Should be administered in same conditions, with the same instructions, as the final product

Question 20

What is the purpose of the different types of item analyses indices - difficulty, reliability, validity, discrimination?

Answer 20

To determine which items are best for which purpose (how reliable and valid are they? how well do they discriminate test takers?)

Question 21

What is the difference between the item-difficulty index and the item-endorsement index?

Answer 21

Difficulty = proportion of respondents answering an item correctly
Endorsement = percentage of agreement (as opposed to percentage “correct”, b/c there may not be a “correct” response)

Question 22

What is the d-value on the item-discrimination index?

Answer 22

The proportion of high scorers answering an item correctly versus the proportion of low scorers answering the item correctly

This value can NEVER be negative
We generally want it to be higher, but there’s no particular threshold

Question 23

What are the a and b parameters on an item characteristic curve?

Answer 23

a parameter = the relatedness of the item to the latent construct (the slope of the line); this can NEVER be negative

b parameter = the point of the latent construct where the probability of endorsing the item equals 0.50

Question 24

How do qualitative methods differ from more quantitative methods?

Answer 24

Data generation and analysis relies primarily on verbal rather than mathematical procedures

Question 25

What are some examples of qualitative methods of item analysis?

Answer 25

Think-aloud test administration (verbalize thoughts) Expert panels Sensitivity review (examines fairness by checking for offensive language or stereotypes)

Question 26

When/why would we want to revise a test?

Answer 26

- outdated or offensive words - norms no longer represent the population - psychometric properties need improvement - underlying theory has changed

Question 27

What is Cross-Validation in test revision?

Answer 27

The revalidation of a test on a new sample of test takers, other than those on whom performance was originally found to be valid Can have the same or new inclusion criteria

Question 28

What is Co-Validation in test revision? What are the benefits of it?

Answer 28

Two or more tests are validated using the same sample of test takers - economical for test developers - minimizes sampling error

Question 29

What are the 3 applications of IRT in test building/revising?

Answer 29

- Evaluating existing tests for the purpose of mapping test revisions - Determining measurement equivalence across test-taker populations - Developing item banks