6 Steps of Test Construction
- Define Test’s Purpose
- Preliminary Design Issues
- Item Preparation
- Item Analysis
- Standardization and Ancillary Research
- Preparation of Final Materials and Publication
Test Purpose
- What will be measured?
* Who is the target audience and does the construct match the group?
Preliminary Design Issues definition
Anything that introduces error. Must strike a balance between efficiency and accuracy as well as meet breadth and depth
Examples of Preliminary Design Issues
- Mode of administration
- Length–longer is more reliable (up to ~15min)
- Item format (T/F, multiple choice, essay)
- # of scores (For example, “Depression is multi-faceted and has many scales)
- Training
- Background research
What is the most important Preliminary Design Issue?
Background research!
4 parts of Item Preparation
- Stimulus
- Response
- Conditions governing responses
- Scoring procedures
Stimulus
The question itself is a stimulus.
*We are trying to provoke a specific response correlated with the construct driven ONLY by the stimulus
Response
The behavior you are looking for that is correlated to a construct
Conditions governing responses
What are your rules? Is there a time limit? Are they able to ask questions?
Scoring procedures
Formula or rubric used to formulate final scores
*Make sure each facet is represented (weighted)
Types of Test Items
- Selected-Response Items
* Constructed-Response Items
Selected-Response Items
Where you know all possible responses without bias
*T/F, multiple choice, Likert scale, etc.
Constructed-Response Items
Responses are unknown/more nebulous
*Essays, oral responses, performance assessment
Benefits of Selected-Response Items
- One clear answer
* scoring reliability and efficiency
Benefits of Constructed-Response Items
- No agreed-upon answer
- Bx can give further context
- Goes deeper into construct
Item Analysis
- Item Tryout
- Statistical Analysis
- Item Selection
Item Tryout
aka Pilot Test
- Get subjects similar to target population–cannot be same people used in actual survey
- 2-3x the items you think you will need.
Statistical Analysis
- Difficulty
- Discrimination
- Distractor Analysis
Item Difficulty
% of subjects taking the test who answered correctly
Difficulty formula
p = # people correct // total
What shows good variability for Difficulty?
.5
Difficulty considerations
- Behavioral measure
- Characteristic of the item and the sample
- Extreme p values restrict variability
- More comparative than a ‘cut-off’
Why is Difficulty a behavioral measure
It taps into individual differences in holding the construct
Item Discrimination
- Assumption that a single item and the test measure the same thing–comparing items to other items within the test
- Looks at how well any single item is good at discerning who does/does not have a trait
- You want a high rate!
