Chapter 6

Question 1

Major Steps in Test Development

Answer 1

• Define purpose
• Preliminary design issues
• Item preparation
• Item analysis
• Standardization/research
• Final materials and publication

Question 2

Statement of Purpose

Answer 2

• Just one sentence

- Focus on trait and scores/interpretation

Question 3

Mode of Administration

Answer 3

• Group versus individual issue to consider

- Understand processes as person is taking

Question 4

Length of Exam

Answer 4

• Short limits reliability

- Long makes it more reliable

Question 5

Item Format

Answer 5

• Type of question to consider when addressing issues

- Some subjectivity in essay responses, etc.

Question 6

Number of Scores

Answer 6

• Issue to consider when designing
• Bigger test developers with wider application
• Improves marketability

Question 7

Administrator Training

Answer 7

• How much to be considered when designing exam

Question 8

Background Research

Answer 8

• Literature search issue

- Discussions with practitioners if used by clinicians

Question 9

Item Preparation

Answer 9

• Stimulus/item stem (question, apparatus, etc.)
• Response format (M-C, T-F)
• Scoring procedures (correct/incorrect, partial credit, etc.)
• Conditions governing response (time limit, probing of responses, etc.)

Question 10

Selected Response Items (Fixed Response)

Answer 10

• T/F, M/C, Likert, etc.
• Objectively scored
• Assigning of points
• Keep content right, simple, and don’t be too obvious

Question 11

Constructed Response Items (Free Response)

Answer 11

Fill in the blank style test items

Question 12

Inter-Rater Reliability

Answer 12

Scoring requires judgment and a certain degree of agreement is crucial so items are evaluated in the same way or similar way

Question 13

Holistic

Answer 13

• Scoring scheme in which a single judgment about quality

- Overall impression of what the paper is

Question 14

Analytic

Answer 14

• Scoring scheme in which it is rated on several different dimensions
• Grammar, organization, vocabulary, criteria, etc.

Question 15

Point System

Answer 15

Scoring scheme in which certain points must be included for a perfect answer or full credit

Question 16

Automated Scoring

Answer 16

• Scoring scheme by computer programs that simulate human judgment
• Comes from the “and now” period
• Used for scoring essays too

Question 17

Suggestions for Writing Constructed Response Items

Answer 17

• Clear directions
• Avoid optional items (chose to answer 3 out of 5 essays)
• Be specific about scoring procedure when preparing questions
• Score anonymously
• Use sufficient number of items to maximize reliability and validity

Question 18

Three Selected-Response Advantages

Answer 18

• Scoring reliability
• Scoring efficiency
• Temporal efficiency

Question 19

Two Constructed Response Advantages

Answer 19

• Easier observation of behavior and processes

- Exploring unusual areas such as covering materials multiple choice cannot address

Question 20

Item Analysis

Answer 20

Involves the statistical analysis of data obtained from an item tryout

Question 21

Three Phases of Item Tryout

Answer 21

• Item tryout
• Statistical analysis
• Item selection

Question 22

Informal Item Tryout

Answer 22

5-10 people similar to those for whom the test is intended comment on the items and directions

Question 23

Formal Item Tryout

Answer 23

Administration of test items to samples of examinees who are representative of the target population for the test

Question 24

Independent Study

Answer 24

• Conducting a study exclusively for the purpose of item analysis
• Most common practice
• Formal practice for item tryout, subjects often paid

Question 25

Attachment

Answer 25

- Including tryout items in the regular administration of an existing test - SATs and GREs for example

Question 26

Item Difficulty (p)

Answer 26

- Percent of examinees answering the item correctly - P value of .95 is very easy (95% got it right) - P value of .15 is very difficult

Question 27

Item Discrimination (D)

Answer 27

- An item's ability to differentiate statistically in a desired way between groups of examinees - D = sample difference in percent correct in the high and low scoring groups - 50% maximum possible scoring differentiation

Question 28

Truman Kelly

Answer 28

- Showed that statistically, the best way to look at D is 27% top/bottom - Has become the "industry standard" for splits

Question 29

Factor Analysis

Answer 29

- Used to select items that will yield relatively independent/meaningful scores - Applications commonly include attitude scales and personality/interest evaluations - Basic approach: iter-correlations among the items are factor analyzed and underlying dimensions (factors) are identified

Question 30

High Loading Item

Answer 30

- 0.3 or higher in factor analysis - Items that are selected for inclusion in the final test - Each correlation between each item and factor - High cross-loading is NOT a good item

Question 31

Five Guidelines for Item Selection

Answer 31

- Number of items - Content considerations - High discrimination indices - Relationship between p-value and D - Average difficulty level

Question 32

Increased Number of Items

Answer 32

Increased reliability

Question 33

Starting with Easier Test Items

Answer 33

Increases motivation

Question 34

High Discrimination Indices

Answer 34

0.3 to 0.5

Question 35

Maximum possible D-value

Answer 35

- Occurs when p-value is at its midpoint | - Maximum D (1.0) when p = 0.5

Question 36

Mean Score

Answer 36

Sum of the p-values

Question 37

To get an easy test...

Answer 37

Use items with high p-values (closer to 1)

Question 38

To get a difficult test...

Answer 38

Use items with low p-values (closer to 0)

Question 39

Discrimination Index

Answer 39

Difference in percent correct between high-scoring and low-scoring groups

Question 40

Standardization

Answer 40

- Used to develop the norms for the test - Should be the exact version that is published (changing items throws off established norms) - Representativeness is key

Question 41

Equating Programs

Answer 41

- Might be conducted at the same time as the standardization program - Alternate forms - Revised editions - Different levels (such as K-12)

Question 42

Final Test Forms

Answer 42

- Test booklets - Technical manuals (psychometrics, how norms obtained, etc.) - Administration and scoring manuals (how to score, etc.) - Score reports and services - Supplementary materials

Question 43

Continuing Research on Published Tests

Answer 43

- Updating norms | - Applicability of the test to various other populations

Question 44

Test Fairness

Answer 44

A test measures a trait with equivalent validity in different groups

Question 45

Test Bias

Answer 45

- A test does not measure the trait in the same way across different groups - Simple difference in average performance does not constitute bias - Difference in averages must NOT correspond to real difference in underlying trait - Group averages should differ if the groups really do differ in the trait we are trying to measure