Week 4 - Test Construction

Question 1

Rational-empirical approach

Answer 1

relies on both reasoning from what is known about psychological construct, and collecting and evaluating data about how the test and items actually behave when administered

Question 2

Empirical approach

Answer 2

relies on collecting and evaluating data about how each of the items from a pool of items discriminates between groups who are through to show or not show the measured attribute

Question 3

Steps (5)

Answer 3

1. Test Conceptualisation
2. Test Construction
3. Test Tryout
4. Item Analysis
5. Test Revision

Question 4

Steps (specific)

Answer 4

1. Specify Attributes
2. Check literature for existing test
3. Choose measurement model
4. Write and edit items
5. Administer and analyse response
6. Select ‘best’ items for test
7. Check reliability and validity
8. Norm
9. Prepare test manual
10. Publish test

Question 5

Specification of the attribute

Answer 5

Attribute, construct, latent trait, test specification

Question 6

Attribute

Answer 6

consistent set of behaviours, thoughts of feelings of a characteristic

Question 7

Construct

Answer 7

a specific idea or concept about a psychological process or underlying trait

Question 8

Latent Trait

Answer 8

involves the strong assumption that there is only one dimension underlying the attribute

Question 9

Test specification

Answer 9

a written statement of the attribute or construct that the test constructer is seeking to measure and the conditions under which it will be used

Question 10

Literature search

Answer 10

• See how others have approached the problem in the past
• Identify theories or other constructs that may be relevant
• Obtain a clear, theory-informed conceptualisation and definition of the target construct

Question 11

Literature search questions

Answer 11

○ Do psychological traits and states exist
○ Can they be measured
○ Test behaviour is predictive
○ What are tests strengths/weaknesses/errors
○ Is it fair and will benefit society

Question 12

Types of measure

Answer 12

• Nominal - classification or categorisation - categories themselves are not meaningful - no relationship between categories (includes yes/no questions)
• Ordinal - classification but in some sort of rank - not units of measurement, not evenly spaced
• Interval - equal intervals between each number
• Ratio - as per interval, but has a true zero

Question 13

Models of measurement

Answer 13

formal statement of observations of objects mapped to numbers that represent relationship among the objects

Question 14

Trace line

Answer 14

a graph of the probability of response to an item

Question 15

Classical test theory, Item Response theory (CTT, IRT)

Answer 15

not actually sure what these mean?

Question 16

Differential item functioning

Answer 16

possibility that a psychological test item will behave differently for different groups of respondents

Question 17

Item writing and editing

Answer 17

• Items should be relevant and representative
• Plan for item writing - a plan of the number and type of items that are required for a test
• Over inclusion of items is recommended at this point
• 5th - 7th reading level

Question 18

Item writing guidelines

Answer 18

• Use straight forward language
• Avoid double barrelled items
• Avoid slang and colloquial expressions that can quickly become obsolete
• Consider if using positive and negative words is a good idea
• Write items that majority can respond to appropriately
• Ask about sensitive issues using straightforward and non-judgemental language
  Phrasing is consistent with response options

Question 19

Likert Scale

Answer 19

Typically provides the test-taker with 5 or 7 possible responses along a continuum

Pros - degree of trait can be measured, lots of information, easy to use, works best with strong statements
Cons - odd vs even number of responses

Question 20

Binary choice scale

Answer 20

two options, ie. True/false, yes/no

Pros - easy to construct and score, quick to administer, a lot of questions
Cons - Allows guessing, only suits dichotomous content, content not as rich

Question 21

Paired comparisons

Answer 21

two options, on a basis of some rule, with each option assigned value (0 or 1)

Question 22

Comparative scaling

Answer 22

sorting or ranking stimuli according to a rule

Question 23

Written/essay formats

Answer 23

Pros - written communication, complex and imaginative, information generated not recognised

Cons - narrow content, bluffing possible, hiding behind good writing, scoring is time consuming, inter-rater reliability issues

Question 24

Test try out

Answer 24

• Administer test on representative sample
• Use standardised instructions
• Data is then used to narrow down number of items

Question 25

Item Analysis

Answer 25

Properties to investigate: Item difficulty/distribution, Dimensionality (i.e. factor analysis), Item reliability, Item validity, Item discrimination

Question 26

Item Validity

Answer 26

the extent to which the score on an item correlates with an external criterion relevant to the attribute

Question 27

Performance of items Item Difficulty Index

Answer 27

Performance on each individual item should differ (i.e. 100% correct is bad) Item difficulty index = examinees who answered correctly/total number of examinees - High index = low difficulty - The probability of guessing correctly is taken into account when deciding the optimal item-difficulty index

Question 28

Item distributions

Answer 28

- Consider removing items with skewed distribution - these are items most people will answer in the same way - Keep items with high variance/distribution - Keep items with a mean close to centre of range of possible scores

Question 29

Dimensionality

Answer 29

Some items may not have a common underlying variable or they may have several underlying variables (go to factor analysis)

Question 30

Factor analysis

Answer 30

- New scale development usually starts with exploratory factor analysis (EFA) to identify a manageable number of factors - Confirmatory factor analysis (CFA) used when number of factors is known - Determine the number of underlying latent variables or constructs - Help condense information - Define the content or meaning of the factors - Helps identify items that are performing better or worse

Question 31

Factor analysis decisions

Answer 31

- Number of factors to extract – Eigenvalues (> 1) – Scree Plot - Rotation - oblique (factors are correlated) or orthogonal (factors are uncorrelated)

Question 32

Item reliability

Answer 32

- a measure of internal consistency Are the items homogeneous? - Correlation between the score for the test item and the scale score (item-scale correlations), inter-relatedness (Cronbach's alpha)

Question 33

Item-discrimination index

Answer 33

- Does the item separate high and low scorers - Comparison of top and bottom performers on the test - Commonly calculated using a point-biserial correlation

Question 34

Test revision

Answer 34

Once test has been revised, it needs to be tried out and go through analysis again Existing tests - tests can 'age' (interpretations, domains, stimuli change; word meanings, test norms, theories behind test) - may need to be reviewed

Question 35

Cross-validation

Answer 35

Collection of additional criterion-related validity data - Is the test applicable to this population? Validity Shrinkage: Often lower validity the second time around ▪ Inevitable ▪ Generally a slight difference ▪ Eliminating chance results ▪ Near enough is good enough!

Question 36

Norming

Answer 36

In a representative population - General population vs specific population Move on to creating a test manual/instructions and publication