Name the four possible measurement modes for typical performance tests
- self report
- other report
- somatic indicators
- physical traces
-> these measurement modes can be reactive (people can influence their score on the construct) or nonreactive
The literature reports different strategies to construct typical performance tests. Three broad classes of strategies are distinguished
- intuitive class: the relation between the construct and the items is of an intuitive nature
- inductive class: the tests are derived from empirical data
- deductive class: the tests start from theoretical or conceptual notions of the construct
The intuitive class consists of …
- the rational method: uses a loose description of the construct that is based on the knowledge of experts or members of the target population
- the prototypical method: asks members of the target population to think of persons having the construct, and write down their behavior that is typical for the construct
The inductive class consists of …
- the internal method: starts from a broad pool of personality or attitude items. The items are administered to a sample of the target population, and the associations are computed between item scores. The test developer searches for clusters of items.
- the external method: starts from a broad pool of items and a criterion that has to be predicted. The items and the criterion are measured in a sample of the target population
The deductive class consists of ….
- the construct method; starts from a explicit theory, and items are derived from this theory.
- the facet design method; starts from a conceptual analysis of the construct.
What are the distinctive item response mode in a typical performance test?
Open ended versus closed ended
Closed ended is divided in: frequency scale and endorsement scale
Endorsement scale is divided in: all or none and intensity
Intensity endorsement scale is divided in: discrete and continuous
When is kappa best to used, and when weighted kappa?
Use kappa when you have dichotomous items
Use weighted kappa when you have ordinal-polytomous items
When do we use the coefficient of identity?
In case of a bounded continuous scale
What are the two versions of the identity coefficient?
- coefficient of identity for interrater agreement
- coefficient of identity for intrarater agreement
How is the identity coefficient for interrater agreement computed?
- Compute Ra * Rb for every item (Ra = score rater A)
- Add the outcomes of step 1
- Multiply the outcome of step 2 by 2
- Square all the individual ratings
- Add all the squared individual ratings
- Divide outcome 3 by outcome 5
What is a response tendency?
The differential application of the response scale
What is a response style?
The differential use of the item response scale by different persons ; this is a person-specific property
What is acquiescence ?
The tendency to agree with an endorsement statement, independently of the content of the statement
What is dissentience?
The opposite of acquiescence
What is a response set?
The differential use of the item response scale by different persons and with different constructs -> a person/construct - specific property
What is the aim of testing?
To yield scores of test takers maximum or typical performance
What are the two types of test scores?
- the observed test score = derived from the item scores by taking the weighted or unweighted sum of the itemscores
- latent variable score = derived from the item responses under the assumption of a latent variable item response model
What is measurement by fiat?
The item scores are assigned to a test takers responses without any theoretical justification
Name two naive strategies for handling item non response
- delete test takers who have not answered one or more items or delete items that are not answered by many test takers
- to substitute the test takers personal mean -> the variance of the test score is affected in this way
An item score consists of two parts, namely …
A fixed component and a random residual
How can you calculate the fixed component of the test taker’s missing item score?
= TWjk
TWjk = PMj + IMk - OM
PMj = the personal mean item score IMk = the mean of item k OM = the overall mean score
What is TWEjk and how is it computed?
TWEjk is the value which is imputed in case of item non response. It is calculated as:
TWEjk = TWjk + Ejk
TWjk = PMj + IMk - OM
Ejk -> a value is randomly selected from a normal distribution
How is the variance computed?
N - 1
- Calculate the mean score
- For each person: calculate item score - mean score
- Square the outcomes of 2 and add them all together
- Divide by n - 1
How can you detect acquiescence and dissentience?
Compute the personal mean scores for the contra-indicative items and the indicative items
A high PMIj & low PMCj = acquiescence
A low PMIj & high PMCj = dissentience
