Chapter 5 - Reliability

Question 1

What are some cultural considerations in test construction and standardization?

Answer 1

1- make sure test norms are appropriate for the targeted test taker population
2- understand the culture and time period of the test taker when interpreting results
3- context should be taken into account

Question 2

What is reliability?

Answer 2

The consistency in measurement
I.e. how much of the observed variance is due to actual variance among the true scores

Question 3

What is a reliability coefficient?

Answer 3

An index of reliability
Ratio between the true score variance and the total variance

Question 4

What is an observed score comprised of?

Answer 4

The “true” score plus error

Question 5

What is error?

Answer 5

A component of the observed score that doesn’t have to do with the trait being measured

Question 6

What is measurement error?

Answer 6

Factors associated with the process of measurement that don’t have to do with the variable being measured (i.e. interference)

Question 7

What are the types of measurement error?

Answer 7

Random: unpredictable fluctuations or inconsistencies (i.e. noise)
Systematic: constant

Question 8

Where might error be introduced into the assessment process?

Answer 8

1 - Construction of a test
2 - Test administration
3 - Interpretation
4 - Sampling

Question 9

How might error be introduced when designing a test?

Answer 9

Variance existing within the items on the test
I.e. issues with the phrasing, item sampling, content sampling, examples given, etc.

Question 10

How might error be introduced during test administration?

Answer 10

Setting, materials, environment
Test-taker: sleep, physical discomfort, personal situations
Test examiner: appearance, demeanor, level of training

Question 11

What is methodological error?

Answer 11

Issues with training, ambiguous wording, biased framing of questions, etc.
Tends to be more systematic than random

Question 12

How might error be introduced during interpretation?

Answer 12

Subjectivity by assessors (especially during behavioral assessment)

Question 13

What is sampling error?

Answer 13

When the sample isn’t actually representative of the population

Question 14

What is Test-Retest reliability?

Answer 14

Having the same test-takers take the same test under two different administrations (some time has passed)

Question 15

When is it appropriate to use test-retest as an estimate of reliability? When is it NOT appropriate?

Answer 15

Appropriate when the variable we’re testing is supposed to be stable over time (ex: personality)
NOT appropriate when the variable is expected to change over time (ex: mood)

Question 16

What is the coefficient of stability?

Answer 16

A type of test-retest in which the time interval between administrations is 6 months or more

Question 17

What is Parallel-Forms or Alternate-Forms reliability?

Answer 17

Having the same group of test-takers take two different versions of a test (typically form A and form B, one right after the other)

Question 18

What is the purpose of Parallel-Forms and Alternate-Forms?

Answer 18

To find the Coefficient of Equivalence, or the degree to which two versions/forms of a test (meant to measure the same construct) are similar or equivalent to each other

Question 19

When is Parallel-Forms or Alternate-Forms reliability appropriate to use?

Answer 19

When we have two versions of a test created to measure the same construct

Question 20

What is a Parallel Form? What does this mean for the coefficient of equivalence?

Answer 20

Each version of the test produces EQUAL means and variances
This results in a higher coefficient of equivalence

Question 21

What is an Alternate Form? What does this mean for the coefficient of equivalence?

Answer 21

Each version of the test has similar item content and difficulty, but they don’t meet the strict requirements for a parallel form (same mean and variance)
This results in a lower coefficient of equivalence

Question 22

What is Split-Half Reliability?

Answer 22

Divide the scores from a single test administration into two halves, then look at the correlation (Pearson r) between the two groups of scores

Question 23

What is the Spearman-Brown formula?

Answer 23

Used to adjust the half-test reliability by changing the way the test is divided in half (obtaining different combinations and thus correlations)
Allows a test developer to estimate the amount of internal consistency of a test and its items

Question 24

When is Split-Half reliability appropriate to use?

Answer 24

When the items on the test are linear and continuous (dichotomous items require a different formula)

Question 25

What is inter-item consistency? How is it affected by the operational definition of the construct?

Answer 25

The degree of relatedness between items within a test Gauges how homogeneous the test is, and thus how narrow the construct is Define a construct more broadly = more heterogeneous test items = less inter-item consistency (this is NOT necessarily a bad thing)

Question 26

What is the Kuder-Richardson formula 20?

Answer 26

Used to determine inter-item consistency for tests that contain dichotomous items such as true/false or yes/no

Question 27

What is the coefficient alpha?

Answer 27

The mean of all possible split-half correlations, corrected using the spearman-brown formula Most popular approach to measuring internal consistency

Question 28

What are the measures for a coefficient alpha? How is this affected by the homogeneity of the construct?

Answer 28

Values range from 0 to 1, with an acceptable score being considered 0.7 and above Homogeneous (narrow) constructs have higher coefficient alphas, while heterogeneous (broad) constructs have lower ones

Question 29

Why might a coefficient alpha over 0.9 not be a good thing?

Answer 29

It might mean that test items are too redundant

Question 30

When does internal consistency need to be established?

Answer 30

EACH time that a test is administered

Question 31

What is inter-rater reliability? When is it appropriate to determine?

Answer 31

The degree of agreement or consistency between two or more scorers, judges, or raters Often determined for behavioral measures (but is not appropriate for self-reports)

Question 32

How does the nature of a test determine the reliability metric that we use? (5 ways)

Answer 32

1- Homogenous or heterogeneous 2- Dynamic or static 3- Range restricted or not 4- Speed or power test 5- Criterion-referenced or not

Question 33

What is a speed test used for? How does this affect internal consistency?

Answer 33

Meant to determine how many items you can get done in a particular amount of time Since some items won't even be answered, internal consistency is not appropriate

Question 34

What is a power test used for? How does this affect internal consistency?

Answer 34

Designed to get harder and harder until you cannot answer the questions accurately anymore Since you WILL eventually fail it, internal consistency is not appropriate

Question 35

What does Classical Test Theory (CTT) say about test scores?

Answer 35

There is a true score that genuinely reflects an individual's ability level as measured by a particular test

Question 36

What are potential advantages and disadvantages of CTT?

Answer 36

Advantage: CTT assumptions are more readily met than other models Disadvantages: 1- the assumption of CTT that test items are equivalent in their ability to measure the construct may be problematic 2 - may yield very long tests

Question 37

What is Generalizability Theory?

Answer 37

States that a person's test scores may vary depending on variables in the testing situation and sampling (i.e. the testing environment)

Question 38

What is Item-Response Theory (IRT)?

Answer 38

Provides a way to model the probability that a person with X ability will be able to perform at a level of Y In other words, it views someone's ability as a prediction of how they'll perform on a test

Question 39

What are the potential advantages and disadvantages of IRT?

Answer 39

Advantages: 1- can provide a lot of information about the usefulness of certain items (via discrimination and difficulty) 2 -can be more powerful in its predictions Disadvantage: it uses more advanced statistical methods with more stringent requirements

Question 40

What is item discrimination (a)?

Answer 40

The degree to which an item differentiates between people with higher and lower levels of a trait In other words, it's a measure of how good an item is at distinguishing between two groups

Question 41

What is item difficulty (b)?

Answer 41

How difficult an item is to be accomplished, solved, or comprehended Indicates where on a continuum of a trait the item gives the best information for In other words, where on the scale of the trait does this item differentiate

Question 42

What is the Standard Error of Measurement? What is it used for?

Answer 42

A measure of the precision of an observed test score. In other words, an estimate of the amount of error inherent in the observed score Can be used to estimate the extent to which an observed score deviates from the true score in CTT

Question 43

What is a Confidence Interval? How do we calculate one?

Answer 43

A range of test scores around the observed score that is likely to contain the true score Observed score +/- confidence level z-score x SEM As a general rule in psych, we use a confidence level of 95%, or z-score of about 2

Question 44

What is Standard Error of the Difference?

Answer 44

Used to determine how large a difference between test scores should be before it's considered statistically significant

Question 45

What can standard error of the difference be used to measure? (3 thing)

Answer 45

1- one person's performance on test 1 vs test 2 2 - one person's performance on test 1 vs another person's performance on test 1 3 - one person's performance on test 1 vs another person's performance on test 2