L3 Reliability

Question 1

Classical Test Theory

Answer 1

• Is a measurement theory that defines the conceptual basis of reliability.
• The observed score (X) is the True score (T) plus Error (E)
• X = T + E

Question 2

True scores

Answer 2

A hypothetical entity devoid of measurement error.

• True scores deal with reliability while construct scores deal with validity.
• True scores may be “perfect” but perfect reliability does not equal perfect validity.
• May get the same consistently wrong score.

Question 3

Observed scores

Answer 3

Are the actual scores obtained from tests or instruments.

• True scores are hypothetical entities which represent the observed scores under the pretence that they are devoid of measurement error.
• We want observed scores to be as close as possible to true scores.

Question 4

Reliability (another way to think about it)

Answer 4

• The correlation between observed scores and true scores

- The difference between true scores and observed scores is due to measurement error.

Question 5

Error scores

Answer 5

Should have a mean of zero.

• This is because an equal amount of people should have an observed score that is too large as too small (and the magnitudes should be the same)
• Effectively, error cancels itself out across cases.
• Error scores should be an independent and random process (not correlate with anything).

Question 6

R^2 between observed and true scores

Answer 6

• the correlation between observed and true scores is known as the reliability index
• squaring the reliability index gives an estimation of reliability.
• look at slide for formula (Rxx = r2ot

Question 7

Interpretations of reliability

Answer 7

.60 = too low
.70 = bare minimum acceptable for beginning stage research
.80 = good level for research purposes
.90+ = Necessary in applied contexts where important decisions are made about individuals.

Question 8

4 different reliabilities

Answer 8

Rxx = r2ot

Rxx = St2/S2o

Rxx = 1 - S2e/S2o

Rxx = 1 - r2oe

Question 9

Parallel Tests

Answer 9

Are identical to each other psychometrically, but differ in the items that make up each test.

Question 10

All tau-equivalence assumptions

Answer 10

Tau-equivalence, in parallel tests, implies that the true scores associated with each test represent the same construct.

• Thus a person’s true score on 1 test is expected to be identical on the other test.
• Assumed equal error variances between the 2 tests.

Question 11

Test-retest interval

Answer 11

• the length of time between test 1 and test 2 matters.
• the magnitude of the interval between the two testing sessions will affect the magnitude of the correlation between the scores.
• Developmental and lifespan changes.

Question 12

stability coefficient

Answer 12

Is test-retest reliability

Question 13

Internal-consistency reliability

Answer 13

• A practical alternative to the test-retest procedure.
• only need to complete 1 test at 1 occasion
• it treats different items within the same test as different forms of the test.

Question 14

Factors that affect internal consistency reliability

Answer 14

• the degree of consistency between the items of the test

- The length of the test

Question 15

Spilt-half reliability problem

Answer 15

• How should the test be split into halves?
• Cronbach saved the day
• Cronbach introduced a reliability formula that represented the reliability of all possible split-halves.
• Cronbach demonstrated a method of estimation based at the item-level.

Question 16

Cronbach’s Alpha

Answer 16

• ratio of true score variance to total variance
• The product of the number of items squared and the mean inter-item covariance divided by the sum of the square variance/covariance matrix.