Week 4: Reliability

Question 1

Give an example of a norm-referenced test.

Answer 1

An IQ test – as the average is dependent upon all participants.

Question 2

Give an example of a criterion-referenced test.

Answer 2

A quiz where the answers are compared to a set of correct answers.

Question 3

What are the advantages and disadvantages of using a norm-referenced or criterion-referenced test?

Answer 3

The advantages of norm-referenced test include scores being protected from a test possibly easier or harder (in the case of writing an examination – if the writer is in a worse mood, and writes a very hard test, a lower proportion will fail it), and it yields a good distribution of scores, which then allows discrimination of good and bad). The disadvantage of norm-referenced tests is that the norm is easily changeable – and scores are changed if the norm is (if the norm is really smart, less likely to get grade deserved). The advantage of criterion referenced tests include scores not changing if the norm changes, and that absolute standards can be set based on what people can actually do. The disadvantage of criterion referenced tests include that scores are affected by test difficulty, and that there is a larger risk of floor or ceiling effects.

Question 4

In a criterion-referenced course, the proportion of higher grades is found to increase from a previous semester. What three reasons could account for this?

Answer 4

The students got smarter, the assessment was easier than previous, or the teaching of the course was better.

Question 5

What is classical test theory?

Answer 5

Classical test theory is the traditional conceptual basis of psychometrics – it is the idea that every measurement we take can be decomposed into two parts: true score (underlying thing trying to be measured) and measurement error.

Question 6

What is true score theory?

Answer 6

Another name for classical test theory

Question 7

What is reliability in terms of the relationship between true and total test score variability?It is a type of test construction.

Answer 7

Reliability is the ratio between the true variability and the total variability.

Question 8

List the various sources of measurement error.

Answer 8

Test construction, test administration, test scoring and then other influences.

Question 9

What is item sampling?

Answer 9

Item sampling is where a small selection of questions are asked out of a great range of possibilities. It is a type of test construction.

Question 10

What is content sampling?

Answer 10

A type of test construction – can be a source of measurement error.

Question 11

What is domain sampling?

Answer 11

A type of test construction – can be a source of measurement error.

Question 12

Why can we only estimate the reliability of a test and not measure it directly?

Answer 12

Reliability is based on the hypothetical construct of a true score, which in turn means that we are never able to say for certain what the reliability is – we can only estimate it.

Question 13

Describe four methods available to us to help estimate the reliability of a test.

Answer 13

Estimating the internal consistency, test-retest reliability, parallel-forms reliability and inter-rater reliability.

Question 14

What is internal consistency?

Answer 14

How much the item scores in a test correlate with one another on average (the average correlation between the items on a scale).

Question 15

If someone describes a psychological scale as having “high internal coherence”, what are they talking about?

Answer 15

They are saying that the psychological scale has high internal consistency – there is a high average correlation between the items on the scale.

Question 16

What is inter-item consistency?

Answer 16

An alternative label for internal consistency.

Question 17

How do you calculate Cronbach’s alpha using JAMOVI?

Answer 17

Select FACTOR then RELIABILITY ANALYSIS. Then, select all the items in scale and move them to the ‘items’ box.

Question 18

Describe the steps involved in calculating Cronbach’s alpha by hand.

Answer 18

The questionnaire is split in half. The total score for each half is calculated. Then the correlation between the total scores for each half is worked out. Steps 1-3 is then repeated for all possible two way splits of the questionnaire. The average of all possible split half correlations is calculated. Finally, the correlation is adjusted to account for the fact that the test has been shortened by applying the Spearman-Brown formula as a correction.

Question 19

What is test-retest reliability?

Answer 19

The correlation between scores on the same test by the same people sone at two different times.

Question 20

What statistic can we use to evaluate test-retest reliability?

Answer 20

r – correlation

Question 21

Test-retest reliability involving giving the same test twice. Give an example of a situation where giving the same test with the same items twice might be a problem.

Answer 21

In a situation where the learning effect is likely/possible – particularly tests which are tests of competency.

Question 22

What is parallel forms reliability?

Answer 22

The correlation between scores on two versions of the same test by the same people done at the same time.

Question 23

What statistic can we use to evaluate parallel forms reliability?

Answer 23

Pearsons r

Question 24

Imagine you create an ability test which involves an examiner making a number of ratings of an individual’s thumb-rolling skill. You test the inter-rater reliability using two examiners and obtain a correlation of 0.87. What does this mean?

Answer 24

The correlation between each examiners is good, however the means and SDs would need to be checked for similarity.

Question 25

For inter-rater reliability, when might we want to examine the means and SD of two raters’ ratings in addition to the correlation between them?

Answer 25

If it is a criterion referenced test.

Question 26

List five issues/situations that might affect which reliability estimate you can use. Give an example of each.

Answer 26

Homogeneity/heterogeneity of the test, static v dynamic characteristics, restriction of range/variance, speed tests v power tests, and criterion-referenced tests.

Question 27

What is a homogeneous test and a heterogeneous test?

Answer 27

A homogeneous test is where all the test items measure the same construct. A heterogeneous test is a test which has subscales which measure different constructs.

Question 28

If you had a heterogeneous test, which estimate of reliability should you AVOID?

Answer 28

Internal consistency – although you could look at the internal consistency of each subscale separately.

Question 29

If you had a test measuring a dynamic characteristic, which estimate of reliability should you AVOID?

Answer 29

Test-retest reliability

Question 30

If you had a dataset of test scores where the range of scores was substantially restricted, which estimate of reliability would be affected (assuming the data could in principle yield statistics on all reliability estimates)?

Answer 30

As the correlation would be affected, all reliability estimates would then be affected.

Question 31

If you had a speed test (as opposed to a power test), which estimate(s) of reliability should you AVOID?

Answer 31

Measuring internal consistency

Question 32

What special considerations do we need to take into account when analysing the psychometric properties of a speed test compared with a power test?

Answer 32

Not all questions will have been answered – which can end up in a spurious correlation if internal consistency is not avoided.

Question 33

What possible feature of some criterion-referenced tests might create problems for estimating reliability?

Answer 33

There may be very little variation in responses – due this restriction of range using any reliability estimates can be problematic.

Question 34

What’s the relationship between reliability and the number of items in a test?

Answer 34

Reliability tends to increase when you have more items and decrease when you have less items.

Question 35

Why should having more items in your test lead to better reliability?

Answer 35

It is the effect of domain sampling, with more items there are more samples of the domains of interest, and the test score then becomes a better representation of the total domain score.

Question 36

How can the Law of Large Numbers apply when trying to predict the behaviour of an individual?

Answer 36

A large and diverse sample of evidence of past behaviour is going to better predict future behaviour than sample consisting of a singular past behavioural event.

Question 37

When might you want to use the Spearman-Brown formula?

Answer 37

To estimate how reliability would change if a test was shortened or lengthened.

Question 38

Imagine you had a questionnaire with 20 items and you were disappointed that its internal consistency was .59. What effect would adding another 30 equivalent items be predicted to have on the reliability?

Answer 38

It would likely increase the reliability

Question 39

True or False: A test designed to yield information about whether or not a student has mastered the ability to multiply two-digit numbers to a specified level of competency could be described as norm-referenced.

Answer 39

False

Question 40

True or False: If an examination is criterion-referenced, then an individual's score will not be affected if the norm for the course changes.

Answer 40

True

Question 41

True or False: If an examination is norm-referenced, then an individual’s score will be less likely to be affected by changes in the quality of teaching

Answer 41

True

Question 42

True or False: If an examination is criterion-referenced, then marks are protected from changes in examination difficulty.

Answer 42

False

Question 43

True or False: If the proportion of higher grades changes substantially between semesters for a course, this probably means the course scoring is criterion-referenced.

Answer 43

True

Question 44

True or False: When I convert your percentage score to your grade score at the end of this course, this is an example of a linear transformation.

Answer 44

False

Question 45

True or False: True Score Theory involves conceptualising test score variability as comprising true test score variation and total test score variation.

Answer 45

False

Question 46

True or False: Classical Test Theory can be described by the formula X = T / E (where X is the observed score, T is the true score, and E is measurement error).

Answer 46

False

Question 47

True or False: According to Classical Test Theory, if a test has very high reliability then the measurement error must be very high.

Answer 47

False

Question 48

True or False: Reliability conceptualised in Classical Test Theory as being total test score variation minus measurement error.

Answer 48

False

Question 49

True or False: According to Classical Test Theory, if a test has very high reliability then the measurement error must be very low.

Answer 49

True

Question 50

True or False: If a test was unreliable then the true test score variability would only be a small proportion of the actual test score variability.

Answer 50

True

Question 51

True or False: The fact that we cannot ask students everything about the course in the quizzes will decrease the proportion of the observed score that can be attributed to the true score (assuming the quiz marks are supposed to reflect students’ overall PSYC3020 knowledge).

Answer 51

True

Question 52

True or False: As part of the process of calculating Cronbach’s alpha, you have to multiply the correlations derived from all possible ways of splitting the test into two.

Answer 52

False

Question 53

True or False: As part of the process of calculating Cronbach’s alpha, you have to average the correlations derived from all possible ways of splitting the test into two.

Answer 53

True

Question 54

True or False: As part of the process of calculating Cronbach’s alpha, you have to adjust for the fact you’ve halved the test by applying the Spearman-Brown formula.

Answer 54

True

Question 55

True or False: As part of the process of calculating Cronbach’s alpha, you have to adjust for the homogeneity of the test by applying the Spearman-Brown formula.

Answer 55

False

Question 56

True or False: Imagine you create an ability test which involves an examiner making a number of ratings of an individual’s acrobatic skill. You test the inter-rater reliability using two examiners and obtain a reliability of -.98 (negative point nine eight). This means one examiner is giving the opposite rating to the other.

Answer 56

True

Question 57

True or False: Imagine you create an ability test which involves an examiner making a number of ratings of an individual’s acrobatic skill. You test the inter-rater reliability using two examiners and obtain a reliability of .88 (point nine eight). This means one examiner’s rating is independent of the other’s rating.

Answer 57

False - they are related to one another (high correlation)

Question 58

True or False: If we double the number of items in a test, then the Spearman Brown formula predicts that we should double its reliability.

Answer 58

False

Question 59

True or False: 0.70 is usually considered the typical minimum threshold for decent reliability.

Answer 59

True

Question 60

True or False: 0.90 is usually considered the typical minimum threshold for decent reliability

Answer 60

False