Reliability

Question 1

Reliable = consistent ≠ perfect
* Top three most reliable cars according to consumers reports
(2019 based on 10 years of service history)?
* Toyota 4Runner
* Toyota Prius
* Toyota Camry

• Is any measurement perfect?
  …
• What is an attribute that makes a measurement good?
  …

Answer 1

• Is any measurement perfect?
  Probably not…
• What is an attribute that makes a measurement good?

If it’s reliable…

Question 2

• Reliability
• what is it?
• If you measure the same thing, would you…
• Can our measure be confirmed by further measurements or observations?
• Reliability
• … in scores
• Sources of errors
• Participant
• Test administrator
• Testing
• Scoring
• Instrumentation
• Environment
• Systematic vs. Unsystematic Errors
• Measurement error
• Bias
• Confounding variability

Answer 2

• Reliability
• Consistency or stability of measurement
• If you measure the same thing, would you get the same score?
• Can our measure be confirmed by further measurements or observations?
• Reliability
• Consistency in scores
• Sources of errors
• Participant
• Test administrator
• Testing
• Scoring
• Instrumentation
• Environment
• Systematic vs. Unsystematic Errors
• Measurement error
• Bias
• Confounding variability

Question 3

Question 4

Measurement Error
* A measurement error is the difference between…
* They almost always …
* Includes systematic and random error
* The estimate of measurement error is Reliability
* Measured score = …

• Define an operational definition that minimizes error.
• Develop protocol specific enough to objectively measure the concept/variable being studied, then check reliability of measurement and measurer(s)
• Method/instruments used to measure something with sufficient detail/standardization to allow for replication of the results using properly trained evaluators/testers
• Standard of measurement conducted by trained individuals who will be applying a standard set of criteria

Answer 4

Measurement Error
* A measurement error is the difference between an observed value, and an actual value
* They almost always occurs
* Includes systematic and random error
* The estimate of measurement error is Reliability
* Measured score = true score + measurement error

Question 5

Systematic Error vs. Random Error
* Systematic Error (… error)
* A … can result in a systematic error sometimes
* Systematic Errors are:
* …able
* One …
* … true score
* Can correct for or recalibrate

• Random Error (… variable present)
• Due to …
• …able
• Can occur for a variety of reasons
• With many trials, it would eventually …, or cancel out, so average score is a good estimate of the true score
• Can be the result of a … variablity

Answer 5

Systematic Error vs. Random Error
* Systematic Error (bias error)
* A bias can result in a systematic error sometimes
* Systematic Errors are:
* Predictable
* One direction
* Over or underestimate true score
* Can correct for or recalibrate

• Random Error (confounding variable present)
• Due to chance
• Unpredictable
• Can occur for a variety of reasons
• With many trials, it would eventually mitigate, or cancel out, so average score is a good estimate of the true score
• Can be the result of a confounding variablity

Question 6

Inter-rater reliability (…)
* A coefficient that assesses the agreement of
observations made by…
Inter-rater reliability =

Answer 6

Inter-rater reliability (between)
* A coefficient that assesses the agreement of observations made by 2 or more raters

Inter-rater reliability = number of agreements X 100% / number of possible

Question 7

Intra-rater reliability (…)
 The stability of data recorded by …
 Reliability is established with … trials
 This could be a source of … Bias

Answer 7

Intra-rater reliability (within the rater)
 The stability of data recorded by one individual
across two or more trials
 Reliability is established with multiple trials
 This could be a source of Rater Bias

Question 8

Inter-rater reliability (…)
Intra-rater reliability (…)

Answer 8

Evaluation by different “measurers” several times
Evaluation by the same “measurer” several times

Question 9

Test-Retest
* The results of both …
* Reliable over …

After a day/week/month
Change the order of the test procedure

Answer 9

Test-Retest
* The results of both test and retest should be consistent if the test is reliable
* Reliable over time

Question 10

Alternate forms reliability
* Use …
* Reliable over … (…)

Answer 10

Alternate forms reliability
* Use alternate forms of the testing instrument for one
measurement
* Reliable over time (body composition - %body fat)

Question 11

Internal Consistency
* Ideal cake should taste …
* Split half reliability…
* Cronbach’s Alpha (similar to split half but more general)
* The average of all possible split halves
* How closely related a set of items are as a group.
* Considered to be a measure of scale reliability

Internal consistency Split Half example
* Quiz 4 stats: Class average 84.02% (corrected)
* Questions 1-13 class average …%
* Questions 14-26 class average …%
* There was very good split half reliability in our quiz

• .00 to 1.0
• .00 …
• 1.0 … consistency in measurement
• 0.70 means that 70% of the variance is reliable variance
• It also indicates that 30% of the variance is due to error.
• How are items correlated
• 80% of people who got question 1 correct, got question 4 and 5 correct, and q 3 wrong. The correlation between those values would be 0.8 and the error would be 0.2

Answer 11

Internal Consistency
* Ideal cake should taste the same throughout
* Split half reliability. compare scores of each half
* Cronbach’s Alpha (similar to split half but more general)
* The average of all possible split halves
* How closely related a set of items are as a group.
* Considered to be a measure of scale reliability

Internal consistency Split Half example
* Quiz 4 stats: Class average 84.02% (corrected)
* Questions 1-13 class average 84.01%
* Questions 14-26 class average 84.03%
* There was very good split half reliability in our quiz

• .00 to 1.0
• .00 no consistency
• 1.0 perfect consistency in measurement
• 0.70 means that 70% of the variance is reliable variance
• It also indicates that 30% of the variance is due to error.
• How are items correlated
• 80% of people who got question 1 correct, got question 4 and 5
  correct, and q 3 wrong. The correlation between those values would
  be 0.8 and the error would be 0.2

Question 12

TEST
Methods of Establishing Reliability

Determining stability:
* …, … over time
* Same-day …—…

Constructing alternate forms:
* Determining … Standard
* Construct … forms and check reliability to … standard

Obtaining internal consistency:
* … technique

Answer 12

Determining stability:
* Test, Retest over time
* Same-day test—retest

Constructing alternate forms:
* Determining Gold Standard
* Construct alternate forms and check reliability to gold standard

Obtaining internal consistency:
* Split-half technique