Reliability is based on…
the porportion of the total observed variance that is attributed to error
Total variation needs to have a certain amount of spread in order to get proper reliability.
____ is crucial for reliabilty!
Variance
Wide range of scores is important
Where does variability come from?
- Examiner
- Subject
- Instrumentation
We want ____ for correlation and reliability
spread
If subjects are homogenous it will result in…
the reliability coefficent to be low because the total variance is small
Reliability formula
Observed Score = True Score ± error score
Variance formula
s2 (score) = s2 (true) + s2 (error)
A measurement is more reliable if a greater proportion of the total observed variance is _____
represented by the true score variance.
What does a reliability of 1 mean?
Measurement has no error and therefore reflects the true score.
Reliability Coefficent Range
- Range 0-1
- 0 = all meausre due to error
- 1 = true score, no error
Why not use a correlation coefficent?
- Correlation do not reflect similarity of repeated scores. Correlations reflect covariance (rank order characteristics within the data) thus not consistency of ranks. Ex: Photo: two different lines but the correlations will be the same because of linear analysis.
- Product moment correlation designed to assess bivariate relationships (only two can be assessed)
Advantages of an ICC
- Can be used to assess reliability of >2 ratings.
- Does not require the same number of raters for each subject (however, better to have all data). Leave box blank if you do not have data.
- Can be used on ordinal data.
- Supports generalizability theory (different testing occasions, characteristics of the raters/subjects, testing conditions and others).
What model and settings do we use to analyze ICC on SPSS?
Model 2
Two-way random effects model - Absolute Agreement
When show I use single vs mean scores for an ICC?
- Single: Most common, for individual raters or ratings are used. Intraclass coefficent
- Mean: measurments are unstable for several raters or ratings. interclass coefficent
3 Data trials on 2 different days
EX: Single: Take the 3 trials on one day and do an ICC
Ex: Mean: Get mean for each day then do an ICC.
When examining a SPSS ICC Model 2 chart what do we need to look at specifically?
Only the Intraclass coefficent for the measure you analyzed. Single or Mean measures.
An ICC of ____ is needed for most clinical measures
0.9 and above
What makes an ICC low?
- Raters or ratings do not agree
- Too little variance, sample is too homogenous. Must have a heterogenous sample to avoid this!
What is standard error of measurement?
- Referred to as typical error for a test
- Expected trial to trial difference swhen the true score does not change
The SEM quantifies the ____ on a test
percision of individual scores
Standard Error of Measurement (SEM) is not the same as ____
Standard error of mean (measure of vairability not necessarily percision)
SEM is a function of the ____ value and ____ of the population
- ICC
- variability
The greater the ICC value the smaller the ____ values. Why is this important?
- SEM
The smaller the SEM, the greater the sensitivity of your instrument to detect change. This is the responsiveness of the instrument, the more responsive the better.
Minimum Detectable Change (MDC)
- Smaller MDC = greater responsiveness = good
- Tells you meaningful change. If I am above the value then real change has occured.
- MDC = 2.77 * SEM
____ reliabe instrument -> ____ measurement error -> ____ MDC -> ____ responsiveness -> ____ capable of detecting change (good)
- Highly
- Low
- Small
- High
- More
