Week 1

Question 1

System Validation

Answer 1

Ensures that measurement tools provide accurate and reliable data

Question 2

Concurrent Validation

Answer 2

Assessment of a new system compared to an existing system while recording simultaneously

Question 3

Accuracy

Answer 3

Ability to measure a quantity close to its true value approximated with ‘gold-standard’ system

Question 4

Reliability

Answer 4

The extent to which measurements can be replicated; consistency or reproducibility

Question 5

Precision

Answer 5

Spread of repeated measurements

Question 6

Measuring accuracy

Answer 6

Agreement and Correlation
- Agreement is the more appropriate estimate of accuracy

Question 7

Agreement

Answer 7

How closely the measurements of two systems match each other across a range of values. Assesses if measurement of the same variable by two different systems produce similar results

Question 8

Pearson Correlation

Answer 8

Measures how strong pairs of variables are related and only interpreted if p-value is significant

Question 9

How is pearson correlation a misleading measurement of agreement

Answer 9

• Only measures the linear relationship of two sets of observations
• Does not measure bias or systematic difference between systems
• Can consistently be off by n number of units and still show a strong correlation

Question 10

Bland-Altman Analysis

Answer 10

Analysis with 95% limits of agreement (LOA) measures the agreement between two measurement systems instead of comparing the new system to a perfect system. This provides interchangeability between the reference and the new system.

Question 11

how to calculate 95% LOA

Answer 11

95% LOA = bias +/- 1.96 x SD of the differences

Question 12

Challenges with BA analysis

Answer 12

• N decisive p-value
• No standard for acceptable LOA
• Researchers must establish a priori acceptability limits based on clinical relevance or prior studies to define the boundaries

Question 13

Priori acceptability limits

Answer 13

• Determined based on an expected difference between healthy and unhealthy populations prior to study
• The comparison between health and unhealthy populations provides some level of quantification of the acceptability criteria used for future use case of the new measurement system
• If the calculated LOA are narrower than the a priori acceptability criteria, then we can conclude acceptable agreement

Question 14

BA Analysis Steps

Answer 14

1. Select a priori acceptability criteria
2. Calculate the difference between system A and system B
3. Calculate the mean of system A and system B for each trial
4. Calculate the mean difference (Bias)
5. Calculate the 95% limits of agreement (LOA)
6. Plot the mean (x-axis) against the difference (y-axis)
7. plot the calculated LOA
8. Evaluate:
   - Bias
   - Calculated LOA vs acceptability criteria

Question 15

Reliability Theory

Answer 15

• All measurements are made with some degree of error
• Any time a score is recorded the observed score is the sum of the true score and error

Question 16

Forms of measurement Error

Answer 16

RANDOM ERROR: noise or unpredictable error - averages out to zero over time/ can be mitigated by multiple measurements
SYSTEMIC ERROR: Scores trend up or down over multiple measurements - directional and usually solved by simple addition or subtraction

Question 17

Test re-test reliability

Answer 17

• To assess if scores are similar overtime
• The measurement is administered to a sample of participants and then repeated at least once at some other time

Question 18

reliability equation

Answer 18

reliability = true variance/ (true variance + error variance)

Question 19

Intraclass correlation coefficient

Answer 19

Quantifies reliability
- calculates an index that comprises the same variable measured on multiple occasions; withing a group/class

Question 20

Interclass correlation coefficient

Answer 20

Correlation which assess if two variables from different classes are correlated

Question 21

What are the 3 models of ICC

Answer 21

Model 1: Some participants are measured by different raters
Model 2: Participants are measured by the same raters, but can generalize the reliability to other raters of the same type
Model 3: Participants are measured by the same set of raters, but the raters are the only raters of interest

Question 22

What are the two types of ICCs

Answer 22

Type 1: single measurement
Type k: mean of k measurement or trials

Question 23

Types of challenges with test re-test reliability

Answer 23

Participant factors
Equipment and measurement factors
Environment and protocol factors

Question 24

Participant factors of test re-test reliability

Answer 24

BIOLOGICAL VARIABILITY
- Natural fluctuations in trial-to-trial movement patterns
- Population considerations, health/skill status
- Solution: Repeated trails, increased sample size, warm-up, inclusion/exclusions criteria
LEARNING EFFECT
- Task adaptation through repeated trials
- Solution: Familiarization trials, randomization of conditions
FATIGUE, RECOVERY, AND PSYCHOLOGICAL:
- cumulative fatigue, motivation/boredom
- solution: spacing of trials, adequate rest between sessions, clear instructions

Question 25

Equipment and measurement factors of test re-test reliability

Answer 25

INTRUMENT PLACEMENT CONSISTENCY - small misalignment of sensors results in different outputs - Solution: use anatomical landmarks, same personnel for set-up, mark placements SYSTEM CALIBRATION SIGNAL DIFT - Incorrect calibration or discounting sensor drift results in different outputs - Solution: pre-collection calibration, drift measurements SAMPLING RATE - low sampling rate may miss movement patterns or events of interest in the signal - Solution: consult literature, pilot collections

Question 26

Environment and protocol factors in test re-test reliability

Answer 26

TESTING ENVIRONMENT - Surface variability, attire, room temp, electrical interference - Solution: consistent environmental conditions and attire, standardized protocol, checklist TIMING - long intervals between testing can introduce physiological changes (e.g. strength, gains, injury) - Solution: minimize spacing between testing, consistent time of day