Validity

Question 1

what questions does validity answer

Answer 1

-is the test or measure meaningful?
-does it measure what it was supposed to measure?

Question 2

reliable vs valid on dart board

Answer 2

-reliable: not on the bullseye but all darts are in the same place
-valid (+ technically reliable): all darts are on the bullseye, + also in the same place

Question 3

types of research design validity

Answer 3

-internal validity
-external validity

Question 4

internal validity

Answer 4

basically protocols that have to be followed
* high rigor
* high repeatability
* highly controlled
* low risk of bias

Question 5

external validity

Answer 5

generalizability
* appliability
* transferability
* less controlled

Question 6

measurement validity

Answer 6

accuracy
-does the instrument measure what it is supposed to?

Question 7

face validity

Answer 7

subjective assessment of degree to which a test appears to measure what it is intended to
* most basic test of validity, not many papers use just this type anymore
* lacks operational definitions + comparison to reference standard
* does it pass the “eye test”?
* RED FLAG if something doesn’t pass this simple form of validity

Question 8

red flag if something doesn’t pass which type of validity

Answer 8

face validity
-most basic test of validity

Question 9

content validity

Answer 9

degree to which test items represent all relevant facets of the variable being measured
* theoretical concept, based on subjective opinion, so it can be difficult to measure directly (not as applicable)
* usually determined by a group of experts

Question 10

construct validity

Answer 10

different from content validity because it is more SPECIFIC + emphasizes an operational definition

Question 11

how do you assess construct validity?

Answer 11

-statistical analysis (simplest)- compare means
-analysis (simple)- correlation
-analysis (complex)- factor analysis

Question 12

how to assess construct validity-
STATISTICAL ANALYSIS

Answer 12

-simplest
-compare means
-comparison of means via t-tests (p-value)
* should expect no differences between “like” groups (p is greater than 0.05)
* should expect significant differences between “unlike” groups (p is less than 0.05)

Question 13

how to assess construct validity-
ANALYSIS (simple)

Answer 13

correlation
* convergent: strong association between “like” items or groups
* divergent (discriminant): weak or no association between “unlike” items or groups
* best when single measure exhibits both convergent + divergent

Question 14

how to assess construct validity-
ANALYSIS (complex)

Answer 14

factor analysis
* used to identify which items on a questionnaire are highly correlated
* convergent items are grouped together (loaded), while divergent factors stay separate from one another (ex: function + symptoms factors contain different items)
* cross-loading is a problem where one item on a questionnaire loads onto multiple factors, so we don’t know what is actually being measured

Question 15

cross-loading

Answer 15

a problem where one item on a questionnaire loads onto multiple factors, so we don’t know what is actually being measured

Question 16

criterion-related validity

Answer 16

degree to which a measure of interest relates to a measure with established validity

Question 17

2 types of criterion-related validity

Answer 17

-concurrent validity
-predictive validity

Question 18

concurrent validity

Answer 18

comparing 2 measures that are taken at the same time

Question 19

predictive validity

Answer 19

reflects the degree to which the results of a specific test can predict a future outcome

Question 19

responsiveness

Answer 19

ability of a measure to detect change; clinical meaningfulness

Question 20

example of responsiveness

Answer 20

-scores for a certain measure decrease for those that worsen
-scores for that same measure increase for those that improve

Question 21

what 2 things are clinically practical

Answer 21

-criterion-related validity
-responsiveness

Question 22

quantifying validity-
we normally report “r” for

Answer 22

association (relationship)

Question 23

general guideline for validity with “r”

Answer 23

it is valid if r = 0.35 or greater

Question 24

quantifying validity- it is common to do what to "r" for validity

Answer 24

convert to r^2 for validity (shared variance)

Question 25

general guideline for validity with "r^2"

Answer 25

it is valid if r^2 = 10% or more

Question 26

in simple correlation/regression, describe r and r^2

Answer 26

r and r^2 are EQUIVALENT

Question 27

in multiple regression, describe r and r^2

Answer 27

r and r^2 are NOT equivalent

Question 28

quantifying validity- comparing means effect size

Answer 28

* between groups- construct validity * change over time- responsiveness

Question 29

between groups =

Answer 29

construct validity

Question 30

change over time =

Answer 30

responsiveness

Question 31

factor analysis

Answer 31

name factors based on items that load on factor * basically running an analysis to see if all of the items on one questionnaire load onto the same factor * ex: all the items on PCS load onto Pain Catastrophizing factor

Question 32

can you have validity without reliability

Answer 32

NO- you cannot have validity without some degree of reliability

Question 33

something that is reliable is ____

Answer 33

consistent -but not necessarily accurate

Question 34

cautions of validity

Answer 34

* even a well-designed tool could produce invalid data if it is used incorrectly * criterion-related validity + responsiveness are objective, but they are only as strong as the comparison, or gold standard, that is used * no cut-off for a coefficient to determine if it is "valid"; it's more of a spectrum * validity is multidimensional; there are multiple components going into validity

Question 35

can a well-designed tool produce invalid data

Answer 35

yes- if used incorrectly

Question 36

criterion-related validity + responsiveness are subjective/objective

Answer 36

objective -but they are only as strong as the comparison, or gold standard, that is used

Question 37

is there a cut-off for a coefficient to determine if it is valid

Answer 37

no -it's more of a spectrum

Question 38

validity is ____

Answer 38

multidimensional -there are multiple components going into validity