Chapter 6 - Validity

Question 1

What is validity?

Answer 1

A judgment or estimate of how well a test measures what it’s supposed to measure in a particular context

Question 2

What is the relationship between validity and reliability?

Answer 2

Reliability is required by not sufficient for validity

Question 3

What is validation? Who plays a role in it?

Answer 3

The process of gathering and evaluating evidence about validity
This can be done by both test developers and test users

Question 4

What is local validation?

Answer 4

When test users aim to determine the validity of a test within their own local settings or conditions, using their own group of test takers

Question 5

What are the 3 main categories of validity (from easiest to hardest to establish)?

Answer 5

Content –> Criterion-Related –> Construct

Question 6

What is Content Validity?

Answer 6

How well a test samples behaviors that are representative of the broader set of behaviors that it’s designed to measure

In other words, it measures how well test items/topics adequately represent the content that should be included based on the operational definition being used

Question 7

What is Face Validity?

Answer 7

A form of content validity, it is a judgment concerning how relevant the test items appear to be on the face of it
This is the simplest form of validity to prove, but some tests are intentionally designed to have low levels of it

Question 8

What is a Test Blueprint?

Answer 8

Part of the process of creating content validity, it is a plan regarding the types of information covered by the items, the number of items tapping into each area of coverage, and the organization of the items in the test

Question 9

How do we typically establish content validity?

Answer 9

Expert panels: obtain expert ratings on the degree of item importance and scrutinize what is missing from the measure
Focus Groups: having the general population react to the measure

Question 10

What is Criterion-Related Validity?

Answer 10

Evaluates the relationship between scores obtained on one test and scores obtained on other tests or measures

Question 11

What is a criterion? What does it need in order to be adequate?

Answer 11

A standard against which a test or test score is evaluated
Must be…
1- relevant to the matter at hand
2- valid for the purpose that it’s being used
3- uncontaminated, as in it cannot be a part of the predictor

Question 12

What ways can we establish criterion-related validity (in order from easiest to hardest to establish)?

Answer 12

1 - Concurrent validity
2 - Predictive validity
3 - Incremental validity

Question 13

What is Concurrent Validity?

Answer 13

The degree to which a test score is related to some criterion measure obtained at the SAME time

Question 14

What is Predictive Validity?

Answer 14

The degree to which a test score predicts some criterion measure (or outcome) obtained at a FUTURE time

Question 15

What is a Base Rate? How does it influence predictive validity?

Answer 15

The extent to which a phenomenon exists in the population
The less frequent it is, the more difficult it would be to show predictive validity

Question 16

What is the Hit Rate when establishing predictive validity? What are the two kinds?

Answer 16

The ability of the measure to accurately predict results

Two possibilities…
1- true positive
2- true negative

Question 17

What is the Miss Rate when establishing predictive validity? What are the two kinds?

Answer 17

Failure to identify something accurately

Two possibilities…
1- False positive or Type I error: saying that something will happen and then it does not
2- False negative or Type II error: saying that something will not happen and then it does

Question 18

What is the Validity Coefficient? What is it affected by?

Answer 18

A correlation coefficient between test scores and scores on the criterion measure
Affected by restriction or inflation of range

Question 19

What is Incremental Validity?

Answer 19

The degree to which an additional predictor explains something about the criterion measures that is not explained by predictors already being used
Essentially saying “this test predicts the criterion better than other tests”

Question 20

What is Construct Validity? How do we acquire evidence for it?

Answer 20

The ability of a test to measure a theorized construct. Essentially, does the measure map onto the THEORY the way we would expect it to (as in, do high scorers and low scorers behave as theorized?)

Establishing content and criterion-related validity will also provide evidence for construct validity, but it requires a little bit more than just that

Question 21

What are the different forms of evidence for construct validity? (7 things)

Answer 21

1- evidence of homogeneity
2- evidence of changes
3- evidence of retest changes
4- evidence from distinct groups
5- convergent evidence
6- discriminant evidence
7- factor analysis

Question 22

What is evidence of homogeneity?

Answer 22

How uniform a test is in measuring a single construct (established using evidence from internal reliability)

Ex: if I believe that my construct is narrow, then my internal consistency should be high

Question 23

What is evidence of changes?

Answer 23

How the construct changes over time in the way it’s expected to, established using evidence from test-retest reliability

Question 24

What is evidence of posttest or retest changes?

Answer 24

Test scores change as a result of some kind of experience or intervention between pretest and posttest, established using evidence from dynamic assessment

Question 25

What is evidence from distinct groups?

Answer 25

Scores on the test vary in a predictable way based on membership in some group

Question 26

What is Convergent Evidence? How is it distinct from concurrent validity?

Answer 26

Scores from a test correlate highly in the predicted direction with scores on older, more established tests designed to measure the same or a similar construct This is similar to concurrent validity, except we're looking at correlations with other measures rather than with criterion in general

Question 27

What is Discriminant Evidence?

Answer 27

Scores from a test have little to no relationship with variables that it should NOT be correlated with Ex: in most cases, test scores should NOT be correlated with measures of social desirability

Question 28

What is Factor Analysis?

Answer 28

A family of statistical tests that looks at how items are correlated with one another and develop into factors or groupings A test should only measure ONE common factor unless it's an inventory or intentionally divided into different domains

Question 29

What is Bias?

Answer 29

A factor inherent in a test that systematically prevents accurate, impartial measurement

Question 30

What is Rating Error?

Answer 30

A judgment resulting from the intentional or unintentional misuse of a rating scale Raters may be either too lenient, too severe, or reluctant to give ratings at either extreme (i.e. central tendency error)

Question 31

What is the halo effect?

Answer 31

The tendency to give a particular person a higher rating than he or she objectively deserves because of an overall favorable impression of the person

Question 32

What is fairness?

Answer 32

The extent to which a test is used in an impartial, just, and equitable way

Question 33

Can a test be unbiased and still be unfair? Why?

Answer 33

Unbiased tests can still be unfair due to random error or aspects of the test's administration/application (rather than aspects of the test itself)