Week 5: Validity

Question 1

Why is it not strictly accurate to talk about the validity of a test (hint: one test could be used in more than one context)?

Answer 1

It is not strictly accurate to talk about the validity of a test because test scores are able to be used and interpreted in more than one way, therefore making validity evaluations more about the context in which you are using a test, along with the persuasiveness of the test in the context measuring what it is said it is.

Question 2

What are constructs?

Answer 2

Constructs are unobservable, underlying, hypothetical traits or characteristics that we can try and measure indirectly using tests.

Question 3

What is construct underrepresentation?

Answer 3

The part of variation in the underlying construct being measured that is not captured.

Question 4

What is construct irrelevant variance?

Answer 4

The part of variation that a measure captures that is not related to the construct.

Question 5

What’s the difference between content and face validity?

Answer 5

There is no true difference between content and face validity – the only difference is that content validity is often based on the opinions of experts whereas face validity tends to be based on the opinion of the person taking the test.

Question 6

Give your own example of a testing situation where content validity would be of central importance.

Answer 6

For a competence test – a hazard perception competence test for example

Question 7

How could I create a university examination that had great empirical validity but poor content validity?

Answer 7

By including questions that successfully predict students understanding of the course, without being about the content from the lectures.

Question 8

Why would it not be okay for the PSYC3020 quizzes to have poor content validity (even if they could reliably tell apart high and low scoring students)?

Answer 8

Because it isn’t fair

Question 9

How could you go about evaluating content validity?

Answer 9

By asking a group of experts their opinions as to if the test measures what it is supposed to.

Question 10

List five types of validity.

Answer 10

Content validity, face validity, convergent validity, discriminant validity, and criterion validity

Question 11

What does hypothesis testing have to do with evaluating the validity of a test?

Answer 11

To design an empirical validation study, hypotheses need to be created regarding how measures ought to perform if it is valid.

Question 12

Is reliability necessary for validity?

Answer 12

Yes

Question 13

Is validity necessary for reliability?

Answer 13

No

Question 14

Explain why reliability is necessary for validity but validity is not necessary for reliability.

Answer 14

It is problematic if there is no reliability, as there is nothing stable enough to be valid or not valid, however, reliability is not affected by validity.

Question 15

Give two examples of things that might restrict the range of scores in a test and indicate what influence this could have on the validity coefficient.

Answer 15

Non-random attrition and self selection

Question 16

Give an example of how self-selection during participant recruitment might restrict the range of test scores.

Answer 16

Self-selection may restrict the range of test scores, as there is a possibility for there not to be a distribution which mimics the real world (missing a part of the population)

Question 17

How big does a validity coefficient have to be for a test to be considered valid?

Answer 17

The required magnitude depends entirely on the context – there is no specific cutoff.

Question 18

What is criterion validity?

Answer 18

Criterion validity is the evidence that a test score corresponds to an accurate measure of interest (the criterion).

Question 19

Give your own example of criterion validity.

Answer 19

Giving a new test to 50 surgeons, get them to do it, find real world patient outcomes for each surgeon, and compare the test score to real life.

Question 20

Give three examples of criterion measures that could be used to evaluate the criterion validity of relevant tests.

Answer 20

GPA at end of 1st year to measure university admissions test, supervisors ratings of job performance to measure clerical aptitude, panel rating of creativity displayed in artistic products to measure creative thinking.

Question 21

What is the criterion variable used to evaluate criterion validity?

Answer 21

The criterion is the standard against which the test is evaluated.

Question 22

What is the method of contrasted groups?

Answer 22

An approach to criterion validity, where it is determined if test scores of groups of people vary as expected.

Question 23

What is criterion contamination?

Answer 23

Where the criterion used to assess the validity of a test is pre-determined by the test, thereby undermining the logic of criterion validity.

Question 24

Give an example of criterion contamination when evaluating criterion validity.

Answer 24

Validating a test of schizophrenic people by seeing if it can tell those diagnosed from those not, and then discover that those people have been diagnosed using the same test looking to be validated (circular).

Question 25

Does it matter if a criterion used to validate a test is not that reliable? Why?

Answer 25

Yes – because the reliability limits the size of how big the validity coefficient can be.

Question 26

What’s the relationship between the reliability of the test/criterion and the validity coefficient?

Answer 26

If the reliability of the test is low then there is limited chance for the validity coefficient to be high or even accurate.

Question 27

What is concurrent validity?

Answer 27

Concurrent validity is a subset of criterion validity where test and criterion are measured at the same time.

Question 28

Give an example of concurrent validity.

Answer 28

If the test and criterion are administered in the same session

Question 29

What is predictive validity?

Answer 29

Predictive validity is a subset of criterion validity where the test is trying to predict what the criterion will be at some future time

Question 30

Give an example of predictive validity.

Answer 30

Waiting a period of time between obtaining the test score and criterion score.

Question 31

What is incremental validity?

Answer 31

Incremental validity is how much each individual predictor adds to predicting the criterion in addition to the effect of other predictors – it is a type of criterion validity, and can be predictive or concurrent.

Question 32

Give an example of incremental validity.

Answer 32

Measuring sensation-seeking, fear or heights, and susceptibility to peer pressure when predicting if someone would do a bungee jump.

Question 33

What are the different ways you could go about testing the empirical validity of a test?

Answer 33

Criterion validity, convergent validity, construct validity, discriminant validity, measuring expected developmental changes, experimental effects and internal structure.

Question 34

Give examples of convergent and discriminant validity.

Answer 34

An example of convergent validity is if test scores correlate with other measures of the same (or similar) thing. An example of discriminant validity is if test scores do not correlate very highly with measures that you would expect them not to correlate with.

Question 35

How can we evaluate the validity of a test by inspecting its internal structure?

Answer 35

We can investigate the predictions about whether the internal structure of our test maps onto the internal structure of our construct using factor analysis.

Question 36

What is factor analysis?

Answer 36

Factor analysis uses mathematical techniques to group items into clusters (factors or components) on the basis of how much they correlate with each other.

Question 37

Explain what factor analysis does.

Answer 37

Factor analysis picks out the distinct clusters of items in our data based on their inter-correlations.

Question 38

What is construct validity?

Answer 38

Whether or not the test measures the construct it is meant to.

Question 39

True or False: Construct underrepresentation refers to the variance in a construct that does not covary with test scores.

Answer 39

True

Question 40

True or False: Construct irrelevant variance refers to the variance in a test that does not covary with the construct of interest.

Answer 40

True

Question 41

True or False: If the variances of a test and the construct it is attempting to measure overlap substantially, then the test is likely to have a high degree of validity.

Answer 41

True

Question 42

True or False: If the variances of a test and the construct it is attempting to measure only overlap by a small degree, then the test is likely to have low reliability.

Answer 42

False - it is likely to have low validity

Question 43

True or False: Fifty participants take a new test and then complete a questionnaire in which they rate the perceived validity of the test on a number of quantitatively-scored scales. We can use the rating data obtained from this questionnaire to evaluate the face validity of the test.

Answer 43

True

Question 44

True or False: Face validity is ONLY really useful for persuading people to do the test in the first place.

Answer 44

False

Question 45

True or False: Having good face validity is always an advantage for a test.

Answer 45

False

Question 45

True or False: One useful thing about high face validity is that it might help to persuade people to do the test in the first place.

Answer 45

True

Question 46

True or False: When students complain that, in a course examination, a lecturer did not ask any questions from a particular presentation, they are effectively complaining that the examination has poor content validity.

Answer 46

True

Question 47

True or False: It is possible to have a test with excellent empirically-based support for its validity without having any content validity.

Answer 47

True

Question 48

True or False: One way of evaluating a test’s content validity would be to ask a group of experts to complete a test and then examine their performance on each item for its validity.

Answer 48

False – their opinion hasn’t been asked which is key for content validity

Question 49

True or False: One way of evaluating a test’s content validity would be to ask a group of experts to complete a test and then ask their opinions on whether the test content adequately covered the domain it was designed to cover.

Answer 49

True

Question 50

True or False: If a test is reliable then we can infer that it must also be valid.

Answer 50

False

Question 51

True or False: If we want to calculate a correlation involving a test and a validating measure, then any restrictions of range on the variables could be a problem.

Answer 51

False

Question 52

True or False: If the reliability of both a test and a criterion measure is high, then this means the correlation between them should also be high.

Answer 52

False

Question 53

True or False: Factors that may affect a predictive validity coefficient include certain types of people dropping out of the sample between the original test and when the criterion is measured.

Answer 53

True

Question 54

True or False: You can evaluate the incremental validity of a test by seeing whether it can predict some relevant criterion measure IN ISOLATION from other measures.

Answer 54

False – it is in addition to other measures

Question 55

True or False: Incremental validity refers to the total validity achieved by combining multiple tests to predict a single outcome.

Answer 55

False – referring specifically to what happens when you add an item to test battery, not the overall battery together

Question 56

True or False: A zero correlation between a test score and a relevant validating scale for that test always means that the test is not valid.

Answer 56

False

Question 57

True or False: Examining whether developmental changes occur as expected is one way of evaluating the validity of a test.

Answer 57

True

Question 58

True or False: Examining whether developmental changes occur as expected is one way of evaluating the content validity of a test.

Answer 58

False

Question 59

True or False: If we had an established intervention known to reduce state anxiety, then we potentially could use this to test the validity of a new measure of state anxiety.

Answer 59

True

Question 60

True or False: Relating test scores to the framework of a theoretical construct is one method of evaluating the validity of a test.

Answer 60

True

Question 61

True or False: A factor analysis involves mathematically grouping items according to their intercorrelations.

Answer 61

True

Question 62

True or False: A factor analysis involves mathematically grouping items according to the similarity of the item content.

Answer 62

False

Question 63

True or False: Face validity can be regarded as a type of construct validity.

Answer 63

False

Question 64

True or False: Convergent validity evidence can be used to evaluate construct validity.

Answer 64

True