Reliability and validitiy

Question 1

What is reliability?

Answer 1

• the test measures on and only one thing precisely.

Question 2

How do we measure reliability?

Answer 2

• it has CONSISTENCY ACROSS ITEMS: All items measure the same thing (internal consistency, alternate forms, split-half reliability)
• it has CONSISTENCY ACROSS TIME: The test measures the same thing every time (test-retest)
• it has CONSISTENCY ACROSS OTHER SOURCES: (e.g. raters-inter-rater reliability)
• GENERALIZABILITY (G-theory): looks at all the different sources as part of the same analysis- looks at the amount of inconsistency de to each source of error.

Question 3

What is validity?

Answer 3

• whether the test measures what it is supposed to measure

Question 4

How do we examine validity?

Answer 4

• Evidence from item CONTENT
• Evidence from PROCESS/MANIPULATIONS (E.g. whether the test taker is using heuristics or actually being tested on what the test is trying to measure)
• Evidence from INTERNAL STRUCTURE
• Evidence from RELATIONSHIP TO OTHER VARIABLES: (discriminant- don’t correlate highly with something that the test isn’t measuring, convergent (does correlate highly with something that the test is designed to measure), criterion (concurrent + predictive)
• Evidence from CONSEQUENCES OF TEST USE.

Question 5

What are the test standards?

Answer 5

• recommendations for using and interpreting test scores

- allows a standard of which to go by

Question 6

What is validity?

Answer 6

• validity refers to the extent to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests:

Question 7

What are some way of evaluating validity, according to the Test Standards, 2014?

Answer 7

EVIDENCE (from empirical observations)
THEORY
INTERPRETATION (meaning that the test users derive from them)
USE OF TESTS (outcome to evaluate validity)

Question 8

What was the criterion-based view of validity?

Answer 8

• that a test is valid for anything with which it correlates

Question 9

What is the Tripartate view of validity? (1966 Standards)

Answer 9

• CONTENT VALIDITY
• CRITERION VALIDITY (concurrent & predictive)
• CONSTRUCT VALIDITY (convergent & discriminant)

Question 10

What is the Tripartate + outcomes view of validity?(1985 Standards)

Answer 10

• examines intended or unintended consequences of the test)

Question 11

What is the 1999 Standards view of validity?

Answer 11

• A unitary form of validity, based on evidence from multiple sources to support an argument for what the rest scores actually mean

Question 12

How does the 1999 Standards view of validity examine validity? And what was their aim?

Answer 12

A unitary form of validity, based on evidence from multiple sources to support an argument of what test scores actually mean.

1. Evidence from the CONTENT of the test
2. Evidence from RESPONSE PROCESSES
3. Evidence from the INTERNAL STRUCTURE
4. Evidence from the RELATIONSHIP TO OTHER VARIABLES
5. Evidence regarding the CONSEQUENCES OF TESTING.

Question 13

What is the 1950’s criterion view of validity?

Answer 13

• a test is used to predict an outcome. How well it predicts this outcome is the validity of the test
• conceptualised as a STATIC PROPERTY OF THE MEASURE: a test is either valid or not valid.
• assumes it will always be used for one purpose.

Question 14

What are some problems with the criterion view of validity?

Answer 14

• there is not always one obvious criterion variable (e.g. criterion for test of self-control? aggression?)
• some tests are used for DIFFERENT purposes, in DIFFERENT GROUPS e.g. English- reading comprehension test: valid indicator of 6th grade academic achievement, but poor indicator of intelligence of adult migrants from non-English speaking countries)
• validity is dependent on: test purpose and use and characteristics of the test takers

Question 15

What are the three components of the Tripartate view?

Answer 15

1. Criterion validity:
   - concurrent: criterion measured as same time as the test administered
   - predictive: criterion measured as some time after the test administered
2. content validity: content of the test is both RELEVANT to domain and REPRESENTATIVE of domain
3. Construct validity:
   - convergent: concepts that are THEORETICALLY RELATED demonstrate EMPIRICAL R/S
   - Discriminant: concepts that are THEORETICALLY UNRELATED show NO empirical r/s.

Question 16

What is construct validity based on (Cronbach & Meehl, 1955)?

Answer 16

• idea of a nomological network
  “ the interlocking system of laws which constitute a theory”
• the theoretical framework about what the test should measure
• encompasses things like: correlations with other tests and outcomes, group differences, structure, differences over time, differences under different conditions, training, interventions etc.)
• validity= empirical tests of this theoretical framework

Question 17

What are some problems with the idea of construct validity?

Answer 17

• if a research tests validity and does not find the expected outcomes, is the theory mis-specified or is the test invalid?
• no clear specification of how to test this

Question 18

What are some problems with Tripartate view?

Answer 18

• too much emphasis on different forms of validity (e.g. distinction between convergent and concurrent validity not always clear)
• over-emphasis on CORRELATIONS as proof
• no explicit mention of the test use and consequences

Question 19

What are the different sources of evidence as based on the validity in 1999/2014 standards?

Answer 19

1. CONTENT of the test
2. Response processes captured by the test
3. Internal structure of the test
4. The RELATIONSHIP of the test to other variables:
   - convergent and discriminant evidence
   - test-criterion r/s
   - validity generalisation
   - intended and unintended CONSEQUENCES of testing.

Question 20

How did the 1999/2000 standards add to the definition of validity?

Answer 20

• longer conceptualize different “types” of validity
• validity is a property of the INTERPRETATION of the test
  Evidence that derives from:
• content of the test
• response processes captured by the test
• internal structure
• relationship to other variables
• intended/unintended consequences of the test.