Reliability, Validity, Utility

Question 1

TRUE OR FALSE?
Psychological assessments are only useful if the tests we use are consistent, accurate, and practical.

Answer 1

TRUE.

Question 2

THREE MAJOR CONCEPTS
Consistency

Answer 2

Reliability

Question 3

THREE MAJOR CONCEPTS
Accuracy

Answer 3

Validity

Question 4

THREE MAJOR CONCEPTS
Practical usefulness

Answer 4

Utility

Question 5

Consistency of measurement; the degree to which test scores are stable, dependable, and free from random error.

Answer 5

Reliability

Question 6

Key Idea: If I measure the same thing again, will I get the same result?

Answer 6

Reliability

Question 7

TYPES OF RELIABILITY
Same test given at two different times; should produce similar scores.

Example: Taking an IQ test in January and again in February.

Answer 7

Test-Retest Reliability

Question 8

TYPES OF RELIABILITY
Agreement between different scorers/observers.

Example: Two clinicians rating the same patient’s behavior.

Answer 8

Inter-Rater Reliability

Question 9

TYPES OF RELIABILITY
Two different but equivalent versions of a test; should give similar results.

Example: Version A and Version B of an exam.

Answer 9

Parallel-Forms Reliability

Question 10

TYPES OF RELIABILITY
Consistency of items within the same test; measured by Cronbach’s alpha.

Example: All items on a depression scale should relate to depression.

Answer 10

Internal Consistency

Question 11

Accuracy of measurement; the degree to which a test measures what it is supposed to measure.

Answer 11

Validity

Question 12

Key Idea: Am I measuring the right thing?

Answer 12

Validity

Question 13

TYPES OF VALIDITY
Does the test cover all relevant aspects of the concept?

Example: A math test that only asks about addition has poor content validity.

Answer 13

Content Validity

Question 14

TYPES OF VALIDITY
How well a test predicts performance on an external standard.

Answer 14

Criterion-Related Validity

Question 15

TYPES OF VALIDITY | CRITERION-RELATED VALIDITY
Measured at the same time (e.g., depression test vs. clinical diagnosis).

Answer 15

Concurrent Validity

Question 16

TYPES OF VALIDITY | CRITERION-RELATED VALIDITY
Predicts future performance (e.g., SAT predicting college GPA).

Answer 16

Predictive Validity

Question 17

TYPES OF VALIDITY
Does the test really measure this theoretical concept?

Example: Does an anxiety scale truly capture anxiety and not just stress or shyness?

Answer 17

Construct Validity

Question 18

TYPES OF VALIDITY | CONSTRUCT VALIDITY
Test correlates with similar measures.

Answer 18

Convergent Validity

Question 19

TYPES OF VALIDITY | CONSTRUCT VALIDITY
Test does not correlate with unrelated measures.

Answer 19

Discriminant Validity

Question 20

Practical value of the test; practicality. The usefulness of a test in a real-world setting, considering both benefits and costs.

Answer 20

Utility

Question 21

Key Idea: Is this test worth using?

Answer 21

Utility

Question 22

FACTORS AFFECTING UTILITY
A test that isn’t consistent or accurate won’t be useful.

Answer 22

Reliability & Validity

Question 23

FACTORS AFFECTING UTILITY
Time, money, effort vs. value gained.

Answer 23

Costs vs. Benefits

Question 24

FACTORS AFFECTING UTILITY
Accessible and fair across groups.

Answer 24

Fairness

Question 25

**FACTORS AFFECTING UTILITY** Easy to administer, score, and interpret.

Answer 25

Practicality

Question 26

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* An IQ test that gives widely different scores to the same student each time won't be useful.

Answer 26

Reliability

Question 27

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A "creativity test" that actually measures vocabulary is not useful, even if it's consistent.

Answer 27

Validity

Question 28

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A company considers a complex personality test for hiring. It costs P5,000 per applicant but only slightly improves hiring decisions. The cost outweighs the benefit, so the test has low utility.

Answer 28

Costs vs. Benefits

Question 29

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A short, free online survey that predicts job performance almost as well as the expensive one.

Answer 29

Costs vs. Benefits

Question 30

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A math placement test written only in English may disadvantage students who are proficient in math but not fluent in English.

Answer 30

Fairness

Question 31

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A well-designed nonverbal reasoning test avoids language bias, making it fairer and more useful.

Answer 31

Fairness

Question 32

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A test requiring expensive equipment, three hours of administration, and advanced statistical scoring is impractical for most schools.

Answer 32

Practicality

Question 33

**FACTORS AFFECTING UTILITY** *Identify whether this statement relates to Reliability/Validity, Costs vs. Benefits, Fairness, or Practicality:* A 30-minute test with easy-to-score answer sheets is more practical and therefore more useful.

Answer 33

Practicality

Question 34

**TRUE OR FALSE?** A test must be reliable to be useful, but reliability alone is not enough. A test must also be valid to ensure it measures what it claims. Finally, a test must have utility to be practical in real-word settings. *Together, these three concepts determine whether a psychological test is worth using in practice.*

Answer 34

**TRUE.**

Question 35

**OTHER KEY CONCEPTS IN PSYCHOLOGICAL ASSESSMENT** The process of giving a test the same way every time to ensure fairness; includes norms (what's "average" for a group). Example: IQ tests are standardized on large populations to know what an "average" score is.

Answer 35

Standardization

Question 36

**OTHER KEY CONCEPTS IN PSYCHOLOGICAL ASSESSMENT** Reference points to interpret a score.

Answer 36

Norms

Question 37

**TYPES OF NORMS** Compare with same-age peers.

Answer 37

Age Norms

Question 38

**TYPES OF NORMS** Compare with students in the same grade.

Answer 38

Grade Norms

Question 39

**TYPES OF NORMS** Tells the % of people who scored lower.

Answer 39

Percentile Ranks

Question 40

**OTHER KEY CONCEPTS IN PSYCHOLOGICAL ASSESSMENT** No test is perfect; there's always an error. Errors may come from the test itself, the environment, or the test-taker (mood, fatigue). *Observed Score = True Score + Error*

Answer 40

Errors in Measurement

Question 41

**ERRORS IN MEASUREMENT** The score you actually get on a test (e.g., 85/100).

Answer 41

Observed Score

Question 42

**ERRORS IN MEASUREMENT** Your real level of the trait being measured (e.g., your actual intelligence, ability, or depression level).

Answer 42

True Score

Question 43

**ERRORS IN MEASUREMENT** Everything else that affects your score but is not part of the true ability (e.g., guessing, bad instructions, being tired, noise in the testing room).

Answer 43

Error

Question 44

**OTHER KEY CONCEPTS IN PSYCHOLOGICAL ASSESSMENT** Examining test questions to see if they are good measures; includes difficulty index (easy vs. hard items) and discrimination index (does the item distinguish between high and low scorers?).

Answer 44

Item Analysis

Question 45

**ITEM ANALYSIS** Easy vs. hard items.

Answer 45

Difficulty Index

Question 46

**ITEM ANALYSIS** Does the item distinguish between high and low scores?

Answer 46

Discrimination Index

Question 47

**OTHER KEY CONCEPTS IN PSYCHOLOGICAL ASSESSMENT** A test must not be unfair to groups based on culture, language, gender, or socioeconomic status. Example: A math test with word problems written in complex English may disadvantage non-native speakers.

Answer 47

Fairness and Bias

Question 48

**OTHER KEY CONCEPTS IN PSYCHOLOGICAL ASSESSMENT** Informed consent, confidentiality, appropriate use, test security. Tests are powerful tools; misuse can harm people.

Answer 48

Ethical Issues in Testing