Test Construction

Question 1

Classical test theory

Answer 1

• measurement that is used to develop and evaluate tests
• framework
• assumes that obtain test scores (X) are due to the combination of true variability (T) and measurement error (E)

X=T+E

Question 2

True score variability

Answer 2

• result of actual differences among examinees in regards to whatever the test is measuring
• assumed to be consistent

Question 3

Measurement error

Answer 3

• due to random factors that affect test performance of examinees in unpredictable ways

Example: distractions, ambiguously worded test items, and examinee fatigue

Question 4

Test reliability

Answer 4

• The extent to which a test provides consistent information
• there are several methods for evaluating this and each is appropriate for different circumstances

+ Most methods provide a reliability coefficient

Question 5

Reliability coefficient

Answer 5

• type of correlation coefficient
• ranges from 0 to 1.0
• designated with the letter r with the subscript of two of the same letters or numbers
• always interpreted as the the direct amount of variability obtained in test scores that’s due to true score variability
• .70 or higher is considered the minimally acceptable level, but 90 is usually required for higher stakes tests used to select employees, assign diagnoses, or other important decisions about individuals

Question 6

The acceptable level of reliability

Answer 6

• depends on the type of test and its purpose

Question 7

Standardized cognitive ability tests versus personality test

Answer 7

• cognitive ability test have a higher reliability coefficient

Question 8

Standardized cognitive ability tests versus personality test

Answer 8

• cognitive ability test have a higher reliability coefficient

Question 9

Test retest reliability

Answer 9

• provides information on the consistency of scores over time
• involves administering the test to a sample of examinees and readministering the test to the same examinees at a later time than correlating the two sets of scores
• useful for tests that are designed to measure a characteristic that is stable over time

Question 10

Alternative forms reliability

Answer 10

,- provides information about the consistency of scores over different forms of the test and when the second form is administered at a later time, the consistency of scores over time

• involves administering one forum to a sample of examinees administering the other forms to the same sample of examinees and correlating the sets of scores
• important whenever a test has more than one form

Question 11

Internal consistency reliability

Answer 11

• provides information on the consistency of scores over different test items
• useful for tests that are designed to measure a single content, domain or aspect of behavior

,- not useful for speed tests because it overestimates their reliability

Question 12

Speed test

Answer 12

• test retest reliability and alternative forms reliability are appropriate

Question 13

Coefficient alpha

Answer 13

-aka cronbach’s alpha

• involves administering the test to a sample of examinees and calculating the average inter-item consistency

Question 14

Coefficient alpha

Answer 14

-aka cronbach’s alpha

• involves administering the test to a sample of examinees and calculating the average inter-item consistency

Question 15

Kuder Richardson’s 20( kr-20)

Answer 15

• alternative to coefficient alpha
• can be used when test items are dichotomously scored

Question 16

Split half reliability

Answer 16

• involves administering the test to a sample of examinees splitting the test in half, usually in terms of even and odd numbers and correlating the scores on the two halves

Question 17

Problem with split half reliability

Answer 17

• You’re required to calculate reliability coefficient for two forms of the test that are half as long as the original test and shorter tests tend to be less reliable than longer ones
• It usually underestimates a test’s reliability and is usually corrected with the spearman Brown prophecy formula

Question 18

Inter rater reliability

Answer 18

• important for measures that are subjectively scored
• provides info on the consistency of scores or ratings assigned by different raters

Question 19

Percent agreement and cohens alpha coefficient

Answer 19

• methods for calculating inter rater reliability

Question 20

Percent agreement

Answer 20

• can be calculated on two or more raters
• does not take chance agreement into account which can result in an ever overestimate of reliability

Question 21

Cohen’s kappa coefficient

Answer 21

• aka the kappa statistic
• One of several inter-rator reliability coefficients that is corrected for chance agreement between raters
• used to assess the consistency of ratings assigned by two raters when the ratings represent a nominal scale

Question 22

Factors that affect Reliability of subjective ratings

Answer 22

• can be affected by consensual observer drift

Question 23

Consensual observer drift

Answer 23

• occurs when two or more raters communicate to each other while assigning ratings which results in an increased consistency but decreased accuracy in ratings and an overestimate of iterator reliability

Question 24

Ways to reduce consensual observer drift

Answer 24

• not having raiders work together
• providing raiders with adequate training and regular monitoring the accuracy of raters ratings

Question 25

Factors that affect the reliability coefficient

Answer 25

- content homogeneity - range of scores - guessing - reliability index - item analysis

Question 26

Content homogeneity

Answer 26

- tests that are homogeneous tend to have larger reliability coefficients than tests that are heterogeneous - especially true for internal consistency reliability

Question 27

Content homogeneity

Answer 27

- tests that are homogeneous tend to have larger reliability coefficients than tests that are heterogeneous - especially true for internal consistency reliability

Question 28

Range of scores

Answer 28

,- reliability coefficients are larger when the scores are unrestricted in terms of range - when a sample includes examinees who have high, moderate, and low levels of the characteristics

Question 29

Guessing

Answer 29

- reliability coefficients are affected by the likelihood that a test items could be answered correctly by guessing - The easier the questions. The lower the reliability coefficient - true false tests are likely to be less reliable than multiple choice tests with three or more answers

Question 30

Reliability index

Answer 30

- is either the reliability coefficient or the theoretical correlation between the observed test scores and the true test scores calculated by taking the square root of the reliability coefficient

Question 31

Item analysis

Answer 31

- used to determine which items to include on a test and involves determining each item's difficulty level and ability to discriminate between examinees who obtain high and low test scores

Question 32

Standard error of measurement

Answer 32

- used to construct a confidence interval and is calculated by multiplying the test standard deviation times 1- us the reliability coefficient

Question 33

Calculating confidence interval

Answer 33

- for 68%. You add and subtract one standard error of measurement to and from the obtained score - for 95% confidence interval you add and subtract two standard deviations of measurement + And for 99% confidence interval you add and subtract three standard deviations of measurement

Question 34

Adequate reliability

Answer 34

- test scores can be expected to be consistent ,- does not indicate that the test measures what it's designed to measure

Question 35

Validity

Answer 35

- The degree to which the test accurately measures which what it was designed to measure Three types ,- content validity - Construct validity Criterion related validity

Question 36

P content validity

Answer 36

- important for tests that have been designed to measure one or more content or behavioral domain ,- example achievement tests or work samples - established during the development of a test by clearly defining the domain to be assessed and including items that are representative samples of that domain ,- test items are then systematically reviewed by subject matter experts to ensure the item's address all important aspects of the domain

Question 37

Construct validity

Answer 37

- important for Tess that have been designed to measure a hypothetical trait. Example intelligence, motivation, or introversion that cannot be directly observed but is inferred by an examinees behavior - involves using several procedures including obtaining evidence of a test, convergent and divergent validity

Question 38

Convergent validity

Answer 38

,- The degree to which scores on a test have high correlations with scores or other measures designed to assess the same or related constructs

Question 39

Convergent validity

Answer 39

,- The degree to which scores on a test have high correlations with scores or other measures designed to assess the same or related constructs

Question 40

Divergent validity

Answer 40

- also known as discriminant validity - The degree to which scores on the test have low correlations with scores on measures of unrelated constructs