Personality testing

Question 1

What are four factors affecting reliability?

Answer 1

• PEOPLE TAKING THE TEST: (that there is adequate variance; need a match between person level and test level i.e. floor and ceiling effects)
• TEST CHARACTERISTICS: (Bandwidth (content coverage) versus fidelity (reliability) - the more specific the test content, the higher the reliability BUT should not sacrifice content coverage to obtain reliability)
• ITEM CHARACTERISTICS: (correlation between items and no. of items affect internal consistency; a reliable test can have a lot of items that show a small correlation or a few items that are strongly correlated)
• METHOD USED TO ESTIMATE RELIABILITY (internal consistency, alternate forms, test-rest)

Question 2

Why not keep increasing test length?

Answer 2

• boredom
• exhaustion
• motivation
• make the test as short as possible within acceptable heuristics

Question 3

What are some practical problems with short tests?

Answer 3

• item exposure
• for high-stakes applications, you don’t want people to have seen the tests before (RPM & selection; educational testing)
• ability to adequately sample the domain of interest (Bandwidth vs. fidelity; breadth of the construct measured)

Question 4

What is computerised adaptive testing?

Answer 4

• a computerised algorithm determines what the individual test-taker sees next
• it is adapted to the individual’s own ability as the test-taker’s responses determines what they see next

Question 5

What are the advantages of CAT? (Computer Adaptive Testing)?

Answer 5

• it makes the test shorter, but just as reliable (less problems with fatigue; economic bottom line i.e. less testing sites booked for fewer hours, less test proctors etc.)
• easier to maintain test security
• motivation factors (very able ppl aren’t getting a large number of too easy items, low ability aren’t failing item after item).
  MOST APPROPRIATE FOR LARGE SCALE TESTING WHERE SECURITY IS AN ISSUE.

Question 6

What are some disadvantages of CAT?

Answer 6

• substantial preparation and outlay needed:
  1. development of VERY LARGE ITEM POOL
  2. Analyses of VERY LARGE ITEM POOL to determine the difficulty of each item
  3. Automated programming of algorithm/decision rule
• requires computerised administration

Question 7

What is CAT most appropriate for?

Answer 7

• large-scale testing where security is an issue