Week 9 - Multiple Choice Testing

Question 1

Why is MC testing used in schools?

Answer 1

• Easy to grade and can be seen as more ‘objective’
• Only requires one marker, rather than two as there wouldn’t be disagreement on blind marks assigned to students work (if it was essay, for example)
• There is no need for a second marker

Question 2

Can multiple choice tests be tested?

Answer 2

Yes - this is another benefit to MCQs.

Question 3

What formats can MC questions be in?

Answer 3

• Most common: single option selected
• Confidence marking: assign confidence to a favourite option.
• Elimination testing: eliminate all options that examinee identifies as incorrect
• Complete ordering: rank options from most to least favorite
• Probability testing: distribute 100 points across the options
• Multiple correct answers: you can select more than one correct answer

Question 4

The most common MC question format is having a single option selected - what is the main limitation of this?

Answer 4

• Partial knowledge: this is where the examinee knows only part of their answer, or is not confident about the answer.

Question 5

List the MC formats that actually test partial knowledge

Answer 5

• Confidence marking
• Elimination testing
• Complete ordering
• Probability testing*
• Multiple correct answers

Question 6

Explain probability testing (an MC format) as a way of testing partial knowledge

Answer 6

• Distribute 100 points across all options
• The final score is a sum of probabilities assigned to correct answers
• It has precision in measuring partial knowledge
• So, rather than getting 0% selecting the incorrect option - they could score more than 0 and account for partial knowledge.

Question 7

What is a limitation of using ‘Multiple correct answers’ as a format for MC questions?

Answer 7

• There is a concern that ‘risk-taking’ students will score differently than ‘risk-aversive’ students
• As a result - rather than measuring knowledge it may inadvertently measure risk behaviour.

Question 8

Apart from partial knowledge, what is another problem that exists for MC testing?

Answer 8

• Correction for guessing
• Having a limited number of options means that correct answers can occur even without knowledge

Question 9

What scoring rules have been developed to tackle the problem of correction for guessing in MC questions?

Answer 9

• Subtract partial score for each incorrect response
• This is 1/(n-1), n = number of options
• Also, you can omit answers to avoid the penalty - they can choose to omit it.
• For example: On a 4-option MC test with 100 questions, the student scores 70 correct responses, 20 incorrect responses, and 10 omissions.
• Score = 70 - 20*(1/[4-1]) = 70 -6.33 = 63.33

Question 10

What are problems with the rules developed to tackle correction for guessing (LOL)

Answer 10

• Problems for formula scoring:
• Examinee’s don’t like being penalized
• Test scores are influenced by metacognition: students must assess their knowledge correctly when deciding which answers to omit
• Risk taking behaviours can influence mark

Question 11

Outline Higham’s study on correction for guessing

Answer 11

• Students were given SAT questions - and were given the standard instructions: such as you can omit.
• They were then asked what they would have said to the questions they omitted.
• The test was then rescored based on these answers.
• Findings: those who omitted answers were generally losing points.
• So.. omitting answers can be bad as they may be of good enough quality to override the penalty, if answered.

Question 12

What did McDaniel do to see if MC questions were the best learning tool?

Answer 12

• Gave a web-based university course on brain and behaviour
• Students were assigned weekly readings and then practiced with: MC quizzing, short-answer quizzing or re-reading.
• Corrective feedback was provided
• Some facts were not practiced at all - these served as new questions on the test (to compare advantage of practice compared to none).
• In their final test at the end of the semester it was found…
• Read only gave a tiny advantage
• MC quizzing tends to produce a bigger increase in performance (retrieval practice effect)
• Short-answer quizzing, was best, it had the biggest effects.
• So.. Daniel concluded short-answer quizzing was best (in opposition to MC)
• However, ANY format for retrieval practice will provide benefits

Question 13

What have meta-analyses found for MC questions vs cued-recall (recall retrieval)?

Answer 13

• Meta-analyses have found that MC often produced better performance compared to cued-recall
• However, some meta-analyses sees the opposite so idk

Question 14

What are the pitfalls of MC practice tests?

Answer 14

• The negative testing effect.
• For example: choosing a wrong answer on a MC quiz can lead to producing this wrong answer in cued-recall later on
• So, some people sometimes respond with a lure from the MC practice test on cued recall tests - the negative testing effect.

Question 15

Marsh et al demonstrated the negative testing effect - related to MC testing - what did he do?

Answer 15

• Participants practiced answering SAT II MC test questions on biology, chem, world history and US history.
• It was formula scored - so there was an option to omit responses
• This was followed by a filler task
• They then answered short-answer questions with: 40 items from earlier and 40 new items
• Undergraduates were more likely to make errors by recalling an earlier item they selected incorrectly - and were more likely to do that if they were tested (they were lured!!!)
• However, the correct answers still overwhelmed the negative testing effect
• For high-school Juniors the negative testing effect is bigger than the correct answers - more lures were recalled than correct answers

Question 16

The negative testing effect is seen with which type of learning strategy?

Answer 16

Multiple Choice (MC) questioning

Question 17

How do you get around the negative testing effect in MC questions?

Answer 17

• Give corrective feedback.
• When the correct answer is offered, rather than just leaving it with incorrect, the negative testing effect is reduced.

Question 18

What is meant by when there are Related questions with Different Correct Answers in MC testing - and why does this happen?

Answer 18

• Say, if you take a MC test which has a certain structure to it like “From what classic movie comes the line (A)” - you may get the answer wrong on MC then receive corrective feedback
• In the final test - there may be a question framed the same way, but with a different correct answer: “From what movie comes the line (B)”
• In the first practice (MC questions) there can be competitive lures (difficult MC question) - you have to go through each of the options and retrieve information about it.
• As a result, all the extra retrieval you did is reflected in the final test, as earlier you may have considered information about other competitive lures.
• This is elaborative retrieval

Question 19

Little et al looked at elaborative retrieval in MC testing - what did they do?

Answer 19

• Participants first completed an online MC practice test with general knowledge questions
• Elimination testing was then given to encourage processing of all lures (elaborative retrieval)
• After a distractor task, they completed a cued-recall test
• The test included: previously tested items (repeated), related questions and new questions (control).
• Findings:
• There is a clear testing effect.
• Repeated had the highest recall, followed by related followed by new (control).
• This is evidence that elimination testing forces considering alternatives which helps them on cued recall tests.
• Cued-recall effects were best in related questions rather than new questions

Question 20

Whether practice of later items faciliates later performance depends on whether the final test is cued-recall or MCQ format - true or false?

Answer 20

• True, performance depends on format of the final test (either MCQ or cued-recall)

Question 21

Outline Alamri’s experiment on whether performance differs on cued-recall final tests or MC final tests

Answer 21

• Followed a design similar to Little et al. (MC general knowledge practice, elimination testing, distractor task and final test)
• She changed the group IVs though.
• Group 1: practice test = MC, final test = cued-recall
• Group 2: practice test = MC, final test = MC.
• Repeated questions, related questions and new questions were involved.
• In repeated questions, MC testing was higher than cued recall, with the same in new questions
• Cued-recall was only better than MC testing when the questions were related.
• Interestingly, if the final test is MC and had new questions - performance was as good as related questions and cued-recall.
• So, what was interesting was MC in related questions impaired performance - yet improved performance in new questions
• Most errors were due to participants selecting the corrective feedback from the practice test, which is no longer correct (in related questions) - participants believed related questions were repeated: it is false recognition.

Question 22

When MC practice tests are given and have lures, when tested - which format of final testing leads to benefits and which format leads to impairment?

Answer 22

• Benefit: MC –> Cued recall
• Impairment: MC –> MC

Question 23

When can final MC tests show benefits, rather than impairments in final tests?

Answer 23

• When the testing occurs in genuine educational environments
• When related MC questions are presented close together on a test

Question 24

Compare the Negative Testing Effect and the Related Questions Effect (negative) - in MC format

Answer 24

• Negative testing effect
• Order: MC —-> Cued-recall
• Cause: Repeated questions with same answers
• Consequence: lures on first test are falsely recalled on test 2.
• Role of feedback: this is reduced by feedback!
• Cause of errors: repeating earlier error
• Related Questions Effect
• Order: MC —-> MC
• Cause: Related questions with different correct answers
• Consequence: feedback on test 1 is erroneously selected again on test 2.
• Role of feedback: problems are caused by feedback D:
• Cause of errors: responding with earlier corrective feedback

Question 25

What is a good way of using MC testing as a learning tool?

Answer 25

Generating MC questions

Question 26

What does PeerWise do with MC testing?

Answer 26

It acts as an online tool to allow learning by: - creating - sharing - answering - discussing MC questions (basically don't worry about memorizing - it just allows you to generate MC questions and collaborate with others who also generate).

Question 27

What 2 learning techniques makes PeerWise successful?

Answer 27

- *Generation* - generating information leads to better memory compared to just reading - *Retrieval practice* - retrieving information leads to better retention rather than restudying it

Question 28

Outline Kelley's study on PeerWise and MC testing

Answer 28

- Kelley looked at 2 learning techniques PeerWise uses separately - *generation* and *retrieval practice* - looking at the role of each - Participants were students on a cognitive psychology course - They were required to both generate and answer questions - In this, for each of the 8 text book chapters - they were required to generate at least 1 question (*generation component*). - They had to back up why the option was right, and why other options were not right and this task was evenly spread through the semester - They were also required to answer and evaluate other students questions (*retrieval practice component*). - At the end of the course.. researchers examined the questions generated and answered by each student, and then looked at their final exam - determining how these overlapped with generated or answered questions - For example: Student generates question on topic X, the exam question is on topic X - The performance was then compared on these questions to control questions, which were questions that didn't overlap with their experiences - **Findings** - For both generation and retrieval practice done on PeerWise, exam accuracy was higher than controls (questions that weren't generated or retrieved from other's tests) - Student generated MCQs produced the highest recall