One student is presented with multiple-choice questions in an order that corresponds to the structure of the lecture series. The other student receives the same questions in random order. Does this affect the exam results? Albert de Vaal, associate professor of International Economics, was curious to find out and conducted research into this. 'When questions are jumbled up, there are fewer advantages.'
For years, Albert de Vaal followed the same steps when compiling an exam with multiple-choice questions. During the weeks in which he gave lectures, suitable questions naturally came to mind, which he noted down. At the end of the course, he then arranged them in order, matching the structure of the lecture series. To prevent students in a classroom from secretly signalling answers to each other, he created a second and third version of the test, for which he shuffled the questions. 'I think a lot of lecturers do it this way. But a few years ago, I suddenly thought: would it matter if, as a student, you were given the first version or one of the other two?
Telescope
Some research has been done on the order of multiple-choice questions, but not much can be concluded from the existing studies. In one study, the order did influence the test results, but in another it did not. De Vaal decided to take another look at it. This led him to venture into a new scientific field: economics in the field of education. Together with Kees Haasnoot and Charan van Krevel, colleagues at the University of Amsterdam who were previously also affiliated with Radboud University, he analysed twelve examinations from his microeconomics course between 2013 and 2019. There were three versions of each examination. A total of 745 students took these exams, resulting in 16,127 individual question-answer combinations. ‘Previous studies only looked at the result of the entire test, not the score per question. That makes a big difference. At the meta level, anything can distort the picture. A colleague compared our method to an improved telescope that allows you to look into the universe.’
Meticulous method
To find out what effect the order of questions has on whether or not a multiple-choice question is answered correctly, De Vaal and his co-authors ensured that as many other factors as possible were controlled for. These included year-specific circumstances such as the average quality of a student cohort, specific deviations in the course of events during an exam, and the difficulty level of a question. This detailed analysis led to a clear conclusion: the order of the questions matters. 'It is beneficial for students if the questions are asked in the order of the course. In fact, the more ordered questions there are in a row, the greater the chance of the correct answer.' A single question in the 'natural' order does not yet yield a visible improvement, but from a series of three onwards, the positive effect is statistically significant. With each doubling of such a series, the chance of the correct answer increases by approximately five percentage points.
Explanations
According to De Vaal, there are several explanations for the better results on test questions with a known order. 'When students study for an exam, they often do so based on the order of the lectures. Furthermore, a previous question can help you with a new question. All you have to do is take a logical step. An earlier question can also help you think in the right direction. You are then already in the place in your head where you have stored the information about a particular theme. If the questions are jumbled up, you don't have these advantages.'
Not harmless
The conclusions of the study are important, says De Vaal. 'Shuffling questions is much less innocent than it seems. It gives an advantage to students who get the questions in the order of the course. And of course, you don't want that.' That doesn't necessarily mean that as a lecturer you always have to shuffle all the questions. ‘Questions can be arranged in a logical order. But make sure that this is the same for each exam variant.’
Since the results of the study were published, De Vaal has been taking this into account in his own examinations. ‘Of course, I now make sure that the advantage is roughly the same for each version. And my co-authors do the same. Now it's up to the rest of the world to follow suit.’
Text: Willem Claassen
Photo: Duncan de Fey