Open questions are questions where the students formulates their own answer. The answer that needs to be given consists of one word, a few sentences or a comprehensive elaboration.
Within the exam you can combine different question types and questions with different cognitive levels well. Of course, a combination with closed exam questions is also possible.
What can be assessed?
With this assessment method you can assess skills as well as knowledge (and insight). Depending on the learning objective, you can, for example, use an essay question, fill in the blank or a question with a numeric answer.
Which levels can you assess?
Within Bloom’s Taxonomy, assessing with open questions primarily lends itself to the cognitive levels of Remembering, Understanding, Applying, Analysing and Evaluating.
When it comes to Miller’s Pyramid, assessing with open questions appeals to the levels of Knows, Knows how and Shows.
Prerequisites for this assessment method
- An exam with open questions is pretty much suitable for all group sizes, but with large groups it leads to a lot of workload when grading.
- An exam with open questions is taken on campus.
- The following question types are possible: essay, fill in the blank, short answer, numeric, financial statement and mathematical questions
Points of attention while taking the exam
While taking a written exam with open questions, several factors can influence the validity and reliability of the assessment results.
Validity
- Contents of the exam: the questions in an exam need to be complete and free of errors.
- Duration: if it is important how quickly a student gives their answer, you need to adjust the available time. In all other cases, you make sure there is enough time. Tip: before students take the exam, give it a try yourself.
Reliability
- Assessment instruction: the instruction needs to be clear.
- Location and circumstances: the circumstances in different rooms need to be the same.
- Tools: before the exam, students need to be informed on what tools are allowed (law bundle, calculator, dictionary).
- Examiners and invigilators: the examiner is almost never present during the exam. So the instructions for the invigilator need to be very clear.
Appraising
When appraising an exam with open questions, you can make several choices, such as horizontally (per question) or vertically (per student). With assessment software this can be done anonymously. While grading, you give points based on the answer model and grading information you established while constructing the question.
With assessment software you can automatically appraise open question types such as fill in the blank, short answer, numeric, financial statement and mathematical questions.
When converting the scores to grades, make sure you take facultary or programme-specific (EER) rules on required cut-off percentages, rounding, and partial or total grades into account.
Feedback to the student
During an inspection you can give plenary feedback by, for example, discussing mistakes that were made often. If you link learning objectives to exam questions in digital assessment software, it is possible to generate individual score sheets for students. This way, a student gains more insight into their performance.
When using assessment software, it is possible to add feedback to the automatically graded questions. With open questions that are not automatically graded, you can add substantive feedback while grading. Students can view the feedback during the inspection.
Evaluating an assessment method
What do you evaluate?
Among other things, you look at:
- The score distribution of the assessment.
- Whether the assessment scores are distributed normally.
- Whether the highest achieved score comes close to the maximum achievable score.
- How individual questions were answered, and whether it happened as you expected. How hard or easy did students find the questions?
- What the score distribution is for the different questions, and whether there are questions for which no one achieved the maximum score. Are there questions where the lowest scoring students performed better than the highest scoring students?
- Additionally it can be helpful to check whether there are differences between the different assessors.
How do you evaluate?
After the assessment it is important to discuss the strengths and weaknesses of the assessment and draw conclusions about the assessment and education. Based on the analysis, you decide whether the assessment design or exam questions need to be adjusted. You can immediately adjust the exam questions in your item bank so they are ready for future (re-)use.
To gain insight into the scores and score distribution, it can help to create an overview of the scores per question per student. With large exams, from approximately 100 students onward, you can make use of a so-called psychometric analysis: a statistical edit of the scores. It gives a lot of information about the quality of the exam and how questions within the exam functioned.