Examination Design
Exams have a significant influence on what and how students learn and how successful they are in their studies. But which examination format is the right one? How can we ensure that students achieve their learning objectives? And what organizational and legal framework conditions need to be taken into account?
The added value of written exams is relatively clear: the framework is the same for everyone, as are the questions and the grading system. In addition, it is possible to obtain proof of performance from many students in a short period of time.
Written examination formats are much more diverse than is commonly thought: from the typical written exam with open and closed or multiple-choice questions to written seminar papers and learning portfolios in which students are supposed to reflect on their own learning process. Within the various examination formats, there are many different question formats - from pure knowledge questions to application-oriented tasks – which, based on the learning objectives taxonomy, enable precise testing of the desired learning outcomes.
When selecting an examination format, it is always important to first compare it with the examination regulations. These specify which examination formats are permissible and how their use is regulated.
In order to make a written examination as fair and objective as possible, the requirements for the examinees and the framework conditions should be made transparent at an early stage. Old exams or sample tasks can be provided for this purpose. One danger of written examinations is that only a comparatively one-sided picture of the examinee is obtained, only a small section of what has been learned can be tested and it is difficult to deal with the particularities of the examinee (test anxiety or similar issues). To ensure objectivity, you as the examiner must define clear assessment criteria in advance and be clear about which answers you expect as a sample solution. You should also consider whether and how the questions are interdependent and how you will deal with consequential errors.
Tips:
- It helps students to know the maximum number of points that can be achieved on an assignment and the form in which the question should be answered, e.g. in bullet points, as a graph or continuous text.
- When creating the exam, keep the correction effort in mind and look for suitable compromises. Many exciting assignments are very time-consuming to correct.
- To ensure a smooth exam, especially in the event of disruptions, at least two people should supervise an exam. The exact number depends, of course, on the size of the room and the number of examinees.
Multiple choice tests allow exams with a large number of participants to be held and evaluated quickly and reliably. Suitable software supports not only the creation of the questions but also the evaluation of the exam, feedback to the students and the checking of the task quality. The quality of MC questions can be checked well by means of statistical evaluations that can be automated and can thus also be secured and improved more easily; test quality criteria such as objectivity and reliability are generally high. One can also easily measure difficulty and validity. And leaving aside the fact that not everyone likes every form of testing, this type of test is relatively fair.
The main disadvantage of this format is that the positive features of this type of test can only be realized with carefully formulated questions and answer alternatives ("distractors"). Otherwise, there is a risk of arriving at the correct solutions by guessing or combining. Moreover, MC questions are not suitable for all levels of learning outcomes. Whereby MC tests can definitely ask for more than just memorized knowledge! There are enough possibilities to ask not only simple true-false questions and knowledge queries but also assignment questions, questions about preconditions and consequences, analogies or questions as small case studies.
Plausible and well-worded distractors, along with interesting questions, are the essence of good MC exams. This way, you may not solve the guessing problem, but you eliminate the hidden clues due to weaknesses in content or language. No MC question should be crackable because you understand the principle of its construction.
Legal problems and implementation guidelines derived from them are always a big topic. In the General Academic and Examination Regulations (APSO) (§ 12a) you can find detailed guidelines for MC exams. For TUM in general the following applies: There may be – with at least three, but recommended four or more answer options – only one correct answer (single choice). There are 0 points for wrong answers, no points ("malus points") may be deducted.
Tips:
- Create a topic grid. List the content to be tested in full and then specify the desired processing depth of the tasks in the columns. In addition, you can specify the weighting of the individual topics.
- Begin formulating the answer alternatives with the correct answer and then continue with the incorrect ones. Arrange the answer alternatives logically, according to solution content in ascending or descending order, alphabetically ...
- Formulate simply, clearly and positively; do not forget the information necessary to answer! Answers should be grammatically appropriate to the question and should not differ in length or differentiation.
- Reduce the likelihood of pure guessing by using a large enough number of questions with as many (4 or more) alternative answers as possible.
- Always pretest with the questions and evaluate for goodness of fit. Also test the questions with non-specialists for comprehensibility, hidden clues, etc.
- Create a pool of questions at the chair and add to it regularly. Check the questions occasionally or evaluate the test results according to question quality. Shuffle the test questions so that they do not appear in the same order.
Oral exams cover a wide spectrum: from the typical individual oral exam to reports or oral presentations to group exams. Regardless of the format used for the oral examination, proof of performance in the form of written documentation such as a protocol is essential. In the case of group examinations, it is also important that the performance of the individual is recognizable and reflected in the individual grade. In addition, when selecting the examination format, it is essential to compare it with the requirements of the examination regulations.
In order to make the examination as fair as possible and to reduce the "examination anxiety" factor, the requirements for the examinee and the general conditions should be made transparent at an early stage. One way of doing this is to provide sample questions or to publicize criteria for determining grades.
In order to make the assessment of examinations as objective as possible, it is important that you as the examiner define clear assessment criteria in advance and that you are clear about which answers you expect as sample solutions and to what extent. Especially in the case of oral examinations such as papers and presentations, the student should be informed about which criteria, in addition to the correct presentation of the content, are included in the evaluation, e.g. eloquence, form of presentation, scope of the explanations ... In addition, it is helpful to reflect on one's own expectations before and during the examination and to pay attention to which criteria actually reflect the student's performance and which rather reflect personal preferences of the examiner.
The added value of oral examinations compared to written examinations lies on the one hand in the possibility of obtaining a comprehensive picture of the examinee. On the other hand, special features of the situation or the examinee (such as test anxiety or stage fright) can be dealt with flexibly.
Tips:
- Offer visualization opportunities such as a sheet of paper with pens.
- Ensure a pleasant atmosphere during the exam: offer a glass of water, give the student time to arrive, introduce the assessor.
- Go from the simple to the complex in terms of task difficulty. By doing this, you allow the examinee to reduce nervousness and "get into the exam." In addition, you can then probe how sound the candidate's knowledge is in the difficult questions.
- For students who have not yet had an oral exam and are therefore very nervous, you can briefly act out an exam in the course together with a colleague; this creates transparency and reduces anxiety.
- Don't take it personally if students strategically learn only parts of the material relevant to the exam. Nevertheless, make it clear that other areas are also required and encourage the student to think further instead of clinging to what they have learned.
- Get feedback from your students after grades are announced (e.g., using a short questionnaire). This will help you to constantly improve your own examination style.
If you think about competency orientation in a consistent way, you quickly end up with practical examinations. Above all, students can prove that they have not only acquired inert knowledge, but are also capable of translating knowledge, skills and attitudes into competent action. Initial experience with these competence-oriented examinations shows that they are very time-consuming – but that they allow reliable statements to be made about the competencies acquired, effectively control the learning process, are instructive, and are often even fun - for both students and instructors.
In order to test competencies, students must be given the opportunity to appropriately deal with a situation that is as close to real life as possible using their knowledge and skills. Creating such situations that are as realistic as possible and translating them into an examination format is unfamiliar and time-consuming. Elements of a practical exam can be:
- Role plays
- Simulations, business games
- Building models, programming
- Circuit stations where skills are demonstrated
In medicine, for example, there is the "Objective Structured Clinical Examination" (OSCE) format. It consists of a course with various standardized stations where students demonstrate practical skills (e.g., applying a plaster cast, placing a catheter, conducting an educational interview) on simulated patients, fellow students, or simulation mannequins and are observed by the examiners and graded on the basis of simple rating scales. Such an exam can take a full day for 100 students and employ around 20 examiners and assistants.
Tips:
- If many examiners and assistants are used, it is advisable to distribute handouts with clear instructions and observation criteria, and to hold a joint exam briefing or training session.
- With this exam format, a test run with all stakeholders is strongly recommended.
- To reduce the expense of such exams in the long run, keep reuse of materials in mind from the beginning.
- Consider whether consequential errors may occur in your format, how to deal with them, and how to factor them into the grade.
- Consider what factors may jeopardize the objectivity of the exam and how you can control or reduce those factors.
- Inform your students about the unfamiliar exam format early on, make the requirements transparent, and give them opportunities to prepare for it.