What are the pros and cons of marking exams one question at a time? IB examiners will tell you. Assessment researcher Katie Schultz explains what the IB calls “Question Item Groups” and shares the results from a 2017 examiner survey on how marking exams as QIGs impacts assessment.
By Katie Schultz – Assessment Research and Design researcher, IB
When you mark student work, do you ever go through and mark responses to one question at a time, marking all of the responses to question one before moving on to question two? Or do you prefer to mark tests one student at a time, marking one student’s responses to all questions before starting on the next student’s work? Marking question-by-question instead of student-by-student can be a hassle, but some teachers say that for them it makes marking easier and more efficient. For some exams, the IB does this too. Though most IB exams are marked as whole papers, all Middle Years Programme (MYP) eAssessment exams and some Diploma Programme (DP) exams (such as physics paper two and economics paper one) are marked as what the IB calls “Question Item Groups” or “QIGs”. The IB creates QIGs by breaking full exams into smaller parts, groups of questions that go together and can be marked as a set.
How does the IB do it?
This is similar to how some teachers mark their students’ work already. The difference is that the IB is able to do this digitally, and by doing so can send responses to different groups of questions to different examiners to be marked, as shown below.
Figure 1: Creating QIGs from a single exam paper to send to multiple examiners for marking.
When the examiners are done marking, the marks for each QIG are added together to determine the student’s total marks for that exam. In most cases this means that, for exams marked as QIGs, multiple examiners contribute to the final mark.
When everything goes right, this can make marking easier for examiners and fairer for students. It’s no secret that not every examiner is the same. The IB uses various processes to ensure examiners mark to the same standard and interpret the markscheme the same way, but there is still some room for differences in opinion. This means some examiners can be slightly more generous in their marking than others. But, this is where marking exams as QIGs comes in. When multiple examiners contribute to a final mark on an exam by marking different parts of it, these differences between examiners are more likely to balance out, leading to the fairest most accurate mark possible.
Pros and cons of marking exams as QIGs
So why does the IB do this for some exams and not others? There are pros and cons to marking exam papers as QIGs (I’m sure you can think of many more than just the few I’ll mention here!).
At first, the IB thought marking exams as QIGs would make things easier for examiners, like it sometimes does for teachers. Instead of having to mark entire exams at once, examiners only have to focus on one small part of an exam at a time. And, for exams where students get to choose which questions they answer and which topics they write about, examiners can choose to only mark questions about topics they know a lot about.
However, we soon found out that dividing an exam paper into QIGs is a lot of work—for the IB and for our examiners. Instead of qualifying once to mark a full exam, examiners who mark QIGs have to go through the qualification process multiple times, once for each individual QIG. Because different QIGs are sent to different examiners, it can also be difficult for the IB to make sure every QIG gets marked around the same time. The sooner every QIG gets marked, the sooner QIG marks can be added up, mark totals for exams can be calculated and final marks can be converted into final grades for students.
“What’s interesting is that even taking these differences into account, the examiners we heard from seemed to like marking exams as QIGs”.
In 2017, the IB Assessment Research & Design team conducted a survey to find out how examiners felt about QIGs. Every examiner who marked QIGs for the May 2017 exam session was invited to participate. We asked examiners who had experience marking both QIGs and whole exam papers about the pros and cons of marking exams as QIGs. What we found was that examiners from different subjects had some differences in opinion. This makes sense because not all QIGs are exactly the same—there’s a big difference between marking a DP physics QIG with a few short-answer questions and marking a DP language A literature QIG with long essays!
What’s interesting is that, even taking these differences into account, the examiners we heard from seemed to like marking exams as QIGs. More than 80% said that focusing on one part of an exam at a time helped them get to know the markscheme better and more than 70% said it made it easier for them to concentrate while marking. Fewer examiners identified drawbacks of marking exams as QIGs. Of those that did, 35% acknowledged that it’s more work because needing to qualify for multiple QIGs and repeat the qualification process several times takes longer.
For now, we acknowledge that marking exams as QIGs can be beneficial (but also complicated!) and that it works better for some subjects than for others. What about you? What do you think some of the other pros and cons of marking exams as QIGs might be? Let us know in the comments down below👇🏼
Katie Schultz is a researcher with the IB Assessment Research & Design team (ARD). She contributes to IB internal research focused on improving IB assessment processes and supports curriculum development by analysing the results of assessment trials. Her work helps to ensure that newly developed assessment tasks are manageable, valid, and can be marked reliably. Katie is also a Diploma Progamme (DP) graduate. She graduated from Portage Northern High School in 2009.
This post refers to the results of a 2017 examiner survey conducted by the Assessment Research and Design team (ARD), which sought to understand how IB examiners experience marking student responses at the item level and by Question Item Groups (QIGs). The results of this survey were presented at the 2018 IB Chief Examiner’s conference in The Hague, as well as the 19th annual AEA Europe conference in Nijmegen-Lent.
If you enjoyed this story, consider reading more below:
