Marking Errors

In a recent post, I mentioned the need to deal better with errors and statistical analysis amongst theoreticians and modellers in nuclear physics than we habitually do at present.  Apart from the fact that I'm co-oraganising a workshop on the topic, my post was spurred by a conversation I had at a workshop in Poland.  One of the comments in the recent post, about assumptions in methodology, made me think of something that I usually think about at this time of year, namely: How accurately can we measure student achievement?

We (and I mean teachers and academics in general, rather than my department at my University) set a lot of different kind of assignments for students and generally give them a numerical score for each assignment.  To make the case simple to discuss, let's restrict it to my field of Physics, where most assignments have pre-determined mark schemes, that can be very detailed and prescriptive, written at the time of setting the assignment.  I have little doubt that we can mark (quite) faithfully according to the mark scheme, and therefore give a "precise" mark (in the usual language distinguishing precision from accuracy).  On the other hand, I'm not so sure that marks are so accurate.

For example, let's say one student takes one final year module, examined by me, and another takes a different optional module examined by my fictional colleague Dr Cruel.  We have both followed the guidelines to the best of our ability in terms of matching up our assessment to learning outcomes and level descriptors, and all the other sorts of things that the educationalists have made us think about, yet the mark a student gets in my course is much higher than in Dr Cruel's course.  The student put in the same amount of work in both courses.  Is it just that they revised the wrong thing in my course?  That they found the material differently difficult?  That I am a kinder marker?  

There are probably many reasons, and to some extent, they are all just part of the system, and students (and everyone else) must accept this.  And accept that there are factors like the luck of the style of the 2013 exam being one that a certain student was hoping for and another not, or vice versa.  Still, I certainly find it hard to argue that a student having a mark of 59.2% means that a lower second degree is absolutely the obvious outcome, whereas one with 60.1% absolutely has an upper second.  Actually - I guess I don't find that hard - they are rules which are set down, but I get really frustrated by people who are adamant that we mark to an accuracy of anything below a few percent.