One of the problems I found with my law degree is that I was never quite sure why I received the marks I did. So I am very sympathetic to students when they consult me for exam feedback. I go through the exam with the student and point out where they lost marks, and what could have been done better. Usually, with hypothetical problems, if the student actually studied, it’s a question of timing (student ran out of time) or emphasis (student emphasised issues which were not relevant). Occasionally, someone knows his or her stuff, but gets the wrong end of the stick, which is always upsetting. With essays, I mark down pieces which are unstructured, cursory, lack argument or cite bold propositions with no authority whatsoever.
There have been increasing incidences of students challenging their mark, particularly given the very competitive nature of the job market in areas like law and commerce. I do dislike it when people hint (or outright suggest) that their mark should be higher because I’m very thorough and careful with my marking, and give extensive feedback. Very rarely, students argue with me. I would never have had the cheek to do that when I was a student. I deal with them politely, but very firmly.
Recently, a student successfully applied for a university to release its marking guide under FOI procedures. The university appealed the decision of the FOI Tribunal, but to no avail. A decision confirming the decision of the FOI Tribunal was handed down last week in University of Melbourne v Zane McKean [2008] VSC 235.
The case arose in 2006 when Mr McKean sought certain documents from the university in relation to two commerce subjects, ‘Investments’ and ‘Intermediate Personal Finance’. These documents included the Investments and Intermediate Personal Finance marking guides (which set out sample correct answers for examiners in relation to those particular exams) and his examination paper in Intermediate Personal Finance.
The university sought to exempt the marking guides pursuant to s 34(4)(c) of the Freedom of Information Act 1982 (Vic). That section provides an exemption for ‘an examination paper, a paper submitted by a student in the course of an examination, an examiner’s report or similar document and the use or uses for which the document was prepared have not been completed.’ On 31 July 2007, the FOI Tribunal ruled that the marking guides should be released, and were not subject to the s 34(4)(c) exemption.
Naturally enough, the reason why the student sought the examination guides was not only to ascertain why he received certain marks, but he also has an eye to challenging the result in two subjects. The article in The Australian explained as follows:
Mr McKean said he might use the guides to challenge his marks in the two subjects, even though he finished his arts-commerce degree in 2006. “I wanted value for money. Not only that, but I wanted good marks, and if I didn’t get them, I wanted a good reason why,” he told The Australian.
The main question was whether the use for which each document had been prepared was completed. The University attempted to argue that each marking guide could be broken down into discrete documents representing each question, and that some discrete questions might be recycled in future years. Accordingly, it submitted that the Tribunal failed to consider whether the use for which each document had been prepared was completed.
However, Kyrou J confirmed the Tribunal’s finding that the particular marking guides in question were prepared solely for use in regard to that particular examination, and thus the particular use had been completed, regardless of the fact that some questions may be recycled in future years. Therefore, the University had to release the marking guides.
Prediction: following this decision, universities will try to minimise the use of marking guides to try and reduce the opportunity for students to challenge marks. This could reduce the amount of consistency between markers and thus result in less fair marks for all students. It could result in a less satisfactory outcome for students overall.
I note also that Mr McKean seems to have a touch of the attitude that he was somehow entitled to good marks because he was paying for his degree, and if he didn’t get good marks, then he seems to imply that it must have been the university’s fault. This is erroneous. Good teaching can make an immense difference, but ultimately, how students do in an exam is up to them. You can do all kinds of things to help them and make a subject more palatable, but in the end, the final responsibility lies with the student. That’s one of the hardest things about being a teacher.
Update
Tobias from Not a Hedgehog has alerted me in comments below to the equivalent NSW provision. He notes that Clause 16(a)(i) to Schedule 1 of the Freedom of Information Act 1989 (NSW) has an exemption for documents the disclosure of which could ‘reasonably be expected to prejudice the effectiveness of any method or procedure for the conduct of tests, examinations or audits by an agency.’ That seems like a more appropriate provision to me.
Sounds to me like the University of Melbourne and other Victorian universities should be lobbying for an amendment in those terms to the Victorian FOI Act.
28 Comments
Great post, LE. I read about the case in the Australian yesterday and then checked out the decision as well. I have the same concerns as you appear to – it could actually compromise the way exam assessment is planned and conducted, which would be a net loss for academic integrity.
I think NSW’s Freedom of Information Act seems to have a more appropriate way of handling exam-related documents. As opposed to a, “has the use of the document been completed?” approach, it exempts documents if the disclosure could be expected to prejudice the effectiveness or objects of tests, examinations or audits (clause 16 of Schedule 1).
Under that provision, I would think it can be argued that effective assessment requires the use of reliable and valid assessment procedures, which includes (i) the re-use of questions that have been shown to measure performance appropriately in previous exams, and (ii) objective and explicit guidelines for evaluating responses to those questions.
He sounds like a very presumptous person! You’d want to be VERY confident in your abilities to be challenging the marking in that manner…
The other thing it will lead to is written questions being scrapped and the use of just multi-choice ones, where there are no arguments about right and wrong (especially if it is really true that it is simply not possible to set different questions each year). Given that decent multi-choice questions are hard to construct good sets of, I’m not sure what will happen if that gets challenged that too! (or: Complainer 1, Learning -1).
Thank you, Tobias. I agree that the NSW legislation sounds like it deals with the situation far better. I’ll add an update to that effect.
Ms Laurie, I can’t really fathom that level of confidence – my attitude was to take the mark I received on the chin, although I did sometimes look at the paper to see where I went wrong just so I could learn to do better next time.
Conrad, I do hope you’re not right, but I suspect that you are. I loathe multi-choice unless it’s a factual question with a clear answer. It’s useless for subjects which are more interpretative.
This is not a problem, and the reuse of questions from previous exams is just poor practice. If you reuse questions you can be guaranteed that there is a cabal of students who have recorded those questions and are passing them on to future students, in fact there are websites which have been set up to record these exams, some even pay money for them, more in the USA than in Australia, but this is an area which is rapidly globalising (e.g. the outsourcing of programming assignments to India and Eastern Europe). Once you are known to be the type of examiner who repeats questions, then this will inevitably happen. Do you think that the average student, let alone law students aren’t cunning little so-and-so’s?
I ran the same course for 11 consecutive semesters, each semester had full access to the past exams and the sample solutions for each exam, did this mean that the exam was easier? No, the marks spread was basically the same after the first semester where I made the exam too hard, so unless they were gradually getting stupider, having access to the exam and sample solutions didn’t help them at all.
Why not? The exam didn’t attempt to test memorisation, it tested their ability to complete the tasks and show the skills that they had learnt over the semester. Knowing the answers to the particular scenario for a question from a previous semester does not help you answer the same question in your exam, because the scenario is different. Knowing and even understanding an answer is not the same thing as being able to create the answer from scratch. In fact, it was often quite sad when students reproduced from memory vast swathes of previous exams and I gave them zero or very low marks for not having answered the questions that I actually asked.
Multiple choice doesn’t help with the problem either, students are quite able to memorise vast quantities of multiple choice questions and other material.
As to the assertion that it is not possible to set new questions each year, I am extremely sceptical that this is true in any discipline and seems more likely to be the sort of excuse for avoiding work that I hear too often from students
Martin, I have to say that I didn’t much like the sound of the fact that they reused questions either. Certainly, in subjects I have taught, we always write the exam afresh each year.
The faculty concerned in this case was Commerce, and I don’t really know how exams operate in that area. I’m an Arts/Law grad myself. Still, it does seem like asking for trouble to repeat questions. I think they may have made a big deal of it in order to fall within the s 34(4)(c) exemption.
Law used to make sample answers available too, although I gather they don’t any more, which is a pity. I found it tremendously useful as a student to read sample answers. You can always tell when someone just recycles a sample answer.
I don’t think giving access to sample answers per se is a problem at all – indeed, I would advocate such a practice. You don’t even have to use a past exam: make up a problem and do an answer for it.
The problem is that the student is now going to go over the marking guide with a fine tooth comb in order to find reasons to challenge his mark. This sets a precedent for students to ask for the production of marking guides to challenge their marks. I suspect this might be why Law no longer provides sample answers to past exams as well, which is really sad.
I think this decision might encourage universities to restrict access to resources rather than make material such as sample answers more available, and that would be a great pity.
Legal Eagle,
I’m in a Faculty of Business, so yes, our students are often are looking for that extra mark to get them over the grade barrier (CR to DI, DI to HD, NN to PA etc.), I wouldn’t have thought that Law students would be much different.
From my experience, I don’t think sample answers to past exams have any impact on these sorts of discussions. I’ve never had a student validly bring up, the sample answer for last year’s question was this, why did I only get 2/10 for this question for this year’s exam. The answer is generally because you answered last year’s question, not this year’s question
In terms of the marking guides, I can see your point, I don’t run in to this as I adopt the following process:
1) have a tough marking guide
2) when marking, err a little bit on the side of generosity in terms of the marking guide, so that the tough marking guide is balanced out.
3) when student comes to look at paper and you show them the marking guide, it is obvious that rather than doing them out of marks, you’ve been a bit generous based on the marking guide and if the marking scheme was adhered to strictly, then their mark would go down not up
4) Student goes away satisfied if not happy
On the other hand, a key principle of my assessment practices is transparency. As a student I always hated getting some mark at the end of semester which seemed to bear no comparison to what I’d achieved or felt that I’d achieved, so providing the marking guide for an assessment item to students is a necessary step if you want to be transparent in your assessment practices.
The mark a student gets in a course is a very important thing for the student and it is necessary that as the academic we can be held accountable for that mark. Having sat on quite a number of appeals against assessment results, in a minority of cases, there have been some egregious decisions/mistakes by academics and without the marking guide the student would have had a very difficult time to make their case against the mark. If all you can look at is an exam with the mark 4/10 scrawled on it and you can’t see the marking guide, how can the student make any judgment as to why they received that particular mark for that question.
To be honest, in a few cases, when students have challenged my marks, they’ve been right and I’ve been wrong and I’ve ended up giving them a better final mark. if you are marking 200 plus exams and you’ve got a week to mark them and they aren’t multiple choice exams, then it is inevitable that mistakes will be made, probably many more mistakes are made currently than are challenged.
Now, we always double mark fails, so I’m pretty confident that not many mistakes are made there, but I’m not so confident at the other borders (and the research on consistency of marking is pretty alarming and doesn’t justify much confidence that a 59 is actually better than a 62 or that a 78 is worse than an 81) are quite as successfully managed. I do look to see if student performance in the exam differs greatly from student performance in the assignments and double-check those who stand out, but it’s probably not sufficient. My opinion is that you should either abolish grades or abolish marks, both together are a problem.
To me the argument that we can’t give the students the marking guides because they will challenge their marks reflects a lack of confidence in the standards of marking.
So, my argument for making marking guides available is twofold:
1) It’s a matter of procedural fairness that is necessary to ensure mistakes are corrected
2) More importantly, it enables the student to learn from the mistakes that they made. It should be standard practice that all students receive copies of their exams with the sample answers and the marking guides to give them the opportunity to learn what they got wrong before moving on the next course and hopefully do something about it. Of course, trying to find the time, resources and energy to do this is not easy, in fact I’m still working out how to do this.
cheers
Martin
PS. Great Blog, I always look forward to yours and Skeptic Lawyer’s posts.
Thank you, Martin.
It goes without saying that if someone challenged a mark and I had made a mistake, I would change it! So far, it hasn’t happened, but as you say, it can be very hectic marking those exams.
I agree with you that sometimes it’s difficult to justify why someone got a 68 rather than a 71. I generally find that there’s no argument about really good papers or really bad papers, but differentiating the ones in the middle can be very difficult. I always have a rationale as to why I marked as I did. And, like you, I err on the side of generosity, so generally students go away satisfied.
I do sometimes feel very dissatisfied with marks as a measure of performance, as they are such a chancy thing. And one’s mark doesn’t necessarily reflect one’s knowledge of the subject. There must be a better way, I’m just not sure how.
I’ve been thinking further about your comments, Martin. I am all for an informal procedure where a student goes to a lecturer and says “I think you might have gotten it wrong”. Then the student and the lecturer can discuss it, and if appropriate, the lecturer can adjust the mark. As I said above, mistakes can happen, and lecturers are human.
But what happens when a student challenges every single mark he has received? (I have heard anecdotal evidence of a student who apparently did this).
There are a few things that worry me about opening the can of worms and creating a culture where people regularly challenge marks.
1. Pushy, confident people will be more likely to challenge and get a mark upgrade. Therefore they will be advantaged. As I said in my post, I would never have had the chutzpah to go and challenge my grade. Being a humble person is a disadvantage.
2. It undermines the basic integrity of the marking process. Yes, marks are somewhat chancy and random, but to an extent, unless a clear error can be shown, the umpire’s decision should be final.
One lecturer might give 68 for an answer, and another might have given 71 for the same answer – but in the end, tough cookies – that’s the marker’s discretion. If one would have given a 68 and one would have given an 82, then you have a problem. But where the margin is pretty similar (say within 4 – 5 marks), you live with the fact that you had the person who thought your question was worth 68 mark your exam. That’s life.
If students regularly challenge marks, then what is the point of getting someone to mark it at all? It suggests that there’s really no point, because the mark isn’t indicative of anything.
I agree with the last comment by Legal Eagle (9.48am).
I am sick of being assessed by lecturers and tutors when they are not always right. The majority of assessors are right the majority of the time, but not always. I should be able to check this. University staff do not like it WHEN I START ASSESSING THEM.
Grades, when assessed by idiots, are not indicative of anything at all.
Zane McKean
Questions in say contract are like a women’s dress. There is essentially only one but it can be presented in a great variety of ways.
On marking. The comments by Martin on marking confidence are spot on. I find it relatively straightforward to mark in grades (usually with a 5 mark range) as there is normally some guide that an “A requires …..standard “etc . In subjects such as law it is ridiculous to think you can do much better if you want to consider the logic of an argument, coherence etc rather than tick off a list of points.
In my classes (usually 100+) I would get no more than 5 complaints and many of these tend to pull their head in when I suggest they read a couple of answers from the top students that I post on-line. Finally I note that even with a take home test and a week to do it the final results tend to have much the same spread as an exam.
Legal Eagle,
we have to balance out the two competing factors:
1) If we don’t allow students the right to challenge their marks, then we are going to have a range of situations where students are disadvantaged unfairly
2) we do allow students to challenge their marks and then we have the possibility of students abusing the facility and challenging all their marks.
My experience at a University which grants students the right to challenge any of their marks and take it to a formal process is that the problem with scenario 2 is largely hypothetical and and the problem with scenario 1 does actually occur, so on a pragmatic basis alone, I tend strongly to a system that is open about marking guides.
Remember, the facility to challenge the mark was there and still is in Melbourne Uni’s policies, they just didn’t want to reveal the basis on which the mark was granted, making the ability to challenge rather moot. IANAL, but to me this seems much like disallowing discovery to one side in a law case.
Let’s ignore all the highly valid educational reasons for providing marking schemes to students and concentrate on the possible problems with allowing it.
So we need to make sure that our process provides students with a reasonable opportunity to challenge marks but does not provide an incentive for students to routinely challenge marks just on the off chance they may improve their mark. IMO, this is relatively easy to achieve:
1) Students can request an informal explanation of their mark, which includes access to the marking guide and their exam paper
2) A student can request a change of marks, the lecturer can agree or disagree.
3) In the event that the lecturer disagrees and the student wishes to take it further, the student can request a formal remark. In the remark, the student’s result can increase or decrease or stay the same.
4) If they still disagree, they can take it to a formal University panel.
So the key is that the remark or the panel can either confirm the mark, increase the mark or decrease the mark, with the possibility that the mark can decrease, you immediately rule the vast majority of students using the system opportunistically. Very few students are willing to risk a remark if their mark can go down and they are not serious in believing there was a problem with the marking.
As far as the ones like the one you mentioned that challenges every mark, in over 10 years I’ve never seen this, formal challenges are very rare, even where the university already allows this and really unless you ban challenges entirely, then students like this (who often have issues independent of the marks) are just one end of the spectrum that makes up our student population. As a degree leader, I work on the 5/95 rule, 5% of my students take up 95% of my time, that student is clearly one of the 5%.
cheers
Martin
Legal Eagle,
on the issue of marks vs grades vs something else, I’m very happy to mark to grades, I can clearly distinguish between Fail, Pass, Credit, Distinction and High Distinction work with very little effort nowadays, though I would like the option of a Very High Distinction for the few very exceptional pieces of work. But, students and employers both want to see actual marks on their transcript. The problem arises when we combine both and a student can see that they have an x9 mark and 1% more would push them over to the next grade. And they have a very good case that there is no real difference between an x9 mark and 1% more and that it is almost certainly random statistical variation that caused the difference.
Th grade is important to them, as a lot of Unis maintain a GPA, quite a few employers and scholarships have a GPA requirement and a HD looks better on the transcript than a DI. It thus becomes a matter of importance for them to get that 1%. I know many lecturers who won’t give a x9 or even an x8 mark, because of students badgering them about being marked up and who just move them up to 50, 60, 70 or 80.
So while I would prefer just giving grades, I can’t see that happening in the competitive business environment of the modern University, so I think the second best option is to get rid of grades altogether and GPA and just have marks and an average mark for your studies.
Other options, like descriptive comments in combination with a Pass/Fail don’t meet the needs of employers who want a quick and dirty means of comparing students and aren’t really worried about the validity of the comparison. These methods are also far more time-consuming.
cheers
Martin
Zane,
assuming you’re the relevant Zane McKean
,
have you got hold of the marking guides yet and if so, did they reveal anything that makes you feel that your marks were incorrect?
cheers
Martin
The canny lecturer would start taking 10% off their ‘real’ mark before the inevitable rounding up. Everyone wins: the mark is what the lecture wants and the student thinks they pulled a swifty.
The best assessment scheme I’ve come across is the International Baccalaureate’s.
Grades are divided up by assessment area — eg in Arts, grades for artwork are done independently of reviews etc etc — then in each area it’s out of 5. Each of the 5 levels has a detailed explanation of what it requires, which students have access to.
Also available upon request is complete syllabi, so that students can identify, with great precision, what they are expected to know for their assignments and exams.
Most assessment is not performed by the teacher. Instead work is forwarded to other teachers elsewhere in the world, using the same framework.
It’s a really good system. I felt it was fair and transparent. Because there were only 5 levels for any measurement there was no pressure to just add one or two percent. The criteria were very well written and very useful for self-assessment.
The IB, in many ways, shows that decoupling teaching, syllabus development and assessment can improve education overall.
Jacques, there are a few problems with comparing the IB and university stuff:
1) The IB has lots of people to work on it. Most university subjects have 1, and that 1 person has 23 other responsibilities also.
2) I would love not have to do the assessment for my subjects but it’s not up to me (and I’m sure everyone else teaching would too!!)
3) Most IB subjects hang around for years and years. Many university subjects need to be updated from year to year. Doing assessment for new things is always tricky if you care about psychometric validity (most people don’t, and wouldn’t know where to start), and thus it’s not surprising people cannot come up with such good guides.
See Re James et al and ANU - the 1984 case which has been referred to in some of the press reports.
Thanks Marcellous – interesting case. I think that students should DEFINITELY be supplied with examiner’s notes on their exam, essay or thesis if they are available.
Martin, I do support the formal process of challenging marks. If someone were severely unhappy with my marking, I would get someone else to mark it as a first option, and then I would let them go to university admin as a second option.
I just find it regrettable that students wish to use marking guides, not to learn where they went wrong, but to lobby for better marks right from the word go. If Mr McKean had simply said that he just wanted to know why he received the mark he did, and the marking guide would help him do that, I would have been 100% behind him. If the marking guide then showed that an error had been made, I would also have supported him getting that mark changed.
But I inferred from his remarks that his idea was to lobby for a better mark right from the start, which I do not think is a good attitude. It suggests that marks are negotiable if one makes enough of a fuss. Lobby for a mark to be changed only if you find an error – don’t take it as the default position when you’re merely unhappy with your grade. Unhappiness with one’s grade is not a reason for a mark to be changed. Otherwise I’d have asked for all my grades to be changed to High Distinctions!
I think that there are more people who challenge marks these days than there was when I was a student. I suggest that the phenomenon is going to increase rather than decrease. People get very unhappy if they are paying a fortune for a degree and don’t do well.
I think one thing universities could do to help lecturers and students is to have uniform policies with regard to disclosure of marking guides, examiner’s reports, what to do in the event of a dispute etc. I don’t think my university has that in place at the moment. I could be wrong – but I doubt it.
It seems to me that the whole point of exams and learning is to learn from one’s mistakes. So one does need to have access to whatever will help you do that.
LE, I think you would enjoy this.
PC – tee hee…love it!
Thanks PC – I enjoyed it too. Now, back to the marking, which one or two will challenge, and another argue that plagiarised text still demonstrates intensive research because it took two hours to locate the best info on the web…
Legal Eagle,
been at a conference.
Yes, it is regrettable that students use the marking guides not to learn, but whenever you give people the power to use something for improvement, someone will work out a way to abuse that power, so Melbourne Uni’s attempt to stop Zane or any other student accessing the marking guide was highly regrettable and as it turned out pointless and very expensive.
In terms of number of students challenging their marks, I started lecturing in 1994 and I haven’t really seen any change in the numbers over the year. Consistently, a very small number of students have challenged their marks for my courses. Maybe that’s because I’ve always had a very transparent assessment policy.
I agree that a key point of learning is learning from your mistakes, but exams have never been about learning from your mistakes. No university that I’ve heard of makes it the default process that students see their exams, let alone be given feedback on their exam. It’s okay you got 50% plus, then you can go on. You want to see your exam? Okay, if you turn up at 3pm next Thursday, I’ll give you 15 minutes to look at, I won’t discuss with you any of the marking etc. etc.
I think all students should get their exams back with a detailed description of where they went wrong, so they can improve upon their performance in following courses, but that’s expensive to do and doesn’t fit within the normal parameters of a Uni course. As well, it will increase the number of students challenging their marks, which many lecturers will abhor, so probably not going to happen.
As I’ve got a small cohort this semester, I’m now inspired to practice what I’m preaching, so at a minimum, marking guide and copy of the exam posted out to all my students and I’ll see how it goes!
cheers
Martin
“Why did I get this mark?” Because you’re a lazy slug who didn’t study.
Well, duck me…
LE
I have never questioned a grade. To be fair, I think I am graded too generously most of the time. In a 1st year History course, I slipped into total mature-aged-student stereotype mode and bounded down the front of the class after the lecture to engage my lecturer in the finer points of the Jericho’s settlement pattern, the secondary products revolution, or some such.
As we chatted, he said, “I’ve just marked your review of Guns, Germs, and Steel But now (after our chat) I realise I have graded you unfarily. Would you like me to get a second opinion!?.
Well, yes please!
Marking guides? I would give my left nut for a History lecturer to provide us with sample essays that have received grades of 90+ in exams!
Well done Zane! You need to be given a hero status man. I an overseas graduate and had to sit for exams to qualify here in Australia. Its taken me 4 years and i am yet to complete the exams. Just to give u guys a breif summary: we have three parts to the exam (U pass in two ..u can carry oneupler) I passed in two sections in my first attempt. Wrote the supplimentary…they said i was just bad in one part and asked me to sit again. Results…fail…i had to sit the entire exam. Results: fail. We have counselling for 10 minutes and they do have an appeal process with last line reading,” U results won’t change”. Why do we appeal then?
I have no clue how they assess or mark…I just cannot believe this happens in Australia. I need someone to please tell me how I should approach this whole thing. What sort of a lawyer do i look for? Please someone help me…..
In my experience the students who challenge their grades are the ones who don’t pay attention to instructions or marking guides or rubrics in the first place.
I’d be fascinated to know what the student in this case had been doing BEFORE the exam to ensure his success. Was he asking for help, using office hours to consult with his instructors and making sure he did all his assignments correctly the first time?
Because, usually this grade complaining comes as a result of the clash between entitlement (which this guy clearly has in spades) and instructor expectations.
My duck piece is based on the fact that it doesn’t matter how hard I try to explain my criteria, there are still students who ignore them until they get a bad grade.