It’s that time of year…

By Legal Eagle

…when I wonder yet again whether exams are really the best way of judging students’ knowledge, and whether our teaching methods are the most efficient way of getting knowledge across. I’m half way through my giant pile, so as a reward to myself, I’ve allowed myself a quick post.

One of the things which is frustrating as a lecturer is watching people make the same mistake over and over again, even though you told everyone in class not to make that mistake. (Eg, I literally jumped up and down and pleaded with the students not to mix up personal and proprietary remedies; you lose a stack of marks immediately if you elide the two.) With some papers, you can tell the student hasn’t made much of an effort to study and has probably cribbed off someone else’s notes from a previous year (because there are various inaccuracies and odd phrases that were never used in class which come up in successive papers). However, the most tragic ones are where the student clearly studied hard and knew his or her stuff, but just missed the point on one of the questions. Or where the student has done two of the questions really thoroughly and well, but left no time for the third question.

For non-lawyers, I should explain law exams tend to be a mixture of hypotheticals and essays. ‘Hypotheticals’ are, as the name suggests, invented scenarios which bear a definite resemblance to some of the cases and statutes in the course, but where the lecturers have varied the facts somewhat. The student has to identify the issue which arises on those facts, explain the law which applies to it, and explain how it is similar or different to the cases studied throughout the semester. I always felt more comfortable with hypotheticals than essays, but strangely enough, I often did well with essays in exams. I was best at hypotheticals which were really, really hard. Strange but true – the more complex the factual scenario was, the better I did.

The difficulty of law exams is that they are trying to fulfill two aims which are not necessarily commensurate. First, they are designed to give the student an incentive to learn the material. Secondly, they are designed to enable the student’s skill in the law to be assessed so that the we can know whether they will be a good lawyer. I think sometimes students do need fear of a test before they will be impelled to learn material. I was a law school nerd. I was one of those people who always did the reading and always started researching my essays early. A lot of students don’t do that: they try to “cram” it all in at the end. But being good at exams or at essays doesn’t necessarily guarantee that one will be a good lawyer in practice. Yes, one needs to be able to identify the issues, and to apply the law to them, but the exam paradigm is unrealistic. Usually, in practice, one specialises in a particular area (so you get a deep knowledge of that area), and if a client comes to you with a problem with which you are not familiar, you go off, research the area and consult with your colleagues.

I was discussing with a colleague our view that perhaps oral exams would be a better way of measuring a student’s true knowledge. You could give the student a hypothetical and a theoretical question a week before the exam. Sure, they can consult with other students about it if they want (just like in practice). Then, the day before the exam, they have to hand in a “brief” outlining their answer (which tests their writing and research skills). The day of the exam, they are questioned closely by two lecturers on their answers. That way you could find out whether the student really has depth of knowledge. It doesn’t depend on exam timing skills, and if a student has misread the question, the lecturers could point this out and ask the student what their response was. That way, the student is allowed to make a mistake and learn from it with guidance from a more experienced practitioner (again, just like in practice). I think oral exams would provide a good way of forcing law students to learn stuff, simply because most people don’t like looking foolish in public. I have found that lecturing forces one to learn the law in great depth because you know students will question you on it, and you have to stand up there and talk about it for two hours. If only I’d treated all my law subjects like I’d had to give a lecture on it. Any person who was unable to communicate their answer effectively to lecturers in oral exams would be unlikely to be a very good lawyer. The essence of being a lawyer is communication: communicating the law to clients, judges, other lawyers and the world at large.

I don’t know that any universities would accept this (too labour intensive perhaps?). But I think it’s an interesting idea.

24 Comments

  1. Sinclair Davidson
    Posted November 17, 2010 at 7:24 am | Permalink

    While I understand your concerns and fraustrations, I think there is a problem with your second aim of examination. You cannot determine whether or not any individual will be a good lawyer (or economist or what not) at university. That is not the aim of a university education. The purpose of a university education is to provide a body of knowledge and a set of skills. Hopefully proficiency in those skills and knowledge will be highly correlated with later professional life (if studying a professional degree). Proficiency in those skills and knowledge is tested in a known and standardised way. That usually consists of an examination under severe time constraint. In a closed book exam students must know the law, in an open book exam they must know where to find the law. Students who make mistakes or who cannot manage their time well must be penalised.

    By all means let’s experiment with different assessment techniques but I think we are too defensive of the existing techniques that work well for their purpose. Testing knowledge in a particular way.

    Our business model is providing an education that equips for a career – we are not selling careers. So we cannot access whether they would be any good at that career.

    Anyway – just some thoughts. I finished marking last week. 🙂

  2. Peter Patton
    Posted November 17, 2010 at 7:37 am | Permalink

    LE

    Yeah, I agree with Sinc. I know that Law is probably the most angst-ridden of modern uni faculties when it comes to pedagogy and assessment. You – not just you, but Law schools in general – if you accepted you are not in the business of teaching fine lawyers, but graduates who know as much of the law and its incredible intricacies as one possibly can in 3 years. Relieve yourself of the burden of producing the lawyer with 5 years under their belt! As a Property lecturer, surely you want your graduates to be able to answer a real life property problem with a nuanced exposition of the law, and not draw breath for 60 minutes while they’re at it! 🙂

    Though I understand modern Law schools have cut back quite a bit on the fixed curriculum so you could spend your entire third year doing internships, writing Research papers on anything you want, mooting, and so on. So maybe what they are not getting in your Property lectures, they might be getting through these other channels.

    But LE, despite all the angst over assessment, do you – and your colleagues – feel that academic justice is ultimately done. Or does the idiosyncratic Law school exam system routinely reward dullards, while punishing the brilliant and hard working?

  3. conrad
    Posted November 17, 2010 at 8:01 am | Permalink

    I don’t like exams or spoken exams. The problem with the second is that too many people have high levels of anxiety, especially when they’re 19 years old. The problem with the first is that most students cram and hence that is a big factor in what you are measuring, as you have noted.

    I’m not entirely in agreement with Sinclair — I think it really depends on the subjects, and perhaps all law ones are too broad. In part of the course I teach on, there are one or two subjects that potentially equip students for a career, and the people that employ them are very happy we do them (it’s why students doing a particular major get good outcomes). If those students don’t like or can’t do the subject well, they won’t like the career.

  4. Jacques Chester
    Posted November 17, 2010 at 8:12 am | Permalink

    I had a previous stab at a Grand Reform Proposal, but I’m not sure how well it would map to law.

    In any subject, I suppose that there are different kinds of learning taking place.

    One is to develop fast, efficient recall of many small facts. This is ‘easy’ to do, if the facts are sufficiently discrete to allow learning through scheduled repetition. Legal ‘headnotes’ might fall into this category.

    Another is to develop comfort with the mental tools of a profession – its analytical methods. I was taught the IRAC method by law school. These can only be obtained through practice under the critical supervision of an experienced thinker.

    Then there is developing the particular supporting professional skills — drafting, negotiating, standing in court etc. I think most law schools nod in this direction.

  5. Posted November 17, 2010 at 9:13 am | Permalink

    I’m pretty happy with the legal education I’ve received, both in Australia and over here. Maybe there are better ways of assessing it, but I don’t know what they are, and I do think there is something to be said for a mixture of exams and assignments (which seems to be pretty universal). Realistically, all the exams should probably be open book (closer to practice), but I must admit I’ve always preferred closed book exams, as I tend to have an absolute advantage when it comes to detailed recall.

    On wider issues of course content and design, I do like that law requires the mastery of lots of information and the construction of logical, focussed arguments. I like its acceptance of human reasonableness, the possibility of objectivity, its argument that there are facts, and that they can be known.

    I keep reading that people are turning to law as a ‘foundation’ degree (if they have the results to get in), whereas in days gone by, it was the humanities. This is sad, but I can see why: (a) people like to learn useful stuff, and (b) if you’re a Westerner, then law is very much ‘western civ’ writ large.

  6. Posted November 17, 2010 at 11:05 am | Permalink

    Not on legal… but…. there are probably some things with orals that are subject-invariant.

    Oh hell… flashbacks to some horrible orals – 10 minutes per table, wildly varying styles by examiners. Some could send you into a panic either because they were the type of person who just wanted to fail everyone, or they (without intent) just put you off (the lack of “bedside manner” explaining why they turned away from clinical practice), or you’d had a run-in with them (“it’s giving off portal congestion and biliary cirrhosis type vibes – you know – it can’t possibly be anything else”).

    Some would dive right into the detail, with what looked like a predatory smile, pointing to what looks like a grubby bit of mangled wet string inside an obscurely prepared bit of body that could be from anywhere, anyway up, and you couldn’t tell if it was a nerve, blood vessel or sloppy bit of connective tissue without having to grab the probe from the examiner and check the consistency, as it could vary widely between people and preparation methods.

    Others would either help you orient yourself first, or if they didn’t, but got into trouble with an initial question, ask “ok, so do you see anything you do recognize? ok, so what are some of the things you’d expect to find nearby? ok, so what type of tissue is that? ok, so what is it called?”. Such examiners might or might not be hard markers, but were good at finding out whether you knew something or not.

    So you had the luck of the draw as to which table you went to first – get a nice examiner first, you were probably fine for the others. If the first was the type of person to send you into a panic, particularly if that table was your weak spot (“please, please, let my first table be the hand not the foot”), you could be a blubbering mess for the rest of the oral.

    And of course, when looking down a microscope and the examiner started with “OK… so what are those red whatchamacallits….”, and you went “ummmm…. what red things are you talking about…. I’m color blind”, “the little dark dots”, “ummmm, what do you mean by little, relative to what?”… panic!

  7. Patrick
    Posted November 17, 2010 at 1:49 pm | Permalink

    I like oral exams because they provide the best opportunity to actually test your understanding of the subject. I note this article which is acutely relevant to the topic!!

    A law degree which did not require yards of sweated-over written documents would not, however, be a law degree worthy of the name.

  8. Thomas Buckingham
    Posted November 17, 2010 at 2:37 pm | Permalink

    Having just finished my exams I feel particularly posed to comment. Some of the tutors at my University and I share the view that exams are highly contrived and unrealistic and are by no means the best way to judge a law student. They are, however, efficient. They are cheap and, when heavily weighted, tend to “sort the men from the boys.” But this does not vindicate there design principles, nor does it validate them as an effective means of grading students.

    Exams with problem questions – hypotheticals as you call them – promote short-term memory recollection over a nuanced application of knowledge to the factual scenario. They reward “exam technique” inasmuch as they reward ability. I have struggled developing my “exam technique” during my degree and I’m only finally (fourth year, no less) getting the hang of it. I have consistently entered exam periods with great marks only to come out of the semester with a C or a P.

    My primary concerns are threefold: first, as mentioned, exams test the wrong kinds of skill and knowledge. Second, they are contrived and unrealistic. These factors are exacerbated when the questions (hypotheticals) have been poorly conceived. Lecturers have admitted that some are designed in such a way that there it is close to impossible for the student to give complete answers to all the problem questions (they’re basically just forced to write for three hours straight); third, exams are quite often too heavily weighted. Most of mine are about 60%, anything over 65% is, imho, lazy and unfair (personally I’d prefer if there were cap of 50% on any single assessment item).

    An ancillary concern of mine is the hand-writing aspect. I have hand-writing that is legible enough to me and anyone who cares to read it in any usual context but it is far from perfect. Put me under exam conditions where I’m trying to relay as much information as possible constrained by time and limited resources and my hand-writing inevitably suffers. Some examiners claim that they will not mark illegible work and, while I’ve never had any formal complaints, I can’t help but wonder whether my writing has been detrimental to my marks in the past (it’s not like they grade and return exam papers, so I wouldn’t know).

    I do have one question: is it true that exam marks are often scaled?

  9. Peter Patton
    Posted November 17, 2010 at 3:01 pm | Permalink

    SL

    All my friends who did Combined Law at Sydney Uni back when every course was assessed by one 3 hour final say it was the best education of all their degrees by far in terms of cognitive value-added and sheer thoroughness. This was back when Sydney Uni was still on the Oxford system three terms per year, so these courses were a full year long not just 1 semester. The course was very humbling for a group of people who need a little humility at that stage. But for just about all of 3rd term they were at their desks 24/7.

    LE

    YOU would have hated it. They had to do a full year of compulsory Equity! 🙂

  10. Thomas Bland
    Posted November 17, 2010 at 5:47 pm | Permalink

    Hi Legal Eagle,

    I’m a JD student at MLS (hooray!), and just finished a thrilling set of exams.

    I agree that oral assessment is an excellent way to ensure that students can really grasp legal material. I moot a fair bit at law school, and one of my favourite aspects is that a successful oral argument requires one to really get the law ‘in’, in a way that writing a comprehensive set of notes does not.

    The point of this comment, however, is to mention that this semesters JD Constitutional Law class had an oral assessment in the form of a moot. Perhaps you could speak to some of the ConLaw faculty, to find out whether they thought the assessment was effective? Personally, I loved it, but then again, I’m a real law school/mooting nerd.

    Cheers, Tom Bland

  11. Hugivza
    Posted November 17, 2010 at 6:12 pm | Permalink

    Coming into this post rather late, and reading [email protected]’scomment a that “The purpose of a university education is to provide a body of knowledge and a set of skills,” I was reminded of the comment of a maths professor during my first year of engineering, who contrastingly stated that the purpose of a university was to “teach you how to think, not what to think.” The only skill to be developed was thinking, driven by a sense of curiosity for the subject matter. Perhaps he might have been a throw back to the days of natural scientists and philosophers where learning was for its own sake, and any vocational slant was frowned upon.

    Engineering by its very nature has a vocational perspective which is similar to the law, where the requirement to think and pull areas of knowledge into a coherent focus is fundamental to both. Engineering was taught as a discipline with a structure like an inverted cone: dealing with broad elementary but largely mathematical concepts initially, and then narrowing the scope and focus of these and applying mathematics as a tool to deal with more complex concepts in depth. When I studied law much later, I found that by and large it dealt with a broad range of concepts which were all at a similar level of complexity, save perhaps for the more esoteric areas like jurisprudence. Perhaps this is a consequence of the profession requiring I think eleven subjects as being fundamental requirements: equity, property, contract, torts etc which leaves little scope within the compass of a normal degree programme.

    Measuring the ultimate success of how students will perform in practice can only be by analogy: whether it is by oral or written exams, or set pieces. Whereas these seem fundamentally doomed to ultimate failure, there are probably not alternative effective measures, which despite their analogous shortcomings that universities can economically apply that will provide objective outcomes. I favour having a response to hypothetical scenarios which require detailed analysis and original thought over say a weekend, but whether these could be sufficiently broad year after year to test student performance may be wishful thinking on my part, particularly as I am not an academic. I personally have never found exams to be a problem as an event, but I can remember gifted students going into a complete nervous funk over exams and performing well below their inherent capabilties.

    I guess the answer is 42.

  12. Patrick
    Posted November 17, 2010 at 6:29 pm | Permalink

    PP, was that ironic? Although a year of equity is standard, I think LE might actually hate a year of compulsory equity at Sydney University.

  13. Posted November 17, 2010 at 6:57 pm | Permalink

    Thinking that because of later use and abuse of the 6-minute billable unit, the person who manages to wangle the longest extensions for overdue assignments ought to get /some/ form of brownie points at the end of the year.

  14. pete m
    Posted November 17, 2010 at 7:35 pm | Permalink

    Exams are a necessary evil. Took me a year to figure out how to pass them! I had nightmares for years after law school about missing an exam, or more recently, forgetting I had enrolled in another course! Talk about burned!!!

    There was an elder gentleman at my year who wrote slowly. He knew the law better than most lecturers due to his civil engineering background, but constantly failed to get all points on paper as he was too slow. He was kicked out for failing 2 years in a row. Sad, but in real life, lawyers don’t have time to spend 5 times longer on what another lawyer can produce, unless s/he is a judge 😉

    Good luck marking – it seems you have it down pat. Just don’t pigeon hole people. I found lecturers were too forgiving of poor efforts from their A students, while harder on their C students for the same work. I mentioned that to my brother when he started and encouraged him to work hard in his first year, which he did – then in 3rd year he had life issues and managed credits when he really barely should have passed. And this in a Justice course!

    Re whether uni equips one for practice – sorry, but it doesn’t even come close. The skills needed to practice law are too many – uni just teaches an outline of the law and examples of it in real life. I’m still learning despite 20 years of practice! Best advice I’ve received is re-read the legislation for each case, even if you think you know it.

    Re communication – I took it as an elective, but it seemed to be mostly focused on advertising, which sucked. Agree it is essential. Amazing how many people cannot explain the basics to lay people.

    The best education post uni I’ve found has simply been to read judgments as they are published across all areas, despite solely practising in 1 area. That and legislation as it changes!

    I also wish every lawyer was forced to spend their first 2 years in a general law firm (or in a larger firm but moved around different areas). The rush to specialise is misguided, as every area of law borrows from another.

  15. Posted November 17, 2010 at 8:21 pm | Permalink

    I’ve just had to let a great tribe of people out of the spammer, apologies for your comment not turning up earlier. Your first comment goes into automatic moderation so that we have some idea of who (or what) we’re letting in.

    You’ll be able to comment normally now.

  16. Patrick
    Posted November 18, 2010 at 5:24 am | Permalink

    Pete, whilst I agree about generalist early experience, how do you possibly avoid specialisation? I can’t read enough judgements in my own area let alone every other area!!

  17. Posted November 18, 2010 at 8:53 am | Permalink

    I’m presently on a committee at Sydney Uni that is reviewing assessment policy across the Uni. It is quite frightening, really, when you hear how assessment is being done in some disciplines. I’m not a great believer in managerial approaches to academic life, but I think there are situations where it’s important that staff understand policy and abide by it, for reasons of equity etc. Very few academics have any interest in or understanding of pedagogy, and thus haven’t really got good theoretical tools to plan their assessment. Sydney also has a lot of staff who are much more focussed on research than on teaching, and this leads to some odd, err, anomalies when it comes to assessing and reporting student progress.

    But I find that often younger staff are much more interested in talking about what they want students to learn and how they might test them, and this post is a good example of that kind of thinking.

  18. Posted November 18, 2010 at 7:32 pm | Permalink

    Everyone should read Patrick’s link @8. Even if it’s only 25% true, it’s absolutely terrifying, and makes me think that law’s insistence on exams under time pressure where a portion of the marks are allocated for accurate expression is the way to go.

  19. Posted November 19, 2010 at 7:55 am | Permalink

    Two more links of different forms of cheating:
    1 an appeal to student’s sense of ethics
    2 a really scarey site advertising ‘ghost writing’

  20. pete m
    Posted November 19, 2010 at 5:58 pm | Permalink

    Patrick – I did not say don’t ever specialise – just avoid it for a couple of years.

    Wish I had a ghost writer, as I sucked at assignments – sigh!

  21. Posted June 9, 2011 at 10:38 pm | Permalink

    The oral exam is an interesting idea if as you said you have a topic with a week to research it. I think the capacity for a lawyer to test their knowledge in context is an important confidence booster and so, I’d have to go with a debate type environment on a more regular basis.

    In the bigger scheme of things though, I am an absolute oponent to the use of exams, I feel they’re really an inaccurate picture of what people know.

    So far as testing how students react under pressure, conditioning against feeling pressure would be more productive, surely? Healthier in the long-run too, if what they say about the rates of depression among lawyers is right?

4 Trackbacks

  1. […] This post was mentioned on Twitter by John Hacking, Legal Eagle. Legal Eagle said: It’s that time of year…: …when I wonder yet again whether exams are really the best way of judging students’ kno… http://bit.ly/aJkCDO […]

  2. By Skepticlawyer » When there is demand… on November 18, 2010 at 9:45 pm

    […] Legal Eagle’s thread on alternative forms of assessment, Patrick dropped a link in the comments thread that exposed the extent of a problem I’ve long known […]

  3. By Rafe’s Roundup 19 Nov 2010 at Catallaxy Files on November 19, 2010 at 2:53 pm

    […] So be sure to check out Don’s Missing Link, even if you only read Skepticlawyer and Legal Eagle. […]

  4. By Skepticlawyer » Marking exams… on June 9, 2011 at 8:17 am

    […] …so if I’m grumpy, don’t be surprised. As I always do at this time of year, I am wondering yet again if there’s a better way of testing people’s knowledge. […]

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*