Can multiple-choice questions be used effectively for assessment in any academic subject? Having worked mainly in arts and humanities, I admit I’d never seriously considered this. But what with one thing and another over the last couple of years (!) I found myself grappling with this exact question about … errr… questions.
Recent moves by universities towards more blended and online learning contexts have necessitated more consideration of online assessment and the trickle down consequences this might entail. For the managerially oriented, this offers the intoxicating waft of increased efficiency and even automated marking in the virtual air. You can almost sense the technology vendors circling, pitching Martin Weller’s Silicon Valley Narrative on how education is broken, and only private sector solutions can fix it. Check out Weller’s ed tech pitch generator to get inside knowledge on the next big ed tech tornado coming our way!
So it was with some chastened surprise that I learned about the nuance, challenge, and even validity (insert shocked emoji if required) that MCQs can offer, according to a range of very credible advocates. I started with Phil Race’s Lecturers Toolkit, which lists a host of advantages, many perhaps unsurprising:
- Reliability
- Time efficiency
- Availability of multiple-choice and multiple response options
Other strengths of MCQs were, to me, less obvious. As mentioned above, Race contends that good question items can be meaningful and valid – they test what you want them to – whilst also covering a greater extent of the syllabus (Race, 2015, p. 61). Having close correspondence to much real world decision-making was a further benefit listed. If well-constructed MCQs can be combined with other forms of assessment, it seems even greater validity, syllabus coverage and efficiency might be possible.
At this point, a request for advice on the SEDA mailing list produced the usual very generous and well-informed response. Colleagues in the fields of science, medicine, agriculture and educational development (among others*) have been producing great work in this area for some time. Of course, the community produced the same answer to my question as to virtually any in HE: ‘it depends … [how deep you want to go?]’.
Dr David Smith at Sheffield Hallam devises unGoogleable exam questions and gives tips and resources for teasing out those higher order thinking skills including “the ability to apply, analyse and evaluate information”. Rebecca McCarter and Dr Janette Myers note that single best answer questions (select the ‘most correct’ option) can test application of concepts over simple recall, whilst Peter Grossman uses degree of difficulty estimation (“think scoring of Olympic diving”). His idea of setting in-class tasks for students to construct their own questions is also valuable for a range of educational reasons. Colleagues Linda Sullivan (MTU Cork) and Ruth Brown (SEDA consultant) added further nuance by highlighting how we can ask students to rank or confidence-weight their responses. They also link to psychological research by Butler (2018) outlining 5 best practices in MCQ construction. It might, however, be challenging to balance Butler’s overall recommendation for simple item formats with the more nuanced demands of confidence weightings, rankings and explanations of answers discussed above.
This tension between simplicity and effectiveness gets to the heart of the issue on writing MCQs, and to the conclusion of my brief research into this area. Doing this well will take substantial time, expertise and input from a range of stakeholders. The usual suspects will be required: lecturers, educational developer, learning technologist, student voice, quality and standards and more I’m sure. Good items take time to produce; banks of items much longer. Quality control and piloting is essential – I was amazed how many ways there are to get this stuff obviously wrong, in ways which aren’t obvious during question construction. Colleagues from Harper Adams highlighted the analysis techniques needed to assess both the difficulty of items and the extent to which each one discriminates student level, correlating student scores for each question to the overall assessment mark. Fascinating, but tricky and time-consuming. So, MCQs – better than I thought as a tool, even more challenging to make than I realised. Reassuringly, Butler (2018, p.323) notes that aside from simply measuring things, MCQs can “cause learning”. Phew. I’ll try that line next time someone asks what I do: “I cause learning”. I wonder if the OfS will accept that as evidence when they next come knocking?
Steve White has been lurking in teaching, learning and research-related third spaces in HE for about 20 years. Most relevant for this article, he’s dabbled in online materials and test item writing for Oxford University Press. He worked in intriguingly ill-defined roles while developing online MA courses and MOOCs for the University of Southampton, leading him to complete PhD research on the third space in HE. More recent roles have straddled Learning Development and Educational Development at Arts University Bournemouth and Southampton.
*Many thanks to contributors to the discussion on MCQs: Ruth Brown, Dr David Smith, Peter Grossman, Dr Janette Myers, Clare Davies, and colleagues at Harper Adams. My apologies if I’ve missed anyone out – I’ve recently changed employer so lost access to some emails.
References
Here’s a list of resources and references I received from the SEDA community, including a number of items suggested by Clara Davies:
Burton, R. F. 2005. Multiple-choice and true/false tests: myths and misapprehensions. Assessment and Evaluation in Higher Education. 30 (1), 65 – 72
Butler, A. C. (2018). Multiple-Choice Testing in Education: Are the Best Practices for Assessment Also Good for Learning? Journal of Applied Research in Memory and Cognition, 7(3), 323–331.
Case, S.M. & Swanson, D.B, 2002. Constructing Written Test Questions For the Basic and Clinical Sciences
Gronlund, N.E. 1991. How to Construct Achievement Tests. Allyn Bacon, 4th Edition.
Race, P. (2020). The Lecturer’s Toolkit (5th ed.). Routledge. (Ch.2) Race, P. (n.d.) Designing Multiple-Choice Questions. Phil Race: Assessment, learning and teaching in higher education.
thesedablog 