There’s no such thing as an objective test, only a reliable one

Written by  Femke Kools Tuesday, 17 April 2007 00:00

Progressive research into information-rich testing programmes

How do you judge whether a medical student is or is going to be a good doctor? This question will be explored by Professor Lambert Schuwirth, who was appointed chair of ‘Education and research into innovative forms of testing’ at the Education Development and Research Department of the Faculty of Health, Medicine and Life Sciences on 1 March 2007. Current testing methods often fall short when this consideration is made. New testing programmes and models will therefore have to be developed. Schuwirth hopes to build these in the coming years. If he succeeds, this will have far-reaching consequences, not only for the way in which medicine is taught, but for education in general.

In its role as a teaching university, Maastricht University (UM) has a good name to keep up in the field of educational innovation. As far as research into education and testing is concerned the educational institute of the former Faculty of Medicine can rightfully be called one of the top institutes worldwide. Lambert Schuwirth’s research into innovative forms of testing will contribute to UM’s pioneering role, as he plans to develop a set of instruments that will enable lecturers to effectively judge whether someone is going to become a good doctor.

Why are these instruments not yet in place? According to Schuwirth because formal testing institutes dominate the way of thinking on the subject of testing too much. They treat tests mainly as the measuring of stable characteristics. ‘It is a view based on test psychology, in which it is assumed that the items that you have to test, such as knowledge and skills, are stable characteristics – like intelligence, for instance – and that they can only be measured properly and expressed in grades. An IQ test is an example of a measurement based on this way of thinking.’ This approach, however – the psychometric model – seems increasingly inadequate. ‘To start off with, when comparable tests are performed consecutively, this model interprets variations in scores mainly as measurement errors (after all, they are assuming that a stable characteristic is being measured). A great amount of variation in scores is subsequently wrongly dismissed as a measurement error. Another problem is that this model takes too little account of the fact that various aspects of medical competence do not develop separately and therefore cannot be measured separately. We should really be looking at how these relate to each other. At the moment we’re just adding up the credits from all the elements students are tested on. It’s actually absurd! That’s not the way to determine whether someone can become a good doctor.’ The greatest misconception in Schuwirth’s view is that tests are believed to have be objective and that information from a test should be reduced to a grade. This belief in objectivity has led to a fruitless search for the ultimate test. ‘The judgments we pass, the things we think a good doctor ought to know and be able to do, are always subjective, testing therefore is too.’ This doesn’t mean that tests and lecturers’ judgments cannot be reliable. On the contrary, they must be. ‘To make tests or judgments reliable, we need to gather even more information. Random testing is more important than being objective. If many experts arrive at the same subjective judgment independently, then that judgment is not objective, but it is reliable.’

The tragedy of these misconceptions is that much information is lost because of the current way of testing: information about the exact strengths and weaknesses of individual students’. Identifying these strengths and weaknesses is necessary to be able to give these students good guidance. Schuwirth is therefore pleading for an information-rich testing method and for the development of a balanced testing programme rather than a single ultimate test. ‘We shouldn’t want to express everything as a score, but rather develop a way of testing that provides lecturers with information that shows what a medical student has yet to learn and enables them to judge whether someone can become a good doctor. We should start qualifying and individualizing more. This may mean that two students get the same grade but that they get different feedback. When developing this testing programme it is also important that we take note of the fact that matters such as knowledge and skills are not stable characteristics.’

To be able to achieve this, Schuwirth will have to develop design criteria to guarantee the quality of the testing programme. In addition, as a supplement to existing models, new statistical and scientific models will have to be designed that will allow reliability and applicability of testing to be determined. These developments may have wide implications for the way of thinking in other educational institutes, quite apart from the Faculty of Health, Medicine and Life Sciences. People in other study programmes are also aware of the limitations of applying the psychometric model too one-sidedly. The effect can already be seen in the former medical faculty, where the soon to start four-year study programme will put an end to testing in the traditional sense of the word.

Professor Lambert Schuwirth is employed by the Educational Development and Educational Research department of the Faculty of Health, Medicine and Life Sciences. On 1 March 2007 he was appointed professor holding the chair of 'Education and research into innovative forms of testing’. Contact: +31 43 388 5731 or This email address is being protected from spambots. You need JavaScript enabled to view it.

Rate this item
(0 votes)
Read 33009 times
You are here: Home Education There’s no such thing as an objective test, only a reliable one

Maastricht University Webmagazine

Marketing & Communications
Postbus 616, 6200 MD Maastricht
Minderbroedersberg 4-6, 6211 KL  Maastricht
Tel: +31 43 388 5222
This email address is being protected from spambots. You need JavaScript enabled to view it.

Download UM Magazine

Read a digital version of Maastricht University magazine,
or download the PDF.
UM Webmagazine June 2017

Connect with us