Despite its long history, test preparation never fails to grab the attention of the media. In 2009 it has hit the headlines a number of times and then, inevitably, politicans become involved. Below there are links to clips from the radio programme that broke one of the major stories of the year in the UK, the 'undercover data', and the response of one satirical programme. But first of all we try to put test preparation into a historical context to show why it has always been a common educational practice. The main question we consider is why the test prepatation techniques in this particular news story violate our sense of what is 'fair'.
A little bit of history
The Chinese Imperial examination system was put in place during the time of Sui emperors (589 - 618 CE), and it did not come to an end until 1911. The purpose was to select the very best individuals for the Civil Service, and preparation for the various tests was exceptionally rigorous. And since the consequences of the testing were so serious for the individual and the reputation of the teachers, it was inevitable that "test prep" methods were devised. Here is an extended quotation from Miyazaki (1981: 17) that illustrates the extent of test preparation, and how the authorities reacted to it.
Despite repeated official and private injunctions to study the Four Books and Five Classics honestly, rapid-study methods were devised for the sole purpose of preparing candidates for the examinations. Because not very many places in the classics were suitable as subjects for examination questions, similar passages and problems were often repeated. Aware of this, publishers compiled collections of examination answers, and a candidate who, relying on these publications, guessed successfully during the course of his own examination could obtain a good rating without having worked very hard....Reports from perturbed officials caused the government to issue frequent prohibitions of the publication of such collections of model answers, but since it was a profitable business with a steady demand, ways of issuing them surreptitiously were arranged, and time and again the prohibitions rapidly became mere empty formalities.
It was also acknowledged that test preparation was pretty boring, and distracted learners from the real task of learning. Nevertheless, test preparation is endemic wherever the scores determine personal progression and success, in education or work. In the 19th Century the practice came to be called "cramming" - a term that we still use today. The famous statistician Karl Pearson wrote a biography of Sir Francis Galton, who was one of the first 'scientific' testers. He reports that in 1889 Galton had argued the best way to avoid learners cramming is to have lots of tests, vary the content as much as possible, and ensure that what is tested cannot be easily memorised. Sounds quite modern, doesn't it? If you wish to read this, Pearson's Life, Letters and Labours of Francis Galton is now available online.
Listen to the radio programme
The issue at stake is what happens when test preparation practices become so extreme that they may constitute 'cheating', or 'malpractice'. And when the test designers haven't taken the kinds of measures recommended by Galton.
This is a link to The Investigators, originally broadcast on Five Live, 19th April, 2009. The programme considers a training programme offered to GCSE language teachers in the United Kingdom, which consists entirely of test preparation.
[Before listening: If you wish you can download and print out focus questions which structure the programme and help you to focus on salient points as you listen]
Why spend so much time on test prep?
Latham (1877) argues that in any system driven by high stakes tests, teaching will be affected by the test. What we now call 'washback'. This is not always a bad thing in his view. But he also recognizes that it can lead to practices that are questionable. Talking about langauge testing in particular, he says:
"The tutor must consider not what studies or what kind of teaching will do him [the learner] most good, but what studies will yield the highest aggregate in the given time, and he must teach his pupil each subject not with a view to call out his intelligence, but with a view to producing the greatest show on a stated day; for instance he must teach him a language by some sort of Ollendorff process, which shall address itself to the ear and the memory, rather than by a method which involves any grammatical analysis" (Latham, 1877: 5 - 6).
Although Latham would not agree with using the kinds of techniques described in the radio programme, there is a clear similarity in argument here. We can understand why some of the more extreme practices might happen.
Ethical test preparation?
Popper (1991) argued what has now become the accepted position of the educational community and testing professionals: "No test-preparation practice should increase students' test scores without simultaneously increasing student mastery of the content domain tested." He went on to identify five types of test preparation practice, as follows:
Previous forms preparation
Current form preparation
Generalized test taking preparation
Same format preparation
Varied format preparation
Of these, Popper argues that (1) and (2) violate the principle of ethical test preparation practices because both are likely to alter scores without a concomitant increase in ability on what the test is likely to measure. He also argues that (4) is educationally unsound on the grounds that it would limit the item/task types that students would encounter during a programme of study, and hence limit the generalizability of their knowledge and skills. (3) is acceptable because some general preparation regarding item types that might be encountered, timing, and basic test taking strategies, are likely to reduce method effects, thus increasing the likelihood that the test scores will more accurately reflect ability on the construct.
Make your mind up time
Here is a link to a short clip from the data gathered by the investigative journalist in the programme that you have just listened to. She recorded this secretly, so the quality is grainy. However, the audio is reasonable.
What do you think of the advice being given to teachers?
Can anyone be blamed for the use of these kinds of test preparation techniques?
If you were in a position of power over the examination system, what (if any) action would you consider taking?
And what of the journalists?
The media seem to have put their finger on the point; which is why the words 'cheating' and 'malpractice' come up in the discussion. For the media it's about someone getting a test score that doesn't represent what the test taler can actually do, or what they really know. The journalists' instinct is good. For the link between a test score and its meaning is essentially what 'validity' is all about. To the media these practices just seem bizarre.
So I end with this excerpt from the Now Show from 24th April 2009, which documents some of the other weird and wonderful practices to have caught the attention of the media.
Thinking and doing
What test preparation practices have you used (if you are a teacher), or experienced (if you are a language learner)? Do you think any of these techniques work? Why?
And a final suggestion for a group activity. Go to any search engine and type in test preparation. You will be swamped by offers from companies to get you good grades on tests. Most of them, I guarantee, will focus on the market for English as a foreign/second language tests. Many of these will be reputable, but some may not be. If you're working with friends or colleagues, select two or three of these sites and conduct a critical analysis of what they claim to be able to provide. Are any of the practices on offer questionable? What evidence (other than the fact they've been in the business for a long time - which isn't evidence) do they provide to support whatever claims they make?
Web Resources
Does the International Language Testing Association Code of Ethics help you to decide whether the various test preparation practices discussed in this Feature are ethical?
Exam Row Hits Gordonstoun. A teacher at the elite public school is suspended for the test preparation practices described in the programme.
Do whatever it takes. A report into 'assistance' that is often seen as bordering on cheating, from the Times Educational Supplement. Which of these practices do you think are acceptable, and which questionable?
Also see the other Feature on High Stakes Testing, which shows the pressure that learners are sometimes under to perform well on tests.
References
Latham, H. (1877). On the Action of Examinations Considered as a Means of Selection Cambridge: Deighton, Bell. [Page numbers in this Feature refer to the first US edition, which is the copy I have on my shelves. This was published in 1886 in Boston by Willard Small. However, I assume that the page numbers are the same as the 1877 UK edition.]
Miyazaki, I. (1981). China's Examination Hell. The Civil Service Examinations of Imperial China. New Haven and London: Yale University Press.
Popham, W. J. (1991). Appropriateness of Teachers' Test-Preparation Practices. Educational Measurement: Issues and Practices 10(4), 12 - 15.
