bright
bright
bright
bright
bright
tl
 
tr
lunder

Language Testing Review of the Year 2015

This site designed and maintained by
Prof. Glenn Fulcher

@languagetesting.info
runder
lcunder
rcunder
lnavl
Navigation
lnavr
navtop
lnavs
rnavs
lnavs
rnavs
bottomnav
lnavl
Tools
lnavr
navtop
lnavs
rnavs
lnavs
rnavs
bottomnav
 
 

 

 
     
 

Each year since I started doing the yearly reviews I've always been able to begin and end with something highly amusing. And perhaps include some very light hearted pieces between the serious news items. This year I struggled a little. 2015 seems to have been a fairly dark and serious year. A bit like watching one of the early episodes of Dr. Who with Peter Capaldi after the geniality of Matt Smith. Takes a bit of getting used to. The year has been dominated by quite vitriolic debates around standardized testing, league tables and accountability, with a heavy dose of immigration issues, particularly for health professionals. But as you read on, you will see that there have been some lighter moments as well. At least the satirical Now Show returned to the topic of accountability, and there was John Oliver's unforgettable performance on Last Week Tonight. I do hope that you enjoy my selection of stories for the 2015 language testing review of the year.

Standardized Testing. One theme that ran through the testing year was the seemingly endless conflicts that rage over standardized testing. As you'll see below, this emerges in the big issues of finance, cheating, and international migration. So it's not surprising that important figures in linguistics get involved; or that testing becomes a topic for prime-time television. In this YouTube video Noam Chomsky gives his views on the detrimental impact of over-testing in Schools and its impact on learning and teaching. The basic message is that testing is not the same as learning and teaching, and that it can impede (if not destroy) creativity. This is actually an argument that most language testers (but not necessarily psychometricians) have agreed with for many years. The study of washback in particular has taught us that if we are going to use tests in learning contexts we need to design for intended impact, or engage in what Fred Davidson and I call effect-driven testing. So I'm rather ambivalent about the simplicity of such attacks on standardized testing; but when testing is done badly - as it so often is - it is impossible to defend against this kind of criticism.

The Miami Herald published two articles on standardized testing that were a cut above what I normally expect to see in the popular press. Laurie Futterman published Beyond the Classroom: The history and challenges of standardized testing on 4th March, and Beyond the Classroom: Perspectives on and the future of standardized testing on 11th March. These highly intelligent pieces draw on research and assessment literature to critique the overuse of standardized testing in the United States. But the lessons are relevant well beyond those shores. She covers the history and economics of testing, unaccountability of private testing companies, effect on instruction, cheating, cultural bias, and data use, among other topics. So much is achieved in two short articles. Great food for thought. The video covers some of the same ground. This is John Oliver's performance on Last Week Tonight, aired on the US channel HBO in May. As the Huffington Post put it, this was a comedian pointing out everything that is wrong with standardized testing; even taking the risk of litigation by naming some of the most powerful testing service providers. Within days a Pearson spokesman responded with a defence of the company in the Washington Post, which was discussed endlessly on social media. This programme is amusing, but also very serious. It didn't pull any punches in raising the issues of financial gain and monopolies, particularly when the same company produces the tests, the test preparation materials, and may even run training courses for teachers who prepare learners for the test. Oliver also refers to the quality of test materials, and illustrates his point with reference to the Pineapple text. You may recall that I discussed the Pineapple incident in the Review of 2012, and you may wish to go back there, download the questions, and see the related video. Finally, you'll note Oliver's criticism that no one is allowed to comment on test questions, on the grounds of "security". This year Pearson was also criticised for taking "security" to new levels, by "snooping" around social media and reporting children who were commenting on their tests to the Department of Education. As the Huffington Post pointed out, there's a fine line between trying to maintain test security, spying, and just protecting your reputation when poor items have been used in live testing. And of course, there's always the right to protest too, like the sabotage undertaken by opponents of the Common Core Tests by deliberately revealing questions on Facebook! What subversion! It could even bring down Western civilization as we know it.

Keeping the Lawyers in Business. So here we develop the theme raised in the last section. As Oliver pointed out, there is a lot of money involved in high-stakes testing. We learned in May that California had awarded the state testing programme to ETS, which was contested by Pearson. The contract was worth a staggering $240,000,000. Education Week reported that Pearson complained the award of the contract was "invalid and illegal". Then came the news that Pearson had lost both the $500 million contract to provide testing services to Texas, and a New York contract. The sums of money involved, with all the collateral sales of textbooks and training services, does raise the issue of the primary purpose of the testing companies, which Oliver is correct to question.

There is one other legal story that is worth reporting. An IELTS candidate in India took action against the British Council because they had not sent her certificate to Universities in time for her application to be processed. She claimed that she had suffered mental agony as a result. When taken to court, the British council claimed that being a division of the British High Commission in India, it enjoyed the status of a diplomatic mission, and thus could not be subjected to the court's jurisdiction. They lost the case, and rightly so. As a co-owner of the test it would have been a travesty of justice if they could have avoided responsibility by claiming diplomatic immunity, and the attempt to get away with it beggars belief.

Big Business. Two fascinating financial stories hit the press in 2015. The first, and arguably the most interesting, was the decision of IDP Australia to float shares on the Stock Market. IELTS is jointly owned by Cambridge Assessment, the British Council, and IDP. IDP owns 50% of its share, with the other 50% owned by Australian Universities. But 50% of IDP is owned by SEEK, an online job agency. SEEK is now planning to sell off its share in IDP to concentrate on its employment business. Confused? Just like the last story, take a look at the profits in this story from the Sydney Herald. The other story worth telling is the rapid expansion of test prep start-up companies, many of which are launching apps for learners to practice test questions on their mobile devices. The value of these companies is quite astonishing. Over the years I've made a study of the satellite industries created by high-stakes testing, from the marriage agencies in India (select your IELTS girl) to online preparation. The expansion of uncontrolled testing related business never seems to slow down.

Internet Use? Regular readers may remember that a few years ago I wrote a feature on internet access during tests. Denmark is still the only country to allow this in high stakes tests because of the fear of what test takers will get up to if they can access information outside the test engine. This year the Oxford and Cambridge Examination Board suggested that access should be considered in the UK. There was an immediate reaction against the idea. In this extract from the Today Programme you can see just how animated some people can become about the prospect of "dumbing down" standards. Note also the comparison between countries (using test data from PISA) to object to such an innovation. So many assumptions, so little thought! What surprises me is that there has been no published research (that I am aware of) into learners' use of the internet in a testing context.

Cheating - Yet Again! Yes, I know. This is a perennial story. So I'll keep it shorter than usual. In 2015 China started using drones (picture right) to monitor test centres and intercept illegal electronic communications. As The Daily Mail put it, "The devices will use 360 degree rotations to scan testing halls and locate suspicious radio signals created by hidden earpieces used to obtain the answers to exam questions." This is definitely what is needed in India. The video on the right is a clip from the BBC news that demonstrates some of the "cheating devices" that are widely available for sale - not only in India, but around the world. In Egypt this year, for example, the Cairo Post reported the use of metal detectors to stop students sneaking cheating devices into exam halls; and in China the problem of "ghosting" has led to a growing number of fraud cases. Many countries make a big deal out of combating cheating now that language tests are used as a surrogate for immigration policy, and this kind of response to parliamentary questions shows just how little politicians understand about the extent and nature of the international cheating industry. I've made this point many times before, but it's still worth repeating. Finally in this section, in 2013 I reported on the Atlanta cheating scandal. In 2015 The New York Times reported on the end of the trial, educators were duly convicted, and newspapers like The Guardian began to analyse the implications for the state of US education. And Georgia State University studied the effects of the scandal on stakeholders. This seems to wrap up that particular story. But it's probably just a matter of time before the next one starts.


Confiscated Electronic Cheating Glasses

A Cheating Vest

League Tables and Averages. The reason for very sad stories like that from Atlanta is in fact quite simple. If you start evaluating teacher performance by learner achievement on tests, and link evaluations to pay and career prospects (including being fired), there comes a point where it's better to help test takers cheat than face the consequences. Add to that institutional accountability by test scores alone, and there's a real incentive for poor behaviour. And in many countries individual institutions are named and shamed using the ubiquitous league table. In the following audio extract from The Now Show the UK government is subjected to much deserved satire following stupid pronouncements like "all schools should be above average." Again, this is driven by international economic comparisons using yet more tests, like PISA. Anyway, this is absolutely brilliant satire. Enjoy!


Immigration. In the run-up to the UK general election immigration and the ability to speak English became a significant issue for the general public. In this video the Mayor of London, Boris Johnson, addresses the question of language ability in the public services, and particularly the National Health Service (NHS). In fact, the public became even more concerned than usual, and the tabloids even ran the story Foreign vets from EU who can't speak English putting pets at risk by working in Britain. Putting pets at risk! Perhaps they should have run the headline Language tests can save your cat! But it is all above board and legal, as The European Court of Justice ruled that language tests could be used for immigration, integration, and citizenship. The UK had introduced a law forcing the spouses of UK citizens to pass a language test before they could relocate to the UK, which I reported on in the 2011 Review of the Year. The first case brought under this law was referred to the Supreme Court, and the human rights arguments rehearsed in papers like The Daily Mail and The Daily Telegraph. The question is whether language testing in this context breaches Article 8 of the Convention on Human Rights, which guarantees the right to a family life. The challenge was rejected by the Supreme Court, meaning that language tests can still be used for this purpose. The ethics of this are more complex than the popular press can be bothered to engage with. The judgments themselves deal with questions of language and identity, protected characteristics, and the meaning of marriage. The legal documents are well worth engaging with from a language testing perspective, as they raise many fundamental philosophical questions about what we think we're doing, and why. In the meantime, we will see more stories like this one from the Inverness Courier, in which a Scottish man is separated from his Venezuelan wife because she has not passed a government recognised language test.

Button Poetry. As so much of the year has been about standardized testing, its use and misuse, I decided to end my review with a YouTube video of Olivia Fantini reading her poem On Standardized Testing. This is very moving, and when I first heard it, it reminded me of the girl in tears on the John Oliver video above. You can get the words to the poem by following the link. And this allows me to finish with a message that I've used before at the end of my annual review. But it's one that should be repeated again and again until it gets through to the politicians and all those who think that education is improved through the imposition of dumb accountability systems. Poor tests and testing policy harm good educational practice; and useful testing should not be done so frequently that it interferes with quality learning and teaching. It's really the same point Chomsky makes in his YouTube video. With the caveat that I believe language testing is too important to do badly.

Glenn Fulcher
December 2015