|
|
| |
The Concept of Test Architecture |
|
| |
|
|
| |
This is the first of a two-part feature. The second part is available here.
Test Architecture is an approach to test design that treats tests as evolving, organic entities. We developed the concept of test architecture (Fulcher, 2006; Fulcher and Davidson, 2009) as a way to conceptualise and deal with the notion of change in testing. Change is inevitable. But when change is unplanned and does not respect test users, the impact on users can be negative. The notion of test architecture provides a framework for considering change. In this first of a two-part feature we introduce the concept and explain how language testing is similar to architectural design. Below there are a number of thought exercises to help get you into the metaphor.
Some people and institutions who design a test and roll it out for operational use, consider it a 'finished product'. There is a mistaken belief in some quarters that test design happens and then comes to an end. But this is not the case. Just like buildings tests change to suit the needs of their users; and just like good architects language testers should be aware of how their creations can grow to be better than they originally were.
Watch this YouTube video by Steward Brand, based on the book How Buildings Learn: What happens after they're built. As you are watching, consider some of the following questions that bring out parallels between architecture and the practice of language test design and maintenance.
1. At the start of the video Brand says that "Architectural styles came and went. Planning theories came and went." What are the main "architectural styles" or "planning theories" that we have experienced in language testing? Can we see traces of these styles or theories in our existing tests?
Like buildings, we can design tests that have serious faults or are useless for the people who are going to use them because their effect on users is not taken into account. I have argued that "Like architects, test designers imagine the intended effect they wish the test to have, and design with the intended effect in mind" (Fulcher, 2006; Fulcher & Davidson, 2007; Fulcher & Davidson, 2009; Fulcher, 2010). The key to effective design is therefore to be clear about the intended users, the test purpose, and the kinds of decisions that are to be made using test scores, just as architects need to know who will use a building, what they will do there (functions), and what they are expected to achieve.
2. Look at the test(s) with which you work. Is there a clear statement of who is going to take the test? What the purpose of function of the test is (including domains to which we wish to predict)? Who the score users are, and what decisions they will make based on scores? Next, look at the test itself. Can you identify particular features that can be clearly linked to the statement of test purpose?
Brand discusses the problems that are frequently faced in working environments, including (but not limited to):
- space
- security
- air conditioning
- window cleaning
- signage
There are parallels to many of these in language testing. Test security is probably the most obvious.
3. Try to think of a serious threat to security that could be created through test design decisions (which may also impact on how the test is administered or delivered)? Let's take another one - signage. What kinds of problems may arise in test design with regard to the instructions to the students about what to do? How may we limit the chances that design errors like these are made?
The materials that are used in construction can also be a problem. Modern architects are fond of using glass because of its weight and the stunning impressions glass facades can make. But glass is often inconvenient for many reasons, including problems with wind, heat, light and reflection. Another issue raised is the maintenance of building, and its cost.
4. What are the materials of choice for the test(s) you work on? (Items, tasks, technologies)
5. Are the tests easily maintained? Consider, for example, sourcing reading texts, constructing listening materials, developing parallel prompts that do not invite rehearsed responses....add another couple to this list!
 Last but not least, buildings and tests evolve. They change over time. The example in the video is the Palazzo Pubblico in Sienna. The same is true of tests. Testing agencies are constantly trying to make improvements to tests. Hopefully the tests improve with time, but we can also make them worse.
6. What motivates test evolution?
7. I call changes to tests "retrofits", which is the term used in architecture. We usually retrofit buildings to make them more usable or fit for purpose. Have you instigated or implemented a test retrofit in the last year? If yes, what did you expect the retrofit to achieve? How did you evaluate it?
In the second part of this feature - available later in 2012 - we will look a little more at test evolution, how and why it happens, and how we can manage the process to avoid unnecessary harm to test takers.
References and Further Reading
Brand, S. (1994). How buildings learn: What happens after they're built.
New York: Penguin.
Fulcher, G. (2006). Test Architecture. Foreign Language Education Research, 9, 1 - 22.
Fulcher, G. (2010). Practical Language Testing. London: Hodder Education.
Fulcher, G. (forthcoming). Test Design and Retrofit. In Chapelle, C. (Ed.) The Encyclopedia of Applied Linguistics. London: Blackwell.
Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment. London and New York: Routledge.
Fulcher, G. and Davidson, F. (2009). Test Architecture, Test Retrofit. Language Testing 26, 1, 123 - 144
Read the second part of this feature.
Glenn Fulcher
May 2012
|
|
| |
|
|
|
|
| |
| |
|
| |
|
| |
|
|
| |
