Test Architecture, Design and Retrofit II

This site designed and maintained by
Dr Glenn Fulcher


Test Evolution

This is the second part of a two-part feature. The first part is available here.

Brand talks about the six design considerations in architecture. The site is the place and foundation on which a building is constructed. In language testing, the parallel is the model of language ability upon which the test is based. These change little over time, but some plate movement does take place occasionally. The structure is the test purpose, constructs of interest, and discourse domains relevant to the population for whom the test is to be built. The testing equivalent of the skin is the delivery model and presentation model, which tell us how the test is delivered to the test takers, and what it will look like. The space plan tells us what we are going to get in the test, in what order, how much of it there will be, and how it will fit into the time available. The "stuff" is the test tasks and items that we fit into the space plan. Finally, the services of a building give it usability. In language testing this is the validation argument that states what evidence and theoretical rationales make the test usable for its intended purposes. As tests evolve we can change elements in each of these design layers. But as you can imagine, changing the site is extremely difficult. It often means starting again, but as in architecture some extensions are possible. Similarly, changes to structure are quite rare, but can and do take place. The other levels, to various degrees, can be changed more frequently. A test can be made longer (space plan). A computerized version can be developed (a change in skin). New item types may be developed in order to better represent a construct (a change in stuff).

This is a short continuation of the video by Stewart Brand. In this video he talks about how buildings evolve. You will recognize the different layers of change, each of which moves at a different speed: site, structure, skin, space plan, services and stuff. He shows how buildings reveal their history through changes in different layers at different times. One of the interviewees says of the historic building in which he works that if it did not evolve "in a few years we would be in a museum". The same is true of tests.

Watch the video and reflect on these questions. As with the suggested tasks in the first feature on test architecture, you will get more out of these thought-exercises if you do them with colleagues.

1. Identify parts from a test you work with that date from different eras. If you find this difficult you could work with the CPE and consult Weir, C. and Milanovic, M. (Eds.) (2003). Continuity and Innovation: Revising the Cambridge Proficiency in English Examination 1913 - 2002. Cambridge: Cambridge University Press. The test as you see it today reflects Brand's view that the real architect of the test is time!

2. In a test you currently work with, what would you change (if you could) to bring it up to date?

3. Look again at the changes you have listed in 1 & 2. Relate these to the layers. Are the changes alterations to the site, the structure, the skin, the services, space plan, or stuff?

4. What does this tell us about the nature and speed of change in language testing? Which of the changes you have listed represent what the video refers to as "originality and creativity", and which represent "continuity and restraint"? And perhaps most importantly of all, do you think that these are determined by the users or the test designers?

5. The video discusses what I have termed "change retrofit" with reference to churches and banks. Imagine a new user group for a test that you work on. What would you need to change and do to make it usable by that group and the score users? Or would the effort involved justify the design of a new test from the ground up?

Last but not least, download this article on Test Retrofit published in Language Testing in 2009. Do you find the metaphor of architecture useful in thinking about test design and evolution? Happy reading.

Read the first part of this feature.

Glenn Fulcher
September 2012