bright
bright
bright
bright
bright
tl
 
tr
lunder
Recent Content of Assessing Writing
This site designed and maintained by
Prof. Glenn Fulcher

@languagetesting.info

runder
lcunder
rcunder
lnavl
Navigation
lnavr
navtop
lnavs
rnavs
lnavs
rnavs
lnavs
rnavs
lnavs
rnavs
lnavs
rnavs
bottomnav
 
 

 

 
Titles and Abstracts
     
 
ScienceDirect Publication: Assessing Writing

Exploring the relationship between textual characteristics and rating quality in rater-mediated writing assessments: An illustration with L1 and L2 writing assessments
Publication date: October 2017
Source:Assessing Writing, Volume 34
Author(s): Stefanie A. Wind, Catanya Stager, Yogendra J. Patil
Numerous researchers have explored the degree to which specific textual characteristics of student compositions are associated with high and low ratings, as well as differences in these relationships across subgroups of students (e.g., English language learners). These studies provide insight into rater judgments and the development of writing proficiency. However, the degree to which textual characteristics are associated with the psychometric quality of ratings is relatively unexplored. This study illustrates a procedure for exploring the influence of textual characteristics of essays on rating quality in the context of rater-mediated writing performance assessments in order to gain a more-complete understanding of rating quality. Two illustrative datasets are used that reflect writing assessments for native English speakers and English language learners. The CohMetrix software program was used to obtain measures of textual characteristics, and the Partial Credit model was used to obtain indicators of rating quality. The relationship between essay features and rating quality was explored using correlation and profile analyses. Results suggested that rating quality varies across essays with different features, and the relationship between rating quality and essay features is unique to individual writing assessments. Implications are discussed as they relate to research and practice for rater-mediated writing assessments.



Automated formative writing assessment using a levels of language framework
Publication date: October 2017
Source:Assessing Writing, Volume 34
Author(s): Joshua Wilson, Rod Roscoe, Yusra Ahmed
This study investigates a novel approach to conducting formative writing assessment that involves evaluating students' writing skills across three levels of language (word, sentence, and discourse) using automated measures of word choice, syntax, and cohesion. Writing from students in Grades 6 and 8 (n=240 each) was analyzed using Coh-Metrix. Multigroup confirmatory factor analysis evaluated a hypothesized three factor levels of language model, and multigroup structural equation modeling determined if these factors predicted performance on a state writing achievement test comprised of a Direct Assessment of Writing (DAW) and an Editing and Revising test (ER). Results indicated that a subset of 9 Coh-Metrix measures successfully modeled three latent levels of language factors at each grade level. Results also indicated that the DAW test was predicted by the latent Discourse factor and the ER test was predicted by the latent Discourse and Sentence factors. Findings provide a proof of concept for automated formative assessment using a levels of language framework. Furthermore, although not the primary goal of the study, results may lay the groundwork for new levels of language detection algorithms that could be incorporated within automated writing evaluation software programs to expand automated+teacher assessment and feedback approaches.



Assessing C2 writing ability on the Certificate of English Language Proficiency: Rater and examinee age effects
Publication date: October 2017
Source:Assessing Writing, Volume 34
Author(s): Daniel R. Isbell
Differentiating between advanced L2 writers at the higher levels of the Common European Framework of Reference (CEFR) presents a challenge in assessment. The distinction between descriptors at the C1 and C2 levels are fine-grained, and even native speakers of a language may not consistently achieve them. At the same time, the CEFR has generally been conceived with the language abilities and contexts of use of adults in mind, thus making CEFR-based interpretations of young language learner?s abilities problematic. This study examines two issues in the assessment of C2-level writing in the context of the Certificate of English Language Proficiency (CELP) writing task: rater effects and examinee age. Interrater reliability and many-facet Rasch analysis showed that raters varied substantially in severity. CELP scoring procedures for rater disagreement partially mitigated severity differences. Contrary to expectations, age differentiated examinee abilities minimally and defied hypothesized ordering (i.e., that writing ability would increase with age). Additionally, some raters were found to demonstrate bias towards the youngest examinees. Specific implications for the CELP?s validity argument and broader implications for assessing young writers in CEFR terms are discussed.



Integrating assessment as, for, and of learning in a large-scale exam preparation course
Publication date: October 2017
Source:Assessing Writing, Volume 34
Author(s): Karim Sadeghi, Teymour Rahmati
This empirical study examined the validity of arguments regarding assessment integration tensions, strategies, and the potential of an integrated assessment model in enhancing students? writing ability. To this end, an integrated assessment as, for, and of learning model was experimented with a group of learners preparing to take the Cambridge English: Preliminary English Test. Moreover, an assessment for and of (non-integrated) model was used with another group of candidates as the control group. Subsequently, the candidates? writing performances measured by Cambridge Assessment in terms of overall band descriptions were converted into numerical indices. The Mann-Whitney U Test comparison of the participants? converted scores revealed that the integrated assessment group performed better than the non-integrated assessment candidates. Furthermore, classroom observations and a focus-group interview with the integrated assessment group clarified a number of issues concerning assessment integration and AaL implementation tensions and strategies. The results indicated that an integrated assessment model tailored to contextual specifications can contribute both theoretically and practically to teaching and assessing writing.



Assessing Writing, Teaching Writers: Putting the Analytic Writing Continuum to Work in Your Classroom, M.A. Smith, S.S. Swain. Teachers College Press, New York (2017)
Publication date: Available online 22 September 2017
Source:Assessing Writing
Author(s): Les Perelman




Ed.Board/Aims and scope
Publication date: July 2017
Source:Assessing Writing, Volume 33





Editorial for ASSESSING WRITING Vol 33 2017
Publication date: July 2017
Source:Assessing Writing, Volume 33





Tools and Tech: A New Forum
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): Laura L. Aull




The TOEFL iBT writing: Korean students? perceptions of the TOEFL iBT writing test
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): Eun-Young Julia Kim
The TOEFL is one of the most widely recognized language proficiency tests developed to measure international students? level of readiness for degree study. Whereas there exist a number of correlational studies conducted by various affiliates of ETS based on large-scale quantitative data, there is a dearth of studies that explore test-takers? perceptions and experiences concerning the TOEFL iBT. Writing skills have paramount importance for academic success, and high-stakes tests such as the TOEFL have a tendency to influence test-takers? perceptions on what defines good academic writing. To date, no research has specifically focused on test-takers? perceptions on the writing section of the TOEFL iBT. To fill this gap, this study explores Korean students? perceptions of effective strategies for preparing for the TOEFL iBT writing test, challenges they face in the test-taking and test-preparation processes, and implications such findings have for various stakeholders, by analyzing online forum data. Findings indicate that the scores for the writing section of the TOEFL iBT, albeit helpful for the initial benchmarking tool, may conceal more than it reveals about Korean students? academic writing ability. The study suggests that the format, questions, and the scoring of the TOEFL iBT writing test be critically examined from test-takers? perspectives.



Assessing peer and instructor response to writing: A corpus analysis from an expert survey
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): Ian G. Anson, Chris M. Anson
Over the past 30 years, considerable scholarship has critically examined the nature of instructor response on written assignments in the context of higher education (see Straub, 2006). However, as Haswell (2008) has noted, less is currently known about the nature of peer response, especially as it compares with instructor response. In this study, we critically examine some of the properties of instructor and peer response to student writing. Using the results of an expert survey that provided a lexically-based index of high-quality response, we evaluate a corpus of nearly 50,000 peer responses produced at a four-year public university. Combined with the results of this survey, a large-scale automated content analysis shows first that instructors have adopted some of the field's lexical estimation of high-quality response, and second that student peer response reflects the early acquisition of this lexical estimation, although at further remove from their instructors. The results suggest promising directions for the parallel improvement of both instructor and peer response.



Understanding university students? peer feedback practices in EFL writing: Insights from a case study
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): Shulin Yu, Guangwei Hu
While research on peer feedback in the L2 writing classroom has proliferated over the past three decades, only limited attention has been paid to how students respond to their peers? writing in specific contexts and why they respond in the ways they do. As a result, much remains to be known about how individual differences and contextual influences shape L2 students? peer feedback practices. To bridge the research gap, this case study examines two Chinese EFL university students? peer feedback practices and the factors influencing their feedback practices. Analyses of multiple sources of data including interviews, video recordings of peer feedback sessions, stimulated recalls, and texts reveal that the students took markedly different approaches when responding to their peers? writing. The findings also indicate that their peer feedback practices were situated in their own distinct sociocultural context and mediated by a myriad of factors including beliefs and values, motives and goals, secondary school learning and feedback experience, teacher feedback practices, feedback training, feedback group dynamics, as well as learning and assessment culture.



Evaluating rater accuracy and perception for integrated writing assessments using a mixed-methods approach
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): Jue Wang, George Engelhard, Kevin Raczynski, Tian Song, Edward W. Wolfe
Integrated writing (IW) assessments underscore the connections between reading comprehension and writing skills. These assessments typically include rater-mediated components. Our study identified IW type essays that are difficult-to-score accurately, and then investigated reasons based on rater perceptions and judgments. Our data based on IW assessments are used as formative assessments designed to provide information on the developing literacy of students. We used a mixed- methods approach with rater accuracy defined quantitatively based on Rasch measurement theory, and a survey-based qualitative method designed to investigate rater perceptions and judgments toward student essays within the context of IW assessments. The quantitative analyses suggest that the essays and raters vary along a continuum designed to represent rating accuracy. The qualitative analyses suggest that raters had inconsistent perceptions toward certain features of essays compared to the experts, such as the amount of textual borrowing, the development of ideas, and the consistency of the focus. The implications of this study for research and practice of IW assessments are discussed.



Similarities and differences in constructs represented by U.S. States? middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): Ya Mo, Gary A. Troia
Little is known regarding the underlying constructs of writing tests used by U.S. state education authorities and national governments to evaluate the writing performance of their students, especially in middle school grades. Through a content analysis of 78 prompts and 35 rubrics from 27 states? middle school writing assessments from 2001 to 2007, and three representative prompts and rubrics from the United States? 2007 National Assessment of Educational Progress (NAEP) writing test, this study illuminates the writing constructs underlying large-scale writing assessments through examination of features in prompts and rubrics and investigation of the connections between prompts and rubrics in terms of genre demands. We found the content of state writing assessments and the NAEP align with respect to measurement parameters associated with (a) emphasis on writing process, audience awareness, and topic knowledge, (b) availability of procedural facilitators (e.g., checklists, rubrics, dictionaries) to assist students in their writing, and (c) inclusion of assessment criteria focused on organization, structure, content, details, sentence fluency, semantics, and general conventions. However, the NAEP?s writing assessment differs from many state tests of writing by including explicit directions for students to review their writing, giving students two timed writing tasks rather than one, making informational text production one of the three genres assessed, and including genre-specific evaluative components in rubrics. This study contributes to our understanding of the direction and path that large-scale writing assessments in the US are taking and how writing assessments are continually evolving.



Social Justice and Educational Measurement: John Rawls, the History of Testing, and the Future of Education, Z. Stein. Routledge (2016). 220 pp., ISBN 978-1138947009
Publication date: July 2017
Source:Assessing Writing, Volume 33
Author(s): J.W. Hammond




Placement of multilingual writers: Is there a role for student voices?
Publication date: April 2017
Source:Assessing Writing, Volume 32
Author(s): Dana R. Ferris, Katherine Evans, Kendon Kurzer
Directed Self-Placement (DSP) is one placement model that has been implemented in various composition programs in the U.S. but has yet to be investigated thoroughly in second language writing settings. Central to DSP is the belief that, if students are given agency to help determine their educational trajectory, they will be empowered and more motivated to succeed (Crusan, 2011; Royer & Gilles, 1998). In this study, 1067 university L2 students completed both a voluntary self-assessment survey and the locally administered placement examination. We statistically compared the students? placement exam scores and their responses to the final question as to which level of a four-course writing program they thought would best meet their needs. We also examined a stratified random sample of 100 students? standardized test scores to see if there was a statistical relationship between those tests, our locally designed and administered placement test, and students? own self-placement scores. We conclude that student self-assessment might have a legitimate role in our placement process, but it probably cannot be used by itself to accurately place large numbers of multilingual students into a four-level sequence.