Consequences, Complexities, and Multi-disciplinarity: An Outsider’s View of Mindfulness in Measurement

My perspective on mindfulness in measurement has been informed by my outsider status.  I am a writing teacher at heart who has been drawn into the field of measurement because of my commitment to transformative writing pedagogies and the assessment innovations that are needed to support them.

Consequences

My perspectives on measurement were initially grounded in my own experiences as a student, and later in my experiences as a high school English language arts teacher. I come to measurement, then, as someone about whom inferences were being made, and upon whom actions and decisions were being taken.  Consequently, for me, mindfulness always begins with careful attention to the lived consequences of our assessment design choices and the inferences and decisions that flow from them.  In 2016, I published an Integrated Design and Appraisal Framework (IDAF) that guides writing assessment designers and users through a systematic examination of the consequences that flow from our assessment design choices.  The IDAF provides a heuristic that invites exploration of questions such as: How well do we understand the constructs we are measuring?  How well have we captured those constructs through the assessment?  What is the impact of our design choices on the quality of inferences we can make from assessment data across diverse populations? What messages do our design choices convey about what and who we value?  What are the consequences of those messages for students, teachers, classrooms, schools, and systems of education? 

Complexity

Systematic attention to the consequences of design choices, draws our attention to the issue of complexity; a second aspect of my thinking about mindfulness and measurement.  From my perspective as co-editor-in-chief of Assessing Writing, I have a front row seat to the incredible complexity involved in the design and implementation of writing assessments—from classroom to large-scale—used around the world and at all levels of education. When Assessing Writing was founded 27 years ago, much of the scholarship that appeared in the journal was highly critical of the writing assessment designs that came from the field of measurement and that were being imposed on writing classrooms (primarily in the United States).  This critique is perhaps best summed up in the title of one of article published in the first issue of the journal, “The Constant Danger of Sacrificing Validity to Reliability: Making Writing Assessment Serve Writers” (Wiggins, 1994).  The heart of this critique has been that in the interest of achieving stable, reliable measures, many writing assessment programs have stripped away the enormous complexity inherent in the writing construct, writing tasks, the valuing and judgements of writing, and the development of writing ability.  As a consequence of those design choices, the argument continues, these assessment programs rarely serve the needs or interests of teachers and students.   

Because many of the early contributors to Assessing Writing lived out, in their own writing classrooms, the consequences of these design choices, the early focus of the journal was on   advocating for more pedagogically meaningful forms of writing assessment; ones that were built on a robust understanding of the writing construct, that embraced the complexities inherent in how human beings read, process, and value written compositions, and ones that acknowledged the interpersonal, intrapersonal, and ecological factors that shape the development of writing ability. 

Building my research on this body of scholarship has caused me to also see mindfulness in measurement as embracing complexity.  Embracing complexity has meant developing as robust an understanding as possible of the knowledge, skills, and dispositions that support excellence in writing; it has meant trying to map out the sets of bioecological factors that shape human development as they are applied to learning to write, and it has meant trying to understand the complex sets of factors that shape both writing tasks and the valuing of writing across contexts.     

Interdisciplinarity

A research agenda focused on complexity, has informed the third element of my perspective on mindfulness: Interdisciplinarity.  It becomes immediately evident, as one works through the list of complexities involved in the assessment of writing, that expertise across many fields is necessary for the development of robust programs of writing assessment. Huot (2002) observes that one of the primary limitations with earlier generations of writing assessment is that they primarily drew on the theories and perspectives at play in psychology and psychometrics rather than writing studies, literacy education, or composition and rhetoric. He observed that in instances where cross-disciplinary collaboration shaped assessment design, a technocentric focus driven by psychometrics restricted design options available to assessment designers. This observation was supported by Behizadeh & Englehard’s (2011) systematic review of more 100 years of scholarship and practice on writing assessment.  Based on this review they conclude, “intellectual isolation [between psychometric and writing studies communities] needs to be bridged” (p. 206) so that the theory and practice of writing assessment can be advanced in the 21st century. 

The challenge of working across disciplines, especially disciplines that have divergent epistemologies and ontologies, is figuring out how these perspectives integrate and which perspectives frame and lead the others. 

Mindfulness through Collaborative Design

Over the past two years I have had the opportunity to explore this challenge as I co-edited with Maria Elena Oliveri, Andre Rupp, and Norbert Elliot a recently published Special Issue of the Journal of Writing Analytics focused on the design of writing assessment programs of 21st century workplace English communication.  In fact, this special issue exemplifies these three aspects of mindfulness in exquisite detail.     

The project at the heart of the special issue employed principled design frameworks such as eECD, IDAF, and Theory of Action to inform the design of a digital assessment for learning platform for workplace English communication skills.  The first three papers in the SI demonstrate how systematic attention to consequences and design choices are essential for the development of the next generation of assessment platforms.  The final six papers demonstrate how attention to complexities related to the construct model (papers 4, 6, 7, 8) shape approaches to validation (paper 5), score reporting (paper 9), and instructional design (paper 7).  The introduction and afterward highlight the incredible range of expertise that is needed to bring complex assessment designs forward. 

The expertise represented by authors in this special issue span Assessment design, Cognitive science, Curriculum and instructional design, Educational policy, Human-computer interaction, Information visualization, Subject matter, Task design, Psychometrics, and Score report design.  Among the most important insights coming from this special issue is the central role that principled design frameworks play in managing both complexity, and the multidisciplinary expertise required to bring complex assessment design tasks forward. 

An Outsider’s Hope for the Future

The work of this Special issue exemplifies a hopeful future for the field of measurement, where multidisciplinary teams come together to tackle challenging assessment design problems, with a view to embracing the complexities inherent in those problems, and through the application of principled design frameworks, seeking to resolve them in ways that maintain construct integrity, while limiting negative unintended outcomes for educational systems and those working within them.