Mindful Measurement in Writing Assessment

I am a specialist in writing assessment—a research area bicycle narrow and grenade dangerous. Steer toward a single variable such as knowledge of conventions, and you miss the construct target; pedal toward a broad model, and the assessment process explodes because nobody can make a scoring model work.

While the complexities are enormous, so, too, are the rewards. If a writing assessment episode is done well, the assessment becomes a way to create community and advance opportunity to learn. Students benefit from construct-centered feedback, instructors integrate evaluation principles into their classrooms, and administrators obtain information through evidence of fairness, validity, and reliability.

At the present writing—when the news is filled with pandemic fears and racial cruelty—talk about writing assessment may seem far removed from daily concerns. It is nevertheless the case that the ways we define and assess literacy impact educational gatekeeper systems that have long disenfranchised students.  Judging a student solely on command of standard written English or a form of writing unfamiliar to the student can be catastrophic. As the Complete College America initiative has demonstrated, when such reductionistic assessment is used to remediate, students are put on a road to failure. Bad assessments can end a life in systemic ways akin to the way lives are damaged by unjust health care systems or extinguished by police forces. A dream deferred, as Langston Hughes counselled, explodes.

In this environment, is there a role for mindful measurement in writing assessment? If so, how can adopting such a framework help educators abandon diminished perspectives of key constructs such as writing and advance opportunity to learn for all students?

A Particular Way

Historically, the term mindfulness comes late to the English language (in 1899) and is associated with Buddhist enlightenment (in terms of awareness). Philosophically, mindfulness works like this: We become conscious of being in the present moment. As awareness past and future recedes, a new consciousness is accompanied by a still place that, in turn, allows reflection. It is this place of heightened consciousness, and process that brought us to it, that we call mindfulness.

Operationally, mindfulness has been used as a business practice and studied as a psychological phenomenon. Published in Perspectives on Psychological Science, a 2018 critical evaluation of mindfulness revealed that, in 1970, there were no published studies in academic journals. By 2014, that number soared to 1,000 studies. In the popular press during the same period, by 2015, the number rose to nearly 32,000 articles. Following their review, the study authors conclude that “multipronged future endeavors” of rigorous research will help to surmount the prior misunderstandings and past harms caused by “pervasive mindfulness hype” that they have documented.

While it is surely the case that capitalist societies run ideas through the marketplace with great regularity, retaining a promising philosophical concept can surely have value. Masculinist rigor does not always result in value, and loaded language gets us nowhere.

Binaries aside, then, how might mindful measurement contribute to writing assessment?

Let’s begin with a well-known definition of mindfulness offered by Jon Kabat-Zinn, Emeritus Professor of Medicine at the University of Massachusetts Medical School, who asks that we enact mindfulness by “paying attention in a particular way: on purpose, in the present moment, and non-judgmentally.” In work informed by this definition, English professor Christy I. Wenger has coined the term contemplative writing pedagogy, an approach to instruction that invites students to use meditation as they compose with new digital tools. As students think about these two activities, both polar opposites, they explore the dangers of binaristic thinking (the contemplative world of silence versus the frenetic world of the internet)—and the virtues of attunement. Both understanding and expression are advanced, Wenger finds, when mindfulness is used to teach writing.

By extending teaching to assessment, we can find a valuable role for mindful measurement. In his introductory essay on evidentiary reasoning and mindful measurement, André Rupp calls for us to find “islands of calm” in which “either-or distinctions” are kicked to the curb. His theme is at one with the origin, development, and uses of mindfulness.

To get a sense of how mindfulness might work in writing assessment, a brief case study is in order.

SOARing with Mindfulness

To see mindfulness in action, let’s try a SOAR (Situation-Obstacle-Action-Result) thought experiment.

Imagine that a dean of a college of liberal arts wants to institute advanced placement at a diverse, public post-secondary institution. The dean is hip to the fact that the provost and president want to increase retention and time to graduation and believes that allowing students to place out of the initial first-year writing course will advance the institutional mission. The dean is equally hip to the fact that glitches in the Advanced Placement digital testing during the coronavirus pandemic opened doors for colleges to use local tests.

Working with the Department of English, the dean brings a team together of institutional researchers and writing teachers to design a test that has a capacious view of the writing construct in terms of writing performance and a task for students that reflects some of the objectives of the initial course. The test requires students to write an argumentative, source-based essay; the rubric is carefully designed to provide feedback using trait scores and placement using holistic scores. The faculty readers will be paid for their work, and their colleagues from institutional research will help provide construct-related evidence and evidence of scoring reliability. Over time, there will be plans to track the students—both those who have and have not been awarded advanced placement—to study their overall academic performance, as well as their performance in later writing intensive courses. The advanced placement system is ready to go. What could possibly go wrong?

A mindfulness approach would stop the process before the first student sat down to take the test.

Obstacles include the following: a performance-only view of the writing construct that does not include the interpersonal, intrapersonal, and neurological domains traditionally included in the curriculum as collaborative work, self-confidence during writing processes, and disability compliance; restrictive favoring of argument over other forms of writing, with no attention to narration, description, and other modes and aims of discourse associated with academic identity formation; and exclusive focus on evidence of validity and reliability while failing to plan for evidence of fairness, especially in terms of the disparate impact of the test on student subgroups.

As the process is halted under a mindful approach, stakeholders can see the hazards of binaristic thinking (test or no test) and the virtues of attunement (using what exists in the present moment)—and take action.

During this moment, institutional leaders can re-think the values of first-year writing courses to provide experiences with noncognitive domains to students that result in student success—aspects of collaboration, tenacity, and well-being that no test can capture. As well, if it is indeed the case that there are other curricular opportunities for students to practice in these domains, then in-place measures such as high school grades could be used to create prediction models that could, in turn, be used for advanced placement measures. One recent study demonstrated that a multiple measures placement system based on data analytics results in placement determinations that lead to better student outcomes than a system based on test scores alone.

The results of applying mindfulness perspectives to writing assessment seem limitless. But it is hard to find a still place, harder to see structural bias, and almost impossible to do something about it.

Occupying that space of impossibility is exactly what we must now do.