These early posts capture ideas from the founding years of Mindful Measurement.
I’ll continue to share new reflections here periodically.
For my current work and publications, please visit my Center for Assessment page.
Consequences, Complexities, and Multi-disciplinarity: An Outsider’s View of Mindfulness in Measurement
My perspectives on measurement were initially grounded in my own experiences as a student, and later in my experiences as a high school English language arts teacher. I come to measurement, then, as someone about whom inferences were being made, and upon whom actions and decisions were being taken. Consequently, for me, mindfulness always begins with careful attention to the lived consequences of our assessment design choices and the inferences and decisions that flow from them.
On Mindful Measurement, from a Distanced Perspective
The widely used across-person measurement models with trait-based constructs can indeed be useful for purposes like selection, classification, certification, or program evaluation, but they are limited for the purposes of understanding the nature of peoples’ capabilities and how we develop them. Sociocognitive concepts and methods are better suited for those purposes. What’s more, sociocognitive concepts and methods are useful for understanding across-person models in context: when and how they work, and for whom and with what consequences they can be misleading. They are indispensable too, I think, for using the concepts and methods of educational measurement with interactive assessment tasks like simulations, games, and collaboration
The Role of Measurement in Promoting Diversity
The achievement gap concept implies a belief that acquiring skills determined by the dominant culture is a worthy goal. If we look at the skills of existing successful people as a way to predict future success, we get the same people we have now. As Sternberg notes, this leads to less robust selection. Skills and habits of other groups may differ but are needed in society. How can we view diversity as a resource rather than a problem to be solved?
Mindful Measurement and COVID-19
At the moment, I can think of no better context to discuss mindfulness in measurement than the consideration of the novel coronavirus and associated COVID-19 disease. But rather than the impact of COVID on measurement, I’d like to consider the measurement of COVID. Lessons learned in educational measurement can be brought to bear in the measurement of COVID, and vice versa—what we are seeing in the measurement of COVID can inform our understanding of measurement in other domains.
Mindful Measurement: Reflections on Expertise, Power, and Biases in Research
As the world around me becomes more robotic and impersonal, being mindful about what I do professionally, think and feel has become increasingly important. In moments of social strife, mindful measurement takes on an alarming importance. Everyone is looking for accurate measures but more importantly, they are also looking for thoughtful and meaningful interpretation. Mindfulness about how numbers are presented and interpreted becomes vital to ensure that those who depend on us are fully aware of methods of counting, limitations with said methods and more importantly options for courses of action (i.e., risk-benefit and consequential validity).
Mindful Measurement in Writing Assessment
A mindfulness approach would stop the process before the first student sat down to take the test. Obstacles include a performance-only view of the writing construct; restrictive favoring of argument over other forms of writing; and exclusive focus on evidence of validity and reliability while failing to plan for evidence of fairness, especially in terms of the disparate impact of the test on student subgroups.
Mindful Measurement: The Case for Simplicity
I have been doing work in many of these areas for more years than I would like to count now, particularly on implementing these new ideas at scale, and can confirm our attempts at innovation often are at odds with principles of mindfulness. Not only that, but our push for new kinds of tasks, new kinds of evidence, and new kinds of models have yet to yield results that support some of our complex systems are better than our simpler systems created mindfully.
A Shift in Measurement Mindset
When I think about mindful measurement, as the structure of the word ‘mindful’ suggests, I think of what fills the mind with respect to an assessment. What fills students’ minds when they take an assessment? When they see their scores? What fills teachers’ minds when they administer an assessment? When they see their students’ scores? What fills parents’ and guardians’ minds as their children prepare for assessments? When they see their children’s scores? What do we want to fill their minds?
The Erosion of High-Stakes Summative Tests: Adding a New Compass
Rather than frequently interrupting learning with benchmarking tests, “stealth assessments” provide guidance just-in-time, which means many students will do better when high-stakes summative tests are administered. These diagnostic assessments are not substitutes for psychometrically reliable and valid summative tests, but they can accomplish useful functions not offered by those high-stakes measures. I believe that mindful measurement requires these complementary approaches.
Mindful Measurement: A Multidisciplinary Team Leader Perspective
To me, mindful measurement is the practice of thinking about, communicating, and doing psychometric work that’s important, meaningful, and practical. Most critically, it’s about understanding why a project exists. What problem are we trying to solve and for whom? Why do they care, and what does success look like? What happens if we fail to deliver? How do we best deliver given the constraints and affordances in this organization? These questions sound more like the those of business than psychometrics and don’t typically appear in a standard technical reports or presentations. However, I think our work greatly benefits from asking and answering these types of questions because they frame how we build, deliver, score, and report measurement information.
Mindful Measurement: A Generalist Outsider’s View
That is my interest in Mindful Measurement: helping others to change their view of the world of measurement as others helped me. How did I learn? Anyone who’s going to add value as a Chief Learning Officer needs to know something about the core competencies of the organization in which they are lucky enough to reside. Consequently, I found people who were much smarter than me and yet also very kind and most interesting. They filled my mind with multiple dimensions of measurement. The principles gained from all of that tutelage boiled down to three words — error, claims, and fairness.
Mindful Measurement – A Blog about Thoughtful Evidentiary Reasoning
In this blog, I am interested in the seemingly opposite – the islands of calm and quiet amidst a sea of change while thunderstorms are brewing. That is, I am interested in how human beings, with all of their complexities, fallacies, and beautiful characteristics, take the time to “slow down” and reflect on the way they are going about the work in order to be more principled, thoughtful, mindful.