May 19, 2023 | By Rebecca Griffiths
When researchers seek to create a product, program, or intervention to enhance educational outcomes for students, they often start from a theoretical foundation. A theory of change, or a conceptual model based in research of how a specific product1 can lead to a desired change, is often the launchpad for creation.
However, do such theories of change reflect actual needs and problems of practice that educators encounter? We hypothesize that the missing links between traditional research activities and on-the-ground, everyday problems of practice hinder scaling, sustainability, and impact of educational products.
The theory is sound, but does it address an unmet need for users and decision-makers?
Education researchers often develop a theory of change by first identifying a problem and then proposing and testing a potential solution to address that problem. The problem is usually framed from a societal standpoint, such as the need for a more diverse teaching workforce, to reduce biases in disciplinary actions, or to increase educational access for students with particular learning disabilities or differences.
A research-based solution may demonstrate high efficacy, but people will not use it if it does not help them do their jobs better or make their lives easier. And if the product does not address a need in the eyes of end users, implementation could ultimately be a check-the-box activity. Reframing the problem from the perspective of users and decision makers can increase the odds of uptake.
For example, researchers might hypothesize in their theory of change that students will develop deeper learning skills through using an interactive online application that presents scenarios for collaborative problem-solving in math. Their application might prove highly effective in tests, but its widespread use will likely depend on whether it meets a salient need for teachers. Does it help them better engage students in the classroom or differentiate instruction for students with different skills or ability? Does it clearly add value to practice? If not, the product is unlikely to reach a broad audience and fulfill its potential to change outcomes for many children.
To spur growth in adoption and use, researchers must pivot from their theory of change approach and instead investigate the problems of practice their product addresses for educators.
Does the scope of the solution fit with user needs?
Researchers tend to design products that serve specific purposes or that target particular groups of students. Rigorous efficacy testing tends to work better with less “noise” in the form of varying contexts. Similarly, in a complete solution that addresses a full range of needs, it is harder for researchers to pinpoint the “active ingredients” or core components that lead to the intended outcomes.
However, the problem definition underlying a theory of change may lend itself well to research but be too narrow to lead to a scalable product. The target users may be too few or hard to reach to sustain the organizational infrastructure needed to provide and support the product. Or the solution may be a feature rather than a complete product, resulting in a mismatch with what users are seeking. For example, a 4-week supplemental program designed to teach a specific topic like word problems may not meet the needs of districts looking for comprehensive math curricula.
How well do you know what your intended users want?
These pitfalls are not limited to education researchers and developers. A classic Harvard Business Review article encourages entrepreneurs to be clear about the “jobs to be done” by their innovations. The authors argue that developers must understand the experiences their target users wish to create for themselves in a deeply nuanced way, and that the circumstances of adoption can outweigh the characteristics of users and of the product itself. Context is crucial.
Many other factors will affect product adoption besides efficacy. Is it easy to use? Does it integrate seamlessly in the class? Does it require installation of new hardware or software? Does it require a lot of technical support? Is it affordable?
The LEARN Network is using the Invent-Apply-Transition Framework to help researchers address these questions by grounding design and development in the problems of practice that matter to the intended users. In adapting this framework for education, we are considering ways to incorporate principles of Liberatory Design to ensure products are relevant and accessible to the communities they are designed to benefit. Drawing from this work and from new research conducted as part of the network, we will develop a set of publicly available tools and resources to help a broader community of researchers apply these principles from the very beginning of the design process.
Tags: Adaptation Application Research & Developers Transition