August 1, 2024 | By H. Alix Gallagher, Director of Strategic Partnerships, Policy Analysis for California Education, Stanford University

When a new intervention or program is introduced, the conventional wisdom among many seasoned educators is, “This too shall pass.” This attitude doesn’t mean an intervention is not promising or educators are not flexible and innovative. Rather, it emerges from a common pattern of ineffective implementation that doesn’t sufficiently change adult practices or gain the local support necessary to sustain the changes. The promise of the intervention is never realized, and the district moves on to the “next new thing.”

To break this cycle and achieve desired impacts over the long run, evidence-based interventions need to be implemented well at scale in a district. It is important that leaders conceptualize scaling as more than just initially spreading the number of users.¹ Successful scaling involves multiple dimensions:

• Interventions need to lead to a sufficient depth of change in order to make a difference in student outcomes.
• Relevant stakeholders need to develop ownership of the intervention so it is appropriately adapted to the system and has ongoing champions.
• The intervention needs to become embedded in the structures, processes, and culture of a system in ways that foster sustainability.

To achieve this model for successful scaling, a new intervention needs to be piloted. A pilot is the first stage in building support for the intervention among those who will ultimately use it. The pilot should do more than give a few teachers a chance to explore the intervention. If key educators can figure out how to make the intervention fit with the rest of their instruction and gather evidence that it is effective, the pilot builds ownership. These educators are then motivated to help their colleagues see the benefits and support broader implementation.

In short, a successful pilot will determine how a new intervention can be consistently implemented well in the context of a particular system to scale and improve student outcomes.

Conducting a successful pilot

Bring together the right people

The critical first step of a pilot is to convene a cross-functional leadership team. This team should include senior district leaders who have authority to make decisions and allocate necessary resources, as well as representatives of all the role types that would be affected by adoption of the intervention. These roles include the teachers who would use the intervention in their classrooms and the administrators who would need to design professional learning, oversee classroom observations, and provide feedback. Completing an influencer map can help to identify the roles and representatives needed for the team.

When leaders select educators to do the initial pilot, they should consider both the formal roles and informal influence potential team members have. Leaders may often be tempted to recruit “early adopter” types or relative novices with fewer time demands. A better course is to create a balanced team that includes veterans who are willing to pilot, can offer an honest assessment of how receptive their colleagues might be and what evidence from the pilot could increase receptivity, and can ultimately use their influence to build a broader base of support for expansion. This base of support reduces the costs of spread (e.g., administrator energy, fighting against resistance) for all but the most resistant teachers, who can then be more (politically) easily required to implement at a later stage.

Gather the necessary information for decision-making

To be effective, the pilot needs to uncover:

• How educators experience using the intervention, and what this may mean for how easily successful implementation could be spread
• How the intervention is implemented, including how educators fit it into their classroom routines, how it interacts with other materials and resources, and how educators adapted it
• What supports and resources are necessary for effective implementation, and what need to be provided more broadly if the district decides to scale the intervention
• Whether students benefit from the intervention, and, if so, what critical evidence can be used to make the case for broader implementation

After learning this information, district leaders may decide to adopt the intervention and plan for scale-up, engage in additional pilot testing, or abandon the intervention and look for alternatives. If moving forward with adoption, leaders can consider how to leverage the pilot team as champions who can share their experiences to influence others in combating the “this too shall pass” mentality.

On the other hand, district leaders may believe the intervention has great promise but the pilot shows that it fails to benefit students or that teachers struggle with implementation. In this case, leaders can either decide against adoption or use an additional round of small-scale piloting to learn how address those issues.

With the knowledge gained from a thoughtful pilot – and evidence that the intervention can improve student outcomes in the local context — the district is ready to scale. District leaders may choose to expand to a larger group of testers before scaling districtwide. This option helps them learn more about the conditions under which the intervention supports improved student outcomes and about what supports or hinders effective implementation.

Spread the intervention to a larger group

The next step is to spread the intervention from the small team of initial pilot educators to either a larger group of willing educators or within a district structure, such as department, grade level, or school. This is where the influence of champions from the initial pilot team can help convince teachers on the fence to try something new.

The box below shares a teacher’s perspective on how her district went from piloting an intervention to making it a “tight,” [a reference to the idea that there are some practices that are “loose” (teacher discretion) and others where the entire organization is “tight” (i.e., everyone is required to implement with fidelity)]. In this district, there is a history of substantial resistance to new ideas, so it was particularly important to take time to build ownership for the intervention’s successful implementation among key teacher leaders.

At this stage, it is important for district leaders to work with educators to articulate how the intervention fits into existing goals for student learning and how it should change teacher practices or processes (such as how student time is scheduled). The most common pitfall as the intervention spreads is that implementation becomes more superficial as the number of educators using the intervention grows.² Leaders should monitor the quality of implementation as spread happens, recognizing that schools typically underestimate the extent of and types of support needed to enable deep changes in teaching practice.³ Ongoing data collection about both implementation (e.g., as part of regular classroom walkthroughs) and student outcomes can help leaders think about continuing to scale to all adults who should be using the intervention.

Institutionalizing change for sustainability

Ultimately, institutionalizing the changes from an intervention requires ownership from educators who understand how to implement the intervention well and what changes to organizational structures and processes enable successful implementation over the long run. Only when the intervention is institutionalized – becoming the “way things are done” and capable of surviving leadership turnover – is the scaling process complete.

Tags: Education technology Educators and systems leaders Evidence-based Innovation