LEARN to Scale Toolkit: Apply

Satisfy the End User

In the Apply stage, you identify both the features of your product that make your users want to keep using it, and the users who are eager to adopt your innovation and willing to share it with their networks (that is, your champions).

User and decision-maker (often considered your customer) satisfaction is typically derived from four dimensions – performance, reliability, convenience, and cost (PRCC). In the Invent stage, you likely conducted initial PRCC analysis and used that information to begin enhancing your product beyond a prototype. In the Apply stage, you will engage in more rigorous PRCC analysis to further understand and optimize your product, including to learn more about how your innovation stacks up against the PRCC of competing solutions. You might also refine your key performance indicators (KPIs), or measurable outcomes, to further align with end-user needs and goals. Throughout this process, gather community feedback on what does and does not work about your product, taking into account the perspectives of those who are most impacted but furthest from power. This input can help you design iterations to optimize your minimum viable product, or MVP, so that your solution more closely aligns with end-user needs and community values (for more on an MVP, see Imagine a Solution).

Performance	Reliability	Convenience	Cost
refers to what kind of impact the innovation has on educational outcomes. Researchers should consider the innovation’s efficacy and how this compares with the efficacy of existing solutions.	refers to how well the innovation can be implemented in a variety of contexts and whether the innovation will be able to achieve its intended outcomes for all intended users.	refers to how easy the innovation is to implement and how smoothly it fits within the existing educational contexts for which it is intended.	refers to the financial and labor investment needed to implement the innovation.

To inform your PRCC, plan to conduct interviews or focus groups with representative end users to hear more about their needs and values, and learn how your product fits into their existing workflow and what supports they would need for effective implementation. When selecting the type and number of users to speak with, consider finding enough users to yield a diversity of viewpoints so you can identify themes or patterns and reveal variation in how the innovation is experienced across cultures and contexts (a good rule of thumb is to interview 8 to 12 users). You can connect with these end-users through surveys, use cases or scenarios, or user stories. This work can occur at the same time as more rigorous impact testing (such as efficacy testing). You might also consider engaging users as members of your design team, so their expertise, opinions, and needs help guide both the development and refinement of your product.

End-user surveys. End-user surveys should include questions that will help you understand and optimize PRCC from the perspective of your intended or existing users. Consider asking these questions separately for each feature of your product. For example, say you have developed an intervention designed to be delivered through an app. One component of your innovation is the intervention itself, and another is the app software you are using to host that intervention. Consider each question for performance, reliability, convenience, and cost separately for (1) your intervention content and (2) the app software. See resource Questions to Consider for Performance, Reliability, Convenience, and Cost for questions to consider including in surveys.

Use cases/scenarios. You can also ask end-users to describe particular cases or scenarios in which they might use your product. This approach can help you understand the drivers behind specific instances in which a user would decide to use your product, how it fits with their workflow, and what specific need it fills. Getting this feedback from users might also help you think about the features of your innovation that are perceived as most useful to different kinds of end-users (such as part-time teachers, teachers less inclined to use technology, and so on), or even the features that are user-agnostic in particular situations.

User stories. Another great way to understand your user needs is to develop user stories. While there are several different types of user stories, all stories are generally written from the perspective of the user and provide a detailed, sequential description of how the user engages with the product.

Questions to consider for user stories include:

• What is the sequence of steps users take to achieve a goal with your product? Which steps are redundant, and where is a step missing? What are the “pain points” of your product? Where does it really shine?
• What did a “day in the life” of a user trying to achieve a goal look like before versus after your innovation? What does your product do well, and where could it be improved?
• What features of your product are “must haves” that should be prioritized in further refinement of the product? What features “would be nice” but are not necessarily at the top of the list?

Consider collaborating with some users (such as both champions and typical users) to develop stories, and then present the developed stories to other users to discuss whether and how your product aligns with their experiences as well. Getting this feedback from users might also help you think about which features of your innovation are perceived as most useful to different kinds of end-users (such as part-time teachers, teachers less inclined to use technology, and so on), or even the features that are user-agnostic in particular situations. See resource User Stories to Define and Refine Product Features for more information and examples of user stories.

If your product offers a promising way to address an acute need and is being adopted by users (also called user traction), you can say you have demonstrated strong product-user fit. One way you will know you have strong product-user fit is by identifying champions, or users who enthusiastically support your product and go out of their way to help it succeed.

Champions are pivotal figures within the ideal user group or audience of an educational product who not only use the product extensively, but also advocate for its adoption. They are typically well regarded by their peers and have influence in their environments, making them invaluable allies in promoting the product. Their genuine enthusiasm and positive experiences can help convince others of the product’s value, and they often provide critical feedback that can guide product optimization. These individuals are usually early adopters who see the potential in your product before it becomes widely recognized. They are invested in its success because it resonates with their own goals and challenges.

Identifying your champions is an incredibly valuable exercise because they can help you connect with and lobby key decision-makers and help market your product (e.g., testimonials and recommendations to peers). Champions can also give you feedback on marketing materials and suggestions for how your product could better “fit” the local context. They also play an essential role in a product’s lifecycle, particularly during the scaling phase where word-of-mouth and peer recommendations can drive growth.

Stories of Scaling Highlight

InqITS, an AI-based science program that has engaged students in 1.5 million labs to date, leveraged user engagement to identify the critical features of their product.

It’s important to distinguish what’s critical vs. nice to have (and can be added in the future). The team had to be “very scrappy” to set priorities and build the software. Based on teachers’ use of the early versions of the system, they identified the need for an LMS that allows teachers to find and assign labs, give grades, and manage workflows.

InqITS co-founder and CEO Janice Gobert shared that champions were an essential facilitator of scaling.

“There’s nothing more powerful than having a teacher talk about what they’ve done with what you’ve built. … It’s so much more authentic to hear it from an actual teacher.”

Read more of the InqITS story of scaling.

Notice and Reflect

While working to understand and satisfying end-user needs, take time to pause to notice and reflect. Consider the following questions:

• Do the users I am engaging represent different locales, student populations, communities, or cultures? How does their context impact the type of feedback and adaptations I am making?
• What do I still need to better understand about the users and the system to meaningfully address their feedback?

Spotlight Resource

Questions to Consider for PRCC. This resource shares questions to test or gather feedback on an educational product to support PRCC analysis.

User Stories to Define and Refine Product Features. This resource presents three types of user stories that a product developer may use to better engage with and understand the needs of end users and refine and improve products.

LEARN More

Story of Scaling: InqITS. This story of scaling profiles the development and scaling journey of the AI-based science program InqITS, including the importance of user feedback and champions.

Stories of Scaling: Pictoword. This story of scaling profiles the development and scaling journey of the vocabulary development app Pictoword, including lessons learned on user engagement.

Scaling A2i: A Conversation with Jay Connor and Sarah Siegal. This series of video blog posts highlights the story of scaling for the research-based technology and professional development tool A2i, including how the team engaged users to create a scalable product.

Scaling for System-Level Change. This video blog post discusses scaling innovations that disrupt inequity and create system-level change through partnership and centering user needs.

Story Map Concepts. This resource from Comakers LLC provides a process for creating a visual map of your user story.