At first, educators could count the number of edtech programs in their school on one hand—and the number of users wasn’t too much more. Later, schools used spreadsheets and written reports to determine usage, and developers had to wait months, possibly years to get and analyze efficacy data. Now, when usage data is available on a daily basis, developers and educators have a shared interest in that information. During their edWebinar, “Edtech Usage Data: Key to Planning Efficacy Research,” Dr. Denis Newman, Co-founder of Evidentally, Inc., and Kylene Shen, VP of Marketing at Evidentally, Inc., explained why usage data studies are beneficial to edtech companies and schools as well as what types of additional studies can provide relevant insights.

First, research is necessary for educators and developers because it allows them to take advantage of the promises of edtech. While randomized trials were typically the gold standard in the past, starting with usage data has several benefits.

• It shows where a comparison study might be feasible. For instance, in one case study shared by the presenters, a teacher communication tool had an unusual pattern. Upon consulting with the schools, researchers found two distinct types of users and looked deeper into each group’s habits.
• It shows where the study population might be meaningful. The professional development company examined, for example, had a high number of users in New York. The researchers suggested doing an in-depth look at teachers in this group and seeing what other patterns emerge.
• It shows patterns of usage across multiple metrics. In the final case study, a program used widely across California showed achievement gains. The researchers then looked into subgroups to see if there was any difference in gains when broken down by various demographics.

In fact, one of the most important characteristics of a usage study is that it can mitigate the costs of more detailed research. By looking at the usage data, researchers should have a clearer picture of where to go to get meaningful information.

Second, developers and educators should work together to perform these studies because they both have a vested interest in improving edtech. Schools can ask to be part of pilot studies and have any results shared with them. Developers should look for several schools to conduct these studies so that they can accumulate data from a variety of sources. For both, subgroup analysis and avoiding selection bias are key.

Regarding subgroup analysis, Newman reminded attendees that it’s not good enough to know if a product worked on average. If a developer wants to improve the product, they need to know the demographics for each group that shows improvement for each that doesn’t. In addition, several small studies over time are better than one large study. Then, the developer will have enough information over time to make generalizations about for whom their product works and when.

Moreover, Newman warned about collecting data solely from the top users because then it won’t truly reflect how the product is used in the school or results from those who aren’t as tech savvy. “[Selection bias becomes] an issue when you’re using teachers and students who are among the most active users who implemented the program the way it was intended or with fidelity,” said Newman.

This edWebinar was sponsored by Evidentally, Inc.

Watch the Recording

This article was modified and published by EdScoop.

About the Presenters

Dr. Denis Newman, lead author of the software industry’s guidelines for research on edtech impact, is the CEO of both Empirical Education Inc. as well as Evidentally, a spin-off research organization that conducts dozens of RCTs and other evaluations of school programs. He has 35 years of experience improving student-teacher learning processes and instructional technologies, and is a pioneer in applying internet to student learning, professional development, and school administration. His Ph.D. in developmental psychology is from the City University of New York.

Kylene Chinsio Shen brings more than 18 years of experience in the education industry to her role as VP of Marketing for Evidentally, Inc. Since receiving her MSEd from Northwestern University, she worked as a curriculum developer, an elementary school teacher and math specialist, an instructional designer, and product manager. At Empirical Education, she has overseen the development of company products, and at Evidentally, is leading the company’s marketing efforts.

Join the Community

Inside K12 is a free professional learning community on edWeb.net that seeks to empower K-12 stakeholders with knowledge of hidden factors affecting our schools.

Evidentally Inc. was founded in 2018 with the goal of helping edtech providers improve the efficacy of their products, while providing K-12 educators with evidence about what will work in their schools.