Edtech Data Improves Purchases and Student Performance

Evaluating Edtech Programs at the Schoolwide Performance Level edWebinar recording link


Watch the Recording

Administrators selecting educational technology programs for their schools or districts face big decisions due to the time and money at stake, so having accurate and relevant information about the programs’ impact on student performance elsewhere should be a critical part of the decision-making process.

During a recent edWebinar sponsored by MIND Research Institute, Andrew Coulson, Chief Data Science Officer at MIND Research Institute, and Brian LeTendre, Director of Content and Communications, explained a process for finding data that will help administrators make the best choices for their student population, and in doing so “raise expectations about the availability of information” during program evaluations.

Rather than rely solely on randomized controlled trials, generally considered the “gold standard” for medical treatments, the MIND approach emphasizes the use of publicly available schoolwide performance data that can be matched to specific schools and types of students.

Evaluating the Evaluation Process

The advantages of the process Andrew outlined include being able to look at student performance in the same grades over several years. This avoids what’s known as “cherry picking” the data to present an exceptional performance in just one year, while ignoring other years that may be more typical. This approach also includes a wide range of students, so administrators can look at data that relates directly to the types of students in their schools.

While acknowledging that randomized controlled trials have their place in what should be an “edtech efficacy portfolio,” Andrew pointed out that the time and expense required for this type of study usually makes them feasible only once every five or ten years, limiting their ability to show progress over time and include the latest data. Also, this type of study requires a relatively small sample of students in order to control all the variables, so it really provides a snapshot of specific types of students at a specific time.

In contrast, the MIND approach uses schoolwide performance data published annually by the states for language arts and math, which provides a large enough pool of schools for administrators to find an effective match for a particular district or school. Administrators can consider factors such as whether the schools are urban, suburban, and rural, and how many students participate in Title I programs.

States may also break out the data in regard to specific subsets of learners, such as ELs (English Learners) and students with special needs, so the annual publishing of data can allow administrators to see how the same type of student performs in the same grade each year, as well as how much progress occurs as the same students progress through a number of different grades.

Having access to schoolwide performance data over a number of years also allows administrators to look at how students performed before the purchase of the program, and then for years afterward. This helps to determine how much of an impact the program had and how much improvement can be expected as a result of its purchase.

Evaluating Edtech Programs at the Schoolwide Performance Level edWebinar image


Putting Good Data to Good Use

In order for this approach to be effective, there must be grade-wide enrollment in the edtech program being evaluated, so that the test score data published by the state applies to all students. Administrators can have confidence in this type of data because it is derived from rigorous standardized tests that are used statewide, so they provide a uniform basis for comparisons.

This approach has now been used for well over a decade, and Andrew explained that the methodology has been replicated by an independent education research lab, using different sample sizes. The methodology and results have also met What Works Clearinghouse and ESSA standards for these types of studies.

Andrew noted that “other outcomes should also be considered, rather than just what happens on one day in April.” Ideally, an edtech program’s efficacy portfolio would include a randomized controlled test, samples of student work, videos of what happens inside the classroom, and qualitative input from educators and administrators, as well as access to schoolwide performance data that allows administrators to find the right match for their schools.

Meanwhile, Andrew hopes that the approach and results MIND Research Institute has developed will help administrators maximize the impact of an edtech purchase on student performance. He also believes it should create the expectation that every edtech program will have a positive response and supporting evidence when administrators ask, “Do you have a study on a district like mine?”

This edWebinar was sponsored by MIND Research Institute.

Watch the Recording

This article was modified and published by EdScoop.

About the Presenter

Andrew Coulson, Chief Data Science Officer at MIND Research Institute, leads a team of data analysts to conduct evaluations of student and teacher usage and outcomes of ST Math. He previously led MIND’s education division for 12 years, where he helped to execute strategies to scale the organization’s reach to over one million students. Prior to joining MIND, Coulson was the first education program officer for a major Orange County foundation and also worked for 17 years in upper management in high-tech manufacturing engineering. Coulson holds a master’s degree in physics from UCLA.

About the Host

Brian LeTendre serves as Director of Content and Communications at MIND Research Institute. In addition to orchestrating content for MIND, he is an author, podcaster, and avid gamer.

Join the Community

edFocus – An EdTech Industry Network is a free professional learning community on edWeb.net that will help you connect with colleagues in the education industry to share information and resources, raise questions, and get advice.

MIND Research Institute

MIND Research Institute is a nonprofit, social impact organization committed to transforming education for all learners. MIND’s flagship program, ST Math, is a preK-8 visual instructional program that leverages the brain’s innate spatial-temporal reasoning ability. ST Math’s unique, patented approach provides students with more equitable access to deep conceptual learning.