How to Tell if a Product Is Supported by Sufficient Research
This article was originally published in an EdWeek white paper sponsored by Reading Plus.
Determining whether a product has the right kind of research to prove effectiveness is often a confusing process. The Every Student Succeeds Act (ESSA) now requires that the intervention programs purchased by school districts have sufficient evidence to demonstrate that they improve student outcomes. Choosing a program with research-based effectiveness increases the likelihood that it will improve student achievement. Understanding ESSA’s tiers of evidence standards and knowing what to look for will help districts make informed choices about an intervention program’s potential effectiveness.
ESSA Tiers of Evidence
Tier 1. Strong Evidence: from a well-designed experimental study demonstrating statistically significant positive effects on student outcomes.
Tier 2. Moderate Evidence: from a quasi-experimental study that meets ESSA standards. A quasi-experimental study is still an empirical study but without the random assignment that makes Tier 1 studies the gold standard. This level of research also shows significant positive effects.
Tier 3. Promising Evidence: from a well-designed and implemented correlational study, statistically controlled for selection bias.
Tier 4. Demonstrates a Rationale: there is a well-defined logic model based on rigorous research. An effort to study the effects of the intervention is planned or currently underway.
Districts should be able to expect that even brand-new products have been created with solid, peer-reviewed research in either the topic area or pedagogy or both.
How to Make the Best Intervention Selection
It is important that schools and districts understand ESSA’s tiers of evidence so they can quickly determine whether a program’s research claim is truth or fiction. Use the following steps to choose the best solution for your district’s unique situation.
1. Identify Local Needs
First, engage all district stakeholders to determine specific student needs. Then it is important to compare current student outcomes to the district’s performance goals. Creating an inventory of current practices and interventions will inform the types of interventions the district requires. Finally, identify student achievement gaps and determine which of them to prioritize.
2. Select Evidence-Based Interventions
Several clearinghouses help educators find instructional programs that are research-proven. Evidence for ESSA, for instance, provides the most up-to-date and user-friendly review of research based on the ESSA tiers. The What Works Clearinghouse has begun to align its evaluation process with the ESSA tiers, and the National Center for Intensive Intervention provides evaluations of other program components such as academic screeners.
By choosing programs that have rigorous evidence of effectiveness, the intervention is more likely to produce successful results in your district. Studies that meet “strong” and “moderate” evidence are preferred. Because gold-standard studies may not be feasible with all subpopulations, referring to programs meeting “promising” and “demonstrating a rationale” can also be useful when researching solutions for particular sub-populations—for example, English language learners or students with disabilities.
3. Review District Capacities
Funding, technology infrastructure, staffing needs and skills, administration leadership support, and even scheduling requirements are all success factors. It is critical to determine whether the program is a good match to support district goals. Review the district’s ability to implement a given program with fidelity, and certify that there is sufficient allocation of resources.
It is important to note whether the product’s instructional model was originally developed for the reason it is being purchased. For example, there are supplemental ELA/reading programs that originally were not developed as intervention programs. Such a program may not provide effective adaptation and differentiation for students who are significantly below grade level and need to accelerate their reading growth.
It makes the most sense to choose a program that is designed for the purpose you will be using it for.
Reading the Fine Print
Be careful that you read the fine print in marketing materials and when negotiating your contracts. You will sometimes see wording that promises “up to 2X or 3X expected growth.” The red flag here is “up to.” This phrase often implies greater results than the product may actually achieve for most students. If the product’s marketers could say, “On average, students with certain characteristics achieve X% of growth,” they would.
Another red flag is reliance on self-referential data. Proprietary company results and performance data should be correlated with a national metric, such as SBAC, PARCC, or NWEA MAP to be a compelling comparison.
Technology platforms that feature multiple subject areas, such as reading and math, may provide administrative and pricing convenience, but these platforms may not provide equally good instructional support or results for all subjects. Academic experts advise assessing each subject area’s instruction individually against your district’s needs to ensure an “apples to apples” comparison with other programs being considered. The goal is to select the most effective instructional tools to meet your district’s goals.
Resources to Help Your Search
• Using Evidence to Strengthen Education Investments
• School Leadership Interventions Under the Every Student Succeeds Act: Evidence Review – Rand Corporation
• ESSA Tiers of Evidence—What You Need to Know
Want to learn more?
Based on decades of reading science research, Reading Plus assesses reading efficiency and has been shown to produce statistically significant increases in proficiency, comprehension, and motivation. Reading Plus was found to have “strong evidence” to support ESSA.
Request a virtual demo with your local representative to get a closer look at our adaptive literacy program. A virtual demo is an easy way to see how Reading Plus fits into your district’s literacy plans.
