“Students gained 1.9 years of growth in one school year.”
You have seen a line like that on a vendor one-pager. It looks like an answer, and it is one. The question is whether it answers yours.
Vendors answer, “Did this work somewhere?” You need to know, “Will this work here, with my students, my schedule, and my staff?” Those are different questions. Most disappointing purchases live in the gap between them. You do not need a statistics degree to close it. You need six questions and the habit of asking them out loud.
A One-Pager at Lincoln Middle School
Picture Lincoln Middle School, a composite example. The principal, an instructional coach, and a seventh-grade math teacher review a one-pager for a math practice platform. The headline promises 1.9 years of growth. The teacher is ready to sign up. The coach asks to see the fine print.
The fine print changes the picture. The result came from students who logged at least 60 minutes a week. That is a self-selected group, and students who practice an hour a week were probably motivated, supported, or ahead to begin with. There is no comparison group, so nobody knows how those students would have grown without the platform. And the data came from one school with a full-time implementation coach. Lincoln has no such coach.
None of this means the vendor lied. The number may be accurate. It answers a narrower question than the one Lincoln is asking.
Six Questions to Ask Before You Believe the Number
- Who is in the data? Ask how many students, which grades, and how they were chosen. Ask how many started and how many finished. A result that counts only the students who stayed is a result about students who stayed. Then ask whether those students look like yours.
- Compared to what? Growth means little without a comparison group of similar students who did not use the tool. A before-and-after gain cannot separate the product from a normal year of learning.
- Who measured it, and with what? A test built into the product measures what the product teaches. An outside assessment the vendor did not design is stronger evidence. Ask which test was used and who scored it.
- What does “worked” mean? Logins, lessons completed, and satisfaction scores measure usage and opinion, not learning. Ask for the learning outcome, the size of the change, and how many students reached it. An average can hide a lot.
- Under what conditions? Ask about weekly minutes, training, and staff support behind the result. Then ask whether your campus can match them. A result that depends on a coach you do not have will not repeat at your school.
- Has it been repeated? One study is a lead, not a conclusion. Look for the same result at other sites, found by researchers other than the vendor. A finding nobody else can reproduce tells you very little.
That last question connects to the federal Every Student Succeeds Act, which sorts evidence into four tiers. Strong evidence comes from well-designed randomized studies. Moderate evidence comes from studies with a comparison group but no random assignment. Promising evidence comes from correlational studies that control for selection bias. The fourth tier, demonstrates a rationale, means a sound theory with no proof of effect yet. Ask the vendor which tier applies and which study backs it. Then check.
Where to Check (United States)
- What Works Clearinghouse: The federal review of study quality. It rates studies, not whole products.
- Evidence for ESSA: Program reviews from Johns Hopkins University. The site states it has no official status.
- Digital Promise’s Research-Based Design badge: Shows a product was designed from learning research. It does not show students learned more. (Link not working at the time of this posting)
Red flags: “up to” results, “students who used” results, satisfaction surveys offered as proof, data that lives only in the vendor’s dashboard, and “research-based” with no study cited.
Run a Nine-Week Check at Lincoln
Lincoln decides to test the platform before signing a contract. Two comparable seventh-grade sections take part. One uses the platform for 30 minutes, twice a week. The other continues current practice. Both take the same outside benchmark in week one and week nine.
Before anyone looks at results, the team writes its decision rule: expand if the platform section outgrows the other by the margin the team set, and at least 80 percent of students used the platform as planned.
At week nine, the platform section grew a little more, but only about half the students used it as planned. The team holds off on buying. It extends the trial one more semester with protected class time for the platform.
Two sections and nine weeks are not proof. A check like this will not settle whether the product works everywhere. It answers what the vendor cannot: whether your students use it, and whether your benchmark moves. It is also a small test of repeatability, because it asks whether the vendor’s result shows up in your building.
Cautions and Tips
- No evidence is not the same as no effect. New tools, especially AI tools, change faster than studies can keep up. A missing tier is a reason to test locally, not an automatic no. Do not let the evidence standard become a veto on every new idea.
- Ask for the data in writing before the demo. You will read it without a sales conversation steering you.
- Ask, “What would it look like if this didn’t work?” A vendor who knows its product has an answer.
- Set your decision rule first. Decide what counts as success before you see the results.
Paste a vendor’s public marketing material into your AI tool with the prompt below. It finds the gaps and drafts questions. It does not judge whether the claims are true. Never paste student data.
Vendor Claim Check Prompt
Your Next Step
Pick one tool your campus already uses. Run the six questions on its one-pager before the next renewal, PLC meeting, or vendor call.
If you are a principal, ask for the evidence in writing before any pilot begins. If you are an instructional coach, bring the six questions to your next PLC and run them on a tool teachers already like. If you are a teacher, ask the first, second, and fourth questions before you adopt a new tool. You will hear the difference in what comes back.
A note for readers outside the United States: The UK’s Education Endowment Foundation rates how secure its trial findings are with a padlock scale, and England’s Department for Education piloted an EdTech Evidence Board with six core criteria. Both are good models for asking how much confidence a study’s design earns. Neither is a US standard.
