As any good edtech investor does, we speak with a steady flow of entrepreneurs attempting to solve some of the most intractable problems in education today, the bulk of which focus on improving student outcomes either directly or indirectly. Our presupposition at a basic level is that efficacy/improved outcomes help to drive adoption, which drives growth, which drives returns. The point here is that efficacy creates the foundation on which the others build and without that, the rest will face challenges. Of course this is true in theory, but in practice, reality can materialize differently. Of the 100+ companies we speak with every year the vast majority do have a nod toward efficacy in their presentations that show anecdotal data or quotes from a handful of customers that indicate lift in student outcomes. However, very few have statistically significant data that can raise one’s confidence levels of improved outcomes from hunch to near certainty.
There are a couple of reasons CEO’s give for this: (1) It’s really expensive to run these studies. Few startups have the financial backing, bandwidth or, in some cases, a large enough sample set to do this. However, it is typically on the longer-term roadmap to conduct these studies. (2) There is a genuine belief that their product works and that anecdotal feedback is sufficiently evidential of this and customers are satisfied by this. (3) It is acknowledged that the company’s product is early in its build out and there is concern that indeed it may not withstand the scrutiny of a robust study. No one has ever told us that outcomes don’t matter.
The interesting part here is that even without statistically significant results, many of these companies are still scaling and being adopted by educational institutions. I want to be careful to point out that I am NOT saying these decision makers are buying bad products. My sense is many of these products likely do drive better learning outcomes and nearly all of the entrepreneurs we speak with are working hard to continue to drive these upward. However, I do believe the emphasis on this by customers will change.
Perhaps a parallel in how this market may evolve can be drawn from another sector that we are active in – the Internet of Things. Last year we made an investment in Ayla Networks whose customers are large OEMs and manufacturers of devices like thermostats, smoke alarms, water boilers, etc. that are struggling with the first order challenge of just getting devices connected in a secure way that scales. The second order problem that the companies are beginning to ask and demand of vendors like Ayla is what do we do with all this data that is now being streamed to us from these connected devices and how does that inform the way we interact with our customer that buy those products.
I think edtech is going through a similar evolution of sorts. The first order problem here is how do I connect students with infrastructure and devices and make sense of the noise of possible solutions in the market to find those that teachers/faculty and students will adopt and engage with. As the market matures that first order problem will be mere table stakes but to win longer term, companies will have to address the second order problem of “does it actually drive improved learning outcomes and is there statistically significant data that can prove that, particularly with my cohort of students?” K-12 and post-secondary institutions will apply more and more rigor to this question when selecting a vendor. We saw this most recently with Texas’s TEA commissioning a third party study to gauge the efficacy of Think Through Math‘s (an SJF portfolio company) learning outcomes. As we hoped and expected, the results came back exceedingly well. Likewise, another portfolio company, Civitas Learning, has seen explosive growth given the level of rigor they place in showing evidence of things like lift in term-on-term persistence.
I am most excited about the ones that will fare well in this inevitable market landing point. Companies that do this fall into three buckets (1) Those that already have an eye toward this and show an ability to withstand this type of scrutiny (e.g., Think Through Learning and Civitas) (2) Companies where their offering is the actual platform or tool used to empower institutions to figure out which tools they’re using actually work and to help edtech companies determine the efficacy of their products and potentially provides insights on what they can do to their product to drive this upward (3) Those not focused specifically on educational outcomes but in other areas of improving the educational ecosystem that may have less scrutiny applied to outcomes.
For those companies that do not have an eye toward being able to show evidence of real learning outcomes, beware. I think the day is near where that will no longer be acceptable. Schools and users of products will demand more. And perhaps you should, too.