Despite some high profile false-starts, education technology, or ed-tech, has been in vogue in education circles for years. Many educators have high hopes for ed-tech’s potential to be a game-changer in how students at all levels are taught.

A daylong event last week hosted by the Massachusetts Institute of Technology’s Abdul Latif Jameel Poverty Action Lab, or J-PAL, took a hard look at the research illuminating the current state of ed-tech efficacy. While ed-tech’s potential to reshape education delivery models feels intuitive to many of the trend’s proponents, demonstrating results in high-quality controlled studies, and in the real world, has proven to be tricky in some cases.

Throughout the conference, speakers noted that ed-tech on its own is not a “silver bullet” to fix shortcomings in parts of the nation’s education sector. Underpinning the discussions were the numerous cautionary tales illustrating that unless ed-tech solutions are implemented strategically, school districts and universities have at times spent large amounts of time and money to tread water.

Overall, however, ed-tech proponents like keynote speaker, Karen Cator, CEO of the nonprofit Digital Promise, urged educators not to “miss the forest for the trees,” when looking at narrow studies on specific products that might cast a pall on the optimism surrounding education technology.

Cator, the former top ed-tech official in the Department of Education, echoed the need for better research to highlight ed-tech’s success stories. She also stressed the need for ed-tech buyers (typically school districts and universities) to do a better job of voting with their dollars in demanding solid evidence of success before purchasing ed-tech products.

Though Cator didn’t delve into the problem in depth, experts have noted that structural problems in the highly opaque and fractured K-12 purchasing landscape have put a lid on what should be a more dynamic marketplace. Because of the sector’s long purchasing cycles and widespread favoritism toward incumbent vendors, education can be a difficult industry for new companies with superior products to scale quickly.

Earlier in the day, Phil Oreopoulos, an economist at the University of Toronto who helps oversee J-PAL’s work in the education field, gave an overview of where the leading ed-tech research currently stands.

While acknowledging that scholars are still in the “infant stage of the research,” advances in artificial intelligence, machine learning, and big data are good signs for where the field could be headed, according to Oreopoulos. Because ed-tech products typically collect large troves of data, he also suggested that research and product development feedback loops should be quicker than in other sectors, as companies and experts devote more resources to improving ed-tech’s success rates.

Oreopoulos broke down the ed-tech body of research into four categories of studies: those that look at the impact of access to technology, personalized learning, behavioral interventions, and online learning.

Some of the key takeaways included the realization that high-quality personalized learning software, which typically includes using algorithms to adapt lesson plans based on a student’s level of proficiency, is a particularly promising strategy, especially in teaching mathematics.

Again, however, the worry is that school officials sometimes do not have a way of discerning between products that are truly adaptable and those that simply convert old-fashioned work sheets to digital formats. Experts sometimes call the latter, inferior products, “chocolate covered broccoli.” The problem is especially pronounced in the overlapping field of educational gaming.

This is not to say, however, that “gamification” does not have a powerful potential to alter behaviors. The New York Times recently published a feature on how the booming ride-sharing company Uber is taking advantage of the latest in behavioral psychology to influence its workforce.

In a similar vein, Oreopoulos highlighted studies in which text message notifications can be used to stage positive behavioral interventions on parents and students. Studies have shown the statistically significant positive impact of sending parents actionable reminders on how to support the reading skills of their pre-school aged children. Similarly, reminding graduating high school students about financial aid deadlines and procedures has been shown to increase college enrollment rates.

One last important finding from the research review conducted by Oreopoulos and his team was that the face-to-face social component of learning still matters. While he found that incorporating technology into traditional brick-and-mortar classroom structures—a process generally referred to as “blended learning”—can be just as effective as traditional instruction, online-only models are not. Despite the important caveat that online programs tend to service a unique student body, the evidence still shows that educators jettison in-person learning at their peril.

The J-PAL conference was timely, as the Department of Education recently released a large compendium of its ed-tech research from 2002 to 2014. As private capital continues to flow into new ed-tech products and research, and the cost of buying computer hardware platforms goes down while internet connectivity rates go up, one can expect the conversations over how to best leverage the latest in ed-tech to continue to intensify.

Follow Leo on Twitter

Subscribe for the Latest From InsideSources Every Morning