Evaluating the Effectiveness of Online Learning Platforms Using Embedded Experiments in Real-World Settings

Abstract

As online learning platforms become increasingly integrated in educa tion, there is a growing need for scalable methods to evaluate their effective ness in fostering learning. Traditional evaluation methods, such as pre-post tests in classroom settings, are time consuming and difficult to scale. This re search explores the use of platform-embedded experiments as a cost-effective, continuous method for evaluating learning platform effectiveness, using Squla and StudyGo as case studies. In Squla, an experiment was conducted to as sess the impact of practicing relevant topics through the platform on quiz performance. The results showed that for math and language, answers on a new quiz were 3% more likely to be correct after practicing the relevant topic on the platform, compared to practicing off-topic content. However, no significant improvement was observed for spelling and grammar or read ing comprehension. The StudyGo experiment focused on learning within a single attempt at a set of practice questions. The order of questions was manipulated and the results showed that questions were 3% more likely to be answered correctly when placed at the end of a set compared to the be ginning, suggesting that students learned from previous questions. These results demonstrate the potential of platform-embedded methods for scal able and efficient measurement of learning outcomes. Future research should address the limitations of these methods, such as their limited generalizabil ity, and explore their applicability across a broader range of content and different learning platforms.