Research Foundations

Research Foundations of the Enlearn Platform 

Based on Past, Current, and Future Research

The Enlearn Platform was designed and developed based on research in learning science, educational technology, machine learning and knowledge representation; in fact the Enlearn Platform was created by academics at the forefront of education technology research.  Recognizing, however,  that research in these areas can and will be continuously advanced further,  the platform goes significantly beyond deploying pre-existing research findings into the product in three key ways:

  • In order to achieve key features unique just to the Enlearn Platform, the creators initiated, conducted, and published new original research aimed at solving problems that simply could not be solved by the collective body of knowledge in the above mentioned disciplines.  Some examples of these Enlearn-specific research discoveries include:
    1. How to promote productive struggle and growth mindset capable of producing significant change in a digital setting.
    2. Novel machine learning methods uniquely suited to the educational learning setting.  The new methods are capable of dealing with complex learning settings that were not solvable with previous machine learning methods.
    3. Novel ways to represent, encode, and generate learning content at the thought process level, which enables the platform to follow student thinking processes during problem solving, amplification of any curriculum by over 100 fold, and personalizing content, delivery, and learning pathways for each unique learner.
  • Researchers behind the Enlearn Platform realize that the common approach to scientific research of learning is to find principles behind learning methods that work in general terms.  This approach does not, however, provide insight into the ideal learning process for any individual learner, unique classroom setting, or specific teacher.  For this reason, the Enlearn Platform is designed to learn itself and become highly specialized for each classroom and each student.  Insights from every new student learning on the platform are automatically analyzed and applied to future learners.  This self-adaptation ensures that Enlearn is highly personalized and differentiated in ways that general principles in the current state-of-the-art research are not capable of knowing.  Furthermore, it continuously gets more specialized in the continuous rapid process of self-adaptation.
  • A University of Washington team of over 15 researchers, led by Enlearn’s founder, continues to work on further research based exclusively on how the Enlearn Platform can further improve learning outcomes.  The Enlearn Platform research is uniquely pushing new frontiers of research in both learning science as well as machine learning and knowledge representation.  This ensures that the Enlearn Platform will continuously improve over time with the latest state-of-the-art research conducted for the explicit purpose of raising the outcomes of all content already deployed by the platform.

Key Research-based Innovations that

Power the Enlearn Platform

The Enlearn Platform was built based on recent research in not just learning science, but also machine learning, human-computer interaction, and knowledge representation.  They can be broadly grouped in the following categories:

  • Development of productive struggle and growth mindset through micro-level feedback on the problem solving process (that builds meta-cognitive and non-cognitive skills).  University of Washington researchers joined forces with Carol Dweck (the eminent researcher on learning mindset) to publish a series of findings that outline a new way to scaffold and develop practices that promote productive struggle and growth mindset [O’Rourke 2014, O’Rourke 2015, O’Rourke 2016].  Large scale trials of over 50,000 students showed that the novel way to provide feedback to the student with Brainpoints (rewards for demonstrating growth mindset behaviors) helped them solve over 30% more problems than the control group [O’Rourke 2014].  Research also showed that the impact on persistence when facing a difficult challenge is most strongly beneficial to the lowest performing students based on earlier game performance [O’Rourke 2014].
  • Novel representation of learning content capable of tracking student thinking processes and detailed misconceptions, while automatically providing appropriate level of scaffolding through fading worked examples and contrasting cases.  Researchers at the UW have shown that the innovative thought process encoding of conceptual and procedural knowledge can enable generation of highly detailed ontologies of knowledge that are far more detailed than standard manually designed expert maps [Andersen 2013, O’Rourke 2016, Butler 2016, Butler 2016].  This knowledge structure can also generate just the right problem for practically any misconception that a student may be facing, and has been extended to creating complex game levels and even word problems [Butler 2015, Polozov 2015, Chen 2016, Butler 2017]. This new paradigm also enables the platform to track student progress on a thought-level basis, enabling rapid high-precision diagnostics and just-in-time hints and suggestions [O’Rourke 2015, Feldman 2016].
  • Self-improving data-driven methods that optimize learning pathways in complex learning ecosystems by predicting student actions [Liu 2013, Lee 2014] and providing ideal situation-specific activities for both students and teachers.  Enlearn’s founders recently published on newly developed machine learning methods that were specifically designed to deal with complex learning environments that can vary the curriculum depending on individual student histories [Mandel 2016], or in cases where new interventions are constantly being added [Mandel 2016]. These novel reinforcement learning algorithms have been shown to significantly improve the learning outcomes through optimal sequencing of problems for each student. The improvement is particularly stark when compared to the expert-designed progressions [Mandel 2014].
  • Using online software to study and optimize how students react to hints and rewards and other common but poorly-studied elements of educational technology [Andersen 2010, Andersen 2011, Andersen 2011, Liu 2011, Andersen 2012, O’Rourke 2013, O’Rourke 2014]. UW researchers have also developed new data visualization and machine learning methods for investigating student behavior in complex experiments where standard study techniques do not apply [Liu 2014, Liu 2014, Mandel 2015, Mandel 2016]. Much of this work also relies on careful instructional and game design to even have a platform with which to run these experiments, including domains such as fractions, algebraic thinking, biochemistry, and computational thinking (Bauer 2017).
  • Focus on optimizing the entire learning ecosystem rather than just one aspect of learning (which is endemic to practically all research in learning science) [O’Rourke 2016]. Recent work at the UW on large-scale educational campaigns has revealed the importance of accounting for all aspects of the learning ecosystem in order to maximize outcomes at scale. The deployment to over 45,000 students across the state populations of Washington and Minnesota, as well as the entire country of Norway, has shown that when we create the combination of highly parameterized learning content, adaptive learning structures that account for both cognitive and non-cognitive aspects of students, and technology-enabled ways to empower teachers, significant leaps in learning are possible at scale. As just one example, nearly 50% of students in Minnesota’s challenge were able to reach and demonstrate their mastery by solving multiple algebraic equations without feedback. And this includes students who quit early on – that figure rises to 96% demonstrating mastery for students who played at least the requested 1.5 hours. This is made all the more impressive because randomized experiments showed that less than 5% of students given those equations were able to solve them at the outset of the game, showing that students were gaining skill through the challenge [Liu 2015].


Parameterized Problem Space Synthesis

Andersen, E., Gulwani, S., & Popović, Z. (2013, April). A trace-based framework for analyzing and synthesizing educational progressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 773-782). ACM.

Butler, E., Andersen, E., Smith, A. M., Gulwani, S., & Popović, Z. (2015, April). Automatic game progression design through analysis of solution features. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 2407-2416). ACM.

O’Rourke, E., Andersen, E., Gulwani, S., & Popović, Z. (2015, April). A Framework for Automatically Generating Interactive Instructional Scaffolding. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1545-1554). ACM.

Polozov, O., O’Rourke, E., Smith, A. M., Zettlemoyer, L., Gulwani, S., & Popović, Z. (2015, June). Personalized mathematical word problem generation. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI 2015). To appear.

Butler, E., Torlak, E., Popović, Z. (2016). A Framework for Parameterized Design of Rule Systems Applied to Algebra. Proceedings of Intelligent Tutoring Systems (pp 320-326).

Butler, E., Torlak, E., Popović, Z. (2016). Synthesizing Custom Tutoring Rules for Introductory Algebra. In submission.

Butler, E., Torlak, E., & Popović, Z. (2017). Synthesizing Explainable Strategies for Solving Puzzle Games. In Foundations of Digital Games (FDG 2017).

Chen, Y., Mandel, T., Liu, Y. E., & Popović, Z. (2016). Crowdsourcing Accurate and Creative Word Problems and Hints. Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP).

Feldman, M., Andersen, E., Gulwani, S., Popović, Z. (2016). CheckMark: Automatic Diagnosis of Student Misconceptions. In submission.

O’Rourke, E., Butler, E., Tolentino, A.D., Popović, Z. (2016) Automatic Generation of Problems and Explanations for an Intelligent Algebra Tutor. In submission.


Development of Non-cognitive and Metacognitive Behaviors

O’Rourke, E., Haimovitz, K., Ballweber, C., Dweck, C., & Popović, Z. (2014, April). Brain points: a growth mindset incentive structure boosts persistence in an educational game. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems (pp. 3339-3348). ACM.

O’Rourke, E., Chen, Y., Haimovitz, K., Dweck, C. S., & Popović, Z. (2015, March). Demographic Differences in a Growth Mindset Incentive Structure for Educational Games. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale (pp. 331-334). ACM.

O’Rourke, E., Peach, E., Dweck, C. S., & Popović, Z. (2016, April). Brain Points: A Deeper Look at a Growth Mindset Incentive Structure for an Educational Game. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale (pp. 41-50). ACM.


Data-driven Experimental Design and Optimal Learning Pathways

Andersen, E., Liu, Y. E., Apter, E., Boucher-Genesse, F., & Popović, Z. (2010, June). Gameplay analysis through state projection. In Proceedings of the fifth international conference on the foundations of digital games (pp. 1-8). ACM.

Liu, Y. E., Andersen, E., Snider, R., Cooper, S., & Popović, Z. (2011, June). Feature-based projections for effective playtrace analysis. In Proceedings of the 6th international conference on foundations of digital games (pp. 69-76). ACM.

Lee, S. J., Liu, Y. E., & Popović, Z. (2014, July). Learning Individual Behavior in an Educational Game: A Data-Driven Approach. In Educational Data Mining 2014.

Liu, Y. E., Mandel, T., Butler, E., Andersen, E., O’Rourke, E., Brunskill, E., & Popović, Z. (2013, July). Predicting player moves in an educational game: A hybrid approach. In Educational Data Mining 2013.

Liu, Y. E., Mandel, T., Brunskill, E., & Popović, Z. (2014, April). Towards automatic experimentation of educational knowledge. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3349-3358). ACM.

Liu, Y. E., Mandel, T., Brunskill, E., & Popović, Z. (2014, July). Trading off scientific knowledge and user learning with multi-armed bandits. In Educational Data Mining 2014.

Mandel, T., Liu, Y. E., Levine, S., Brunskill, E., & Popović, Z. (2014, May). Offline policy evaluation across representations with applications to educational games. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems (pp. 1077-1084). International Foundation for Autonomous Agents and Multiagent Systems.

Mandel, T., Liu, Y. E., Brunskill, E., & Popović, Z. (2015, January). The Queue Method: Handling Delay, Heuristics, Prior Data, and Evaluation in Bandits. In AAAI (pp. 2849-2856).

Mandel, T., Liu, Y., Brunskill, E., & Popović, Z. (2016, February). Offline evaluation of online reinforcement learning algorithms. In Proceedings of the Thirtieth Conference on Artificial Intelligence.

Mandel, T., Liu, Y. E., Brunskill, E., & Popović, Z. (2016) Efficient Bayesian Clustering for Reinforcement Learning. Proceedings of IJCAI.

Mandel, T., Liu, Y. E., Brunskill, E., & Popović, Z. (2016) Where to Add Actions in Human-in-the-Loop Reinforcement Learning. In submission.


Learning Ecosystem Trials at Scale

Liu, Y. E., Ballweber, C., O’Rourke, E., Butler, E., Thummaphan, P., & Popović, Z. (2015). Large-scale educational campaigns. ACM Transactions on Computer-Human Interaction (TOCHI), 22(2), 8.

O’Rourke, E., Thummaphan, P., Popović, Z. (2016). Personalized Learning for the Classroom: A Hybrid Approach. In submission.


Multivariate Trials

Andersen, E., Liu, Y. E., Snider, R., Szeto, R., & Popović, Z. (2011, May). Placing a value on aesthetics in online casual games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1275-1278). ACM.

Andersen, E., Liu, Y. E., Snider, R., Szeto, R., Cooper, S., & Popović, Z. (2011, June). On the harmfulness of secondary game objectives. In Proceedings of the 6th International Conference on Foundations of Digital Games (pp. 30-37). ACM.

Andersen, E., O’Rourke, E., Liu, Y. E., Snider, R., Lowdermilk, J., Truong, D., … & Popović, Z. (2012, May). The impact of tutorials on games of varying complexity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 59-68). ACM.

O’Rourke, E., Butler, E., Liu, Y. E., Ballweber, C., & Popović, Z. (2013). The effects of age on player behavior in educational games. In FDG (pp. 158-165).

O’Rourke, E., Ballweber, C., & Popović, Z. (2014, March). Hint systems may negatively impact performance in educational games. In Proceedings of the first ACM conference on Learning@Scale conference (pp. 51-60). ACM.


Game and Instructional Design

Bauer, A., Butler, E., & Popović, Z. (2017). Dragon Architect: Open Design Problems for Guided Learning in a Creative Computational Thinking Sandbox Game. In Foundations of Digital Games (FDG 2017).