Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes

Abstract

People are increasingly turning to online resources to learn to code. However, despite their prevalence, it is still unclear how successful these resources are at teaching CS1 programming concepts. Using a pretest-posttest study design, we measured the performance of 60 novices before and after they used one of the following, randomly assigned learning activities: 1) complete a Python course on a website called Codecademy, 2) play through and finish a debugging game called Gidget, or 3) use Gidget's puzzle designer to write programs from scratch. The pre-and posttest exams consisted of 24 multiple choice questions that were selected and validated based on data from 1,494 crowdsourced respondents. All 60 of our novices across the three conditions did poorly on the exams overall in both the pre-tests and post-tests (e.g., the best median post-test score was 50% correct). However, those completing the Codecademy course and those playing through the Gidget game showed over a 100% increase in correct answers when comparing their post-test exam scores to their pretest exam scores. Those playing Gidget, however, achieved these same learning gains in half the time. This was in contrast to novices that used the puzzle designer, who did not show any measurable learning gains. All participants performed similarly within their own conditions, regardless of gender, age, or education. These findings suggest that discretionary online educational technologies can successfully teach novices introductory programming concepts (to a degree) within a few hours when explicitly guided by a curriculum.

An Automated Environment for Teaching Programming Logic on Distance Learning IT Courses

“…The use of tools that allow interaction has been demonstrated to have positive results. A study presented in (Lee and Ko, 2015), demonstrated an improvement of up to 40% in the success rate of programming courses with interactive tools such as Codecademy.…”

Towards Understanding Successful Novice Example Use in Blocks-Based Programming

“…Some blocks-based environments are games-based, such as Blockly Games [64] and Code.org [6]. Games have been shown to be effective learning mechanisms for novice programmers [65], but often do not allow users to create their own project. Furthermore, learning through games does not apply to learning more advanced programming, where example use is critical.…”

Predicting abandonment in online coding tutorials

“…To determine features for predicting abandonment, we considered five sources: 1) factors that appeared related to engagement from previous work on Gidget [2], [3], [25], [29]; 2) factors from studies of MOOCs and in-person CS1 courses; 3) features that we brainstormed based on the available data in the Gidget activity logs; 4) features proven to be significant in existing work on MOOC dropout prediction; and 5) features that are straightforward to interpret and informative to game designers. Table I lists the 12 features from this brainstorming process.…”