Experiences with marmoset

Abstract

We developed Marmoset, an automated submission and testing system, to explore techniques to provide improved feedback to both students and instructors as students work on programming assignments, and to collect data to perform detailed research on the development processes of students.To address the issue of feedback, Marmoset provides students with limited access to the results of the instructor's private test cases using a novel token-based incentive system. This both encourages students to start their work early and to think critically about their work. Because students submit early, instructors can monitor all students' progress on test cases, helping identify challenging or ambiguous test cases early in order to update the project specification or devote additional time in lecture or lab sessions to the difficult test cases.To study and better understand the development process of students, Marmoset can be configured to transparently capture snapshots to a central repository everytime students save their files. These detailed development histories offer a unique, detailed perspective of each student's progress on a programming assignment, from the first line of code written and saved all the way through the final edit before the final submission. This type of data has proven extremely valuable many uses, such as mining new bug patterns and evaluating existing bug-finding tools.In this paper, we describe our initial experiences using Marmoset in several introductory computer science courses, from the perspectives of both instructors and students. We also describe some initial research results from analyzing the student snapshot database.

Addressing drop-out and sustained effort issues with large practical groups using an automated delivery and assessment system

“…Such test cases are usually only visible to the teaching staff, and the student view is limited to the feedback offered by the system (Arnow & Barshay, 1999;Feldman & Jones, 1997;Jackson & Usher, 1997;Jones, 2000;Reek, 1996). A half-way solution is illustrated by the marmoset platform, a testing system that defines different types of test cases: student test, public test, release tests and secret tests (Spacco et al, 2006). A common claim of these studies is that managing a large number of student assignments is only possible through the automation of the grading process.…”

Transitioning to a Large-Scale Distributed Programming Course

“…Malmi and Korhonen [22] describe similar results when explaining the impact of immediate automated feedback in the correction workflow of a data structures and algorithms course. Assessment systems including those such as Online Judge by Cheang et al [14], ArTEMiS by Krusche and Seitz [15], and Marmoset by Spacco et al [16]. These systems rely on a set of secret test cases that can be augmented with public test cases as described in Experiences with Marmoset by Spacco et al [16].…”

“…Assessment systems including those such as Online Judge by Cheang et al [14], ArTEMiS by Krusche and Seitz [15], and Marmoset by Spacco et al [16]. These systems rely on a set of secret test cases that can be augmented with public test cases as described in Experiences with Marmoset by Spacco et al [16]. In addition to unit tests, Liu and Petersen use static code analysis in an online submission system for a Python-based programming course and saw a reduction in the number of repeated errors in the students' submissions [17].…”

Assessing Incremental Testing Practices and Their Impact on Project Outcomes

“…In the classroom, Desai [14] acknowledges the challenges associated with introducing testing into the curriculum, and notes that regular, reinforced learning of testing might be better than only an introduction to it at the start of the semester. Jones [25], Edwards [15,16] and others [35,36] have worked toward introducing testing into the curriculum. Unfortunately, most prior work in this area tends to focus on novice programmers working on small projects, where the benefits of testing are not readily apparent [5], and on 'after-the-fact' feedback.…”