The Quiver system

Ellsworth, Christopher C.; Fenwick, James B.; Kurtz, Barry L.

doi:10.1145/1028174.971374

Cited by 14 publications

(9 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the educational context, the purpose of such frameworks is often to provide students with consistent grading (e.g., in an exam situation, see Watson, Li, & Godwin, 2013;Cali↵ & Goodwin, 2002) or feedback (e.g., when learning to write programs; see Seppälä, 2012). Many technical support frameworks have been developed for use in computer science education (see Cheang, Kurnia, Lim, & Oon, 2003;Ellsworth, Fenwick, & Kurtz, 2004;Summary English, 2002 for examples andAla-Mutka, 2005;Daly & Waldron, 2004;Joy, Gri ths, & Boyatt, 2005; for reviews). In particular, frameworks have been developed to support students in learning test-driven development (see, e.g., Edwards, 2004).…”

Section: Concerning (4) Inmentioning

confidence: 99%

Inferring Skill from Tests of Programming Performance: Combining Time and Quality

Bergersen

Hannay

Sjøberg

et al. 2011

2011 International Symposium on Empirical Software Engineering and Measurement

View full text Add to dashboard Cite

Skilled developers are important to the software industry. In empirical studies in software engineering, knowing the skill level of the participants is also important for correct interpretation of results. The current practice in industry and research for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and e cient way to measure programming skill. Consequently, this thesis aimed to construct a valid instrument for measuring programming skill, where skill is inferred from performance on programming tasks. The Rasch measurement model was used to construct the instrument. Sixty-five professional developers from eight countries participated in validating the instrument, solving 19 Java programming tasks over two days. The validity of the instrument was theoretically investigated through commercial and research-based tests. Programming skill, as measured by the instrument, was also investigated in terms of experience and other background variables. The instrument was found to have desirable psychometric properties, and the overall results appear well aligned with theoretical expectations. This work has shown that acceptable measures of programming skill may be obtained with less than one day of testing. Further work should be directed at reducing the time needed to measure programming skill without a↵ecting the validity of the instrument. The results of the research have already been transferred to the industry through a commercial prototype. iii First and foremost, I am profoundly thankful to my principal supervisor Dag Sjøberg. He has provided me with unwavering support during my research for this thesis, often at odd hours and in strange places. I feel privileged to have had the opportunity to occupy so much of Dag's time over the years. The making of this thesis has been a truly rewarding experience, both professionally and personally, because of him. I am also grateful to my second supervisor, Tore Dybå, for interesting discussions and for inspiring me to look to other research fields for solutions to my research problem. Other researchers have also been important at various stages. I thank Erik Arisholm for his guidance and flexibility during the early years of my work. A special thanks goes to Jo Hannay and Magne Jørgensen, who are highly inspiring researchers. I am also grateful to Jan-Eric Gustafsson for his interest in my work. This thesis would have been impossible without financial support. Simula Research Laboratory gave me the opportunity to carry out the research and Simula Innovation and Simula School of Research and Education provided support as well. The FORNY program at the Norwegian Research Council funded parts of my work, while the University of Oslo allowed me to finish the thesis at a pace that was rewarding for me. This thesis would similarly have been impossible without technical support. Early on, Steinar Haugen shared his passion for programming with me and has since become a...

show abstract

Section: Concerning (4) Inmentioning

confidence: 99%

Inferring Skill from Tests of Programming Performance: Combining Time and Quality

Bergersen

Hannay

Sjøberg

et al. 2011

2011 International Symposium on Empirical Software Engineering and Measurement

View full text Add to dashboard Cite

show abstract

“…Test cases are usually supplied by the teacher and/or randomly generated [47]. Testing is used as an evaluation component of a number of web-based submission and evaluation systems [11,18,23,31,33,50,51,65,72]. Aside from checking functional correctness, testing can also be used for analysing efficiency, memory violations and run-time errors [1].…”

Section: Related Workmentioning

confidence: 99%

“…Concerning evaluation based on automated testing [11,18,23,31,33,50,51,65,72] and automated bug-finding [36,37,71,78], our approach is complementary and it gives an additional confidence on functional correctness of the program. We think that the best way to use it is to apply it on programs where cheaper techniques such as testing and automated bug-finding did not discover any bugs.…”

Section: Relationship To Other Approachesmentioning

confidence: 99%

Regression verification for automated evaluation of students programs

Vujosevic-Janicic

Marić

2020

COMSIS J

View full text Add to dashboard Cite

Regression verification is a form of software verification based on formal static analysis of code, which is used, since recently, in several domains. In this paper we examine potentials of using it in one novel domain-in automated evaluation of students programs. We propose an approach that provides precise assessment of functional correctness of student programs (while it does not address nor affect the teaching methodology). We describe our open-source, publicly available implementation of the approach, which is built on top of the compiler infrastructure LLVM and the software verification tool LAV. The results of evaluating the proposed approach on two real-world corpora of student programs and on a number of classic algorithms show that the proposed approach can be used as a precise and reliable supplementary technique in grading of student programs at introductory programming courses, algorithms courses and programming competitions.

show abstract

“…24 We recognize that published research exists on many comparable tools, but we did not readily find data that addressed our questions. Still, we will continue to track results relevant to other tools, including: ALOHA, 25 ASAP, 26 ASSYST, 27 AutoGradeMe, 28 AWAT, 29 Bottlenose, 30 ClockIt, 31 CloudCoder, 32 CodeAssessor, 33 CodeLab, 34 CTPracticals, 35 EasyAccept, 36 EduComponents, 37 ELP, 38 FitchFork, 39 HoGG, 40 JEWL, 41 Junit, 42 Kattis, 43 Linuxgym, 44 Marmoset, 45 Moe, 46 Online Judge, 47 PASS, 48 Peach, 49 ProgTest, 50 ProtoAPOGEE, 51 Quiver, 52 QuizPACK, 53 Resolver, 54 RoboCode, 55 RoboProf, 56 Scheme-robo, 57 TRY, 58 USACO's, 59 UVA Online Judge, 60 VERKKOKE, 61 WebToTeach, 62 and WeBWorK-JAG. 63 Several excellent surveys exist on automated assessment tools and their behaviors.…”

Section: Related Workmentioning

confidence: 99%

Are Automated Assessment Tools Helpful in Programming Courses?

Pettit

Homer

McMurry

et al.

2015 ASEE Annual Conference and Exposition Proceedings

View full text Add to dashboard Cite

Raymond S. Pettit teaches courses in programming, artificial intelligence, objected oriented design, algorithms, theory of computation, and related subjects in ACU's School of Information Technology and Computing. Prior to joining the ACU faculty, he spent twenty years in software development, research, and training the Air Force Research Lab and NASA's Langley Research Center as well as private industry. His current research focuses on how automated assessment tools interact with student learning in university programming courses. Are Automated Assessment Tools Helpful in ProgrammingCourses? AbstractAutomated assessment tools (AATs) are growing in popularity in introductory programming courses, but researchers may have a difficult time synthesizing valid data to draw conclusions about the tools' usefulness. Our first step addressing this issue was to break down our overriding question-are automated assessment tools helpful in programming courses?-into four more specific questions: (1) Have AATs proven to be helpful in improving student learning? (2) Do students think that AATs have improved their performance? (3) After having used the tools, do instructors think that the tools have improved their teaching experiences? and (4) Is the assessment performed by AATs accurate enough to be helpful? In discussing the many AATs that exist, many researchers have only reported results relevant to one or two of these specific questions. We address each of our four questions separately and draw on data from 24 different tools to arrive at our conclusions. We determine that the literature demonstrates AATs helpfulness in student learning, instructor support, and assessment accuracy. However, we found results about students' opinions regarding the helpfulness of AATs to be inconclusive. Given our findings, we make suggestions both for instructors using these tools and to researchers creating them.

show abstract

The Quiver system

Cited by 14 publications

References 2 publications

Inferring Skill from Tests of Programming Performance: Combining Time and Quality

Inferring Skill from Tests of Programming Performance: Combining Time and Quality

Regression verification for automated evaluation of students programs

Are Automated Assessment Tools Helpful in Programming Courses?

Contact Info

Product

Resources

About