Recommending Tasks in Online Judges

Abstract

Online Judges are e-learning tools used to improve the programming skills, tipically for programming contests such as International Olympiads in Informatics and ACM International Collegiate Programming Contest. In this context, due to the nowadays broad list of programming tasks available in Online Judges, it is crucial to help the learner by recommending a challenging but not unsolvable task. So far, in the literature, few authors focused on Recommender Systems (RSs) for Online Judges; in this paper we discuss some peculiarities of this problem, that prevent the use of standard RSs, and address a first building brick: the assessment of (relative) tasks hardness. We also present the results of a preliminary experimental evaluation of our approach, that proved to be effective against the available dataset, consisting in all the submissions made in the Italian National Online Judge, used to train students for the Italian Olympiads in Informatics.

Towards a Human-AI Hybrid System for Categorising Programming Problems

“…Instead, POJ studies use student data -e.g., the number of accepted problems, wrong answers, or compilation errors -to propose methods to help in learning programming. Such data is used to predict learner performance [1,3,22,31,32,34,36], estimate dropout [21,33], recommend tasks [5,18,39], or detect problem difficulty [9,17,24]. Next, we look at the (very small subset of) POJ studies targeting the same problem as us -that of predicting problem categories (also called problem topics) in online judges.…”

“…POJs are platforms used by many learners who wish to improve their programming skills. These systems have been shown to enhance education, promote competitive programming and support recruitment processes [5,6,35,39,41]. They are widely used in job interviews of large technology companies, such as Google, Amazon, etc.…”

“…Nonetheless, this massive variety of problems are typically piled in volumes which are not organised by categories or difficulty levels [5,17,39,41]. Zhao et al [41] explain that organising problems by volume has low maintenance costs for developing and updating a POJ system, since many problems could be easily collected (e.g., from an annual ACM International Collegiate Programming Contest (ICPC)) and piled into a new volume, without any classification.…”

Recommending Tasks in Online Judges using Autoencoder Neural Networks

“…As observed in (Audrito et al, 2019), there are some peculiarities of Online Judges that prevent the use of a general Recommender System: Users slowly improve their abilities, one task after the other, so the general con-• cept of user preferences does not apply: recommending a movie or a novel differs significantly from recommending a task; a user will probably still like a novel after one year, whilst he might find a task too easy after the same amount of time.…”