The development and operation of Online Judge System (OJS), which is used to evaluate the correctness of programs, is a nontrivial and difficult task due to the various functional and non-functional requirements. However, although many OJSs have been developed and operated, and their usefulness reported, the theory for constructing OJSs has not been sufficiently discussed. In this paper, we present the functional and nonfunctional requirements oriented to OJS as well as demonstrate the internal components and software architecture of an OJS, which has been in operation for over a decade and has evaluated over six million solutions. We also present real-world experiences and challenges encountered during this long journey of our OJS.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.