Traditionally, the assessment of upper limb (UL) motor function in neurorehabilitation is carried out by clinicians using standard clinical tests for objective evaluation, but which could be influenced by the clinician's subjectivity or expertise. The automation of such traditional outcome measures (tests) is an interesting and emerging field in neurorehabilitation. In this paper, a systematic review of systems focused on automation of traditional tests for assessment of UL motor function used in neurological rehabilitation is presented. A systematic search and review of related articles in the literature were conducted. The chosen works were analyzed according to the automation level, the data acquisition systems, the outcome generation method, and the focus of assessment. Finally, a series of technical requirements, guidelines, and challenges that must be considered when designing and implementing fully-automated systems for upper extremity functional assessment are summarized. This paper advocates the use of automated assessment systems (AAS) to build a rehabilitation framework that is more autonomous and objective. INDEX TERMS Automatic assessment, biomedical engineering, motor function, neurorehabilitation, rehabilitation robotics, robotics and automation, upper extremity. TERMINOLOGY To reduce the ambiguity in the clinical terminology, the definition of the terms that will be used along the text are given as follows. • Test or clinical tool: this is understood as the procedure that the patients must perform in order to assess the functionality of the upper extremities. It encompasses a series of steps and rules for its proper administration. It can be single or multi-item. • Item: the movement or single task that the patient must perform. • Outcome measure: the result of a test that is used to objectively determine the UE function.