Multidimensionality through self-evaluation: From theory to practice in the Brazilian graduate system

Abstract

Nearly all science and technology research in Brazil is conducted within a national system of graduate education. Since the 1970s, a graduate program assessment has been an integral part of such a system, and it is currently held on a quadrennial basis. The evaluation model is dynamic, evolving from the experiences of evaluators, policymakers, and the scientific community during each four-year cycle. This study analyses policy initiatives from the 2017-2021 evolving effort, focusing on strategies and recommendations to implement multidimensionality and self-evaluation as integral components of Brazilian evaluation. The paper traces how the idea for a multidimensional assessment was introduced in the country and how U-Multirank, an international ranking of higher education institutions (HEI), has come to inspire an evaluation that is not institutional but of graduate programs instead. The study identified some benefits and limitations of the chosen inspiration and analysed how the Brazilian proposal aligned with the U-Multirank principles. Furthermore, the investigation shows there is little concrete difference from the proposed new model to the one Brazil has already in place. Finally, the last section of this study looks into the once pivotal idea to pursue a self-evaluation component, now relegated to a minor role in the model, but that could be raised to a position supporting the design of an actual multidimensional assessment model.

Evaluation policies for research and graduate education in the Netherlands: lessons on self-evaluation and institutional autonomy

“…As mentioned in Brasil (2022), multidimensional evaluation is one in which the mission or vocation of a research program can be valued through a nonreductionist analysis, distinct from the traditionally adopted evaluation process in Brazil. In evaluations conducted by the Coordination of Improvement of Higher Education Personnel (CAPES) or Brazil's National Institute for Space Research (INEP), postgraduate programs or higher education institutions receive a grade based on the results of their evaluations.…”

As políticas de avaliação da pesquisa e da pós-graduação nos Países Baixos: lições sobre autoavaliação e autonomia institucional

“…Conforme descrito em Brasil (2022), a avaliação multidimensional é aquela em que a missão ou vocação de um programa de pesquisa pode ser valorizada com base em uma análise não reducionista, de forma distinta do processo de avaliação tradicionalmente adotado no Brasil. Nas avaliações realizadas pela Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) ou pelo Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP), programas de pós-graduação ou instituições de ensino superior recebem uma nota consoante ao resultado das suas avaliações.…”

The dynamics of a national evaluation system: The Brazilian model and its role in developing research and graduate education in the country

“…The report of the responsible working group outlined strategies to establish such a system and promote a more multidimensional evaluation process (Verhine et al, 2019). The predicted implementation for the upcoming evaluation of 2021 was modest, reflected in a review of the assessment form that included self-assessment as a subdimension with little consequence on the final grade attributed to the graduate programs (Brasil, 2022). Despite the limited effect at first, the experience of the 2021 evaluation would be valuable to rethink strategies for a greater impact in the future.…”