Metric indicators for the evaluation of graduate programs in Brazil: from Qualis to multi-criteria

Abstract

Brazilian scientific production has been mostly developed (90%) by Brazilian universities, mainly graduate programs, which must be assessed and ranked by the Brazilian government for their maintenance. The Qualis system is used for this classification by valuing the scientific production of graduate programs, stratifying journals and assigning grades. Several Brazilian researchers affirm that the Qualis system is inaccurate and subjective because it is carried out by a group of researchers. This work aimed to propose methods to evaluate Brazilian scientific production in order to improve the assessment of graduate programs through multi-criteria methods in addition to Qualis. The application of different metrics presented results significantly different from Qualis, including the over valuation of journals with a low international impact factor. The proposed metrics considered: (i) web of science impact factor of the journals; (ii) Citations from articles; (iii) Citations of citations, i.e., a new metric; and (iv) H-Index for researchers. It was proposed the multi-criteria method, composed of the mentioned criteria, in addition to Qualis. For the sample of researchers, it was demonstrated that multi-criteria methods can assess scientific production more accurately. Therefore, they are more adequate to assess the Brazilian graduate programs, considering several internationally accepted criteria.