A computational approach to evaluating curricular alignment to the united nations sustainable development goals

Abstract

The United Nations (UN) considers universities to be key actors in the pursuit of the Sustainable Development Goals (SDGs). Yet, efforts to evaluate the embeddedness of the SDGs in university curricula tend to rely on manual analyses of curriculum documents for keywords contained in sustainability lexica, with little consideration for the diverse contexts of such keywords. The efficacy of these efforts, relying on expert co-elicitation in both subject-matter contexts and sustainability, suffers from drawbacks associated with keyword searches, such as limited coverage of key concepts, difficulty in extracting intended meaning and potential for greenwashing through “keyword stuffing.” This paper presents a computational technique, derived from natural language processing (NLP), which develops a sustainability lexicon of root keywords (RKs) of relative importance by adapting the Term Frequency–Inverse Document Frequency (TF-IDF) method to a corpus of sustainability documents. Identifying these RKs in module/course descriptors offers a basis for evaluating the embeddedness of sustainability in 5,773 modules in a university's curricula using classification criteria provided by the Association for the Enhancement of Sustainability in Higher Education's (AASHE). Applying this technique, our analysis of these descriptors found 286 modules (5%) to be “sustainability focused” and a further 769 modules (13%) to be “sustainability inclusive,” which appear to address SDGs 1, 17, 3, 7, and 15. Whilst this technique does not exploit machine learning methods applied to large amounts of trained data, it is, nevertheless, systemic and evolutive. It, therefore, offers an appropriate trade-off, which faculty with limited analytics skills can apply. By supplementing existing approaches to evaluating sustainability in the curriculum, the developed technique offers a contribution to benchmarking curricular alignment to the SDGs, facilitating faculty to pursue meaningful curricular enhancement, whilst complying with sustainability reporting requirements. The technique is useful for first-pass analyses of any university curriculum portfolio. Further testing and validation offer an avenue for future design-science research.

An evaluation of a computational technique for measuring the embeddedness of sustainability in the curriculum aligned to AASHE-STARS and the United Nations Sustainable Development Goals

“…For a list of keywords to address the issues of relevancy and duplicate hits, Lemarchand et al (2022) details a systematic framework, identifying root keywords (RKs) extracted from the 169 targets and 247 indicators describing the SDGs. In summary, an RK is the stem of a word that can be morphologically inflected.…”

“…of times the RK appears in the targets and indicators No. of SDGs in which the RK appears (1) Lemarchand et al (2022) also applied additional filtering to avoid RKs being associated with too many SDGs and hence diluting their meaningful attribution. Equation 2 defines the minimum number of occurrences an RK needs to appear in an SDG to be associated to that SDG.…”

“…One NLP technique, known as Term Frequency-Inverse Document Frequency (TF-IDF), is deployed regularly to identify keywords in documents (Munot and Govilkar, 2014). Lemarchand et al (2022) recognized the potential utility of TF-IDF for measuring sustainability in the curriculum as it aims to attribute a weighted value of terms or document frequency to evaluate the relevance of a keyword (Mishra and Urolagin, 2019). TF-IDF works well for large documents with voluminous texts, such as scientific research and policy documentation (Matsui et al, 2022).…”

“…However, module/course descriptors tend to be much more abridged documents, so the text needs to be much more judiciously written if the TF-IDF is to capture its alignment to the SDGs. Lemarchand et al (2022) use the term frequency component of TF-IDF to identify the relative importance of root keywords associated with each SDG, providing a basis for evaluating the "sustainability importance" of modules/courses. Using the "sustainability importance" concept, 5,773 TU Dublin module/course descriptors are then interrogated to identify potential SDG mappings and to categorize the modules based on AASHE-STARS criteria.…”

“…Using the "sustainability importance" concept, 5,773 TU Dublin module/course descriptors are then interrogated to identify potential SDG mappings and to categorize the modules based on AASHE-STARS criteria. The research objective addressed in this paper is to evaluate the effectiveness of this computational technique (Lemarchand et al, 2022) by comparing its results to those based on expert interpretation of a random sample of 306 of these modules/courses, in which each module/course descriptor is forensically reviewed to identify which (and to what extent) SDGs have been addressed. The objective is underpinned by two research questions: RQ1: how effective is the TF-IDF based technique in assisting with mapping of modules/courses to the SDGs?…”