How to Design Effective Learning Analytics Indicators? A Human-Centered Design Approach

Chatti, Mohamed Amine; Muslim, Arham; Guesmi, Mouadh; Richtscheid, Florian; Nasimi, Dawood; Shahin, Amin; Damera, Ritesh

doi:10.1007/978-3-030-57717-9_22

Cited by 12 publications

(12 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through questionnaires, learners were asked what indicators are relevant for them in an academic advising dashboard [18] or what content should be included in an online self-assessment test for prospective students in online higher education [12]. Drawing from the HCI and information visualisation fields, Chatti et al [9] proposed a human-centred indicator design (HCID) framework for LA to "get the right indicator" and to "get the indicator right" by actively involving learners in needs analysis, ideation, prototyping, and testing. In multiple studies, students expressed interest towards customisable dashboards that could fit their individual needs [2,34].…”

Section: Feedback With Ladsmentioning

confidence: 99%

Quantum of Choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related

Jivet

Wong

Scheffel

et al. 2021

LAK21: 11th International Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

Learning analytics dashboards (LADs) are designed as feedback tools for learners, but until recently, learners rarely have had a say in how LADs are designed and what information they receive through LADs. To overcome this shortcoming, we have developed a customisable LAD for Coursera MOOCs on which learners can set goals and choose indicators to monitor. Following a mixedmethods approach, we analyse 401 learners' indicator selection behaviour in order to understand the decisions they make on the LAD and whether learner goals and self-regulated learning skills influence these decisions. We found that learners overwhelmingly chose indicators about completed activities. Goals are not associated with indicator selection behaviour, while help-seeking skills predict learners' choice of monitoring their engagement in discussions and time management skills predict learners' interest in procrastination indicators. The findings have implications for our understanding of learners' use of LADs and their design.

show abstract

Section: Feedback With Ladsmentioning

confidence: 99%

Quantum of Choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related

Jivet

Wong

Scheffel

et al. 2021

LAK21: 11th International Learning Analytics and Knowledge Conference

View full text Add to dashboard Cite

show abstract

“…The HCD process includes observation of the target group, ideation or generating ideas, prototyping and testing (Norman, 2013). This approach has already been applied in the area of LA, for example, to develop the human-centred indicator design approach (Chatti et al, 2020).…”

Section: Continuous Improvement Cyclementioning

confidence: 99%

Assessment validity and learning analytics as prerequisites for ensuring student‐centred learning design

Divjak

Svetec

Horvat

et al. 2022

Brit J Educational Tech

View full text Add to dashboard Cite

To ensure the validity of an assessment programme, it is essential to align it with the intended learning outcomes (LO). We present a model for ensuring assessment validity which supports this constructive alignment and uses learning analytics (LA). The model is based on LA that include a comparison between ideal LO weights (expressing the prioritization of LOs), actual assessment weights (maximum assessment points per LO), and student assessment results (actually obtained assessment points per LO), as well as clustering and trace data analysis. These analytics are part of a continuous improvement cycle, including strategic planning and learning design (LD) supported by LO prioritization, and monitoring and evaluation supported by LA. To illustrate and test the model, we conducted a study on the example of a graduate‐level higher education course in applied mathematics, by analysing student assessment results and activity in a learning management system. The study showed that the analyses provided valuable insights with practical implications for the development of sound LD, tailored educational interventions, databases of assessment tasks, recommendation systems, and self‐regulated learning. Future research should investigate the possibilities for automation of such LA, to enable full exploitation of their potential and use in everyday teaching and learning. Practitioner notesWhat is already known about this topic To develop sound, student‐centred learning design (LD), it is essential to ensure that assessment is constructively aligned with the intended learning outcomes (LO). This constructive alignment is crucial for ensuring the validity of an assessment program. Learning analytics (LA) can provide insights that help develop valid assessment programs. What this paper adds As not all LOs are equally important, assessment programs should reflect the prioritization of LOs, which can be determined by using various multi‐criteria decision‐making (MCDM) methods. This article presents and illustrates, based on an empirical case, a model of continuous improvement of LD, which uses LA to compare how LOs are reflected in (actual) students' results, in an (actual) assessment program, and in the (ideal) prioritization of LOs based on MCDM. The study presents how clustering of students based on their assessment results can be used in LA to provide insights for educational interventions better targeted to students' needs. Implications for practice and/or policy The proposed LA can provide important insights for the development (or improvement) of LD in line with the intended course LOs, but also study program LOs (if course and study program LOs are properly aligned). The LA can also contribute to the development of databases of assessment tasks aligned with course LOs, with ensured validity, supporting sharing and reusing, as well as to the development of tailored educational interventions (eg, based on clustering). The proposed LA can also contribute to the development of recommendation systems, with recommendations for the improvement of LD for teachers or learning suggestions for students, as well as students' meta‐cognition and self‐regulated learning.

show abstract

“…We adopt the following definition provided by Shaw [29]: "A design space identifies and organizes the decisions to be made, together with the alternatives for those decisions, thereby providing guidance for creating artifacts or a framework for comparing them. " Some authors (e.g., [7]) propose to use 5 W's questions to address such a design space, focusing on the object of interest, from visualisation to dashboards and learning process analysis. In the context of LADs, we formulate the 5 W's as follows: (i) Who?…”

Section: Lad Design Spacementioning

confidence: 99%

“…Visualization and Sensemaking in LADs. HoW is a central question for dashboards, as relevant visualizations are required to gain insights [25,7]. As identified in the "Why" section, dashboard may support a sensemaking process, where the user(s) may navigate across different levels of sensemaking and may need to analyse some data.…”

Section: Lad Design Spacementioning

confidence: 99%

“…Many dashboard designs fail to measure the appropriateness of the incorporated visualizations for their visual literacy levels [27]. Some authors emphasize the importance of empowering the stakeholders [7,17]; and (4) LADs often fail in turning insights into action since the processes by which people use these representations for insight seeking and decision-making are still not well understood [32].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Designing LADs That Promote Sensemaking: A Participatory Tool

Sadallah

Gilliot

Iksal³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

How to Design Effective Learning Analytics Indicators? A Human-Centered Design Approach

Cited by 12 publications

References 19 publications

Quantum of Choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related

Quantum of Choice: How learners’ feedback monitoring decisions, goals and self-regulated learning skills are related

Assessment validity and learning analytics as prerequisites for ensuring student‐centred learning design

Designing LADs That Promote Sensemaking: A Participatory Tool

Contact Info

Product

Resources

About