The Limits of Evaluating Sustainability

Abstract

Designing technology with sustainability in mind is becoming more and more important, especially considering future scenarios of limited resources where the world's current lifestyle of wasteful consumption needs to change. But how can researchers believably argue that their solutions are indeed sustainable? How can consumers and technology users reliably acquire, understand, and apply information about environmental sustainability? Those questions are difficult to answer, especially in research domains where the impact on sustainability is not immediately measurable, such as sustainable HCI. The evaluation of sustainability is an ongoing problem that is often glossed over, but we believe the community needs to intensify its efforts to articulate its evaluation methods to other disciplines and external stakeholders. Even if those disciplines and stakeholders understand the importance of designing for sustainability, we need convincing arguments-such as validation through thorough evaluations-to showcase why a specific design solution works in the real world. In this paper, we analyze this problem by highlighting examples of sustainable HCI research in which evaluation of sustainability failed. We also look at previous research that sought to address this issue and discuss how their solutions can be generalized-and when they might fail. While we do not have the final answer, our intention is to start a discussion as to why sustainable HCI research is oftentimes not doing enough to justify the validity of its solutions. We close our paper by suggesting a few examples of what we believe to be potential ways to address those issues and take action to improve the evaluation of sustainability. CCS Concepts • General and reference → Evaluation • Human-centered computing → HCI design and evaluation methods • Social and professional topics → Sustainability.

Regenerative computing

“…We argue, that the effect of all these limits is that hope has also become limited. While most LIMITS authors take pains to avoid it, the "limited resources in futures of scarcity and collapse represent a future that is often perceived as scary, negative and difficult -and sometimes even described as apocalyptic" [28]. While we may be "helping people recognise immorality of sustainability inaction" [13], we're not offering an alternative vision to guide Silberman's navigation [34].…”

Evaluation Beyond Usability

“…However, existing evaluation processes have not always proven to be suitable for those new and emerging disciplines. Examples of disciplines struggling with evaluation processes are design fiction [55], information visualization [13], HCI for development (HCI4D) [5,11,23,52,53], and sustainable HCI [19,24,26,57,63,77,90]. In the remainder of this paper, we will focus on the latter.…”

“…Some members of the SHCI community have been hoping to define its role and purpose to ensure its future [89]. We believe that the difficulties of evaluation -noted by various researchers [19,24,26,57,77,89] -present an obstacle for researchers looking to engage with the field. Being unsure of how to evaluate research can make it difficult for researchers to communicate the contributions or value of their projects.…”

Evaluating HCI Research beyond Usability

“…In this SIG, we aim to start an active and open-ended debate within the HCI community how to solve those evaluation challenges. We find motivation in recent efforts to identify solutions addressing this issue, such as in sustainable HCI [10,12,13,20] or information visualization [3]. However, this SIG is not limited to one particular application area and the evaluation thereof; we welcome any CHI attendee who has encountered such evaluation challenges in their own research or who is interested in discussing the issue on a broader scope.…”