Information Architecture: Using Best Merge Method, Category Validity, and Multidimensional Scaling for Open Card Sort Data Analysis

Abstract

Open card sorting is a widely used method in HCI for the design of user-centered Information Architectures (IAs). This article proposes a new algorithm that combines the best merge method (BMM), category validity technique (CVT), and multidimensional scaling (MDS) to explore, analyze and visualize open card sort data. A study involving 20 participants and 41 cards explored the IA redesign of a university's website. The collected data were analyzed using two popular methods employed in the quantitative analysis of open card sort data (i.e., hierarchical clustering, K-means) and the proposed algorithm. It was found that the latter provides increased IA insights compared to the existing methods. Specifically, the proposed algorithm can expose hidden patterns and relationships amongst cards and identify complexities. We also found that the proposed algorithm produces better initial clusters, which have a direct effect on the final clustering quality.

Emergency Remote Teaching in Higher Education Institutes: A Taxonomy of Challenges Faced by First-Year Mathematics Students in the Pacific Region

“…We preferred quantitative approaches in this study. Paea et al [27] and Paea et al [18] report a typical step-bystep guideline to successfully apply the OCS method. The next two subsections describe the OCS and how the authors pick the participants of this study.…”

“…This paper employs the Best Merge Category Validity Multidimensional Scaling (BM-CV-MDS) algorithm proposed by Paea et al [18] to analyze the OCS data. The algorithm identifies the mathematically optimal number of categories k, creates and categorizes the initial core categories using the BMM, applies the CV algorithm to categorize the single categories and finally visualizes the clustering results using MDS from the OCS dataset.…”

“…In a card sorting session, participants organize topics (cards) into categories that make sense to them, and they may also help in labeling these groups. There are two primary methods of performing card sorts: Open Card Sort (OCS) and Closed Card Sort (CCS) [17], [18], [27]. In an OCS, each participant is given a stack of cards.…”

“…This is to provide participants with relevant information about the research objectives and the OCS procedure. The current study used individual card sorting with physical cards and a typical step-by-step road map to effectively apply the OCS method described in Paea et al [27] and Paea et al [18]. A stack of 44 cards was placed on the table, and the participants were asked to sort the cards into groups and label these groups.…”

“…The initial set of challenges was collected from a university's in-house monitoring report, literature review [1], [2], [12]- [15] and the authors' experiences. These challenges were organized into a taxonomy by employing Open Card Sorting (OCS) [16], a method of eliciting mental models from participants, and analyzing the collected data using innovative algorithms that have been recently proposed in the literature [17], [18]. The proposed taxonomy can be used to improve ERT in HEIs, especially in the field of mathematics education during a pandemic.…”