Using a concept map to evaluate exploratory search

Abstract

Users often conduct exploratory searches on the Web to solve complex problems, but how for evaluating the effectiveness of a search, especially in a user-centered approach, remains an open question. We propose a user-centered method to evaluate the effectiveness of an exploratory search by focusing on the change in the users' mental representations of a topic during their exploratory search on the Web. This was done by comparing the concept maps depicted before and after each users searching. Thirty-five undergraduate students recruited from various departments and universities participated in our experiment. The participants were instructed to search for and gather Web pages for the task of writing a term paper on two given topics, politics and media, while working in either of two scenarios: "Selective scenario" (identifying only ten Web pages as important) and "As-Many-As scenario" (collecting as many Web pages as possible). The participants were divided into two groups: one group searched for both topics in the Selective scenario, and the other searched for them in the As-Many-As scenario. They composed their concept maps before and after searching. We defined the following measures to show the differences between the maps before and after a search to analyze the concept maps made by the participants: common, new, and lost map components like nodes, links, and link labels. We used these measures to compare the results from different topics, scenarios, and browser types. The results showed that the concept maps greatly changed after searching and that there were more common nodes and link labels in the Selective scenario than in the As-Many-As scenario. A comparative analysis of the concept maps between the pre-and post-search maps indicated that the users significantly changed their knowledge structure of a topic by completing the exploratory search task.

Changes in users' knowledge structures before and after Web search on a topic: Analysis using the concept map

“…Cole et al (2007) focused on how the students' mental model diagrams for a topic were represented in an early exploration stage of the informationseeking process, and they suggested a 12-category classification schema of the mental models. Egusa et al (2010) investigated how a user's concept map differs before and after a search and how the differences between the topics, scenarios, and browser types influence the user's concept map. A comparative analysis of the concept maps between the pre-and post-search maps indicated that users significantly changed their knowledge structures on a topic through an exploratory search.…”

Evaluation in lifelong learning

“…There are various types of concept maps [20]. For example, task demands concept maps demand students to generate their concept maps (students can be asked to fill-in a skeleton map, or construct a map from scratch, or talk about the relation between concepts).…”

“…Task constraints concept maps are characterized by the restrictiveness of the task (e.g., students may or may not be asked to construct a hierarchical map, or to use one or more links between concepts, or to provide the concepts for the map). Task content structure concept maps require the intersection of the task demands and the constraints of the structure of the subject-domain to be mapped (e.g., there is no need to impose a hierarchical structure if the content structure is not hierarchical) [20].…”

How concept maps change if a user does search or not?

“…Egusa et al [3] investigated how a user's concept map differs before and after a search and how the differences between the topics, scenarios, and browser types influence the user's concept map. A comparative analysis of concept maps between pre-search and post-search maps indicated that users significantly changed their knowledge structures for a topic through an exploratory search.…”

“…However, previous studies [3,9,10] have only analyzed the changes of concept maps before and after searching. Therefore, these studies could not exclude the possibility of the influence of instructions or time-dependent changes.…”