Rethinking the Test Collection Methodology for Personal Self-tracking Data

Abstract

While vast volumes of personal data are being gathered daily by individuals, the MMM community has not really been tackling the challenge of developing novel retrieval algorithms for this data, due to the challenges of getting access to the data in the first place. While initial efforts have taken place on a small scale, it is our conjecture that a new evaluation paradigm is required in order to make progress in analysing, modeling and retrieving from personal data archives. In this position paper, we propose a new model of Evaluation-as-a-Service that re-imagines the test collection methodology for personal multimedia data in order to address the many challenges of releasing test collections of personal multimedia data.

Mapping and Taking Stock of the Personal Informatics Literature

“…We identified 23 publications making theoretical contributions, including models of personal informatics [4,77,168,204], frameworks describing how tracking technology can best support a domain or practice (e.g., transformative reflection [266], serious mental illness [202], diabetes [128], productivity [290]), and defining terms (e.g., adherence [275]). The 11 methodological contributions offered new strategies for analyzing selftracked data (e.g., Bayesian analysis for self-experimentation [256], personalized models for event detection from self-tracked data [169]) and approaches for using self-tracked data to understand people's everyday experiences [48,99,117].…”

Experiments in Lifelog Organisation and Retrieval at NTCIR

“…Real-world use cases are likely to either focus on retrieval from longitudinal archives donated by one individual, or across large populations (as in epidemiological studies) and the data gathering and release methodology employed for this task was not ideal, due to the large overhead of effort required to ensure privacy preservation. The evaluation-as-a-service model proposed by Hopfgartner et al (Hopfgartner et al 2020) is one potential way forward, which brings the algorithms to the data, rather than the conventional data-to-algorithm approach. Another potential next step is to encourage more comparative evaluation of interactive systems, since a user of a lifelog tool (either an individual or a professional analyst) is most likely to be using such tools in an interactive manner.…”