Pragmatic evaluation of folksonomies

Abstract

Recently, a number of algorithms have been proposed to obtain hierarchical structures -so-called folksonomies -from social tagging data. Work on these algorithms is in part driven by a belief that folksonomies are useful for tasks such as: (a) Navigating social tagging systems and (b) Acquiring semantic relationships between tags. While the promises and pitfalls of the latter have been studied to some extent, we know very little about the extent to which folksonomies are pragmatically useful for navigating social tagging systems. This paper sets out to address this gap by presenting and applying a pragmatic framework for evaluating folksonomies. We model exploratory navigation of a tagging system as decentralized search on a network of tags. Evaluation is based on the fact that the performance of a decentralized search algorithm depends on the quality of the background knowledge used. The key idea of our approach is to use hierarchical structures learned by folksonomy algorithms as background knowledge for decentralized search. Utilizing decentralized search on tag networks in combination with different folksonomies as hierarchical background knowledge allows us to evaluate navigational tasks in social tagging systems. Our experiments with four state-of-the-art folksonomy algorithms on five different social tagging datasets reveal that existing folksonomy algorithms exhibit significant, previously undiscovered, differences with regard to their utility for navigation. Our results are relevant for engineers aiming to improve navigability of social tagging systems and for scientists aiming to evaluate different folksonomy algorithms from a pragmatic perspective.

Collaborative Computing

“…Therefore, these results are only of a theoretical valuethey shed light on the nature of the greedy navigation process and help to identify potential pitfalls and bottlenecks, as well as ideas for solutions to those problems. Please note that these results are consistent with our previous results on theoretical suitability of tag hierarchies as a navigational support [10] -where here, we extend this investigation by discussing the structure of paths of the greedy navigator.…”

“…From there on, the distance is recursively assigned, e.g., the parent's siblings and the grandchildren nodes are at distance d = 2, the children of the parent's siblings are at distance d = 3, the grandchildren of the parent's siblings are at distance d = 4 and so on. More details and an illustrative example of this particular simulation algorithm can be found in [10]. The algorithm is a variant of the simulation algorithm presented in [1].…”

“…In previous work we have evaluated a series of existing tag hierarchy algorithms on a theoretical level, without taking user interface constraints into account [10]. As we have found that centralitybased algorithms outperform hierarchical clustering algorithms by a large margin (see [10,31] for more details) we select one of these algorithms to conduct further investigations of their usefulness under typical user interface constraints.…”

“…As we have found that centralitybased algorithms outperform hierarchical clustering algorithms by a large margin (see [10,31] for more details) we select one of these algorithms to conduct further investigations of their usefulness under typical user interface constraints. It is reasonable to assume that other centrality-based tag hierarchy algorithms will behave similarly under our constraints.…”

“…Recent theoretical work in this area for example has shown that the ability of tags to efficiently encode resources for later retrieval decreases over time [9]. In our own previous work [10], we evaluated the suitability of different tag hierarchies to support navigation in social tagging systems on a theoretical level -not taking user interface constraints into account. The practicability and usefulness of integrating tag hierarchies into user interfaces of social tagging systems thus remains unclear, and -to the best of our knowledge -has not been investigated so far.…”