Evaluating classification schema and classification decisions

Abstract

Stacy Surla is the Bulletin's associate editor for IA. She serves on the IA Institute Board of Directors and is a past chair of the IA Summit. She can be reached at T his article considers how evaluation pertains to taxonomies. Taxonomies and evaluation are both rich concepts, so it is best to start out with some definitions that help to define our discussion. What do we mean by taxonomy? And what do we mean by evaluation? TaxonomiesFor seasoned information professionals the traditional characterization of a taxonomy is as a hierarchical classification scheme. This characterization has expanded in the last 20 years as the taxonomy community and the information environment have expanded. Today the taxonomy community includes people who design taxonomies, those who build systems that support them and those who use them. Our complex information environment may call for a variety of taxonomic structures, including I flat taxonomies such as lists of languages or lists of countries; I hierarchical taxonomies such as topical or subject classifications, business classifications or service classifications; I faceted taxonomies such as metadata or parametric search structures; I ring taxonomies such as synonyms or authority control data; and I network taxonomies such as fully relationed thesauri or knowledge networks. Each of these structures has its own set of principles and behaviors. And each requires an evaluation method that aligns with those principles and behaviors. This article focuses on the second type of taxonomy -the traditional classification scheme or hierarchical taxonomy. Classification schemes govern the organization of objects into groups according to explicit properties or values. Classification schemes are in widespread use in everyday life -from grocery stores to websites to personal information spaces. C O N T E N T S N E X T P A G E > N E X T A R T I C L E > < P R E V I O U S P A G E Special SectionEDITOR'S SUMMARY Direction on the construction and application of classification schemes such as taxonomies is readily available, but relatively little has been offered on evaluating the schemes themselves and their use to categorize content. A classification scheme can be judged for how well it meets its purpose and complies with standards, and a strong evaluative framework is reflected in S.R. Ranganathan's principles of classification. The degree of certainty of classification decisions depends on objective understanding of the object to be classified, the scope and details of the class and the coverage and organization of the overall classification scheme. The more complete the information about each class, the more reliable the goodness-of-fit for an object to a class is likely to be, whether chosen by human or machine classifiers. This information comes through definitions, examples, prior use and semantic relationships. The risk of misclassification can be reduced by analyzing the goodness-of-fit of objects to classes and the patterns of missed or erroneous selections. KEYWORDS

Introducing an Evaluation Method for Taxonomies

“…Two evaluation criteria stand out as most frequently used: usefulness and applicability. Likewise, in Information Science, Bedford [2] identifies two evaluation issues with regard to classification schemes: evaluation of the classification scheme itself and evaluation of how well the scheme supports classification decisions. Both require their own framework and context for evaluation.…”

“…Orthogonality For taxonomies, Bedford [2] argues that "No two categories should overlap or should have exactly the same scope and boundaries" and, as such, requires orthogonality among a taxonomy's classes. To evaluate orthogonality, we can employ a selfreferencing orthogonality matrix, where the classes of a taxonomy denotes the columns and rows of the matrix (cf.…”

“…Second, the taxonomy's applicability is evaluated. A taxonomy needs to be understandable and usable to be applicable [2]. This can be shown by means of user studies.…”

“…They can serve to (1) classify objects of a research field, (2) provide a common terminology, (3) enable a better understanding of the interrelationships between the classified objects, (4) identify gaps, and (5) support decision making processes [22]. Although taxonomies are typically described as hierarchies of classes, taxonomies can have a wide variety of representations, e.g., a hierarchy with (or without) mutually exclusive classes, a tree, a paradigm, facets, a ring, or a (knowledge) graph [2,18,22]. Most taxonomies in Software Engineering (SE) are rather new and have the practical purpose of classifying among others processes, approaches, and solutions.…”

A taxonomy for sentiment analysis field

“…A taxonomy is commonly known as a hierarchical classification of knowledge (Bedford, 2013). However, for Heather (2010), a taxonomy is any form of organization of knowledge, and a hierarchical taxonomy is just one of them.…”