TopicCheck: Interactive Alignment for Assessing Topic Model Stability

Chuang, J.C.-I.; Roberts, Margaret E.; Stewart, Brandon; Weiss, Rebecca; Tingley, Dustin; Grimmer, Justin; Heer, Jeffrey

doi:10.3115/v1/n15-1018

Cited by 49 publications

(81 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We thus turn to a labeling process mediated by computation known as topic modeling. Topic modeling methods are often used to categorize a set of unlabeled documents into a finite number of coherent topic clusters [7,28,63], in turn helping researchers to navigate a large corpus (HIT documents in our case) [17,18,24,54]. However, unsupervised clustering of unlabeled documents is far from a solved problem [16,18].…”

Section: Types Of Workmentioning

confidence: 99%

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Hara

Adams

Milland

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

366

266

View full text Add to dashboard Cite

A growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 workers performing 3.8 million tasks on Amazon Mechanical Turk. Our task-level analysis revealed that workers earned a median hourly wage of only ~$2/h, and only 4% earned more than $7.25/h. While the average requester pays more than $11/h, lower-paying requesters post much more work. Our wage calculations are influenced by how unpaid work is accounted for, e.g., time spent searching for tasks, working on tasks that are rejected, and working on tasks that are ultimately not submitted. We further explore the characteristics of tasks and working patterns that yield higher hourly wages. Our analysis informs platform design and worker tools to create a more positive future for crowd work. Figure 12. (a) Hourly wage distributions of seven HIT categories provided by Gadiraju et al. [25] (with an additional category Research). (b) Strip plots showing median hourly wages of HITs associated with the topical keywords in Table 4.

show abstract

Section: Types Of Workmentioning

confidence: 99%

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

Hara

Adams

Milland

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

366

266

View full text Add to dashboard Cite

show abstract

“…In particular, for "spreadout" topics that were inherently difficult to interpret, because their tokens were drawn from a wide variety of Newsgroups (similar to a "fused" topic in Chuang et al (2013b)), we expected the proportion of correct responses to be roughly 1/3 no matter the value of λ used to compute relevance. Similarly, for very "pure" topics, whose tokens were drawn almost exclusively from one Newsgroup, we expected the task to be easy for any value of λ.…”

Section: User Studymentioning

confidence: 99%

“…Chang et al (2009) Ramage et al (2009) assert that "characterizing topics is hard" and describe how using the top-k terms for a given topic might not always be best, but offer few concrete alternatives. AlSumait et al (2009), Mimno et al (2011), and Chuang et al (2013b develop quantitative methods for measuring the interpretability of top-ics based on experiments with data sets that come with some notion of topical ground truth, such as document metadata or expert-created topic labels. These methods are useful for understanding, in a global sense, which topics are interpretable (and why), but they don't specifically attempt to aid the user in interpreting individual topics.…”

Section: Topic Interpretation and Coherencementioning

confidence: 99%

LDAvis: A method for visualizing and interpreting topics

Sievert¹,

Shirley²

2014

Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces

948

622

View full text Add to dashboard Cite

We present LDAvis, a web-based interactive visualization of topics estimated using Latent Dirichlet Allocation that is built using a combination of R and D3. Our visualization provides a global view of the topics (and how they differ from each other), while at the same time allowing for a deep inspection of the terms most highly associated with each individual topic. First, we propose a novel method for choosing which terms to present to a user to aid in the task of topic interpretation, in which we define the relevance of a term to a topic. Second, we present results from a user study that suggest that ranking terms purely by their probability under a topic is suboptimal for topic interpretation. Last, we describe LDAvis, our visualization system that allows users to flexibly explore topic-term relationships using relevance to better understand a fitted LDA model.

show abstract

“…Ramage et al (2009) assert that "characterizing topics is hard" and describe how using the top-k terms for a given topic might not always be best, but offer few concrete alternatives. AlSumait et al (2009), and Chuang et al (2013b develop quantitative methods for measuring the interpretability of top-ics based on experiments with data sets that come with some notion of topical ground truth, such as document metadata or expert-created topic labels. These methods are useful for understanding, in a global sense, which topics are interpretable (and why), but they don't specifically attempt to aid the user in interpreting individual topics.…”

Section: Topic Interpretation and Coherencementioning

confidence: 99%

Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces

2014

View full text Add to dashboard Cite

People acquire language through social interaction. Computers learn linguistic models from data, and increasingly, from language-based exchange with people. How do computational linguistic techniques and interactive visualizations work in concert to improve linguistic data processing for humans and computers? How can statistical learning models be best paired with interactive interfaces? How can the increasing quantity of linguistic data be better explored and analyzed? These questions span statistical natural language processing (NLP), human-computer interaction (HCI), and information visualization (Vis), three fields with natural connections but infrequent meetings. Vis and HCI are niches in NLP; Vis and HCI have not fully utilized the statistical techniques developed in NLP. This workshop aims to assemble an interdisciplinary community that promotes collaboration across these fields. Three themes define this first workshop:Active, Online, and Interactive Machine Learning Statistical machine learning (ML) has yielded tremendous gains in coverage and robustness for many tasks, but there is a growing sense that additional error reduction might require a fresh look at the human role. Presently, human inputs are often restricted to passive annotation in ML research. However, the fields of ML and HCI are both developing new techniques-such as active learning, incremental/online learning, and crowdsourcing-that attempt to engage people in novel and productive ways. How do we jointly solve the learning questions that have been the domain of NLP and address research topics in HCI such as managing human workers and increasing the quality of their responses?Language-based user interfaces NLP techniques have entered mainstream use, but the field currently focuses more on building and improving systems and less on understanding how users interact with them in real-world environments. User interface (UI) design decisions can affect the perceived or actual performance of a system. For example, while machine translation (MT) quality improved considerably over the last decade, studies found that human translators disliked MT output for reasons unrelated to translation quality. Many existing systems present sentence-level translations in the absence of relevant context, and disrupt rather than contribute to a translator's workflow. How do we best integrate learning methods, user behavior understanding, and human-centered design methodology?Text Visualization and Analysis The quantity and diversity of linguistic corpora is swelling. Recent work on visualizing text data annotated with linguistic structures (e.g., syntactic trees, hypergraphs, and sequences) has produced tools that enable exploration of thematic and recurrence patterns in text. Visual representations built on the outputs of word-level models (e.g., sentiment classifiers, topic models, and continuous word embedding models) now power exploratory analysis of legal documents, political text, and social media content. Beyond adding analytic value, interactive visualiz...

show abstract

TopicCheck: Interactive Alignment for Assessing Topic Model Stability

Cited by 49 publications

References 32 publications

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

LDAvis: A method for visualizing and interpreting topics

Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces

Contact Info

Product

Resources

About