Learning-to-Rank for Real-Time High-Precision Hashtag Recommendation for Streaming News

Abstract

We address the problem of real-time recommendation of streaming Twitter hashtags to an incoming stream of news articles. The technical challenge can be framed as large scale topic classification where the set of topics (i.e., hashtags) is huge and highly dynamic. Our main applications come from digital journalism, e.g., for promoting original content to Twitter communities and for social indexing of news to enable better retrieval, story tracking and summarisation. In contrast to state-of-the-art methods that focus on modelling each individual hashtag as a topic, we propose a learning-to-rank approach for modelling hashtag relevance, and present methods to extract time-aware features from highly dynamic content. We present the data collection and processing pipeline, as well as our methodology for achieving low latency, high precision recommendations. Our empirical results show that our method outperforms the state-of-theart, delivering more than 80% precision. Our techniques are implemented in a real-time system 1 , and are currently under user trial with a big news organisation.

LOOM: Showing the Dynamics of Power Laws in Twitter Data

“…Twitter has been branded as the social network for news dissemination [6]; indeed, many journalists and news organizations now expend considerable effort tweeting their news articles, in ways to attract maximal attention and engagement [7,8]. In attracting attention to one's tweeted news, the use of the right hashtag for a story has become critical [9]; if a journalist does not use the most-commonlyused hashtag for the story then their news is less likely to reach a target audience [10,11]. Twitter's own TweetDeck application tries to provide broad information on trending hashtags but it is not fine-grained enough to spot the emerging behaviour of competing hashtags on a story.…”

“…In this paper, the data used was collected using a system --called Hashtagger [10] --that extracts trending hashtags from real-time news stories and streamed Twitter data.…”

“…In this 72-hour period the match fell in hours 38 and 39. The dataset used was also collected using Hashtagger [10] and the results were filtered using a series of keywords about the event (e.g., "rugby", "Ireland", "New Zealand"). This filtering was designed to ensure that as many tweets specifically about the rugby match as possible were included in the corpus.…”

Real-time News Story Detection and Tracking with Hashtags

“…The social tags can be manually assigned to an article by a journalist or by an automated hashtag recommender. Hashtagger presented in (Shi et al, 2016) and the method proposed in (Efron, 2010) recommend hashtags to news articles. We build on top of Hashtagger, which recommends up to 10 hashtags to an article, which are updated every 15 minutes over a period of 24 hours from the article publication time.…”

Topy: Real-Time Story Tracking via Social Tags

“…We build on top of the Hashtagger infrastructure [6] for collecting, processing and storing news articles and Twitter data. An article is represented by its headline, subheadline, body, a set of summary keywords and a set of hashtags recommended to the article over a period of 24h from the article publication time [6]. Hashtagger is a recent hashtag recommendation method that achieves Precision of more than 85%.…”