A survey of methods to ease the development of highly multilingual text mining applications

Abstract

Multilingual text processing is useful because the information content found in different languages is complementary, both regarding facts and opinions. While Information Extraction and other text mining software can, in principle, be developed for many languages, most text analysis tools have only been applied to small sets of languages because the development effort per language is large. Self-training tools obviously alleviate the problem, but even the effort of providing training data and of manually tuning the results is usually considerable. In this paper, we gather insights by various multilingual system developers on how to minimise the effort of developing natural language processing applications for many languages. We also explain the main guidelines underlying our own effort to develop complex text mining software for tens of languages. While these guidelines -most of all: extreme simplicity -can be very restrictive and limiting, we believe to have shown the feasibility of the approach through the development of the Europe Media Monitor (EMM) family of applications (http://press.jrc.it/overview.html). EMM is a set of complex media monitoring tools that process and analyse up to 100,000 online news articles per day in between twenty and fifty languages. We will also touch upon the kind of language resources that would make it easier for all to develop highly multilingual text mining applications. We will argue that -to achieve this -the most needed resources would be freely available, simple, parallel and uniform multilingual dictionaries, corpora and software tools.The share of non-English documents on the internet is rising continuously. While many private users will only be interested in finding monolingual information in their own language, the need for multilingual information retrieval, information extraction and cross-lingual information access for professionals, organisations and businesses is rising steadily. Starting from the premise that we need multilingual text mining tools, the question we would like to ask here is: How can we avoid that the development of (any) text mining application for N languages takes N times the effort of developing them for one language. It is generally acknowledged that developers benefit from the experience of having produced tools in one or more languages before, and that the existence of an efficient implementation infrastructure is extremely important (e.g. Maynard et al. 2002). Such software building blocks can include, for instance, a grammar implementation formalism, tools for marking up text, debugging tools, automatic evaluation tools and procedures, etc. Furthermore, simple applications like sentence splitters are typically so similar for different languages that -once one exists -the same tool is usually quickly adapted to new languages. We will thus try to take the effort of developing the infrastructure out of the equation. The question should thus be reformulated:Assuming that you have already developed text mining applications for some languages, ho...

A Survey of Text Mining in Social Media: Facebook and Twitter Perspectives

“…The art of sharing meaningful information with the help of uncommon and meaningless data is truly a good thing. Text mining technique as described by [45] examines the content for extracting the meaningful data which can be used for particular purposes. It looks like text mining that is going to include the overall NLP scheme [46] in its system in order to effectively examine the human language and to structure the unstructured data patterns accordingly.…”

Text Mining: Techniques, Applications and Issues

“…Precision and recall process is used to check and evaluate the relevance of results on the extracted data. In-depth and complete information about the relevant field is required to perform information extraction process to attain more relevant results [19].…”

“…There is a close relationship in text mining and information retrieval for textual data. In IR systems, different algorithms are used to track the user's behavior and search relevant data accordingly [19]. Google and Yahoo search engines are using information retrieval system more frequently to extract relevant documents according to a phrase on Web.…”

DAnIEL: Language Independent Character-Based News Surveillance

“…A major disadvantage arises, however, for the end-user wishing to process a genuine multilingual corpus such as news feed. For most languages, ecient components will be lacking [13]. In recent years, machine learning was successfully used to ll gaps when one can nd sucient training data in a language which has enough common properties with the new one [11].…”