Mining bilingual data from the web with adaptively learnt patterns

Long, Jinlin; Yang, Shiquan; Zhou, Ming; Liu, Xiaohua; Zhu, Qingsheng

doi:10.3115/1690219.1690268

Cited by 23 publications

(23 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But they didn't pay attention to filter out noisy candidates extracted from good wrappers. Compared to [8], we propose to rank all candidates, extracted from all wrappers, by their relevance with seeds, but don't concern on the quality of wrappers. y Our method can be applied to bilingual web pages written in any pair of languages indiscriminately, such as Japanese-English, Korean-English and so on, for that our approach is completely character-based and doesn't limit any language and domain.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using HTML Tags to Improve Parallel Resources Extraction

Feng

Yan

Tang

et al. 2011

2011 International Conference on Asian Language Processing

View full text Add to dashboard Cite

This paper proposes a new approach to extract parallel resources (including bilingual sentences and bilingual terms) from bilingual web pages, which have a primary language and a secondary language (the second language is often the translation to primary language). Our method is composed of four tasks: 1) parsing the web page into a DOM tree and segmenting inner texts of each node into series of monolingual snippets; 2) selecting adjacent snippet pairs in different languages and with higher translation scores as seeds for the next task; 3) constructing comprehensive wrappers from selected seeds, which save both HTML and surface formatting styles; 4) mining candidate instances and selecting good instances by their similarities with seeds. In this paper, we first propose to segment text by HTML tags, and select potential parallel resources by ranking all extracted candidates. According to the experimental results, our method can be applied to bilingual pages written in any other pair of languages. Experimental results also show that our approaches are effective in improving the parallel resources extraction. (Abstract)

show abstract

Section: Introductionmentioning

confidence: 99%

“…y We propose to extract good candidate instances by ranking. In [8], they focus on selecting good wrappers, and all candidate pairs extracted by them are regarded as parallel data. But they didn't pay attention to filter out noisy candidates extracted from good wrappers.…”

Section: Introductionmentioning

confidence: 99%

Using HTML Tags to Improve Parallel Resources Extraction

Feng

Yan

Tang

et al. 2011

2011 International Conference on Asian Language Processing

View full text Add to dashboard Cite

show abstract

“…Jiang et al (2009) used an adaptive pattern-based method to mine interesting bilingual data based on the observation that bilingual data usually appears collectively following similar patterns. They found that bilingual web pages are a promising source of up-to-date bilingual terms/sentences which cover many domains and application scenarios.…”

Section: Related Workmentioning

confidence: 99%

“…In the past decade, there have been extensive studies on parallel resource extraction from the web (e.g., Chen and Nie, 2000;Resnik 2003;Jiang et al, 2009) and many effective Web mining systems have been developed such as STRAND, PTMiner, BITS and WPDE. For most of these mining systems, there is a typical parallel resource mining strategy which involves three steps: (1) locate the bilingual websites (2) identify parallel web pages from these bilingual websites and (3) extract bilingual resources from the parallel web pages.…”

Section: Introductionmentioning

confidence: 99%

An Iterative Link-based Method for Parallel Web Page Mining

Yan

et al. 2014

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Identifying parallel web pages from bilingual web sites is a crucial step of bilingual resource construction for crosslingual information processing. In this paper, we propose a link-based approach to distinguish parallel web pages from bilingual web sites. Compared with the existing methods, which only employ the internal translation similarity (such as content-based similarity and page structural similarity), we hypothesize that the external translation similarity is an effective feature to identify parallel web pages. Within a bilingual web site, web pages are interconnected by hyperlinks. The basic idea of our method is that the translation similarity of two pages can be inferred from their neighbor pages, which can be adopted as an important source of external similarity. Thus, the translation similarity of page pairs will influence each other. An iterative algorithm is developed to estimate the external translation similarity and the final translation similarity. Both internal and external similarity measures are combined in the iterative algorithm. Experiments on six bilingual websites demonstrate that our method is effective and obtains significant improvement (6.2% F-Score) over the baseline which only utilizes internal translation similarity.

show abstract

“…de Souza et al, 2015). Also most of the previous approaches to bilingual data mining/cleaning for statistical MT rely on supervised learning (Resnik and Smith, 2003;Munteanu and Marcu, 2005;Jiang et al, 2009). Unsupervised solutions, like the one proposed by Cui et al (2013) usually rely on redundancy-based approaches that reward parallel segments containing phrase pairs that are frequent in a training corpus.…”

Section: Introductionmentioning

confidence: 99%

An Unsupervised Method for Automatic Translation Memory Cleaning

Sabet

Negri

Turchi

et al. 2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

We address the problem of automatically cleaning a large-scale Translation Memory (TM) in a fully unsupervised fashion, i.e. without human-labelled data. We approach the task by: i) designing a set of features that capture the similarity between two text segments in different languages, ii) use them to induce reliable training labels for a subset of the translation units (TUs) contained in the TM, and iii) use the automatically labelled data to train an ensemble of binary classifiers. We apply our method to clean a test set composed of 1,000 TUs randomly extracted from the English-Italian version of MyMemory, the world's largest public TM. Our results show competitive performance not only against a strong baseline that exploits machine translation, but also against a state-of-the-art method that relies on human-labelled data.

show abstract

Mining bilingual data from the web with adaptively learnt patterns

Cited by 23 publications

References 15 publications

Using HTML Tags to Improve Parallel Resources Extraction

Using HTML Tags to Improve Parallel Resources Extraction

An Iterative Link-based Method for Parallel Web Page Mining

An Unsupervised Method for Automatic Translation Memory Cleaning

Contact Info

Product

Resources

About