TSUBAKI: An Open Search Engine Infrastructure for Developing Information Access Methodology

Shinzato, Keiji; Shibata, Tomohide; Kawahara, Daisuke; Kurohashi, Sadao

doi:10.2197/ipsjjip.20.216

Cited by 18 publications

(13 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the StackFAQ dataset, we further report the result of (Sakata et al, 2019), which serves as the strongest supervised baseline. This baseline combines two methods: TSUBAKI (Shinzato et al, 2008) -a search engine for Q-to-q matching; and a supervised fine-tuned BERT model for Q-to-a matching. We put the results of this work (that were available only on the StackFAQ dataset), just to emphasize that our approach can reach the quality of a supervised approach, and not to directly compare with it.…”

Section: Baselinesmentioning

confidence: 99%

Unsupervised FAQ Retrieval with Question Generation and BERT

Mass¹,

Carmeli²,

Roitman³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

We focus on the task of Frequently Asked Questions (FAQ) retrieval. A given user query can be matched against the questions and/or the answers in the FAQ. We present a fully unsupervised method that exploits the FAQ pairs to train two BERT models. The two models match user queries to FAQ answers and questions, respectively. We alleviate the missing labeled data of the latter by automatically generating high-quality question paraphrases. We show that our model is on par and even outperforms supervised models on existing datasets.

show abstract

Section: Baselinesmentioning

confidence: 99%

Unsupervised FAQ Retrieval with Question Generation and BERT

Mass¹,

Carmeli²,

Roitman³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Target Pattern Pairs We extracted our binary patterns from the TSUBAKI corpus (Shinzato et al, 2008) of 600 million Japanese web pages. Binary patterns are defined as sequences of words on the path of dependency relations connecting two nouns in a sentence and have two variables.…”

Section: Target Data and Baseline Classifiersmentioning

confidence: 99%

Large-Scale Acquisition of Entailment Pattern Pairs by Exploiting Transitivity

Kloetzer

Torisawa

Hashimoto

et al. 2015

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We propose a novel method for acquiring entailment pairs of binary patterns on a large-scale. This method exploits the transitivity of entailment and a self-training scheme to improve the performance of an already strong supervised classifier for entailment, and unlike previous methods that exploit transitivity, it works on a largescale. With it we acquired 138.1 million pattern pairs with 70% precision with such non-trivial lexical substitution as "use Y to distribute X"→"X is available on Y" whose extraction is considered difficult. This represents 50.4 million more pattern pairs (a 57.5% increase) than what our supervised baseline extracted at the same precision.

show abstract

“…We used 100,000 Japanese sentences to evaluate our approach. These sentences were obtained from an open search engine infrastructure TSUB-AKI (Shinzato et al, 2008), which included at least one hiragana character and consisted of more than twenty characters…”

Section: Settingmentioning

confidence: 99%

A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis

Sasano

Kurohashi

Okumura

2014

Journal of Natural Language Processing

Self Cite

View full text Add to dashboard Cite

This paper presents a simple but effective approach to unknown word processing in Japanese morphological analysis, which handles 1) unknown words that are derived from words in a pre-defined lexicon and 2) unknown onomatopoeias. Our approach leverages derivation rules and onomatopoeia patterns, and correctly recognizes certain types of unknown words. Experiments revealed that our approach recognized about 4,500 unknown words in 100,000 Web sentences with only roughly 80 harmful side effects and a 6% loss in speed.

show abstract

TSUBAKI: An Open Search Engine Infrastructure for Developing Information Access Methodology

Cited by 18 publications

References 12 publications

Unsupervised FAQ Retrieval with Question Generation and BERT

Unsupervised FAQ Retrieval with Question Generation and BERT

Large-Scale Acquisition of Entailment Pattern Pairs by Exploiting Transitivity

A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis

Contact Info

Product

Resources

About