A Technical Question Answering System with Transfer Learning

Yu, Wenhao; Wu, Lingfei; Deng, Yu; Mahindru, Ruchi; Zeng, Qingkai; Guven, Sinem; Jiang, Meng

doi:10.18653/v1/2020.emnlp-demos.13

Cited by 19 publications

(13 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The pre-trained model captures rich semantic patterns by training on large-scale corpus. Recent works [25,36] have shown their effectiveness in various natural language processing areas. A notable example is BERT [9], which is built on the transformer architecture [30].…”

Section: Pre-training Techniquesmentioning

confidence: 99%

“…Thus, they pave a way to help specific NLP tasks even when only a small amount of task-specific labeled data is available. In fact, these PLMs have been proven to significantly improve numerous natural language processing tasks such as text classification [28], question answering [36], and single sentence tagging [9]. It is evident from recent works that these PLMs can also help tackle aforementioned two challenges in query understanding [6,21].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Graph Enhanced BERT for Query Understanding

Juanhui¹,

Ma²,

Zeng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Query understanding plays a key role in exploring users' search intents and facilitating users to locate their most desired information. However, it is inherently challenging since it needs to capture semantic information from short and ambiguous queries and often requires massive task-specific labeled data. In recent years, pretrained language models (PLMs) have advanced various natural language processing tasks because they can extract general semantic information from large-scale corpora. Therefore, there are unprecedented opportunities to adopt PLMs for query understanding. However, there is a gap between the goal of query understanding and existing pre-training strategies -the goal of query understanding is to boost search performance while existing strategies rarely consider this goal. Thus, directly applying them to query understanding is sub-optimal. On the other hand, search logs contain user clicks between queries and urls that provide rich users' search behavioral information on queries beyond their content. Therefore, in this paper, we aim to fill this gap by exploring search logs. In particular, to incorporate search logs into pre-training, we first construct a query graph where nodes are queries and two queries are connected if they lead to clicks on the same urls. Then we propose a novel graph-enhanced pre-training framework, GE-BERT, which can leverage both query content and the query graph. In other words, GE-BERT can capture both the semantic information and the users' search behavioral information of queries. Extensive experiments on various query understanding tasks have demonstrated the effectiveness of the proposed framework.

show abstract

Section: Pre-training Techniquesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Graph Enhanced BERT for Query Understanding

Juanhui¹,

Ma²,

Zeng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Yu Wenhao et al [11] developed a TransTQA, which is a novel system that gives automatic responses by retrieving proper answers supported correctly answered similar questions within the past. TransTQA is configured with the ALBERT model consistent with the required arguments that defines the model architecture, producing transformer encoders.…”

Section: Literature Reviewmentioning

confidence: 99%

“…The COBERT system searches a document of 59K corona virus-related literature made accessible through the Coronavirus Open Research Dataset Challenge (CORD-19) [9]. As of late, QA models have made huge advancements as far as both execution and throughput [10][11][12]. Such upgrades can be credited to the presentation of huge scope QA datasets and deep learning models, and the recent focus of the research towards the proficiency of such models.…”

Section: Introductionmentioning

confidence: 99%

COBERT: COVID-19 Question Answering System Using BERT

et al. 2021

View full text Add to dashboard Cite

In the current situation of worldwide pandemic COVID-19, which has infected 62.5 Million people and caused nearly 1.46 Million deaths worldwide as of Nov 2020. The profoundly powerful and quickly advancing circumstance with COVID-19 has made it hard to get precise, on-request latest data with respect to the virus. Especially, the frontline workers of the battle medical services experts, policymakers, clinical scientists, and so on will require expert specific methods to stay aware of this literature for getting scientific knowledge of the latest research findings. The risks are most certainly not trivial, as decisions made on fallacious, answers may endanger trust or general well being and security of the public. But, with thousands of research papers being dispensed on the topic, making it more difficult to keep track of the latest research. Taking these challenges into account we have proposed COBERT: a retriever-reader dual algorithmic system that answers the complex queries by searching a document of 59K corona virus-related literature made accessible through the Coronavirus Open Research Dataset Challenge (CORD-19). The retriever is composed of a TF-IDF vectorizer capturing the top 500 documents with optimal scores. The reader which is pre-trained Bidirectional Encoder Representations from Transformers (BERT) on SQuAD 1.1 dev dataset built on top of the HuggingFace BERT transformers, refines the sentences from the filtered documents, which are then passed into ranker which compares the logits scores to produce a short answer, title of the paper and source article of extraction. The proposed DistilBERT version has outperformed previous pre-trained models obtaining an Exact Match(EM)/F1 score of 80.6/87.3 respectively.

show abstract

“…Recent advances of deep learning technologies with transfer learning has achieved great success in a variety of NLP tasks (Ruder et al, 2019). Several research work in this domain greatly enrich the application and technology of transfer learning on question answering from different perspectives (Min et al, 2017;Deng et al, 2018;Castelli et al, 2020;Yu et al, 2020a). Although transfer learning has been successfully applied to various QA applications, its applicability to technical QA has yet to be investigated.…”

Section: Related Workmentioning

confidence: 99%

Technical Question Answering across Tasks and Domains

Yu¹,

Wu²,

Deng³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

Building automatic technical support system is an important yet challenge task. Conceptually, to answer a user question on a technical forum, a human expert has to first retrieve relevant documents, and then read them carefully to identify the answer snippet. Despite huge success the researchers have achieved in coping with general domain question answering (QA), much less attentions have been paid for investigating technical QA. Specifically, existing methods suffer from several unique challenges (i) the question and answer rarely overlaps substantially and (ii) very limited data size. In this paper, we propose a novel framework of deep transfer learning to effectively address technical QA across tasks and domains. To this end, we present an adjustable joint learning approach for document retrieval and reading comprehension tasks. Our experiments on the TechQA demonstrates superior performance compared with state-of-the-art methods.

show abstract

A Technical Question Answering System with Transfer Learning

Cited by 19 publications

References 27 publications

Graph Enhanced BERT for Query Understanding

Graph Enhanced BERT for Query Understanding

COBERT: COVID-19 Question Answering System Using BERT

Technical Question Answering across Tasks and Domains

Contact Info

Product

Resources

About