Automatic topic labeling using graph-based pre-trained neural embedding

He, Dongbin; Ren, Yuanhang; Khattak, Abdul Mateen; Liu, Xinliang; Tao, Shiquan; Gao, Wanlin

doi:10.1016/j.neucom.2021.08.078

Cited by 44 publications

(67 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Tis work proposes aggregating the related information and adding a cross-attention neural network layer. From the experimental results based on pretrained language models such as DeBERTa-V3-Large [4], RoBERTa-Large [5], and BERT-for-Patent [6], we found that the proposed methods perform better than these general methods. And it has an average of 2-4 percent higher performance, further improving the accuracy and performance of downstream tasks.…”

Section: Introductionmentioning

confidence: 93%

An Enhanced Phrase Matching Method Based on Cross‐Attention

Pang

Chen

et al. 2023

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Text matching is a core problem in the field of natural language understanding. It aims to analyze and judge the semantic relevance or similarity between two texts. In the past, much work on text matching focused on long text and little on phrase optimization. However, phrase matching also has essential application scenarios. Compared with long texts, phrases have difficulties covering less semantics and polysemy, and because of the weak expression ability of phrases, it is hard to match phrases accurately. On the Kaggle patent phrase matching dataset, due to the few words in the phrase and the repeated occurrences under different patent classification numbers, it is difficult to match each other accurately. In the data-processing stage, this work proposes aggregating related targets for data fusion, expanding the background semantic information, and enhancing the expression ability of phrases. In the training stage, this work considers adding a cross-attention network to the model to make the additional related targets better used and learned. The added cross-attention network makes the model pay more attention to the most related information. The proposed method has experimented on pretrained language models such as BERT-for-patent, DeBERTa, and RoBERTa. The results show that the proposed method in this work improves by 2–4 points more than the general method without any information or additional network layers in its evaluation metric called the Pearson correlation, which enhances the performance in short-text matching.

show abstract

Section: Introductionmentioning

confidence: 93%

An Enhanced Phrase Matching Method Based on Cross‐Attention

Pang

Chen

et al. 2023

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…They found 4. Sub-task 2 baseline is available at: https://github.com/nl4opt/nl4opt-subtask2-baseline that implementing adversarial attack using the FGM proposed by Goodfellow et al (2014) on the DeBERTa-large transformer (He et al, 2021) with a CRF layer performed the best on the development set. They trained 9 variations of this model using different random initializations to form their ensemble and leveraged majority voting for the final prediction.…”

Section: Solutionsmentioning

confidence: 99%

NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions

Ramamonjison¹,

Yu²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

The Natural Language for Optimization (NL4Opt) Competition was created to investigate methods of extracting the meaning and formulation of an optimization problem based on its text description. Specifically, the goal of the competition is to increase the accessibility and usability of optimization solvers by allowing non-experts to interface with them using natural language. We separate this challenging goal into two sub-tasks:(1) recognize and label the semantic entities that correspond to the components of the optimization problem; (2) generate a meaning representation (i.e. a logical form) of the problem from its detected problem entities. The first task aims to reduce ambiguity by detecting and tagging the entities of the optimization problems. The second task creates an intermediate representation of the linear programming (LP) problem that is converted into a format that can be used by commercial solvers. In this report, we present the LP word problem dataset and shared tasks for the NeurIPS 2022 competition. Furthermore, we investigate and compare the performance of the ChatGPT large language model against the winning solutions. Through this competition, we hope to bring interest towards the development of novel machine learning applications and datasets for optimization modeling.

show abstract

“…For the reranking we actually used a few different rerankers compared to the known languages track. 9 For Yoruba, we add three models a) the monoT0pp over the translated corpus we had used in English on the known languages track; b) RankT5 using ByT5 [14] 10 as its pretrained language model and trained solely on mMARCO; and c) A cross encoding reranker using Deberta [7] 11 as its pretrained language model. Results are presented in Table 10.…”

Section: Rerankersmentioning

confidence: 99%

Extending English IR methods to multi-lingual IR

Lassance¹

2023

Preprint

View full text Add to dashboard Cite

This paper describes our participation in the 2023 WSDM CUP -MIRACL challenge. Via a combination of i) document translation; ii) multilingual SPLADE and Contriever; and iii) multilingual RankT5 and many other models, we were able to get first place in both the known and surprise languages tracks. Our strategy mostly revolved around getting the most diverse runs for the first stage and then throwing all possible reranking techniques. While this was not a first for many techniques, we had some things that we believe were never tried before, for example, we train the first SPLADE model that is effectively capable of working in more than 10 languages. However, a more careful study of the results is needed in order to verify if we were able to get first place just due to brute force or if the hybrids we developed really brought improvements over the other team's solutions.

show abstract

Automatic topic labeling using graph-based pre-trained neural embedding

Cited by 44 publications

References 8 publications

An Enhanced Phrase Matching Method Based on Cross‐Attention

An Enhanced Phrase Matching Method Based on Cross‐Attention

NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions

Extending English IR methods to multi-lingual IR

Contact Info

Product

Resources

About