WWW'18 Open Challenge

Maia, Macedo; Handschuh, Siegfried; Freitas, André; Davis, Brian; McDermott, Ross; Zarrouk, Manel; Balahur, Alexandra

doi:10.1145/3184558.3192301

Cited by 65 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2: Illustration of the five-step sequential inference pipeline in SPRINT. The example comes from the FiQA dataset [28] within the BEIR benchmark [40].…”

Section: Sprint Toolkit Overviewmentioning

confidence: 99%

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

Nandan

Wang

Gurevych

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Traditionally, sparse retrieval systems relied on lexical representations to retrieve documents, such as BM25, dominated information retrieval tasks. With the onset of pre-trained transformer models such as BERT, neural sparse retrieval has led to a new paradigm within retrieval. Despite the success, there has been limited software supporting different sparse retrievers running in a unified, common environment. This hinders practitioners from fairly comparing different sparse models and obtaining realistic evaluation results. Another missing piece is, that a majority of prior work evaluates sparse retrieval models on in-domain retrieval, i.e. on a single dataset: MS MARCO. However, a key requirement in practical retrieval systems requires models that can generalize well to unseen out-of-domain, i.e. zero-shot retrieval tasks. In this work, we provide SPRINT, a unified python toolkit based on Pyserini and Lucene, supporting a common interface for evaluating neural sparse retrieval. The toolkit currently includes five built-in models: uni-COIL, DeepImpact, SPARTA, TILDEv2 and SPLADEv2. Users can also easily add customized models by defining their term weighting method. Using our toolkit, we establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. Our results demonstrate that SPLADEv2 achieves the best average score of 0.470 nDCG@10 on BEIR amongst all neural sparse retrievers. In this work, we further uncover the reasons behind its performance gain. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document which is often crucial for its performance gains, i.e. a limitation among its other sparse counterparts. We provide our SPRINT toolkit, models, and data used in our experiments publicly here: https://github.com/thakur-nandan/sprint.

show abstract

“…2: Illustration of the five-step sequential inference pipeline in SPRINT. The example comes from the FiQA dataset [28] within the BEIR benchmark [40].…”

Section: Sprint Toolkit Overviewmentioning

confidence: 99%

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

Nandan

Wang

Gurevych

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…• Open-domain Question Answering (QA): Hot-potQA (Yang et al, 2018), NQ (Kwiatkowski et al, 2019), and FiQA (Maia et al, 2018).…”

Section: A Appendixmentioning

confidence: 99%

Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories

Ge¹,

Xiong²,

Rosset³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora ("external memories"), with the option to "plug in" new memory at inference time. We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We instantiate the model in a zero-shot dense retrieval setting by augmenting a strong T5-based retriever with MoMA. Our model, MoMA, obtains strong zero-shot retrieval accuracy on the eighteen tasks included in the standard BEIR benchmark. It outperforms systems that seek generalization from increased model parameters and computation steps. Our analysis further illustrates the necessity of augmenting with mixture-of-memory for robust generalization, the benefits of augmentation learning, and how MoMA utilizes the plug-in memory at inference time without changing its parameters. We plan to open source our code.

show abstract

“…Currently, the main financial dataset sets include a variety of tasks. FPB (Malo et al, 2014) and FiQA-SA (Maia et al, 2018) are mainly used for emotion analysis. The Headline (Sinha and Khandait, 2021) dataset is primarily utilized for news headline identification.…”

Section: Financial Datasetsmentioning

confidence: 99%

The South China Sea in Song-Yuan-Ming-Qing China

Li¹

2020

China’s Maritime Boundaries in the South China Sea

View full text Add to dashboard Cite

The task of financial analysis primarily encompasses two key areas: stock trend prediction and the corresponding financial question answering. Currently, machine learning and deep learning algorithms (ML&DL) have been widely applied for stock trend predictions, leading to significant progress. However, these methods fail to provide reasons for predictions, lacking interpretability and reasoning processes. Also, they can not integrate textual information such as financial news or reports. Meanwhile, large language models (LLMs) have remarkable textual understanding and generation ability. But due to the scarcity of financial training datasets and limited integration with real-time knowledge, LLMs still suffer from hallucinations and are unable to keep up with the latest information. To tackle these challenges, we first release AlphaFin datasets, combining traditional research datasets, real-time financial data, and handwritten chain-of-thought (CoT) data. It has a positive impact on training LLMs for completing financial analysis. We then use AlphaFin datasets to benchmark a state-of-the-art method, called Stock-Chain, for effectively tackling the financial analysis task, which integrates retrieval-augmented generation (RAG) techniques. Extensive experiments are conducted to demonstrate the effectiveness of our framework on financial analysis.

show abstract

WWW'18 Open Challenge

Cited by 65 publications

References 0 publications

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories

The South China Sea in Song-Yuan-Ming-Qing China

Contact Info

Product

Resources

About