Multi-layer features ablation of BERT model and its application in stock trend prediction

Zhao, Feng; Li, Xinning; Gao, Yating; Li, Ying; Feng, Zhiquan; Zhang, Caiming

doi:10.1016/j.eswa.2022.117958

Cited by 17 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The techniques suggested in this category rely merely on news and information related to social and political topics for identifying fluctuations in prices (Serafini et al, 2020;Critien et al, 2022;Yu et al, 2023) (Zhao et al, 2022) introduced a technique for multi-layer feature ablation that utilizes the BERT model. This approach is well-suited for in-depth analysis of lengthy texts and can comprehensively capture various text features, to apply it to stock-related texts.…”

Section: Models Based On News and Social Media Informationmentioning

confidence: 99%

See 1 more Smart Citation

Decentralized Storage Cryptocurrencies: An Innovative Network-Based Model for Identifying Effective Entities and Forecasting Future Price Trends

Davoudi

Ghavipour

Sargolzaei-Javan³

et al. 2023

Preprint

View full text Add to dashboard Cite

This study focuses on analyzing four of the most significant cryptocurrencies in the field of decentralized storage, including Filecoin, Arweave, Storj, and Siacoin. Our method consists of three main components: Network Analysis, Textual Analysis, and Market Analysis. Network Analysis involves identifying relevant entities associated with the target cryptocurrencies to construct a network of entities. During this component, the embeddings of each entity are then extracted using node2vec which are fed into a convolutional neural network. In the second component, Textual Analysis, we first employ the T5 summarization model to encapsulate the content of related news articles. Subsequently, by utilizing the FinBert model the sentiment of news articles and tweets associated with the identified entities are extracted. We then use transformer encoders to process the resulting feature vectors. Ultimately, similar to the Textual component, by leveraging the transformer encoders the financial market information of target cryptocurrencies is evaluated during the Market Analysis component. As the final step, the outputs of these components are combined to predict the price trend of the target cryptocurrencies within a specified time frame. The proposed model’s accuracy in forecasting the future price trend of Filecoin, Storj, Arweave, and Siacoin is 76%, 83%, 61%, and 74% respectively.

show abstract

Section: Models Based On News and Social Media Informationmentioning

confidence: 99%

“…We use the collected information within a time frame that is composed of 𝑑 days, 𝑑 ∈ [7,10,14] to predict the price trend of the next 𝑖 days, 𝑖 ∈ [1,3,7,10]. These parameters were selected experimentally and based on recent work (Zhao et al, 2022).…”

Section: Problem Definitionmentioning

confidence: 99%

Decentralized Storage Cryptocurrencies: An Innovative Network-Based Model for Identifying Effective Entities and Forecasting Future Price Trends

Davoudi

Ghavipour

Sargolzaei-Javan³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The exploited model achieved an accuracy of 82.00%. The researchers have used BERT for several tasks such as stock trend prediction (Zhao et al , 2022), automatic drug labeling (Shi et al , 2023), toxic comment analysis (Zhao et al , 2021), etc.…”

Section: Related Workmentioning

confidence: 99%

A novel multi-layer feature fusion-based BERT-CNN for sentence representation learning and classification

Alyoubi,

Alotaibi,

Kumar

et al. 2023

RIA

View full text Add to dashboard Cite

Purpose The purpose of this paper is to describe a new approach to sentence representation learning leading to text classification using Bidirectional Encoder Representations from Transformers (BERT) embeddings. This work proposes a novel BERT-convolutional neural network (CNN)-based model for sentence representation learning and text classification. The proposed model can be used by industries that work in the area of classification of similarity scores between the texts and sentiments and opinion analysis. Design/methodology/approach The approach developed is based on the use of the BERT model to provide distinct features from its transformer encoder layers to the CNNs to achieve multi-layer feature fusion. To achieve multi-layer feature fusion, the distinct feature vectors of the last three layers of the BERT are passed to three separate CNN layers to generate a rich feature representation that can be used for extracting the keywords in the sentences. For sentence representation learning and text classification, the proposed model is trained and tested on the Stanford Sentiment Treebank-2 (SST-2) data set for sentiment analysis and the Quora Question Pair (QQP) data set for sentence classification. To obtain benchmark results, a selective training approach has been applied with the proposed model. Findings On the SST-2 data set, the proposed model achieved an accuracy of 92.90%, whereas, on the QQP data set, it achieved an accuracy of 91.51%. For other evaluation metrics such as precision, recall and F1 Score, the results obtained are overwhelming. The results with the proposed model are 1.17%–1.2% better as compared to the original BERT model on the SST-2 and QQP data sets. Originality/value The novelty of the proposed model lies in the multi-layer feature fusion between the last three layers of the BERT model with CNN layers and the selective training approach based on gated pruning to achieve benchmark results.

show abstract

“…We utilize a pre-trained uncased RoBERT a large model to evaluate comments and their responses, assigning emotional scores that correspond to trends like Approval, Toxicity, and Neutral, among others. Zhao et al [4] highlight the advantages of the BERT model in detecting these trends. We then map the RoBERT a large output to these seven identified trends and train a dataset to predict the likelihood of each trend, using a variety of machine learning models.…”

Section: Introductionmentioning

confidence: 99%

A Visual Approach to Tracking Emotional Sentiment Dynamics in Social Network Commentaries

Hossain,

Puppala,

Alam

et al. 2024

Preprint

View full text Add to dashboard Cite

The expansion of social media has unlocked a real-time barometer of public opinion. This paper introduces a novel framework to analyze sentiment shifts in social network comment sections, a reflection of the broader public discourse over time. Leveraging a pre-trained uncased RoBERTa model, we predict emotional scores from user comments, mapping these to key sentiment trends such as Approval, Toxicity, Obscenity, Threat, Hate, Offensive, and Neutral. Our methodology employs machine learning techniques to train a dataset that connects emotional scores with these trends, generating trend probability scores. We utilize a bottom-up recursive algorithm to aggregate emotional scores within comment threads, enabling the prediction of trend scores using three distinct aggregation methods. The results demonstrate that our emotional prediction model achieves an AUC of 0.92, and XGBoost stands out with an F1 score exceeding 0.40. Our research elucidates the temporal evolution of online public sentiment, enhancing the understanding of digital social dynamics and offering insights for strategic online interaction, intervention, and content moderation.

show abstract

Multi-layer features ablation of BERT model and its application in stock trend prediction

Cited by 17 publications

References 18 publications

Decentralized Storage Cryptocurrencies: An Innovative Network-Based Model for Identifying Effective Entities and Forecasting Future Price Trends

Decentralized Storage Cryptocurrencies: An Innovative Network-Based Model for Identifying Effective Entities and Forecasting Future Price Trends

A novel multi-layer feature fusion-based BERT-CNN for sentence representation learning and classification

A Visual Approach to Tracking Emotional Sentiment Dynamics in Social Network Commentaries

Contact Info

Product

Resources

About