HeBERT and HebEMO: A Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition

Abstract

Sentiment analysis of user-generated content (UGC) can provide valuable information across numerous domains, including marketing, psychology, and public health. Currently, there are very few Hebrew models for natural language processing in general, and for sentiment analysis in particular; indeed, it is not straightforward to develop such models because Hebrew is a morphologically rich language (MRL) with challenging characteristics. Moreover, the only available Hebrew sentiment analysis model, based on a recurrent neural network, was developed for polarity analysis (classifying text as positive, negative, or neutral) and was not used for detection of finer-grained emotions (e.g., anger, fear, or joy). To address these gaps, this paper introduces HeBERT and HebEMO. HeBERT is a transformer-based model for modern Hebrew text, which relies on a BERT (bidirectional encoder representations from transformers) architecture. BERT has been shown to outperform alternative architectures in sentiment analysis and is suggested to be particularly appropriate for MRLs. Analyzing multiple BERT specifications, we find that whereas model complexity correlates with high performance on language tasks that aim to understand terms in a sentence, a more parsimonious model better captures the sentiment of an entire sentence. Notably, regardless of the complexity of the BERT specification, our BERT-based language model outperforms all existing Hebrew alternatives on all language tasks examined. HebEMO is a tool that uses HeBERT to detect polarity and extract emotions from Hebrew UGC. HebEMO is trained on a unique COVID-19-related UGC data set that we collected and annotated for this study. Data collection and annotation followed an active learning procedure that aimed to maximize predictability. We show that HebEMO yields a better performance accuracy for polarity classification. Emotion detection reaches high performance for various target emotions, with the exception of surprise, which the model failed to capture. These results are better than the best reported performance, even among English-language models of emotion detection.

Leveraging natural language processing to study emotional coherence in psychotherapy.

“…Compared to fully fine-tuned models, adapter models only incorporate a few task-specific parameters for each new task. The BERT-based experiments were conducted on three pretrained language models, each capable of handling Hebrew text, that is, (a) XLM-RoBERTa (Conneau et al, 2019), a multilingual language model based on the RoBERTa architecture (Liu et al, 2019), (b) HeBERT (Chriqui & Yahav, 2021), a monolingual BERT model trained on Hebrew data, and (c) AlephBERT (Seker et al, 2022), another monolingual BERT-based model trained on a large Hebrew vocabulary of 52K tokens optimized via masked-token prediction. Corresponding variants of these BERT models with lightweight adapter solutions focused on a small number of task-specific parameters for training using bottleneck adapters (Houlsby et al, 2019) and mix-and-match (MAM) adapters (He et al, 2021).…”

Combining Psychological Theory with Language Models for Suicide Risk Detection

“…In general, NLP models and solutions for lowresource languages are extremely limited. In Hebrew, two pre-trained language models were published, HeBERT (Chriqui and Yahav, 2021) and AlephBERT (Seker et al, 2022). We used Aleph-BERT which is freely available and was trained on a larger dataset than HeBERT and was able to outperform HeBERT on a variety of natural language tasks.…”

Temporal Relation Classification in Hebrew

“…The second event is marked similarly, but replacing 1 with 2. We use four Hebrew language models: AlephBERT (Seker et al, 2022), HeBERT (Chriqui and Yahav, 2022), mBERT cased (bertbase-multilingual) (Devlin et al, 2019), and AlephBERTGimmel (Guetta et al, 2022), all having 110M parameters, obtained directly from Hugging Face's transformer library. Inspired by Soares et al ( 2019), we experiment with three sequence classification architectures:…”