Very Deep Convolutional Networks for Text Classification

Conneau, Alexis; Schwenk, Holger; Barrault, Loïc; LeCun, Yann

doi:10.48550/arxiv.1606.01781

Cited by 98 publications

(179 citation statements)

References 0 publications

Supporting

Mentioning

177

Contrasting

Order By: Relevance

“…After 2010 the usage of deep learning models started to grow up, some research papers discussed models based on RNN and Long Short-Term Memory (LSTM) [20], Tree-LSTM [21], Multi-Timescale [22], Bidirectional-LSTM with two-dimensional max-pooling [23], some papers used the impact of CNN architectures with word embeddings, researchers [24] presented a Very Deep CNN model for text processing inspired by VGG [25] and ResNets [26]. One of other interesting CNN-based models [27] presented a tree-based CNN to capture sentence level semantics.…”

Section: Related Workmentioning

confidence: 99%

New Arabic Medical Dataset for Diseases Classification

Hammoud¹,

Vatian²,

Dobrenko³

et al. 2021

Intelligent Data Engineering and Automated Learning – IDEAL 2021

View full text Add to dashboard Cite

The Arabic language suffers from a great shortage of datasets suitable for training deep learning models, and the existing ones include general non-specialized classifications. In this work, we introduce a new Arab medical dataset, which includes two thousand medical documents collected from several Arabic medical websites, in addition to the Arab Medical Encyclopedia. The dataset was built for the task of classifying texts and includes 10 classes (Blood, Bone, Cardiovascular, Ear, Endocrine, Eye, Gastrointestinal, Immune, Liver and Nephrological) diseases. Experiments on the dataset were performed by fine-tuning three pre-trained models: BERT from Google, Arabert that based on BERT with large Arabic corpus, and AraBioNER that based on Arabert with Arabic medical corpus.

show abstract

Section: Related Workmentioning

confidence: 99%

New Arabic Medical Dataset for Diseases Classification

Hammoud¹,

Vatian²,

Dobrenko³

et al. 2021

Intelligent Data Engineering and Automated Learning – IDEAL 2021

View full text Add to dashboard Cite

show abstract

“…One form of models called a character-level CNN learns a text embedding from individual characters, and these embeddings are then processed using a sequential CNN and one or more dense layers depending on the task. Recent examples of character-level CNNs include [6,45]. In particular, Conneau et al [6] investigated very deep architectures for the purpose of classifying natural language text.…”

Section: Related Workmentioning

confidence: 99%

URLTran: Improving Phishing URL Detection Using Transformers

Maneriker¹,

Stokes²,

Lazo³

et al. 2021

Preprint

View full text Add to dashboard Cite

Browsers often include security features to detect phishing web pages. In the past, some browsers evaluated an unknown URL for inclusion in a list of known phishing pages. However, as the number of URLs and known phishing pages continued to increase at a rapid pace, browsers started to include one or more machine learning classifiers as part of their security services that aim to better protect end users from harm. While additional information could be used, browsers typically evaluate every unknown URL using some classifier in order to quickly detect these phishing pages. Early phishing detection used standard machine learning classifiers, but recent research has instead proposed the use of deep learning models for the phishing URL detection task. Concurrently, text embedding research using transformers has led to state-of-the-art results in many natural language processing tasks. In this work, we perform a comprehensive analysis of transformer models on the phishing URL detection task. We consider standard masked language model and additional domain-specific pre-training tasks, and compare these models to fine-tuned BERT and RoBERTa models. Combining the insights from these experiments, we propose URLTran which uses transformers to significantly improve the performance of phishing URL detection over a wide range of very low false positive rates (FPRs) compared to other deep learning-based methods. For example, URLTran yields a true positive rate (TPR) of 86.80% compared to 71.20% for the next best baseline at an FPR of 0.01%, resulting in a relative improvement of over 21.9%. Further, we consider some classical adversarial black-box phishing attacks such as those based on homoglyphs and compound word splits to improve the robustness of URLTran. We consider additional fine tuning with these adversarial samples and demonstrate that URLTran can maintain low FPRs under these scenarios.

show abstract

“…Convolutional networks are a cornerstone of modern deep learning: they're widely used in the visual domain [1,2,3], speech recognition [4], text classification [5], and time series classification [6] basically, in any system that is approximately translation invariant. They have also played an important role in enhancing our understanding of the visual stream [7].…”

Section: Introductionmentioning

confidence: 99%

Towards Biologically Plausible Convolutional Networks

Pogodin¹,

Mehta²,

Lillicrap³

et al. 2021

Preprint

View full text Add to dashboard Cite

Convolutional networks are ubiquitous in deep learning. They are particularly useful for images, as they reduce the number of parameters, reduce training time, and increase accuracy. However, as a model of the brain they are seriously problematic, since they require weight sharing -something real neurons simply cannot do. Consequently, while neurons in the brain can be locally connected (one of the features of convolutional networks), they cannot be convolutional. Locally connected but non-convolutional networks, however, significantly underperform convolutional ones. This is troublesome for studies that use convolutional networks to explain activity in the visual system. Here we study plausible alternatives to weight sharing that aim at the same regularization principle, which is to make each neuron within a pool react similarly to identical inputs. The most natural way to do that is by showing the network multiple translations of the same image, akin to saccades in animal vision. However, this approach requires many translations, and doesn't remove the performance gap. We propose instead to add lateral connectivity to a locally connected network, and allow learning via Hebbian plasticity. This requires the network to pause occasionally for a sleep-like phase of "weight sharing". This method enables locally connected networks to achieve nearly convolutional performance on ImageNet, thus supporting convolutional networks as a model of the visual stream.Preprint. Under review.

show abstract

Very Deep Convolutional Networks for Text Classification

Cited by 98 publications

References 0 publications

New Arabic Medical Dataset for Diseases Classification

New Arabic Medical Dataset for Diseases Classification

URLTran: Improving Phishing URL Detection Using Transformers

Towards Biologically Plausible Convolutional Networks

Contact Info

Product

Resources

About