Marrying Up Regular Expressions with Neural Networks: A Case Study for Spoken Language Understanding

Luo, Bingfeng; Feng, Yansong; Wang, Zheng; Huang, Songfang; Yan, Rui; Zhao, Dongyan

doi:10.18653/v1/p18-1194

Cited by 43 publications

(36 citation statements)

References 27 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also compare our method with the basic neural networks enhanced by existing methods of combining rules and neural networks. Luo et al (2018) propose three ways to utilize RE matching results in a neural model: 1) use the results as additional input features; 2) use the results to guide attention; 3) use the results to directly tune the output logits. As our basic networks do not involve attention, we enhance them using 1), 3) or both, denoted as +i, +o and +io respectively.…”

Section: Re-enhanced Basic Networkmentioning

confidence: 99%

“…Nevertheless, symbolic rules are still an indispensable tool in various industrial NLP applications. Regular expressions (RE) are one of the most representative and useful forms of symbolic rules and are widely used for solving tasks such as pattern matching (Hosoya and Pierce, 2001;Zhang et al, 2018) and intent classification (Luo et al, 2018). RE-based systems are highly interpretable * Corresponding author.…”

Section: Introductionmentioning

confidence: 99%

“…How to combine the advantages of symbolic rules and neural networks is an open question and is drawing increasing attention recently. One possible way is to use rules to constrain neural networks, usually in the manner of regularization via knowledge distillation (Hu et al, 2016) and multi-task learning (Awasthi et al, 2020;Xu et al, 2018), or by tuning the output logits of neural networks (Li and Srikumar, 2019;Luo et al, 2018). In this way, information from rules can be injected into neural networks, though the neural networks still require training and remain black boxes that are hard to interpret and manipulate.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks

Jiang¹,

Zhao²,

Chu³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Neural networks can achieve impressive performance on many natural language processing applications, but they typically need large labeled data for training and are not easily interpretable. On the other hand, symbolic rules such as regular expressions are interpretable, require no training, and often achieve decent accuracy; but rules cannot benefit from labeled data when available and hence underperform neural networks in rich-resource scenarios. In this paper, we propose a type of recurrent neural networks called FA-RNNs that combine the advantages of neural networks and regular expression rules. An FA-RNN can be converted from regular expressions and deployed in zero-shot and cold-start scenarios. It can also utilize labeled data for training to achieve improved prediction accuracy. After training, an FA-RNN often remains interpretable and can be converted back into regular expressions. We apply FA-RNNs to text classification and observe that FA-RNNs significantly outperform previous neural approaches in both zeroshot and low-resource settings and remain very competitive in rich-resource settings.

show abstract

Section: Re-enhanced Basic Networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks

Jiang¹,

Zhao²,

Chu³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…Instead of massive labeling, human experts only need to label a small set of documents to fine-tune the neural network. Luo et al [49] incorporated knowledge of regular expressions into the training of neural networks to solve typical spoken language understanding (SLU) tasks. Experiments demonstrate that the learning performance can be significantly improved by the implicit knowledge encoded within regular expressions.…”

Section: Related Workmentioning

confidence: 99%

Regular Expression Based Medical Text Classification Using Constructive Heuristic Approach

Cui

Bai

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Medical text classification assigns medical related text into different categories such as topics or disease types. Machine learning based techniques have been widely used to perform such tasks despite the obvious drawback in such ''black box'' approach, leaving no easy way to fine-tune the resultant model for better performance. We propose a novel constructive heuristic approach to generate a set of regular expressions that can be used as effective text classifiers. The main innovation of our approach is that we develop a novel regular expression based text classifier with both satisfactory classification performance and excellent interpretability. We evaluate our framework on real-world medical data provided by our collaborator, one of the largest online healthcare providers in the market, and observe the high performance and consistency of this approach. Experimental results show that the machine-generated regular expressions can be effectively used in conjunction with machine learning techniques to perform medical text classification tasks. The proposed methodology improves the performance of baseline methods (Naive Bayes and Support Vector Machines) by 9% in precision and 4.5% in recall. We also evaluate the performance of modified regular expressions by human experts and demonstrate the potential of practical applications using the proposed method. INDEX TERMS Regular expressions, text classification, constructive heuristic method.

show abstract

“…Locascio et al (2016) proposed training LSTM NN to generate REs from sample pieces of text. Luo et al (2018) incorporates knowledge of REs into training of NNs at three different levels: as the input features to NNs, as regularizations of the outputs of NN layers, or as a reward/penalty in the loss functions in NNs.…”

Section: Related Workmentioning

confidence: 99%

Regular Expression Guided Entity Mention Mining from Noisy Web Data

Zhang¹,

He²,

Vučetić³

et al. 2018

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Many important entity types in web documents, such as dates, times, email addresses, and course numbers, follow or closely resemble patterns that can be described by Regular Expressions (REs). Due to a vast diversity of web documents and ways in which they are being generated, even seemingly straightforward tasks such as identifying mentions of date in a document become very challenging. It is reasonable to claim that it is impossible to create a RE that is capable of identifying such entities from web documents with perfect precision and recall. Rather than abandoning REs as a go-to approach for entity detection, this paper explores ways to combine the expressive power of REs, ability of deep learning to learn from large data, and human-in-the loop approach into a new integrated framework for entity identification from web data. The framework starts by creating or collecting the existing REs for a particular type of an entity. Those REs are then used over a large document corpus to collect weak labels for the entity mentions and a neural network is trained to predict those REgenerated weak labels. Finally, a human expert is asked to label a small set of documents and the neural network is fine tuned on those documents. The experimental evaluation on several entity identification problems shows that the proposed framework achieves impressive accuracy, while requiring very modest human effort.

show abstract

Marrying Up Regular Expressions with Neural Networks: A Case Study for Spoken Language Understanding

Cited by 43 publications

References 27 publications

Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks

Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks

Regular Expression Based Medical Text Classification Using Constructive Heuristic Approach

Regular Expression Guided Entity Mention Mining from Noisy Web Data

Contact Info

Product

Resources

About