Shortcut Sequence Tagging

Abstract

Deep stacked RNNs are usually hard to train. Adding shortcut connections across different layers is a common way to ease the training of stacked networks. However, extra shortcuts make the recurrent step more complicated. To simply the stacked architecture, we propose a framework called shortcut block, which is a marriage of the gating mechanism and shortcuts, while discarding the selfconnected part in LSTM cell. We present extensive empirical experiments showing that this design makes training easy and improves generalization. We propose various shortcut block topologies and compositions to explore its effectiveness. Based on this architecture, we obtain a 6% relatively improvement over the state-of-the-art on CCGbank supertagging dataset. We also get comparable results on POS tagging task.

Generating CCG Categories

“…Clark and Curran (2007) propose C&C tagger which uses a log-linear model to build the supertagger. Recent works have applied neural networks to supertagging (Xu, Auli, and Clark 2015;Vaswani et al 2016;Wu, Zhang, and Zong 2017b). These works perform a multi-class classification on pre-defined category sets and they can't capture the inside connections between categories because categories are independent of each other.…”

Generating CCG Categories

“…propose C&C tagger which uses a log-linear model to build the supertagger. Recent works have applied neural networks to supertagging (Xu, Auli, and Clark 2015;Vaswani et al 2016;Wu, Zhang, and Zong 2017b). These works perform a multi-class classification on pre-defined category sets and they can't capture the inside connections between categories because categories are independent of each other.…”