Text Categorization Using Hybrid Neural Network Model Based on Attention Mechanism

• 1、Hunan University,College of Computer Science and Electronic Engineering,changsha,410006

Abstract：Text categorization is one of important tasks in natural language processing(NLP). In text categorization, sentence modeling is critical. In exsisting schemes, local correlation of spatial can be effectively captured by convolutional neural network (CNN), and long-term dependencies, historical information and follwing information can obtained by BiLSTM a variant of recurrent neural network (RNN). For their different modeling capabilities, this paper proposes a hybrid neural network model based on the attention mechanism. It first introduces a contribution rate based on BiLSTM to adjust the different influences on historical information and following information and applies the attention mechanism to combine CNN with BiLSTM to figure out the weight of key information for the relief of the loss of text information. The experimental results show that the introduction of attention mechanism and the contribution rate can effectively improve the accuracy of text categorization.

Keywords： Text categorization Convolutional Neural Network Recurrent Neural Network Long Short-Term Memory Bidirectional LSTM Attention mechanism Contribution rate