Click here to flash read.
We propose a novel framework based on the attention mechanism to identify the
sentiment of a movie review document. Previous efforts on deep neural networks
with attention mechanisms focus on encoder and decoder with fixed numbers of
multi-head attention. Therefore, we need a mechanism to stop the attention
process automatically if no more useful information can be read from the
memory.In this paper, we propose an adaptive multi-head attention architecture
(AdaptAttn) which varies the number of attention heads based on length of
sentences. AdaptAttn has a data preprocessing step where each document is
classified into any one of the three bins small, medium or large based on
length of the sentence. The document classified as small goes through two heads
in each layer, the medium group passes four heads and the large group is
processed by eight heads. We examine the merit of our model on the Stanford
large movie review dataset. The experimental results show that the F1 score
from our model is on par with the baseline model.
No creative common's license