Different types of Attention

Table of Contents

\(s_t\) and \(h_i\) are source hidden states and target hidden state, the shape is (n,1). \(c_t\) is the final context vector, and \(\alpha_{t,s}\) is alignment score.

\[\begin{aligned} c_t&=\sum_{i=1}^n \alpha_{t,s}h_i \\ \alpha_{t,s}&= \frac{\exp(score(s_t,h_i))}{\sum_{i=1}^n \exp(score(s_t,h_i))} \end{aligned}\]

Global(Soft) VS Local(Hard) #

Global Attention takes all source hidden states into account, and local attention only use part of the source hidden states.

Content-based VS Location-based #

Content-based Attention uses both source hidden states and target hidden states, but location-based attention only use source hidden states.

Here are several popular attention mechanisms:

Dot-Product #

\[score(s_t,h_i)=s_t^Th_i\]

Scaled Dot-Product #

\[score(s_t,h_i)=\frac{s_t^Th_i}{\sqrt{n}}\] where n is the vectors dimension. Google’s Transformer model has similar scaling factor when calculate self-attention: \(score=\frac{KQ^T}{\sqrt{n}}\)

Location-Base #

\[socre(s_t,h_i)=softmax(W_as_t)\]

General #

\[score(s_t,h_i)=s_t^TW_ah_i\]

\(Wa\)’s shape is (n,n)

Concat #

\[score(s_t,h_i)=v_a^Ttanh(W_a[s_t,h_i])\]

\(v_a\)’s shape is (x,1), and \(Wa\) ’s shape is (x,x). This is similar to a neural network with one hidden layer.

When I doing a slot filling project, I compare these mechanisms. Concat attention produce the best result.