Attention Mechanisms

• Attention? Attention! Blog post by Weng
• Brief Introduction to Attention Models
• Chaudhari et al 2019 - An Attentive Survey of Attention Models

From Attention? Attention! Blog post by Weng:

• Graves 2013 - Generating Sequences With Recurrent Neural Networks Uses an alignment mechanism for handwriting generation, similar to the attention mechanism. The Deep Learning Book p. 415 at the end of Ch 10 says “The idea of attention mechanisms for neural networks was introduced even earlier, in the context of handwriting generation (Graves, 2013), with an attention mechanism that was constrained to move only forward in time through the sequence.”
• Bahdanau et al 2014 - Neural Machine Translation by Jointly Learning to Align and Translate The paper that started it all. Introduced the attention mechanism and initiated the deep learning revolution in NLP. Basically, the paper that got neural machine translation to actually work.
• Ba et al 2014 - Multiple Object Recognition with Visual Attention
• Luong et al 2015 - Effective Approaches to Attention-based Neural Machine Translation This paper introduced dot product attention.
• Cheng et al 2016 - Long Short-Term Memory-Networks for Machine Reading This paper introduced self-attention, called intra-attention (introduced in the Long Short-Term Memory-Networks, LSTMNs, section 3.2). See Fig 1 for a picture.
• Parikh et al 2016 - A Decomposable Attention Model for Natural Language Inference This paper, uses intra-attention from Cheng 2016, and according to the Transformer paper, was the inspiration of self-attention in the Transformer.
• Lin et al 2017 - A Structured Self-Attentive Sentence Embedding Introduces the term “self-attention,” which they say is slightly different than Cheng et al's intra-attention.
• Vaswani et al 2017 - Attention Is All You Need Introduced multi-head attention and the Transformer architecture
• LSH Attention
  • Kitaev et al 2020 - Reformer: The Efficient Transformer Uses LSH to speed up attention
• Linearized Attention
  • Katharopoulos et al 2020 - Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Misleading name. This paper linearizes the softmax in the attention layers which makes it O(n) to compute
  • Wang et al 2020 - Linformer: Self-Attention with Linear Complexity
• Random Feature Attention: Peng et al 2021 - Random Feature Attention Uses random features to approximate the softmax, making it O(1). Drop in replacement for standard attention. Shows that it works in the Transformer, and is twice as fast.
• Dao et al 2022 - FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  • Early related work: Rabe & Staats 2021 - Self-attention Does Not Need $O(n^2)$ Memory
• Single-Headed Gated Attention (SHGA): Ma et al 2022 - Mega: Moving Average Equipped Gated Attention Shows that single-headed gated attention can simulate multi-head attention, and is more expressive (see section 3.3 and Theorem 1).
• Sparse Attention
  • Longformer
  • BigBird
  • Hierarchical Attention Transformers (HAT): Chalkidis et al 2022 - An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
• MoE Sparse Attention
  • Zhang et al 2022 - Mixture of Attention Heads: Selecting Attention Heads Per Token
  • Csordas et al 2023 - SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
  • Jin et al 2024 - SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

• Ni et al 2023 - Finding the Pillars of Strength for Multi-Head Attention

• Neural Network Architectures
• Transformers