Differences

This shows you the differences between two versions of the page.

--- ml:nn_architectures [2024/05/24 22:37] – [Feedforward Networks] jmflanig
+++ ml:nn_architectures [2025/03/25 07:34] (current) – [Sequence Networks] jmflanig
@@ Line 45: / Line 45: @@
   * [[https://arxiv.org/pdf/2105.03824.pdf|FNet]] A faster, attention-free Transformer architecture based on Fourier transforms
   * [[https://arxiv.org/pdf/2305.10991.pdf|Anthe: Less is More! A slim architecture for optimal language translation]]
+  * [[https://arxiv.org/pdf/2305.13048|RWKV (Receptance Weighted Key Value) Network]] Information is passed across positions using a positional weight decay which gates the information. Allows parallel training like the transformer, but more efficient inference like the RNN
   * [[https://arxiv.org/pdf/2307.08621.pdf|RetNet (Retentive Network)]]
@@ Line 73: / Line 74: @@
   * [[https://arxiv.org/pdf/2102.11972.pdf|Narang et al 2021]] Compares activation functions in the Transformer
+===== Matrices ===
+Various representations of matrices, such as sparse, or low-dimensional ones.
+  * Tensor networks
+  * [[https://arxiv.org/pdf/2106.09685|LoRA]]
+  * [[https://arxiv.org/pdf/2204.00595|Monarch Matrices]]
 ===== Set and Pooling Networks =====