• Zhang et al 2019 - A Lightweight Recurrent Network for Sequence Modeling Related to the Transformer, LRNs are a drop-in replacement to other RNNs, which remove the sequential natural of RNN processing. It essentially uses a Key-Query-Value attention mechanism instead of the recurrence.

• Weiss et al 2018 - On the Practical Computational Power of Finite Precision RNNs for Language Recognition
• Merrill et al 2020 - A Formal Hierarchy of RNN Architectures Interesting, but possibly incorrect in practice because the theoretical analysis is for saturated RNNs (related follow-up here).

See Converting RNNs to WFSAs