Partial list of open problems in NLP.

• Koehn & Knowles 2017 - Six Challenges for Neural Machine Translation

• Are there simpler, faster methods for contextualized representations than pre-training Transformers? Historically, complex methods have been invented before researchers find simpler ways to do a similar thing. For example, there were methods for pre-training word embeddings that did not scale well until Tomas Mikolov asked the question “Is there a more efficient way to do this?” and invented the skip-gram model (Mikolov 2013 - Efficient Estimation of Word Representations in Vector Space and follow-up work). It is an open question if we are in a similar situation with Transformer models and contextualized pre-training today.

• Transformers are hard to train (Liu et al 2020 - Understanding the Difficulty of Training Transformers), and often we can't even get them to overfit and just train a long as we can. This isn't a good situation, and shows there is some issue with the normalization, the initializer, or (less likely) the optimizer. Feedforward, CNN, and RNNS had this issue for a long time, and these issues were fixed with Glorot initialization, batch normalization, and layer normalization. The open problem is: what is the correct initialization and normalization procedures for the Transformer?

• Explainability is an open problem in machine learning and NLP. See explainability.