Sometimes called tiny machine learning (TinyML) which is machine learning for resourced constrained devices.

• Voghoei et al 2019 - Deep Learning at the Edge
• 2019 - Deep Learning With Edge Computing: A Review
• Wang et al 2019 - Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Xu et al 2020 - Edge Intelligence: Architectures, Challenges, and Applications

• Iandola et al 2020 - SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
• 2017 - Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• Sankar et al 2020 - ProFormer: Towards On-Device LSH Projection Based Transformers
• Zhao et al 2021 - Extremely Small BERT Models from Mixed-Vocabulary Training Compresses BERT for edge devices by compressing the input word embeddings

• PyTorch Mobile
• TensorFlow Lite
• TensorFlow Lite for Microcontrollers

• MLSys - Has some mobile, on-device, or edge computing papers
• TinyML (Educational - I don't think they publish papers)

• Model Compression