This page is about vision and language tasks that are distinct from visual question answering (which only deals with question answering) or grounded language learning (which includes a learning component to the task).

• Uppal et al 2020 - Multimodal Research in Vision and Language: A Review of Current and Emerging Trends

• CLIP: Radford et al 2021 - Learning Transferable Visual Models From Natural Language Supervision github
• Li et al 2022 - BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Alayrac et al 2022 - Flamingo: a Visual Language Model for Few-Shot Learning (Open model)
• Li et al 2023 - BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

• Overviews
  • Sundar & Heck 2022 - Multimodal Conversational AI: A Survey of Datasets and Approaches
• Diana
  • Narayana et al 2018 - Cooperating with Avatars Through Gesture, Language and Action
  • McNeely-White et al 2019 - User-Aware Shared Perception for Embodied Agents
  • Krishnaswamy et al 2020 - Diana's World: A Situated Multimodal Interactive Agent
• Krishnaswamy & Pustejovsky 2020 - A Formal Analysis of Multimodal Referring Strategies Under Common Ground

See also this bibliography.

• Fried et al 2018 - Speaker-Follower Models for Vision-and-Language Navigation

See also Awesome Vision & Language Pretraining Papers.

• Bugliarello et al 2021 - Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

• Language and Behavior Learning
• Vision-and-Language A curated list of vision and language resources.
• Vision-Language Navigation
• Embodied Vision

• Dhruv Batra
• Jason Baldridge
• Tamara Berg
• Yejin Choi
• Justin Johnson
• Ray Mooney
• James Pustejovsky
• Jesse Thomason
• Xin Eric Wang
• Mark Yatskar

• Embodied AI
• Grounded Language Learning
• Image Captioning
• Video Captioning
• Visual Question Answering