Vision and Language

This page is about vision and language tasks that are distinct from visual question answering (which only deals with question answering) or grounded language learning (which includes a learning component to the task).

• Uppal et al 2020 - Multimodal Research in Vision and Language: A Review of Current and Emerging Trends
• Multimodal Large Language Models (MLLMs)
  • Yin et al 2023 - A Survey on Multimodal Large Language Models Comprehensive github (continuously updated)
  • Li et al 2025 - A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  • For Visual QA:
    • Kuang et al 2024 - Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  • Evaluation of MLLMs:
    • Huang & Zhang 2024 - A Survey on Evaluation of Multimodal Large Language Models

• CLIP: Radford et al 2021 - Learning Transferable Visual Models From Natural Language Supervision github
• Li et al 2022 - BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation github
• Alayrac et al 2022 - Flamingo: a Visual Language Model for Few-Shot Learning (Open model)
• Li et al 2023 - BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models github
• LLaVA: Liu et al 2023 - Visual Instruction Tuning Github: LLaVA: Large Language and Vision Assistant
• Zhu et al 2023 - MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• Gemini-team 2023 - Gemini: A Family of Highly Capable Multimodal Models
• Qiao et al 2024 - VL-Mamba: Exploring State Space Models for Multimodal Learning
• Gemini-team 2024 - Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context
• Meta 2024 - Chameleon: Mixed-Modal Early-Fusion Foundation Models

• Yang et al 2023 - Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V

• Overviews
  • Sundar & Heck 2022 - Multimodal Conversational AI: A Survey of Datasets and Approaches
• Diana
  • Narayana et al 2018 - Cooperating with Avatars Through Gesture, Language and Action
  • McNeely-White et al 2019 - User-Aware Shared Perception for Embodied Agents
  • Krishnaswamy et al 2020 - Diana's World: A Situated Multimodal Interactive Agent
• Krishnaswamy & Pustejovsky 2020 - A Formal Analysis of Multimodal Referring Strategies Under Common Ground

See also this bibliography.

• Fried et al 2018 - Speaker-Follower Models for Vision-and-Language Navigation

See also Awesome Vision & Language Pretraining Papers.

• Bugliarello et al 2021 - Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

• Language and Behavior Learning
• Vision-and-Language A curated list of vision and language resources.
• Vision-Language Navigation
• Embodied Vision

• Dhruv Batra
• Jason Baldridge
• Tamara Berg
• Yejin Choi
• Justin Johnson
• Ray Mooney
• James Pustejovsky
• Jesse Thomason
• Xin Eric Wang
• Mark Yatskar

• Embodied AI
• Grounding
• Grounded Language Learning
• Image Captioning
• Video Captioning
• Visual Question Answering