Stanford CS25: Transformers United V6 I From Language Models to Native Multimodal Intelligence

Native multimodal AI systems are evolving by applying transformer-based text processing techniques to other media like images and audio, though achieving a unified, efficient architecture that excels at both generation and understanding remains an unsolved research challenge.

• • Universal tokenization — Models convert images, audio, and video into sequence tokens similar to how text is handled [04:46].
• • Model output types — Two main classes exist: those that accept multimodal input but only output text, and "Omni" models that generate both text and non-text media [07:26].
• • Discrete vs. continuous — Approaches like Chameleon convert image patches into discrete indices, though this can lead to information loss, while methods like Transfusion use diffusion to improve generation quality [13:01].
• • Modality-dedicated weights — The Mixture of Transformers architecture assigns dedicated model layers to each data type, boosting generation quality without sacrificing text performance [21:49].
• • Language as abstraction — Language is a highly compressed form of human cognition, which makes it superior for reasoning compared to raw sensory data like images [37:20].
• • Generation vs. understanding — While strong understanding capabilities improve generation quality, training on generation does not reliably improve a model's ability to reason [35:30].

• How does the Mixture of Transformers architecture allow for training different modalities asynchronously?

• Why does training on language improve reasoning performance while training on raw video data does not?