Artificial intelligence learns to connect sound and images without human intervention
Scientists from MIT and their colleagues have developed a new machine learning system that can independently discover how audio and visual information are related, representing an advance in artificial intelligence and multimodal learning. The research team introduced a system called CAV-MAE Sync, which brings several major improvements over previous approaches and achieves better results than much more complex architectures.
People perceive the world multimodally, with auditory and visual perception being very closely interconnected. This natural ability inspired scientists from the Massachusetts Institute of Technology (MIT), IBM Research, Goethe University Frankfurt, and other institutions to create a system capable of connecting sound and images more effectively in a digital environment. Their research, published in May 2025, represents an advance in unsupervised learning, in which AI systems learn without the need for labeled data or human instructions. Existing approaches to audiovisual learning typically used global audio representations, which in practice meant that, for example, 10 seconds of audio were matched to a single video frame. However, this approach could not capture finer temporal relationships between sound and images. Moreover, these methods struggled with conflicting optimization objectives when attempting to learn reconstruction and cross-modal alignment simultaneously.
A new approach to learning
The new CAV-MAE Sync system addresses these problems in an original way by processing audio as a temporal sequence aligned with individual video frames instead of using global representations. "In our new approach, we address three key challenges," the research team explains in its publication. "First, we address the granularity mismatch between modalities by treating audio as a temporal sequence aligned with video frames. Second, we address conflicting optimization objectives by separating contrastive and reconstruction tasks through dedicated global tokens. And third, we improve spatial localization by introducing learnable register tokens that reduce the semantic burden on tokens representing image patches."
The researchers, including Yuan Gong and Andrew Rouditchenko from MIT, Edson Araujo from Goethe University Frankfurt, and others, tested their system on several standard datasets, including AudioSet, VGG Sound, and ADE20K Sound. They evaluated its performance on tasks such as zero-shot retrieval, classification, and localization. The results show that CAV-MAE Sync achieves state-of-the-art results and outperforms more complex architectures. One of the key improvements is the way the system handles the temporal alignment of sound and images. Instead of using a single audio representation for the entire video, CAV-MAE Sync extracts audio segments corresponding to individual video frames. This ensures that each audio segment is temporally aligned with the corresponding video frame, increasing coherence between modalities and enabling more accurate learning of relationships.
Another significant contribution is the introduction of global tokens for contrastive tasks. While traditional approaches aggregate patch tokens to create global representations, CAV-MAE Sync introduces dedicated global tokens specifically for contrastive tasks. This reduces the information burden on tokens representing image patches, which can now focus on reconstruction, while the global tokens aggregate information during both the individual modality encoding phases and their joint processing. The researchers also implemented so-called register tokens, which help keep patch tokens focused on reconstruction tasks and allow global tokens to concentrate on contrastive tasks. This separation improves the model's ability to capture semantic information and perform localization. Register tokens are added to the token sequences and are processed in the shared layer in the same way as global tokens.
The experimental results are impressive—the system achieved 35.2% accuracy in retrieving the correct matching audio for visual queries on the AudioSet dataset, representing a significant improvement over previous methods. In classification tasks, it achieved 30.5 mAP on AudioSet-20K and 52.7% accuracy on VGGSound, again outperforming comparable approaches. "Our work shows that by improving temporal alignment between audio and visual modalities and separating conflicting optimization objectives, we can achieve better results with a simpler architecture," the researchers conclude. This advance has potential applications across a wide range of areas, from sound- and image-based retrieval and multimedia content classification to the localization of sound sources in visual scenes.
The research was supported by the German Federal Ministry of Education and Research and the MIT-IBM Watson AI Lab. The scientists also made their code available on GitHub to enable further research and development in this field.
