Over the past few years, artificial intelligence has made tremendous progress. First, large language models, trained on massive volumes of text, mastered the ability to perform “intelligent Q&A.” This was followed by multimodal world models, which, trained on vast amounts of images and videos, achieved a more advanced capability: “spatial intelligence.”

For a time, this made us believe that AI was smart enough. Yet today, it still cannot do one thing: listen to the world.

A multimodal robustness study, MMA-Bench, conducted by Boston University and Google DeepMind found that mainstream multimodal large models exhibit a clear modality preference: text receives the highest attention weight, followed by vision, while audio is the easiest modality to overlook.

AI training has been heavily biased toward text, images, and video, while sound has consistently been treated as a secondary modality. Its value has been seriously underestimated—we still imagine sound mainly in terms of listening to music and voice interaction, and rarely consider that sound can also perceive the physical world, predict intentions, and infer how events will unfold.

In many situations, people perceive events through hearing before vision. For example, when someone approaches from around a corner, you hear their footsteps first and can identify their presence before seeing them; in a dark room, if someone falls, you may be unable to see them but can still determine the direction from the sound. Even before birth, when the eyes have not yet opened, humans first perceive the world through hearing.

After large language models taught AI to “answer questions” and multimodal world models taught machines to “see the world,” the next frontier for AI will very likely be the auditory world model. 52 AUDIO calls on everyone working in AI to rethink the value of sound.

In this article, I will explain why sound matters, what cognitive acoustics is, how hearing can be used to predict, listen to the world, and understand the physical world, how an auditory world model can be built, and what disruptive changes and impacts it will bring to the consumer electronics and embodied intelligence industries in the future.

This article is based on a three-hour in-depth conversation between Mr. Yang Jun, Deputy Director of the Institute of Acoustics, Chinese Academy of Sciences, and Cai Qianxun (Boss Dou), founder of I Love Audio Network. Many thanks to Mr. Yang for his insights, which have opened the door to the “world of sound” for us. The development of AI agents therefore faces a new set of challenges. We offer this article as an initial exploration and invite everyone to join us in exploring the largely unknown “auditory world model.”

Why Does Sound Matter in the AI Era?

Humans First Perceive the World Through Hearing

The developmental sequence of human senses directly determines the underlying logic of how AI perceives the physical world. During fetal development, hearing is the first sense used to perceive the “world.” At around 18 weeks of pregnancy, the auditory structures become fully formed, allowing the fetus to perceive the mother’s heartbeat and blood flow. At 24 to 26 weeks, the fetus can perceive external sounds and respond with movement. Vision develops relatively later, as the fetal eyelids remain closed for a long period and gradually begin to open only after around 26 weeks.

After birth, newborns can only capture blurred light and shadows at close range and cannot clearly see the world. They rely almost entirely on hearing to recognize familiar people, perceive their surroundings, and assess danger. Unlike vision and language, which are acquired after birth, hearing is an innate human sense that engages with the world earliest and is the first sensory channel through which humans perceive the physical world.

Hearing Is the Only Sense That Is Omnidirectional, All-Weather, Long-Range, and Able to Penetrate Obstacles

Humans are generally considered to have five classic senses: sight, hearing, smell, taste, and touch. Vision uses the eyes to see the world, perceive space, and recognize the shape of objects, but it has a limited field of view and blind spots and depends heavily on light; in darkness, vision is largely ineffective. Smell and taste are forms of chemical sensing, while touch can perceive temperature and pain. These sensory dimensions are relatively limited.

Only hearing can perceive dynamic events in all directions—360 degrees—without requiring us to turn our heads, even through walls and from distances of dozens of meters or more. Hearing provides omnidirectional, all-weather, long-range, and penetrating environmental perception. Even more uniquely, among the five senses, hearing is the only one that can use sound to distinguish a person’s tone, emotions, and intentions. It can even determine whether someone is sincere or insincere from the same words, such as “he said yes.” This will be crucial for future AI agents to understand human “intentions” and provide AI-powered services.

In the Past, Sound Was Mainly Used for Music, Recording, and Voice Interaction

Over the decades and even centuries of audio development, our use of sound had been largely limited to listening to music. In pursuit of the highest possible sound quality and faithful waveform reproduction, acoustic research had consistently focused on optimizing frequency response, signal-to-noise ratio, and noise reduction. Later, digital voice recorders emerged, and our use of sound evolved toward recording the content of meetings and conversations. With the development of AI technology, headphones began to implement AI noise reduction, echo cancellation, and beamforming, further expanding the use of sound into AI voice interaction. Yet today, our use of sound still remains largely at the level of listening to music, recording, and voice interaction, and its deeper value has not yet been fully explored.

Hearing Is Not About Recording Sound, but Predicting

In the AI era, a counterintuitive proposition is becoming a focus of acoustic and AI research: hearing is not about recording sound, but about predicting the future that sound indicates. In the past, we pursued the preservation of complete waveforms, but overlooked the fact that hearing has never been a passive recording of external sound signals. Instead, it is an active prediction of the world made by humans for survival and action.

Why can hearing make active predictions? Sound waves generated by the vibration of everything are produced synchronously with physical events and change in real time. Glass shattering, an object falling, and footsteps moving—each sound corresponds to a physical event that is actually occurring and carries temporal, spatial, and causal information. Vision tells you only “what is here,” while hearing can tell you “what is happening” and “what is about to happen.” The auditory system can continuously generate predictions from the sound-wave information it receives and update its model based on prediction errors.

Classic mismatch negativity experiments  (MMN) in neuroscience have already demonstrated that the human brain forms expectations about sound sequences in advance. When an external sound deviates from the prediction, an EEG response is automatically generated within 100–250 ms, without requiring conscious attention. Hearing, therefore, is not passive reception of sound-wave signals; it is active prediction of how events will unfold.

Compared with the other senses, hearing has an exceptionally strong ability to predict the course of events and perceive the physical world, and this ability will play an important role in the AI era. Yet we currently tend to train AI for conversation using speech corpora, and AI remains far behind humans in auditory capabilities. What we should truly be doing today is training AI models with the spatial information, physical laws, and causal logic embedded in sound, teaching AI how to “listen to the world.”

Listening to the “World”: Time, Space, and Causality

In Buddhist scriptures, “shi” refers to the flow of time and the sequence of events, while “jie” refers to direction and space. “Listening to the world” does not mean merely “hearing sounds”; it means understanding the time and space in which acoustic events occur. Traditional speech AI uses only half the value of sound, compressing it into a sequence of words along the event timeline (a sound record: when you said what), while completely discarding spatial information. It does not know where the sound source is located, what the surrounding space is like, or exactly who is speaking.

The Acoustic Industry Is Entering the Era of Cognitive Acoustics

Acoustics has already gone through three waves of development: physical acoustics addresses “what sound is,” computational acoustics addresses “how sound is computed,” and intelligent acoustics addresses “how sound is used.” The AI era will usher in a fourth wave, moving from intelligent acoustics to cognitive acoustics. There is a clear distinction between the two. Intelligent acoustics follows a logic of “full-scale collection + back-end processing,” using AI algorithms to perform noise reduction, enhancement, and recognition after acquisition. Cognitive acoustics, by contrast, studies how the human ear perceives sound and seeks to “understand” the world through hearing. Cognitive acoustics no longer pursues perfect waveform reproduction or perfect noise removal. Counterintuitively, it treats the acoustic information contained in noise as a “clue” and uses it to infer the spatial environment in which the speaker is located. Cognitive acoustics seeks effective information extraction, selective perception, and active prediction.

The field of optics has already leaped from “seeing clearly” to “understanding what is seen.” Event cameras no longer capture complete images; they only capture changes in light intensity and reconstruct the world through “optical encoding + algorithmic decoding.” Following the same logic, Fei-Fei Li has pushed “understanding what is seen” further toward spatial intelligence, using multimodal world models to teach machines to “see the world.” Acoustics can fully follow this path: event microphones record only sudden changes in the sound field; cognitive arrays use beamforming and semantic filtering; acoustic sensors integrating storage and computation perform preprocessing at the physical layer; and an auditory world model can then be built to teach AI to “listen to the world.”

Cognitive Acoustics Is Still Missing Two Pieces of the Puzzle

Cognitive acoustics is still missing two key pieces of the puzzle. The first is an embodied perception loop: robots need to distinguish whether the sound source itself is moving or whether changes in the sound field are caused by their own movement. This is a prerequisite for advanced physical AI perception. The second is a foundational dataset. In the current audio field, AudioSet is only a classification benchmark and lacks scene, causal, and intention annotations. The industry needs an “ImageNet” equivalent for the auditory domain before auditory intelligence can catch up with the pace of spatial intelligence.

Looking back, Fei-Fei Li, the creator of ImageNet, initially used a simple but labor-intensive method to teach machines to “see”: she manually labeled 14 million images to build the ImageNet dataset. Today, all image-recognition models have grown up on this foundation. In the auditory field, we urgently need a SoundNet—an “ImageNet for Sound.”

The Ultimate Direction of Auditory Intelligence

The evolution of auditory intelligence can be divided into seven levels: signal processing (noise reduction), speech recognition (words), speaker understanding (identity), spatial hearing (3D localization), acoustic scene understanding (non-verbal events), contextual awareness (intent), and the auditory world model (prediction of future states). The first three levels are “hearing sounds,” the middle two are “hearing events,” the sixth is “hearing intent,” and the seventh and highest level is “hearing potential”—using acoustic signals to anticipate the location of sound sources and the future course of events. The industry currently remains largely at the first three levels, while higher-order cognitive hearing is only beginning to emerge.

How Do We Build an Auditory World Model?

Traditional acoustic systems remain at the perception level. The auditory world model we will build in the future, however, will need higher-level AI capabilities, enabling a leap from “hearing” to “hearing clearly,” then to “understanding,” and ultimately to “prediction.”

The auditory world model must have three core capabilities:

Decoding Physical Events: No longer treating sound as an isolated waveform, the model will reconstruct the physical process that produced it. It will need to understand whether a particular spectral change indicates metal rubbing against something or glass breaking.

Inverting the Environmental Field: By using the reflection and attenuation of sound waves, the model will reconstruct an invisible spatial map and understand whether sound is propagating through a spacious hall or a narrow pipe.

Predicting Future States: Based on an understanding of current physical events and spatial structures, the model will infer what will happen in the next second. When it hears the sound of braking, it will not only recognize it as braking; it will also be able to predict from the sound where the vehicle is likely to stop or whether it may skid.

The Disruptive Impact of Auditory World Models on Industry

The auditory world model will completely break the underlying logic of traditional acoustics centered on “collection, recording, and sound quality,” returning instead to the essence of human hearing. It will use acoustic information to understand and perceive the physical world, anticipate events, and discern human emotions and intentions. This will profoundly reshape the consumer electronics, robotics, and embodied intelligence industries.

Consumer Electronics

For a long time, consumer audio products such as headphones and speakers have been trapped in an arms race over ultimate sound quality, resulting in severe product homogenization. The emergence of acoustic world models will completely rewrite the competitive logic of consumer electronics, upgrading audio hardware from “audio-visual devices” to “AI perception terminals.”

In the future, audio hardware will no longer focus solely on noise reduction and sound-quality optimization. Instead, powered by an auditory world model, it will achieve scene understanding, intention recognition, event anticipation, and instruction-free proactive interaction. Headphones will be able to determine a user’s travel context from environmental sounds, infer emotions from the user’s voice, understand the user’s intentions, and provide AI-powered services.

Robotics

Humanoid robots, service robots, and industrial robots are developing rapidly today, imitating human forms and movements. When they run fast or jump high, we are amazed and may assume that robots have already become sufficiently intelligent. But frankly, the AI capabilities of robots remain far below the human level.

Perception solutions that rely only on vision and lidar have inherent blind spots and cannot adapt well to complex, dynamic, and occluded real-world environments. The robotics industry needs to think more deeply. In particular, solving the auditory shortcoming certainly does not mean simply equipping a robot with a high-fidelity voice recorder to record sounds.

The deployment of auditory world models will complete the final missing piece of the perception puzzle for embodied intelligence. Leveraging the temporal and causal perception capabilities of hearing, robots will be able to overcome the limitations of vision and, in darkness, smoke, or occluded environments, use footsteps, object friction, equipment vibrations, and abnormal environmental sounds to accurately determine spatial states, physical motion trajectories, and potential risks, and infer how physical events will develop.

More importantly, auditory world models will enable robots to break free from the limitation of “passively executing instructions” and achieve active perception, autonomous prediction, and closed-loop action. The brain uses prediction to interpret sound; physics encodes causality through sound; machines will use world models to predict the future. In the future, auditory world models will inevitably move toward embodied intelligence, helping embodied intelligence finally break free from the predicament of merely “looking human.”

Auditory Intelligence: AI’s Next Decade-Defining Frontier

“If today’s AI can only ‘describe the world,’ why can it not ‘understand the physical world and predict the world’?” This question is worth reconsidering by everyone.

Large language models have taught AI to answer; visual world models have taught AI to see; but AI still has not learned to “listen to the world.”

Auditory world models will teach AI to understand, anticipate, and perceive the causality and future states of the real physical world through sound.

In the past, we underestimated the value of sound, limiting it to shallow scenarios such as listening to music, recording, and voice interaction. But this completely goes against the underlying logic of how human hearing uses sound.

As the fourth wave of cognitive acoustics fully arrives, acoustics will finally move beyond its marginal position and stand alongside vision and language as a core pillar of AI perception. Auditory world models will fundamentally reshape key industries such as consumer electronics, robotics, and embodied intelligence, enabling machines to truly “hear the world, understand causality, and anticipate the future.”

In the future, the ultimate competition in AI will no longer be a contest between language and vision, but a competition in cognitive acoustics, causal prediction, and active perception. AI’s next frontier will ultimately be the auditory world model.

We welcome everyone to discuss and share the ideas in this article. We hope more partners will join us in exploring the world of cognitive acoustics and investigating how AI agents can understand and anticipate.

A very limited number of speaking opportunities and exhibition booths are still available. For inquiries, contact Xiaole at +86 177 0406 1087 (WeChat: same number).