JD Open Sources Long Audio and Video Generation Framework JoyAI-Echo

World's First! JD Fully Open Sources JoyAI-VL-Interaction, Moving Large Models from "Q&A" to "Watch and Speak"

The moment a fire breaks out, the monitoring system can issue an alarm in real time; when an elderly person living alone falls at home, AI can immediately alert distant relatives; when visually impaired people go out, smart glasses can interpret the nearby environment and point the way at any time... These seemingly sci-fi scenarios may soon become reality in the AI era.

Recently, JD open-sourced the real-time video visual language interaction model JoyAI-VL-Interaction. This is also the world's first fully open-source interaction model and system, receiving day-0 native support from vLLM-Omni. It moves large models from "Q&A" to "watch and speak." Based on this framework, developers can quickly build real-scene AI assistants capable of continuous observation, autonomous judgment, and immediate response, expected to promote AI's significant role in the physical world and bring new transformations to human production and life.

Code

🔗https://github.com/jd-opensource/JoyAI-VL-Interaction

Model

🔗https://huggingface.co/jdopensource/JoyAI-VL-Interaction-Preview

Dataset

🔗https://huggingface.co/datasets/jdopensource/JoyAI-VL-Interaction

Not Only Understanding the Past, But Also Understanding the "Present"

Today, many multimodal models focus on competing in parameters, knowledge, and reasoning. Essentially, they are still "Q&A," meaning users upload images or videos, ask questions, and the model provides answers. This method is sufficient for image-text Q&A, video review, and content analysis. However, when AI enters the real world, the model must not only be smart but also "present." In the real world happening right now, countless moments change in an instant, and missing them is hard to remedy.

The JoyAI-VL-Interaction open-sourced by JD allows AI to be continuously "present" like humans: watching, remembering, and judging simultaneously, and actively responding at critical moments, or selectively handing over to background Agents. Compared to traditional models, JoyAI-VL-Interaction has three breakthroughs:

1. Active judgment, rather than passive answering. Traditional models usually wait for users to initiate questions before processing the current screen, while JoyAI-VL-Interaction can continuously observe video streams and autonomously judge when to speak and when to remain silent.

For example, if a user sets "Remind me when the referee shows a red card," the model will continuously monitor the screen and automatically alert when the event occurs, rather than waiting for the user to ask again "What just happened."

2. Real-time response, rather than post-event summary. Traditional video understanding mostly involves analyzing after uploading a complete video. However, in scenarios like security alerts, real-time translation, live commentary, and operation guidance, a few seconds late makes a difference in experience and value. JoyAI-VL-Interaction targets ongoing video streams and can respond when the screen changes.

3. Timely Agent delegation while maintaining observation and interaction. JoyAI-VL-Interaction also possesses background task delegation capabilities and related mechanisms. When the model encounters tasks such as generating code, calling tools, or complex reasoning, it can hand them over to background large models or Agents. The foreground model continues to observe the scene, while the background model processes complex tasks. After the results return, the conversation is naturally resumed. It is more like a collaborative system of "foreground real-time assistant + background intelligent brain": the foreground is responsible for being present, the background for doing heavy work, potentially opening a new paradigm of AI-human collaboration.

In real-time video streams, JoyAI-VL-Interaction makes a judgment every second, such as continuing to observe, remaining silent, detecting key events, actively responding, or handing over complex tasks to background Agents for processing. This means "when to speak" is no longer reliant solely on external rules or timed triggers, but becomes a capability the model learns itself.

For real-time interaction, knowing how to speak is important, but knowing how to remain silent is equally important. A good AI assistant should not disturb the user constantly, but should know when to appear, when to stay quiet, when to solve problems itself, and when to hand over to an agent.

Many open-source models only provide basic reasoning capabilities. If developers want to truly use them, they still need to handle engineering issues such as video access, voice interaction, memory modules, and front-back end coordination themselves. JoyAI-VL-Interaction open-sources a complete technology stack, including model weights, interaction datasets, training solutions, and a fully deployable system, helping developers move faster from model research to real-scene implementation.

JoyAI-VL-Interaction supports various video inputs such as cameras, live streams, and monitoring streams, and also supports voice input/output, visual interfaces, long-term memory, background model interfaces, and vLLM deployment solutions. ASR, TTS, visual interfaces, background models, external tools, and business modules can be replaced as needed. Developers can integrate their own voice services, Agents, APIs, business systems, or front-end interfaces.

In other words, JoyAI-VL-Interaction is not a closed product, but an open framework. It can be used for research, or adapted into real-time AI assistants for security monitoring, elderly and child care, live commentary, e-commerce shopping guidance, operation guidance, AI glasses, accessibility assistance, etc.

In evaluations, JoyAI-VL-Interaction covers real streaming scenarios such as monitoring alerts, real-time counting, real-time translation, time perception, and live tour commentary. In these 58 human blind test cases highly related to visual-triggered active response and real-time performance, JoyAI-VL-Interaction achieved an overall win rate of 77.6% compared to Doubao Video Call Assistant, and 87.9% compared to Gemini Video Call Assistant. Especially in monitoring alert scenarios, it achieved a 100% win rate against both baselines. This stems from the natural advantage of interaction models over traditional "Q&A" turn-based models: autonomous interactivity is built into the model internally, rather than relying on external triggers.

From Generation to Interaction, AI Moves Towards the Physical World. Since the beginning of this year, JD has made several important progress in model infrastructure. In March, JD open-sourced the Instruct version of the basic large model JoyAI-LLM Flash, breaking the dilemma of large model parameter involution; in April, JD open-sourced the image model JoyAI-Image-Edit, reaching world-class levels in spatial understanding and editing capabilities. On June 3, JD again open-sourced the long video generation model JoyAI-Echo, promoting the arrival of the "what you think is what you get" era for long video generation, marking JD's entry into the global first tier in the above fields.

From "Q&A" to "watch, remember, and respond," from offline video understanding to real-time streaming interaction, from AI on screens to AI in the physical world—the full-stack open sourcing of JoyAI-VL-Interaction this time is another step for JD in pushing AI from the digital world to the physical world.

Having cultivated physical industries such as retail, logistics, health, and industry for over twenty years, JD possesses a globally leading physical world operation network, covering massive real scenarios including warehousing, delivery, stores, live streaming, customer service, and after-sales, where real-time interactions between people, goods, and scenes occur every day. For AI, these are not abstract data, but natural training grounds and application fields for entering the physical world, providing solid guarantees for JD to build the "world's largest physical world operation center." In the future, JD will continue to increase R&D investment, open up technical capabilities, and promote AI from thousands of industries into thousands of households.

JoyAI-VL-Interaction has received

vLLM-Omni's day-0 support🎉🎉🎉

It has been natively merged into the vLLM-Omni mainline

Everyone can launch the service with one click on vLLM-Omni to experience it

Or start it with one click in our repository~.

This page provides an editorial summary based on publicly available information. It is not a republished article. Use the source link below for the original report.