NVIDIA on June 16, 2026 released XR AI, an open-source library that connects AR glasses, AI glasses, and XR headsets to GPU-accelerated AI services. Now in public beta, it gives developers a reusable foundation for agents that can see what a wearer sees, understand spoken or typed intent, call enterprise tools, and respond inside the same XR session.
What Happened
NVIDIA published XR AI as a public beta with an open repository on GitHub. The library handles the plumbing that has made XR agents hard to build: routing live camera frames and microphone audio to AI models, grounding responses in what the user is looking at, and returning spatial content to the headset. NVIDIA framed it as a foundation for intelligent XR agents across field service, remote assistance, manufacturing, healthcare, and training, and paired the developer release with a companion overview of the hands-free use cases.
Why It Matters
Until now, wiring a multimodal model into a headset meant building the camera, audio, and rendering pipeline yourself. XR AI ships that stack as reusable parts, so a small team can prototype a glasses agent in days instead of months. It leans on open models rather than a closed API: visual grounding runs on NVIDIA's Cosmos-Reason1-7B vision-language model, which means the perception layer is inspectable and swappable. For creators and studios experimenting with spatial computing, that lowers the cost of trying real ideas, and it puts the same vision and speech models used in cloud agents into a wearable form factor.
Key Details
XR AI is built from named, public components. An XR Media Hub routes camera and mic streams, Cosmos models handle visual grounding, Nemotron models handle language and tool calling, Model Context Protocol servers connect enterprise data, and the NVIDIA NeMo Agent Toolkit orchestrates the agent. The default model stack pairs Parakeet-TDT-0.6B-V3 for speech-to-text with Cosmos-Reason1-7B for vision reasoning and Llama-3.1-Nemotron-Nano-8B-V1 and Nemotron-3-Nano-30B for language, while NVIDIA CloudXR delivers rendered spatial content. Because every model in the default stack is published openly, teams can audit the pipeline and substitute their own weights at any stage.
What to Do Next
If you build for headsets, or just want to see how a modern XR agent is assembled, clone the beta and read the XR AI documentation to wire up the media hub and model stack. Start with the speech-to-text and vision-grounding sample before adding tool calls, since those two pieces define whether the agent feels responsive. Keep the model choices modular so you can swap in lighter or heavier Nemotron variants as your latency budget demands.
