The Next Leap for Autonomous Robots

Zero-shot robot navigation is at the heart of the latest advances in autonomous robotics. Open Scene Graphs for Open-World Object-Goal Navigation introduces a revolutionary approach that enables robots to enter environments they’ve never seen before, receive verbal commands like “find fever medicine” in a pharmacy, and successfully locate their targets—all without any prior map or training in that specific space.

Imagine a robot entering a pharmacy it’s never visited before, receiving a verbal command to “find fever medicine,” and successfully locating it—all without any prior map. This isn’t science fiction: it’s the ambition behind the research paper Open Scene Graphs for Open-World Object-Goal Navigation introduces a revolutionary approach that enables robots to enter environments they’ve never seen before, receive verbal commands like “find fever medicine” in a pharmacy, and successfully locate their targets—all without any prior map or training in that specific space.

Traditional robotics required exhaustive environmental mapping and hand-coded rules. But in the real world, environments are unpredictable and diverse. This paper proposes a modular system called OSG Navigator, capable of generalizing across countless indoor environments, tasks, and even robot types—all driven by the power of foundation models and a groundbreaking spatial memory structure: the Open Scene Graph (OSG).

The Core Challenge: Open-World Semantic Navigation

Robots are increasingly expected to perform “ObjectNav”—searching for a target object specified in natural language, within novel spaces. This task in an “open-world” setting is very challenging due to:

• Diverse Goals: Robots must handle any natural language object description, even for objects never seen in training.
• Diverse Environments: Robots should work in homes, offices, supermarkets, and more—all without custom engineering.
• Diverse Embodiments: The system should function for both wheeled and legged robots.

The Solution: OSG Navigator & Open Scene Graphs

What Is OSG Navigator?

OSG Navigator is a modular navigation system built from several types of “foundation models,” each excelling in different areas:

• Large Language Models (LLMs): For semantic reasoning and task planning.
• Vision Foundation Models (VFMs): For recognizing objects and regions within camera feeds (e.g., GroundingDINO, BLIP-2).
• General Navigation Models (GNMs): For low-level robot control (e.g., ViNT).

These components create a neuro-symbolic architecture for “ObjectNav” in dynamic and unknown environments.

The Open Scene Graph (OSG): Memory for Robots

At the heart of the system is the Open Scene Graph (OSG), a hierarchically organized spatial memory that encodes:

• Objects, places, connectors (like doors/entrances), and higher-level region abstractions (e.g., rooms, corridors, floors).
• Relations such as “connects to,” “is near,” “contains,” enabling efficient reasoning about space.

Key innovation:

• OSG Schemas are templates describing general structure for classes of environments (home, supermarket, etc.).
• These schemas can be automatically generated by prompting an LLM with simple environment labels—enabling zero-shot adaptation to new spaces!

How Does OSG Navigator Work?

System Pipeline

1. Image Parsing: VFMs detect and describe objects and places from the robot’s RGB images.
2. Mapping and Localization: An incremental mapping routine constructs the OSG as the robot navigates, using the OSG schema for that environment type.
3. Semantic Reasoning: LLMs reason over the OSG to select promising subgoals and generate navigation plans.
4. Low-level Control: GNMs use visual goals from the OSG to generate control commands—driving the robot in real time toward targets.

Robust Generalization

Unlike systems that only work in “known” scenes, OSG Navigator:

• Builds scene graphs on the fly in unfamiliar places.
• Works for any object described in natural language.
• Supports varied robot hardware with a common high-level intelligence.

Key Experimental Results for Zero-Shot Robot Navigation

• Tested on Fetch and Boston Dynamics Spot robots, both in simulation and real-world environments.
• State-of-the-art performance on standard ObjectNav benchmarks.
• Zero-shot generalization to new goals, spaces, and robot embodiments.
• Detailed analyses show the OSG structure is critical for this success.

Why Is This a Breakthrough?

• Structured semantic memory (OSG) unlocks the true potential of foundation models in robotics.
• Navigating by semantics and relational maps, not just geometry, leads to flexibility and adaptability previously unseen.
• The schema-driven approach supports fast onboarding to new spaces—just tell the robot “this is a supermarket” or “this is a hospital,” and it will create a new OSG structure instantly.
• Empirical lessons reveal that foundation models, especially LLMs, work best when grounded in structured memory (like OSG), rather than raw sensor feeds alone.

Current Limitations and Future Directions

• OSG Navigator, so far, is evaluated only in indoor navigation—outdoor environments remain a challenge.
• The system currently uses a topo-semantic approach, not embedding full metric geometry (distances/shapes).
• Integrating multimodal, multilingual, or edge-optimized/small models could further boost versatility.
• Adding explicit metric geometry to OSGs promises even more robust navigation.

Open Scene Graphs for Open-World Object-Goal Navigation mark a milestone toward universally adaptable, semantically aware robot intelligence. With OSG Navigator, foundation models are no longer limited by their lack of spatial memory—robots can reason, plan, and act in any environment, for any goal.Open Scene Graphs for Open-World Object-Goal Navigation mark a milestone toward universally adaptable, semantically aware robot intelligence. With OSG Navigator, foundation models are no longer limited by their lack of spatial memory—robots can reason, plan, and act in any environment, for any goal.ReferenceReferenceReference

Reference

This article is based on: Loo, J., Wu, Z., & Hsu, D. (2024). “Open Scene Graphs for Open-World Object-Goal Navigation.” The International Journal of Robotics Research. arXiv:2508.04678v1 [cs.RO] https://arxiv.org/abs/2508.04678