Digital Twin meets Apple Vision Pro in AR/VR

The core objective was clear: to develop virtual twin scenarios using Apple’s hardware and software without relying on external platforms such as NVIDIA’s Omniverse or Unreal Engine. Instead, we wanted to take full advantage of Apple’s own features and functionality within Vision Pro, pushing the boundaries of what’s possible without reinventing the wheel.

Key development challenges

1. Mastering robotics and motion dynamics

As non-3D or game developers, our team had to delve deeply into the fundamentals of robotics and motion:

• Understanding the complexity of joint movements and mathematical constraints.
• Translating these physical dynamics into Apple’s AR/VR/MR environments.

2. Navigating the new visionOS 2 framework

Working with the newly released visionOS 2 beta presented its own set of challenges:

• Limited documentation meant our team had to rely on trial and innovation.
• Building a truly immersive AR experience tailored for Apple devices.

3. Dual approach to motion control

We pursued two advanced approaches for robot joint control:

• Inverse kinematics: Leveraging undocumented visionOS 2 features, we were at the forefront of experimenting with advanced functionality in an evolving API.
• Individual joint control: This required precision in manipulating the rotation of each joint in AR space – a complex task requiring a strong understanding of linear algebra. Blender, an open-source 3D creation software, was essential to rework initial models to meet these precise requirements.

4. Integrating the ‘Real World’

Integrating IoT data and synchronizing interactions between virtual and physical environments posed unique challenges:

• Ingesting and visualizing IoT data: How to ingest real-time data from the robot (physical twin) via Microsoft Azure IoT and visualize it on the Apple Vision Pro.
• Triggering SAP business functions: How to activate business functions, such as plant maintenance notifications, directly within the SAP system.
• Synchronizing digital and physical twins: How to synchronize movements of the digital twin, initiated by gestures in VR, with the physical twin to ensure it mirrors virtual commands.
• Multi-device communication: How to facilitate communication between multiple Apple Vision Pros interacting with a single digital twin, managing both control and receive modes.

Our team developed two primary scenarios that showcase the powerful capabilities of Apple Vision Pro in synchronizing digital and physical assets:

Live control of robot arm (physical twin) with moving digital twin in virtual space

Utilizing the Apple Vision Pro, users can interact directly with a digital twin of a robotic arm by simply pinching and moving the model in virtual space. These digital movements are mirrored precisely by the physical robotic arm, allowing for seamless and responsive control of the real-world asset from a virtual environment.

• Plant maintenance notification integration with SAP

Through our Event-2-Business Framework on the SAP Business Technology Platform (BTP), users can create Plant Maintenance Notifications in their on-premises SAP system. This integration facilitates immediate and efficient response to maintenance needs, streamlining operations and ensuring rapid resolution of potential issues.

• Decision-making based on live telemetry data

Informed decisions are supported by live telemetry data transmitted from Microsoft Azure IoT. Key metrics such as temperature and energy consumption are displayed graphically, providing real-time insights that enhance operational efficiency and enable proactive management.

Working with digital twin across several Apple Vision Pros

In this configuration, Apple Vision Pro #1 serves as the primary controller, while Vision Pro #2 through #99 function as receivers. Through Vision Pro #1, users can actively manipulate the digital twin, with joint coordinates being transmitted to the MQTT Broker. These movements are then mirrored across the additional Apple Vision Pro devices by receiving the corresponding coordinates and joint information.

Moreover, all features available in the initial scenario can be configured in this setup as well, providing a fully synchronized, collaborative virtual environment.

Optimizing and expanding scenarios

As we move forward, our goal is to refine current scenarios and explore new functionalities, such as voice-controlled movement of digital and physical twins. By doing so, we aim to expand the potential of AR/VR applications for real-world business use.

We have already engaged in discussions with our customers and partners on integrating production machinery, production lines, and automated guided vehicle (AGV) systems. These conversations have also explored the establishment of scenarios for shop floor operations and simulations. Furthermore, AR/VR can significantly facilitate the training of new employees. In the design and construction sector, we are evaluating how AR/VR can enhance the design and build phases, particularly in improving interactions with clients.

Building a framework for flexible management of scenarios

In our current iteration, we have developed a separate application for each digital twin scenario. We plan to build a comprehensive framework to manage various types of physical and digital twins (such as robots, machines, and production lines), along with a central object repository for VR model data with low latency, powered by the Microsoft Azure Platform. All IoT traffic will be managed within an IoT Hub, featuring SAP integration. This system will be operated using the Apple Vision Pro via a master app with a harmonized user interface, as well as unified dashboards and control planes. Our goal is to industrialize and streamline the onboarding process.

Embedded AI

Simulations are essential for optimizing the routing of robots or AGVs. Changes to the shop floor layout, the relocation of machinery, or new constructions make it necessary to simulate these adjustments before implementing any physical changes. We aim to support this through the integration of AI capabilities.

When using the Apple Vision Pro for service and maintenance, you often need additional information, such as zooming in to view deeper layers of a model or accessing detailed instructions from a manual. Generative AI enhances this process by facilitating communication and retrieving information efficiently through vectorization and smart querying.

As AR/VR technology continues to evolve, NTT DATA Business Solutions remains committed to driving innovation and real-world impact. We invite you to join us on this journey, transforming how businesses operate through groundbreaking digital twin integrations and immersive experiences.

Learn more about our Innovation & Technology Services