In this article, you will learn how NTT DATA Business Solutions advances AR/VR technologies for real-world applications, leveraging the powerful capabilities of digital twins integrated with Apple’s Vision Pro. You will gain insights into our innovative approach to overcoming development challenges, such as mastering motion dynamics and integrating IoT data, while synchronizing virtual and physical environments. The article further illustrates practical scenarios, such as real-time control of robotic arms and SAP system integration, and outlines our forward-thinking strategy for expanding the use of AR/VR in business contexts.
Digital twin meets Apple Vision Pro - AR/VR solution from NTT DATA Business Solutions
From vision to reality: Practical AR/VR innovations
NTT DATA Business Solutions has a longstanding commitment to advancing augmented reality (AR) and virtual reality (VR) solutions, innovating for real-world applications. Since 2018, we have pioneered AR/VR projects, including AR-enabled data center visualization on iPad with SAP integration, machine recognition through image analysis with live dashboards, and using drones to identify invasive plant species.
Recognizing the potential of Apple’s ecosystem early on, we spent the last year exploring metaverse scenarios and facilitating virtual workshops, training, and onboarding. However, one significant hurdle remained: live, real-world integration.
With the introduction of Apple’s Vision Pro, we saw an opportunity to revisit and enhance our existing AR/VR frameworks by leveraging Vision Pro’s powerful immersive capabilities. This initiative aimed to bridge digital twins – virtual representations of physical assets – with real-time interaction through Apple Vision Pro and key business applications such as SAP and Microsoft Azure IoT.
The mission: Digital twin scenarios on Apple Vision Pro
The core objective was clear: to develop virtual twin scenarios using Apple’s hardware and software without relying on external platforms such as NVIDIA’s Omniverse or Unreal Engine. Instead, we wanted to take full advantage of Apple’s own features and functionality within Vision Pro, pushing the boundaries of what’s possible without reinventing the wheel.
Key development challenges
1. Mastering robotics and motion dynamics
As non-3D or game developers, our team had to delve deeply into the fundamentals of robotics and motion:
- Understanding the complexity of joint movements and mathematical constraints.
- Translating these physical dynamics into Apple’s AR/VR/MR environments.
2. Navigating the new visionOS 2 framework
Working with the newly released visionOS 2 beta presented its own set of challenges:
- Limited documentation meant our team had to rely on trial and innovation.
- Building a truly immersive AR experience tailored for Apple devices.
3. Dual approach to motion control
We pursued two advanced approaches for robot joint control:
- Inverse kinematics: Leveraging undocumented visionOS 2 features, we were at the forefront of experimenting with advanced functionality in an evolving API.
- Individual joint control: This required precision in manipulating the rotation of each joint in AR space – a complex task requiring a strong understanding of linear algebra. Blender, an open-source 3D creation software, was essential to rework initial models to meet these precise requirements.
4. Integrating the ‘Real World’
Integrating IoT data and synchronizing interactions between virtual and physical environments posed unique challenges:
- Ingesting and visualizing IoT data: How to ingest real-time data from the robot (physical twin) via Microsoft Azure IoT and visualize it on the Apple Vision Pro.
- Triggering SAP business functions: How to activate business functions, such as plant maintenance notifications, directly within the SAP system.
- Synchronizing digital and physical twins: How to synchronize movements of the digital twin, initiated by gestures in VR, with the physical twin to ensure it mirrors virtual commands.
- Multi-device communication: How to facilitate communication between multiple Apple Vision Pros interacting with a single digital twin, managing both control and receive modes.
Real-world application scenarios
Our team developed two primary scenarios that showcase the powerful capabilities of Apple Vision Pro in synchronizing digital and physical assets:
Live control of robot arm (physical twin) with moving digital twin in virtual space
Utilizing the Apple Vision Pro, users can interact directly with a digital twin of a robotic arm by simply pinching and moving the model in virtual space. These digital movements are mirrored precisely by the physical robotic arm, allowing for seamless and responsive control of the real-world asset from a virtual environment.
-
Plant maintenance notification integration with SAP
Through our Event-2-Business Framework on the SAP Business Technology Platform (BTP), users can create Plant Maintenance Notifications in their on-premises SAP system. This integration facilitates immediate and efficient response to maintenance needs, streamlining operations and ensuring rapid resolution of potential issues.
-
Decision-making based on live telemetry data
Informed decisions are supported by live telemetry data transmitted from Microsoft Azure IoT. Key metrics such as temperature and energy consumption are displayed graphically, providing real-time insights that enhance operational efficiency and enable proactive management.
Working with digital twin across several Apple Vision Pros
In this configuration, Apple Vision Pro #1 serves as the primary controller, while Vision Pro #2 through #99 function as receivers. Through Vision Pro #1, users can actively manipulate the digital twin, with joint coordinates being transmitted to the MQTT Broker. These movements are then mirrored across the additional Apple Vision Pro devices by receiving the corresponding coordinates and joint information.
Moreover, all features available in the initial scenario can be configured in this setup as well, providing a fully synchronized, collaborative virtual environment.
The road ahead
Optimizing and expanding scenarios
As we move forward, our goal is to refine current scenarios and explore new functionalities, such as voice-controlled movement of digital and physical twins. By doing so, we aim to expand the potential of AR/VR applications for real-world business use.
We have already engaged in discussions with our customers and partners on integrating production machinery, production lines, and automated guided vehicle (AGV) systems. These conversations have also explored the establishment of scenarios for shop floor operations and simulations. Furthermore, AR/VR can significantly facilitate the training of new employees. In the design and construction sector, we are evaluating how AR/VR can enhance the design and build phases, particularly in improving interactions with clients.
Building a framework for flexible management of scenarios
In our current iteration, we have developed a separate application for each digital twin scenario. We plan to build a comprehensive framework to manage various types of physical and digital twins (such as robots, machines, and production lines), along with a central object repository for VR model data with low latency, powered by the Microsoft Azure Platform. All IoT traffic will be managed within an IoT Hub, featuring SAP integration. This system will be operated using the Apple Vision Pro via a master app with a harmonized user interface, as well as unified dashboards and control planes. Our goal is to industrialize and streamline the onboarding process.
Embedded AI
Simulations are essential for optimizing the routing of robots or AGVs. Changes to the shop floor layout, the relocation of machinery, or new constructions make it necessary to simulate these adjustments before implementing any physical changes. We aim to support this through the integration of AI capabilities.
When using the Apple Vision Pro for service and maintenance, you often need additional information, such as zooming in to view deeper layers of a model or accessing detailed instructions from a manual. Generative AI enhances this process by facilitating communication and retrieving information efficiently through vectorization and smart querying.
As AR/VR technology continues to evolve, NTT DATA Business Solutions remains committed to driving innovation and real-world impact. We invite you to join us on this journey, transforming how businesses operate through groundbreaking digital twin integrations and immersive experiences.
Read more blog articles related to innovation
Maria Eisner Pelch
Dec 04, 2024
Wolfgang Möller
Nov 21, 2024
NTT DATA Business Solutions
Oct 31, 2024
Jonas Winther Kvist
Oct 15, 2024
Andy Steer
Oct 07, 2024
Steve Niesman
Aug 21, 2024