A5.2 - Building Bricks Integration

Once the XR Platform Integrated and the XR project created and assessed, then the Building Bricks of the application i.e. the selected manufacturing use case, should be integrated. These Building Bricks include: A1 - Scene Creation, A2 - Simulation, A3 - Interaction and A4 - UI Creation. This activity is technically extensive in terms of the need to ensure the successful import of functional modules developed separately, using another development platforms, independent of the Unity development environment. The import process usually requires adjustments, necessary to meet the specific requirements of the Unity project, as well as adjustments to match the function of the imported Building Bricks.

In order to provide the necessary high level of geometric accuracy / fidelity of the manufacturing system CAD model, a dedicated 3D modeller should be used such as SolidWorks or similar CAD modellers optimized for engineering design use. This raises a question of exporting CAD generated models into a Unity project. There are several approaches, but probably the most convenient one is based on the use of so called GL Transmission Format - glTF. The glTF is an open and royalty-free file format for 3D models designed to be lightweight, efficient, and optimized for real-time rendering across different platforms and applications. It is developed and maintained by the Khronos Group, a consortium of technology companies focused on creating open standards. Here are some key features and characteristics of the glTF file format:

1. Structure and Content: glTF files are structured using JSON (JavaScript Object Notation), a human-readable data format, making it easy to understand and parse. They typically consist of a combination of binary data (geometry, animations, etc.) and textual data (metadata, scene hierarchy, etc.).
2. Geometry and Materials: glTF supports various geometry types, including triangle meshes and lines, allowing for the representation of complex 3D shapes. It also supports materials and textures, allowing for the application of surface appearance and visual details to the models.
3. Animation and Rigging: glTF can store skeletal animations, allowing for the representation of articulated and animated models (such as kinematic chains of robots, kinematic chains of machine tools, or the XR model of the human body which is usually expressed in the form of a rigid skeletal structure whose kinematics is actually a branched kinematic chain composed of parallelized serial sequences representing human limbs, attached to the central serial articular substructure of human body and head). It supports keyframe animations, skeletal animations based on joints and links, or joints and bones, and morph target animations for shape interpolation.
4. PBR Materials: glTF provides support for Physically-Based Rendering (PBR) materials. PBR allows for the creation of realistic material properties, including parameters such as base color, metallic, roughness, emissive, and more. This enables consistent rendering across different platforms and engines.
5. Compression and Optimization: glTF files are designed to be compact and efficient. They can be compressed using industry-standard compression techniques like gzip or Draco, reducing file size while preserving visual quality. Additionally, glTF supports techniques such as Level of Detail (LOD) and binary buffer formats to optimize rendering performance.
6. Extensibility: The glTF format is extensible, allowing for additional features and custom data to be incorporated into the files. This extensibility enables the inclusion of specific metadata, custom attributes, or application-specific properties within the glTF file.
7. Interoperability: glTF promotes interoperability among different software applications and platforms. It is supported by a wide range of 3D modeling and rendering software, game engines, and VR/AR platforms. This compatibility facilitates the exchange and sharing of 3D assets across different tools and environments. The glTF file format aims to simplify the process of creating, sharing, and rendering 3D content, making it an ideal choice for real-time applications like VR and AR. Its lightweight nature, support for PBR materials, animations, and extensibility make it a versatile and widely adopted format in the 3D modelling industry. The execution of this activity requires the implementation of the following high-level workflow:

A5.2.1 Create or Import Assets – the application Building Blocks:

• a. or A5.2.1.a - Create 3D Models: Import, upgrade and adjust 3D geometric and kinematic models, textures, and other visual assets you want to use in your VR application. Ensure they are compatible with Unity’s supported file formats (assumed is that the CAD model is prepared to be exported in glTF wich requires its own set of subactivities of the lower order, but this is the part of the A1 Building Block).
• b. Interaction Assets: If your VR application requires hand controllers or other input devices, ensure you have the necessary assets or import the corresponding packages.
• c. Audio Assets (if used): Prepare or import audio files for sound effects, background music, and spatial audio if required.

A5.2.2 Set Up the Scene:

• a. Create a new scene in Unity or modify an existing one.
• b. Set up the camera: Add a VR camera rig or modify the existing camera to support VR rendering. Configure camera settings for proper stereoscopic rendering and head tracking.
• c. Set up lighting: Configure lighting in the scene to ensure a visually appealing and immersive environment.

A5.2.3 Implement VR Interactions:

• a. Input Handling: Utilize the VR SDK’s input system to handle user interactions such as head movement, hand gestures, and button presses.
• b. Physics and Collision: Implement collision detection and physics interactions between objects in the virtual environment, allowing users to interact with the world realistically.
• c. User Interface: Create and optimize the UI elements, including menus, buttons, and tooltips, for VR interaction. Ensure they are easy to read and interact with in a VR environment.

A5.2.4 Optimize Performance:

• a. VR Rendering: Adjust graphics settings to optimize performance in VR. Use techniques such as level of detail (LOD) rendering, occlusion culling, and efficient shaders to maintain a smooth and immersive experience.
• b. Frame Rate: Optimize the application to maintain a consistent and high frame rate (e.g., 90 FPS) to minimize motion sickness and ensure smooth head tracking.

A5.2.5 Test and Debug:

• a. Test on Target Devices: Connect your VR headset to your development machine and test your application directly on the target VR device to ensure everything functions as intended.
• b. Debug: Use Unity’s debugging tools, such as the Console and Profiler, to identify and fix any issues or errors in your VR application.

A5.2.6 Deploy and Distribute:

• a. Build the Application: Generate a build of your VR application for the specific VR platform you are targeting (e.g., Windows, Android). Unity will package the necessary files and dependencies into a standalone executable or APK file.
• b. Publish or Distribute: Depending on your goals, publish your VR application on appropriate platforms or distribute it to users through other means, including exchange between the student groups from different partner universities. Another important issue is ensuring real-time bidirectional interaction between the CAD modeller and the Unity project. This feature is crucial to enable users to experiment and modify the model in an engineering context. Assuming the CAD model is imported successfully into Unity, real-time bidirectional interaction could be achieved by implementing the dedicated interfacing mechanism (a software routines codded in C#) embedded in the CAD modeller and in Unity. The following workflow outlines the steps to achieve this kind of interaction. This workflow should be treated as a subworkflow of an additional activity designated as A5.2.1.d:
  1. Continuous Monitoring for Updates: Set up a monitoring mechanism in Unity that continuously checks for updates to the CAD model file. This can be achieved using file system watchers or through network-based communication.
  2. Update the Unity Scene: When changes are detected in the CAD model file, trigger a process to update the corresponding GameObjects in the Unity scene. This process can involve applying changes to an existing model or even removing an existing model and re-importing an updated one.
  3. Coordinate Systems Alignment: Ensure that the coordinate systems between the CAD modeller and Unity match to ensure accurate positioning and alignment of the imported model in the Unity scene.
  4. Data Synchronization: For real-time bidirectional interaction, establish a data synchronization mechanism between the CAD modeller and Unity. This can involve exchanging data through files, a local network, or cloud-based services.
  5. User Interaction and Edits: Implement user interaction in Unity to manipulate the imported CAD model. For example, users could interact with the model’s components, change materials, or adjust its parameters.
  6. Feedback to CAD Modeller: If the CAD modeller supports it, provide feedback from the Unity project to the CAD modeller. For instance, if users modify the model in Unity, send information back to the CAD modeller to update the original file.

Input

Integrated XR Platform and XR Project created, Executable Building Bricks A1 to A4.

Output

Fully functional XR System with integrated Building Bricks A1 to A4, XR scene created, XR interactions created, XR System tested, debagged and optimized.

Control

Detailed assessment of the integrated XR Building Bricks by checking its functional performances, debugging, and additional settings or providing necessary corrections in imported models or other components of the Basic Building Blocks A1 to A4.

Resource

Building Bricks ready to be imported, SW development tools for Bricks installation, integration and interlinking.