Joint Learning Lab 2022 - JLL 2022

The Second Joint Learning Lab - JLL 2022, was held at the University of Belgrade. It was hosted by the Faculty of Mechanical Engineering (UBelg FME), Production Engineering Department, Laboratory for CyberManufacturing Systems (CMSysLab), in a RTDI environment well aligned with Industry 4.0 and Smart Manufacturing. The event took place from Monday, May 9 to Friday, May 13, 2022.

The JLL 2022 programme can be downloaded from this link in pdf format. Link

JLL 2022 brought together 23 students from three partner universities and three countries (5 from France, 12 from Italy and 6 from Serbia) accompained with 10 professors and instructors (2 from France, 5 from Italy and 3 from Serbia). Students are divided into 3 mixed project teams, with an even distribution by country in order to maximize the international character of the interactions (following the model explained in the Student teams section of chapter organisation). The composition of students project teams wass as follows:

Student Project Team #1

Student Project Team #2

Student Project Team #3

Welcoming and hosting

The first working activity of the JLL 2022 was a welcome session, held in the Ceremonial Hall of the Faculty of Mechanical Engineering. All students, professors and instructors were warmly welcomed by professor Marko Miloš, Vice-dean for teaching at the Faculty of Mechanical Engineering, and professor Bojan Babić, head of the Department of Mechanical Engineering. Professor Frederic Noel also addressed the audience with a short presentation of the AVATAR project.

The session continued with an extensive presentation, delivered by professor Božica Bojović, the purpose of which was to bring guests more familiar with Serbia, its capital Belgrade, the University of Belgrade (leading state university with the longest history spanning over 2 centuries, comprehensive in its structure, and with more than 100,000 students enrolled at all three levels of higher education studies), as well as at the host institution - the Faculty of Mechanical Engineering (4500 students at three levels of engineering studies - bachelor, master and doctoral, 25 departments, 22 modules of specializations at master level), all through statistical, historical and cultural facts. This presentation concludes the welcome session.

Further student activities continued in CMSysLab, where professor Petar B. Petrović, head of the CMSysLab, presented general information about the laboratory, its fields of research and teaching, key RTDI programmes, national and international research projects, and cooperation activities with industry in manufacturing innovation. Professor Petrović also presented in detail the key laboratory equipment, in particular robot workstations, XR technology and manufacturing mechatronics.

Finally, before the lunch break, the students were introduced to the entire organization of JLL 2022, followed by the accommodation of all guests (students and teachers) in the space dedicated to this event.

Overall organisation of the joint learning lab

The task set for the students is a complex engineering challenge, requiring a high level of knowledge spanning several engineering disciplines, and equally important, requiring multidisciplinary engineering skills in the use of various software tools and techniques, especially for computational modeling, design and programming.

In order to reduce the complexity and physical volume of work that the project teams have to do in order to fulfill the assigned task in a very limited time frame (5 working days!), a customized didactic approach was used. The essence of this approach is that every student activity is carefully freed from routine and unproductive work, while only those activities are retained that are creative, closely related to the thematic framework of the AVATAR project and converge to the solution through the shortest technically feasible pathway. An approach that satisfies Occam’s razor principle in a very rigorous sense. Specifically, this approach included:

1. Extensive preparatory activities:

   a) preparatory activities for students for partner universities from France and Italy in their local environments, supported by local teachers and, particularly important, extensive interaction with students from the University of Belgrade as JLL 2022 host institution;
   b) students from the University of Belgrade are intensively trained locally, well before JLL 2022, to reach such a level of knowledge and skills that will enable them to simultaneously play the role of student tutoring and thus help visiting students to bridge the gap and quickly familiarize themselves with the unknown laboratory environment and equipment;
   c) additionally, interested students from the previous AVATAR generation are engaged as tutors to share their more matured experience and knowledge.
   

2. Preparatory activities at the level of laboratory hardware - the experimental / educational RWC-DT system is physically integrated to a degree of completion that allows student teams to:

   a) easily understand the existing system,
   b) easily recognise and understand a missing content, and
   c) easily devise a solution for the missing content and integrates it into the existing system basedon their own knowledge (dominantly acquired prior the JLL 2022) and by their own hands-on work,and thus achieve the required functionalities or behaviors at the level of the subsystems or the entire RWC-DT system.
   

3. Dual time use - three project teams work in an intertwined mode, so that:

   a) one student team works in the laboratory space and conducts experimental activities on the RWC-DT physical hardware, while
   b) two student teams work in parallel in a computer classroom equipped with all the necessary software tools and prepare themself for experimental activities in the laboratory space and additionally, work on writing project reports in parallel.
   

4. Extensive tutoring support during the JLL 2022 in the laboratory and in computer classroom, with available carefully selected teaching materials.

5. Strong motivational component – form the begining the work activities of student teams are methodologically directed towards the so-called Game-based Learning, thus turning very demanding and dense engineering work into a game that motivates students to engage emotionally and intellectually to their limits. The problem of technical complexity was the biggest didactic challenge of the University of Belgrade project team. The implementation of the mentioned approach required a very extensive preparation of a team of local teachers and students, a detailed plan and a perfect organization.

Student challenge

The primary educational objective set as an engineering challenge to the three student teams at JLL 2022 was to:

• INTEGRATE hardware and software components,
• bring them to a state of FULL FUNCTIONALITY, and
• DEMONSTRATE operation of a given manufacturing digital twin system empowered with virtual reality technology.

In this particular case it was a Robotic Work-Cell Digital Twin (RWC-DT) with embedded virtual reality system dedicated to provide human robot interaction functionality.

In parallel, an additional objective was set to the student teams was to experience the practical implementation of the concept of robot Programming by Demonstration (PbD) using different variants of human-robot interfaces based on Virtual Reality and Digital Twin technology. This secondary objective is a technological objective, which is undoubtedly highly relevant to the domain of Industry 4.0 and the next generation manufacturing. From a pedagogical perspective, a secondary goal gives students a sense of practical purpose for their efforts and offers a deeper understanding of the role and value of XR technology in the next generation of manufacturing. From a didactic perspective, the secondary goal does not represent an additional burden because it is carefully integrated (embedded) into the aforementioned primary educational goal as an inherent component.

The third goal refers to the development of teamwork skills, communication skills, report writing skills and a specific sense of working in a culturally heterogeneous environment - a corpus of so called SOFT skills. Students will be asked to work as a team during that week and achieve the planned results by joint efforts and collective knowledge and skills;

Work organization

The combination of primary and secondary goals is divided into a set of sub-goals that represent a coherent set of topics that are ordered and organized in such a way that they are strictly aligned with AVATAR workflows. These topics, that student teams should accomplish within the JLL 2022, are as follows:

• Topic #1: DIGITAL MODEL: CAD-based XR SCENE CREATION
• Topic #2: DIGITAL MODEL: XR SCENE ANIMATION - RWC Kinematics Modeling
• Topic #3: DIGITAL MODEL: XR SCENE ANIMATION – HMD Installation and interactions
• Topic #4: DIGITAL TWIN: Step I - RWC DIGITAL SHADOW Integration
• Topic #5: DIGITAL TWIN: Step II - RWC DIGITAL TWIN Integration
• Topic #6: OPERATION: RWC DIGITAL TWIN - Robot PbD using kinesthetic haptic controller
• Topic #7: OPERATION: RWC DIGITAL TWIN - Robot PbD using teleoperation haptic controller
• Topic #8: OPERATION: RWC DIGITAL TWIN - Robot PbD using XR haptic controller

The above described agenda is decomposed into eight topic is embedded into the ten half days sessions as demonstrated in the following table.

for each of the three student groups, each of the eight half-day sessions dedicated to a work topic is organized in the following way:

• the half-day session starts with a theory reminder which lasts about 30 minutes and ends with a concise overview of expected work progress;
• preparatory work of student teams on computers in the computer classroom (separate from the laboratory space), with the support of a tutor and auxiliary teaching material, lasting 1 hour and 30 minutes;
• a description of security rules and instructions on hardware about 30 minutes;
• a hands-on work on robotic and control hardware, XR hardware and software, HMI hardware and software and DT hardware and software, tutored by local personnel, lasting 1 hour and 30 minutes.

The following photos show moments from the preparatory activities in CMSisLab, which preceded JLL 2022. Joint work of teachers, associates (doctoral students) and local group of students parcitipating the AVATAR spring semester focused on integration of three experimental building blocks. First, the hardware and software platform of the robotic system RWC-DT, which consists of two anthropomorphic robots. (YASKAWA SIA 10F redundant robot with 7 dof, specialized for robot assembly operations, and YASKAWA MA 1440 with DX100 controller, robot specialized for welding operations) equipped with open architecture control system, suitable for conducting complex experiments. Second, the computer hardware and software tools for integrating Digital Twin and XR system (SolidWorks CAD 3d modeler, UNITY XR modeler and HMD virtual reality hardware and software). And third, a game-based learning platform that includes a kinesthetic HMI and a teleoperation HMI system, open to be integratd with XR equipment for 3D visual perception and real-time human operator-robot interaction in a virtual space. As an example, the third photo shows a computer screen with a 3D graph representing the recorded robot motions, commanded by a human operator through one of the HMI modes of its interaction with the robot in physical and virtual space. This functionality is technically quite complex to implement and is only possible if the robot control system is equipped with an open architecture control module and a dedicated control software which was dedeveloped in Matlab exclusively for JLL 2022. Recorded data can be further used for quantitative analysis of student behavior in operation of the integrated RWC-DT system.

Topic #1: DIGITAL MODEL: CAD-based XR SCENE CREATION

From the PHYSICAL system, through the CAD model, and to the XRSCENE model

• CAD modelling sequence -Sketch, Part creation, Assembly, topological relations between geometric entities are PARAMETERIZED!
• Problem of INTEROPERATION between different modeling platforms
• CAD assembly model -.sldasm, must be converted into file the .glTFformat which can be imported in UNITY
• UNITY PROJECT creation, importing CAD model, adaptation of the CAD model, adding features …
• AVATAR approach is that we model the scene usingCAD engineering tool because we ned highest possible fidelity of the scene
• Final result is a HIGH FIDELITY STATIC XRSCENE MODEL of the RWC

Topic #2: DIGITAL MODEL: XR SCENE ANIMATION - RWC Kinematics Modeling

From the PHYSICAL kinematic system to the XRSCENE ANIMATION model

• Robot kinematics, robot joint and robot arm linkage / mechanism
• Robot motion demonstration using TEACH PENDANT (online programming method), single joint motion, robot arm motion
• Adding kinematic features to the XRrobot models and demonstration of robot joints motion in virtual space
• Inverse kinematic transform using RoboDKsimulation software, JOB TASK creation and TARGET POINT concept
• Integration of RoboDKin UNITY and demonstration of robot arm motion
• Linking CAD model with XRUNITY model using SW API SDK and UNITY scripts written in C#

Topic #3: DIGITAL MODEL: XR SCENE ANIMATION – HMD Installation and interactions

From the 2D SCREEN visualization to the XRSTEREO SCENE VISUAL PERCEPTION and INTERACTION

• Integration of the XRHMD system in UNITY project
• XRHMD system description (HTC Vive and OCULUS Rift S), hardware components, software components / driver, basic principles
• Installation of XRHMD PACKAGE of drivers in UNITY project + OVRplayer controller setup –system ready for operation
• Configuration of safe work space –OCULUS guardian setup;
• First personal experience with RWCXRSPATIAL model –each student wears the OCULUS XRHMD
• Demonstration of robot arm control by XRHMD –each student animates both robot arms in task-space (robot TCP motion)

Topic #4: DIGITAL TWIN: Step I - RWC DIGITAL SHADOW Integration

AUTOMATIC MIRRORING of the physical into virtual space in REAL TIME

• Concept of the MIRRORED spaces –automatic mirroring of the physical system states/behaviors into its virtual replica
• Robot Open Architecture Control system (Robot OAC) and concept of robot joint angles in real-time –YASKAWA MotoPlusSDK
• Networking of Robot OACand UNITY project for robot joint angles transfer via UDPcommunication protocol
• C# script class in UNITY for robot joint angles acquisition and robot posture updating in real-time
• C# script class for robot joint angles RECORDING and C# function for robot recorded motion PLAYBACK (reading an ASCII file)
• Demonstration of the integrated RWCDigital Shadow functionality

Topic #5: DIGITAL TWIN: Step II - RWC DIGITAL TWIN Integration

AUTOMATIC MIRRORING of the virtual/digital into physical space in REAL TIME and OFF-LINE

• Concept of the MIRRORED spaces –automatic mirroring of the virtual system states/behaviors into its physical counterpart:
• Two options: a)mirroring in real-time mode, which is problematic!, b)mirroring in off-line mode whichis safe and allow optimization
• Off-line mirroring is based on creation of a robot job task (programmed in robot programming language) and uploading the code via UDPto the robot controller using Matlabapplication for preprocessing recorded data (ASCII file) of the virtual robot motion
• Demonstration of the integrated RWCDigital Twin functionality
• Robot Programming by Demonstration using physical and virtual haptic interfaces

Use of integrated RWCDigital Twin to create robot applications.

Two very general concepts of robot programming to execute specific manufacturing task:

1. Programming by explicit code writing –conventional approach, very complex and time consuming!
2. Programming by Demonstration (PbD). Within the JLL2022 students will experience and explore PbDthrough Human-Robot Interaction using three types of user interfaces:

The developed AVATAR XR–Based Programming by Demonstration (XR-PbD) concept combines different modes of Human-Robot Interaction in robotic task programming WITHOUT explicit code writing and with integrated XR – based system for visual perception, i.e., visual feedback.

General system architecture of developed AVATAR XR–Based Programming by Demonstration (XR-PbD) concept.

Topic #6: OPERATION: RWC DIGITAL TWIN Robot PbD using Digital Shadow I

XRProgramming by Demonstration using KINESTHETIC haptic controller and XRHMD visual feedback –a HYBRID human-robot interaction e.g., physical commands to the physical robot & XRvisual feedback.

• Each student will program a physical robot to complete the Touch and Jump task
• Real-time monitoring of student behavior, implemented collision avoidance in the physical space of the robot
• The learned robot trajectory (Skill Transfer) will be RECORDED and then REPRODUCED by the virtual robot in its virtual task space (after processing) and by the physical robot in its physical task space (after processing, creating a job task and uploading into the robot controller). The circle is CLOSED!

The sequence of AVATAR Touch-and-Jump task (Game-based Learning):

The CHALLENGE: Touch all targets (balls) with the highest possible accuracy, by jumping from one to another, in shortest possible time!

NO EXPLICIT CODDING of the robot TCP motion. You will use a haptic device to control the robot TCP motion.You will guide the robot by XR visual feedback (Robot Digital Shadow) which looks as it is shown in figure below. When you touch the target properly,it changes its color and another ball appears. You have to touch all targets and then the task is successfully accomplished.Robot motion is RECORDED and can be REPLAYED!

Topic #7: OPERATION: RWC DIGITAL TWIN Robot PbD using Digital Shadow II

Operation of the integrated RWCDigital Twin and GAME Based LEARNING XRProgramming by Demonstration using TELEOPERATIONhaptic controller and XRHMD visual feedback –a HYBRID human-robot interaction e.g., physical commands to the physical robot & XRvisual feedback.

• Each student will program a physical robot to complete the Touch and Jump task
• Real-time monitoring of student behavior, implemented collision avoidance in the physical space of the robot
• The learned robot trajectory (Skill Transfer) will be RECORDED and then REPRODUCED by the virtual robot in its virtual task space (after processing) and by the physical robot in its physical task space (after processing, creating a job task and uploading into the robot controller). The circle is CLOSED!

Topic #8: OPERATION: RWC DIGITAL TWIN XR Robot PbD using full Digital Twin

Operation of the integrated RWCDigital Twin and GAME Based LEARNING XR Programming by Demonstration in XR SPACE using XR haptic controllers and XR HMD visual feedback –a fully VIRTUALIZED human-robot interaction e.g., physical commands to the virtual robot & XR visual feedback.

Each student completed at least two attempts to achieve the Touch and Jump task, for each of the FIVE HMI modalities of interaction with the robot. Each trial is recorded (over 300 recordings) and the saved data is used to further analyze student behaviour, student team behavior and the behavior of the entire JLL 2022 cohort.

Direct or remote contol from physical world

Direct or remote contol from virtual world

Full VR interactive control of either the real robot or its digital twin

Final presentation

The last session, Friday afternoon, was designated for the final presentation of the student report. Each student team presented its report. Then the teachers asked questions and evaluated their work. You can see the presentations of each of the three student teams here:

---