Offre de thèse Mom Packaging & LINEACT-CESI

PhD Dissertation Proposal

Learning-based Object Detection and Registration for Augmented Reality Integrated Manufacturing.

Keywords: Deep Learning, Object Registration, Augmented Reality, Assembly line, Digital Twin, Human-Robot Collaboration.

Context

In the context of the industry 4.0 and digital factory, digital twins, virtual reality and augmented reality are key technologies to design, simulate and optimize cyber-physical production systems (CPPS). They also allow to interact with the CPPS, whether directly, remotely or collaboratively (Uhlemann, Lehmann and Steinhilper, 2017; Vincent Havard et al., 2019; Ke et al., 2019; Sjarov et al., 2020). These technologies open up new possibilities in the design, study or implementation of industrial systems and workstations integrating Industry 4.0 components such as cobots and requiring interaction with the physical system and the virtual system. Indeed, with the evolution of robotic systems and artificial intelligence, humans and robots can collaborate in the same workspace (Rückert, Wohlfromm and Tracht, 2018; Vincent Havard et al., 2019; Pérez et al., 2019).

Human-machine interactions between the operator and the industrial system can be based on augmented reality devices. They are interfaced with the digital twin and allows to visualize information or instructions on the operations to perform, such as assembly or maintenance tasks (Rabah et al., 2018; Gattullo et al., 2019; V Havard et al., 2019; de Souza Cardoso, Mariano and Zorzal, 2020; Lai et al., 2020; Ojer et al., 2020; Havard et al., 2021). These AR devices can also be used to inform the operator about the robots' intentions. Finally, these technologies are also useful for training operators in these professional situations (Matsas and Vosniakos, 2017; Fratczak et al., 2019; Vincent Havard et al., 2019; Badets et al., 2020).

To support its research and training activities, LINEACT CESI has Factory of the Future demonstrators including a metallic additive manufacturing unit, a flexible production system, manual and cobots workstations, mobile robots and robotic arms and prototyping or machining equipment. A first digital twin associated with human machine interfaces based on virtual or augmented reality has been initiated and allows, for example, to perform simulations, in an immersive environment, of assembly operations on manual workstations integrating a cobot station. These physical and digital twins will be exploited within the framework of the development of the proofs of concept of this thesis.

MOM packaging is a company which designs, builds and produces filling machines for many types of products (liquid, powder, …) and containers. As each machine is a unique, MOM needs to keep their know-how. As augmented Reality (AR) is a key technology to train people to assemble complex systems in a reduced amount of time (Havard et al., 2021), MOM packaging is aiming at using this technology to train its employees to assemble its machine as well as its consumers for remote support.

Still, spare parts for industrial machine introduced many challenges. Parts can be symmetric or confused due their size or appearance. Variants of techniques used for AR in domestic environments must be proposed.

Research Topic

Augmented reality relies on adding virtual data to real object in order to give contextualized information to the user. The association of real and virtual data can be performed at different levels. Still, feature matching is the most popular approaches. Features from the real objects in the scene and a reference one are matched in order to register the virtual model in the real scene. Such method is prone to many challenges (Zeng et al., 2017; Xiang et al., 2018):

• Texture-less or small objects: such objects do not provide enough stable features.

• Occlusion and cluttered environments: objects and features cannot be properly detected.

• Deformable objects: the registration is not rigid anymore.

Most of the literature focuses on standalone objects. Assembly lines and Industry 4.0 brings extra challenges defined as follows (Li et al., 2021):

• Incremental completion: parts or sub-assemblies are added step after step sometimes with complex kinematics.

• Occlusions: the operator or the single parts can occlude other parts in the camera field of view.

• Symmetric objects: parts may have large size differences. If the small part is not properly detected, the registration may fail or be inversed.

• Stability: the registration must be as stable as possible for the AR user acceptance.

Expert-engineered registration methods have shown to be outperformed by learning-based approaches. Learning-based approaches requires sufficient data to perform well. Digital twin provides a powerful tool to generate training data to deploy learning-based solution. Moreover, the variety of parts of assembly lines may requires an even larger dataset. Tackling registration and the different viewpoints may even burden the data collection task.

This thesis aims at tackling the challenges specific to the industries 4.0.

Contributions on the following topics are expected:

• Learning from digital twins data and challenge the reality gap (realistic interaction and visual appearance) of digital twins.

• 6 Degrees of Freedom (DoF) spare part registrations for proper assembly.

The PhD thesis major steps can be described as follows:

1. State of the Art of AR for assembly line.

2. Usage of Industry 4.0 platforms and AR tools available at CESI and MOM packaging

3. Contribution in Sim2sim 6 DoF spare part detection and assembly registration in the context of MOM packaging assembly lines.

4. Contribution in Sim2real 6 DoF spare part detection and assembly registration with the help of digital twins in the context of MOM packaging assembly lines.

The expected deliverables are:

• At least, one communication in major conferences and one JCR journal paper.

• The deployment of the algorithms compatible with MOM AR platforms (commodity and embedded hardware).

• An evaluation of the algorithms performance on a real system.

This work is in line with the research themes of the LINEACT CESI laboratory and MOM Packaging R&D roadmap. LINEACT CESI is organized around two interdisciplinary scientific themes and two application areas. The themes "Learning and Innovation" and "Engineering and Digital Tools" develop and cross their research in the application areas of the Industry of the Future and the City of the Future.

Within the framework of this research project, the PhD student will be able to rely on the laboratory's expertise in the architecture conception, modeling, development and evaluation of augmented environments or dynamic and collaborative virtual environments applied in the field of Industry 5.0. He will also be able to rely on the laboratory's skills and work on industrial cyber-physical systems and more specifically on digital twins, their architectures and Artificial Intelligence integrated decision system.

Laboratory: LINEACT (Digital Innovation Laboratory for Companies and Apprenticeships for the Competitiveness of Territories), https://lineact.cesi.fr/en/

Company: MOM packaging, https://www.mom-packaging.com/

Thesis Director: Yohan DUPUIS (ydupuis@cesi.fr), Research Director CESI – LINEACT Lab.

Thesis Advisor: Vincent HAVARD (vhavard@cesi.fr), Researcher-Lecturer CESI –LINEACT Lab.

Company Advisor: Louis DERANGÈRE, MOM Packaging CEO.

Funding: CIFRE PhD grant.

Workplace: MOM packaging, Villepinte and CESI Nanterre Campus, Nanterre.

Start date: September 2022.

Duration: 3 years.

Hiring procedure: Application files and interview.

Your application must be sent to Yohan DUPUIS (ydupuis@cesi.fr) and Vincent HAVARD (vhavard@cesi.fr) :

« [Application] PhD Thesis IA for AR Integrated Manufacturing »

Your application files must include:

• A Detailed Resume, please provide explanations in case of gap period in your resume;

• A cover letter, a focus should be given on your experience related to the topic and your motivations to pursue a PhD thesis;

• Your master grades, the candidate must hold a Master in Computer Science, Artifical Intelligence, Computer Vision or Robotics;

• Any related document that could help evaluating your application.

Please send the document as a zip file entitled « LASTNAME_Firstname.zip »

Applicant skills:

Hard Skills:

· Industrial Robotics or Industrial Automation.

· Computer Vision.

· Deep Learning.

· Augmented reality

· Programming in Python

· Scientific Writing.

· Good English speaking and writing proficiency.

Soft skills:

· Autonomy.

· Adaptability.

· Communication.

· Creativity.

· Problem solving.

· Teamwork.

Bibliography

Badets, A. et al. (2020) ‘Using collaborative VR technology for Lean Manufacturing Training: a case study’, in Proc. VRIC. VRIC ConVRgence 2020: 22nd Virtual Reality International Conference - Laval Virtual, Laval, France, pp. 118–127.

Fratczak, P. et al. (2019) ‘Understanding Human Behaviour in Industrial Human-Robot Interaction by Means of Virtual Reality’, in Proceedings of the Halfway to the Future Symposium 2019. HTTF 2019: Halfway to the Future, Nottingham United Kingdom: ACM, pp. 1–7. doi: 10.1145/3363384.3363403.

Gattullo, M. et al. (2019) ‘Towards augmented reality manuals for industry 4.0: A methodology’, Robotics and Computer-Integrated Manufacturing, 56, pp. 276–286. doi: 10.1016/j.rcim.2018.10.001.

Havard, V et al. (2019) ‘An architecture for data management, visualisation and supervision of Cyber- Physical Production Systems’, in Proceedings of the 17th International Conference on Manufacturing Research. ICMR 2019, Belfast, p. 6. doi: 10.3233/ATDE190015.

Havard, Vincent et al. (2019) ‘Digital twin and virtual reality: a co-simulation environment for design and assessment of industrial workstations’, Production & Manufacturing Research, 7(1), pp. 472–489. doi: 10.1080/21693277.2019.1660283.

Havard, V. et al. (2021) ‘A use case study comparing augmented reality (AR) and electronic document-based maintenance instructions considering tasks complexity and operator competency level’, Virtual Reality. doi: 10.1007/s10055-020-00493-z.

Ke, S. et al. (2019) ‘A enhanced interaction framework based on VR, AR and MR in digital twin’, Procedia CIRP, 83, pp. 753–758. doi: 10.1016/j.procir.2019.04.103.

Lai, Z.-H. et al. (2020) ‘Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing’, Journal of Manufacturing Systems, 55, pp. 69–81. doi: 10.1016/j.jmsy.2020.02.010.

Li, W. et al. (2021) ‘Fully convolutional network-based registration for augmented assembly systems’, Journal of Manufacturing Systems, p. S0278612520302314. doi: 10.1016/j.jmsy.2020.12.017.

Matsas, E. and Vosniakos, G.-C. (2017) ‘Design of a virtual reality training system for human–robot collaboration in manufacturing tasks’, International Journal on Interactive Design and Manufacturing (IJIDeM), 11(2), pp. 139–153. doi: 10.1007/s12008-015-0259-2.

Ojer, M. et al. (2020) ‘Projection-Based Augmented Reality Assistance for Manual Electronic Component Assembly Processes’, Applied Sciences, 10(3), p. 796. doi: 10.3390/app10030796.

Pérez, L. et al. (2019) ‘Industrial robot control and operator training using virtual reality interfaces’, Computers in Industry, 109, pp. 114–120. doi: 10.1016/j.compind.2019.05.001.

Rabah, S. et al. (2018) ‘Towards improving the future of manufacturing through digital twin and augmented reality technologies’, Procedia Manufacturing, 17, pp. 460–467. doi: 10.1016/j.promfg.2018.10.070.

Rückert, P., Wohlfromm, L. and Tracht, K. (2018) ‘Implementation of virtual reality systems for simulation of human-robot collaboration’, Procedia Manufacturing, 19, pp. 164–170. doi: 10.1016/j.promfg.2018.01.023.

Sjarov, M. et al. (2020) ‘The Digital Twin Concept in Industry – A Review and Systematization’, p. 8.

de Souza Cardoso, L. F., Mariano, F. C. M. Q. and Zorzal, E. R. (2020) ‘A survey of industrial augmented reality’, Computers & Industrial Engineering, 139, p. 106159. doi: 10.1016/j.cie.2019.106159.

Uhlemann, T. H.-J., Lehmann, C. and Steinhilper, R. (2017) ‘The digital twin: Realizing the cyber-physical production system for industry 4.0’, Procedia Cirp, 61, pp. 335–340.

Xiang, Y. et al. (2018) ‘PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes’, arXiv:1711.00199 [cs]. Available at: http://arxiv.org/abs/1711.00199 (Accessed: 21 April 2021).

Zeng, A. et al. (2017) ‘Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge’, in 2017 IEEE International Conference on Robotics and Automation (ICRA). 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, Singapore: IEEE, pp. 1386–1383. doi: 10.1109/ICRA.2017.7989165.