Thèse : Development of trustworthy intelligent avatars in virtual immersion

1. Context

Virtual reality is getting democratized in many application fields, from industry to education, from healthcare to cultural heritage. However, several major issues still remain for a total acceptance of the technology by the most. Among them, cybersickness and interaction in virtual environments are evocative of the difficulties to reach acceptable usability of the technology. Past work tried to propose numerous solutions to get rid of these issues or at least diminish their negative effects (Kemeny et al., 2020). However, most of them are integrated from a pure computer scientist perspective, regardless of the user and the context of use of these technologies. In this context, this thesis is positioned in a perspective of proposing a radically different approach from the one adopted until now, mixing computer science, mathematics and human factors.

Virtual reality avatars are among the most promising concepts of modern computer science. Generally speaking, an avatar is supposed to be a representation of the user. But does it really represent the user? In addition, is it helpful, and is it safe for a human user? Could it be the key for solving long-life virtual reality issues?

In this joint thesis, we aim to develop a novel class of trustworthy virtual reality avatars, capable of semi-autonomous action in virtual, augmented, or mixed reality. We will propose the foundational conceptual framework and the first systematic methodology to treat constructs of this type, and we will perform experiments and applications of intelligent virtual reality avatars to advanced manufacturing.

A huge piece of work exists on embodiment (for example, Kilteni et al. (2012)). However, for our topic of intelligent avatars of the new generation, the solution will be more complex as we want to integrate the ability to learn and adapt according to the user and the context, and a high level of autonomy to the avatar. Therefore, the avatar would not be just a representation of the user, but an entity with a dual life. In parallel, this will create extended reality of a new generation: intelligent extended reality.

2. Objective and research questions

The current thesis lays a foundation of an innovative field of research where we revise and update the concept of avatar itself. Our aim is to create avatars that are equipped with trustworthy, human-centered AI, in accordance with the latest regulations by the European Commission (HLEG 2019). The research questions addressed are as follows:

• How far can a virtual avatar represent the user of a virtual reality system?

• Which features and behaviors are to be integrated in virtual avatars?

• How to develop a customized virtual twin of the user that can help improve experience in virtual immersion?

Our solution is to consider the psychological, physiological and behavioral state of the users, as we started to do in adaptive navigation (Plouzeau et al., 2018), integrating objective parameters such as postural sway, electrodermal activity, eye movement, to build a first customized model of the user's virtual companion. This model will represent a true virtual twin of the user and will be used then to predict user intentions. The evolution of his/her state depending on his/her actions in the virtual environment and the navigation context will allow to improve the companion's behavior. The model is therefore aimed at being auto-adaptive.

The prediction and evolution of the companion's behavior will be based on tools from artificial intelligence such as deep learning or topological approaches used for example to classify movements (Bensekka et al., 2017). This is where the concept of intelligent extended reality as integrating intelligence (Luck and Aylett, 2000) and not just being judicious as done for example in Plouzeau et al. (2018) takes place. We therefore wish to question the user's role in virtual reality to best center virtual reality on the user. As a result, this will create a virtual reality of a new generation, safe and equipped with human-centered intelligence.

3. Proposed approach

The cornerstone of our theoretical framework is at the intersection of topological data analysis and machine learning, as well as deep learning.

Persistent homology, which captures the "shape of the data" from representations of datasets by finite point clouds in finite metric spaces (Edelsbrunner et al., 2008), (Carlsson 2009), (Ghrist 2008), has proven over the last decade to be an effective tool for topological data analysis (TDA) in a growing number of scientific domains (Buchet et al., 2018). In particular, the stability of the persistent homology of Rips complexes built on metric spaces (Chazal et al., 2009), (Chazal et al., 2014) motivates the choice of this type of constructions to study metric spaces involved in applications.

While applications of TDA to static metric data have been numerous, there are not many studies on dynamic metric data from an TDA perspective (Kim et al., 2017) as they require relying on the theory of multidimensional persistence, an extension of the concept of persistent homology (Carlsson et al., 2009) which does not have a stable full invariant. Our aim is to test this new theory in practice, and to adapt them to facilitate applications to VR and time series data.

Machine learning is the work horse in the decision-making of our intelligent avatars. We have already demonstrated that the application of persistent homology combined with machine learning methods allows to identify and classify movements in a robust way (Bensekka et al., 2017). We will add unique insights to the promising field of semi-supervised machine learning and deep learning, in particular, by applying and extending the recent results on generalized active learning in graphical environments (Langovoy 2015, 2019) to active and online learning on VR data and on topological invariants.

Moreover, there have been initial experiments with application of topological data analysis to semi-automation of VR data processing (McDonald 2019), and applications of persistent homologies to the analysis of 3D computer graphics data (Giorgi et al, 2017). Applications of topological data analysis to machine learning are relatively scarce yet, but recently a few promising discoveries have been made that can greatly facilitate these developments. In particular, (Hofer et al, 2017) established a link between topological signatures and deep learning, while (Kwitt et al, 2015) paved the way for application of powerful kernel methods to topological data analysis.

The following steps will be conducted as follows:

• Thorough literature review on user experience in virtual immersion, virtual avatars, behavioral agents and topological data analysis

• Development of predictive models based on topological data analysis and machine learning methods

• Development of an adaptive virtual twin of the user based on objective measurements and criteria

• Development of a demonstrator in the field of advanced manufacturing

• Experimental validation with a panel of users, integrating both objective and subjective measurements

• Scientific publications in high-rank international journals and conferences

4. Expected results

The main expected results are a systematic methodology to build novel trustworthy virtual reality avatars capable of semi-autonomous action in virtual, augmented, or mixed reality. With these avatars of the new generation, safe and equipped with human-centered intelligence, it is expected that user experience in immersive environments will be greatly improved while avoiding the current pitfalls of immersive technologies.

Experiments will be performed and applications of the proposed methodology will be developed first to advanced manufacturing, and will be extended to other fields including healthcare and education.

5. Expected profile

Master of Science in mathematics with interest in applications, or M. Sc. level in computer science or Engineering, with a strong background in mathematics and interest in computer graphics.

The following fields of competence are a strong plus:

• Good knowledge of mathematics, especially topology

• Knowledge of Artificial Intelligence (including data analysis, machine learning, statistics, applied probability, optimization methods, algorithms)

• Knowledge or interest in Virtual or Extended Reality (XR), including programming in Python, C++ or C#

The candidate is expected to have sufficient writing skills (English), and to be interested to work on multidisciplinary scientific projects.

6. Place of work

The Ph.D. will take place both in France and Germany with a provisional 50-50 ratio between each place. In France, the candidate will be hosted by ENSAM-LISPEN/Institut Image in Chalon-sur-Saône. In Germany, the candidate will be hosted by KIT-IMI in Karlsruhe.

7. Supervision team and contacts

ENSAM-LISPEN (France)

Dr. habil. Jean-Rémy CHARDONNET

jean-remy.chardonnet@ensam.eu

Dr. Christophe GUILLET

christophe.guillet@u-bourgogne.fr

KIT-IMI (Germany)

Prof. Jivka OVTCHAROVA

jivka.ovtcharova@kit.edu

Dr. Mikhail LANGOVOY

mikhail.langovoy@kit.edu

8. References

[Bensekka et al., 2017] C. Bensekka, C. Guillet, F. Merienne, and T. Pozzo, “A topological approach for human movement classification and anticipation,” in Gait & Posture, 2017, vol. 57, pp. 229–230.

[Buchet et al., 2018] Buchet, Mickaël, Yasuaki Hiraoka, and Ippei Obayashi. "Persistent homology and materials informatics." Nanoinformatics. Springer Science+ Business Media Singapore Private Limited, 2018. 75-95.

[Carlsson, 2009] G. Carlsson. Topology and data. Bull. Amer.Math. Soc., 46:255–308, 2009.

[Carlsson et al., 2009] Gunnar Carlsson and Afra Zomorodian. 2007. The theory of multidimensional persistence. In Proceedings of the twenty-third annual symposium on Computational geometry (SCG '07). Association for Computing Machinery, New York, NY, USA, 184–193.

[Chazal et al., 2009] F. Chazal, D. Cohen-Steiner, L. J. Guibas, F. Mémoli, and S. Y. Oudot. Gromov-Hausdorff stable signatures for shapes using persistence. In Proc. of SGP, 2009.

[Chazal et al., 2014] F. Chazal, V. De Silva, and S. Oudot. Persistence stability for geometric complexes. Geometriae Dedicata, 173(1):193–214, 2014.

[Edelsbrunner et al., 2008] H. Edelsbrunner and J. Harer. Persistent homology-a survey. Contemporary mathematics, 453:257–282, 2008.

[Ghrist, 2008] Ghrist, Robert. (2008). Barcodes: The persistent topology of data. BULLETIN (New Series) OF THE AMERICAN MATHEMATICAL SOCIETY. 45. 10.1090/S0273-0979-07-01191-3.

[HLEG, 2019] The High-Level Expert Group on Artificial Intelligence, set up by the European Commission, “Ethics guidelines for Trustworthy AI”, April 2019, pp. 1-41.

[Hofer et al, 2017] Hofer, C., Kwitt, R., Niethammer, M., and Uhl, A. Deep Learning with Topological Signatures. In Advances in Neural Information Processing Systems (NIPS), 2017.

[Giorgi et al, 2017] Giorgi, D., Pascali, M. A., Henriquez, P., Matuszewski, B. J., Colantonio, S., and Salvetti, O. Persistent homology to analyse 3D faces and assess body weight gain. The Visual Computer, 33(5), 549-563, 2017.

[Kemeny et al., 2020] A. Kemeny, J.-R. Chardonnet, and F. Colombet, Getting Rid of Cybersickness: In Virtual Reality, Augmented Reality and Simulators. Springer International Publishing, 2020.

[Kilteni et al., 2012] K. Kilteni, R. Groten, and M. Slater. The sense of embodiment in virtual reality. Presence : Teleoperators and Virtual Environments, 21(4) :373–387, 2012.

[Kim et al., 2017] Kim, W., & Memoli, F. (2017). Stable signatures for dynamic graphs and dynamic metric spaces via zigzag persistence. arXiv preprint arXiv:1712.04064.

[Kwitt et al, 2015] Kwitt, R., Huber, S., Niethammer, M., Lin, W., and Bauer, U. Statistical topological data analysis - a kernel perspective. In Advances in Neural Information Processing Systems (NIPS), pp. 3070-3078, 2015.

[Langovoy, 2015] Langovoy, M. “A Unified Approach to Data Analysis and Modeling of the Appearance of Materials for Computer Graphics and Multidimensional Reflectometry”. In Transactions on Engineering Technologies, pp. 15-29. Springer, Dordrecht, 2015.

[Langovoy, 2019] Langovoy, M., Generalized active learning and design of statistical experiments for manifold-valued data. Proceedings of the 62-nd World Statistical Congress (2019), Vol. 4, Session: Data Science, pp. 208 – 215.

[Luck and Aylett, 2000] Michael Luck and Ruth Aylett. Applying artificial intelligence to virtual reality : Intelligent virtual environments. Applied Artificial Intelligence, 14(1) :3–32, 2000.

[McDonald, 2019] McDonald, T. “A Topological Framework for Semi-Automatic Neuron Tracing in Virtual Reality”. University of Utah, 2019.

[Plouzeau et al., 2018] J. Plouzeau, J.-R. Chardonnet, and F. Merienne, “Using cybersickness indicators to adapt navigation in virtual reality: a pre-study,” in IEEE Virtual Reality (VR), 2018, pp. 661–662.