• Citer

From multi-modal capture to photo-realistic view synthesis: A high-quality and real-time multiview approach

par Bonatto, Daniele
Président du jury Milojevic, Dragomir
Promoteur Lafruit, Gauthier
Co-Promoteur Munteanu, Adrian
Publication Non publié, 2024-03-07

Thèse de doctorat

• Accès en ligne
• Détails
• Contenu
• Statistiques

Résumé :

Interactive view synthesis is a fundamental task in computer vision, aiming at recreating a natural scene from any viewpoint using a sparse set of images. The primary focus of this Ph.D. thesis is to explore the acquisition processes and the ability to render high-quality novel views dynamically to a user. Furthermore, this research targets real-time rendering as the second objective. In this thesis, we explore two different ways a scene can be reconstructed. The first option is to estimate the three-dimensional (3D) structure of the scene, by means of a set of points in space called a point cloud (PC). Such PC can be captured by a variety of devices and algorithms such as Time of Flight (ToF) cameras, stereo matching, or structure-from-motion (SfM). Alternatively, the scene can be represented by a set of input views with their associated depth maps, which can be used in depth image-based rendering (DIBR) to synthesize new images.We explore depth image-based rendering algorithms, using pictures of a scene and their associated depth maps. These algorithms project the color values at the novel view position using the depth information. However, the quality of the depth map highly impacts the accuracy of the synthesized views. Therefore, we started by improving the Depth Estimation Reference Software (DERS) of the Moving Picture Experts Group (MPEG), a worldwide standardization committee for video compression. Unlike DERS, our Reference Depth Estimation (RDE) software can take any number of input views, leading to more robust results. It is currently used to generate novel depth maps for standardized datasets. The depth estimation did not reach real-time generation; it takes minutes to hours to create a depth map depending on the input views. We therefore explored active depth sensing devices, such as Microsoft Kinect, to acquire color data and depth maps simultaneously. With the availability of these depth maps, we address the DIBR problem by providing a novel algorithm that seamlessly blends several views together. We focus on obtaining a real-time rendering method; in particular, we exploited the Open Graphics Library (OpenGL) pipeline to rasterize novel views and customize dynamic video loading algorithms to provide frames from video data to the software pipeline. The developed Reference View Synthesizer (RVS) software achieves 2x90 frames per second in a head-mounted display while rendering natural scenes. RVS was initially the default rendering tool in the MPEG-Immersive (MPEG-I) community. Over time, it has evolved to function as the encoding tool and continues to play a crucial role as the reference verification tool during experiments.We tested our methods on conventional, head-mounted, and holographic displays. Finally, we explored advanced acquisition devices and display mediums, such as (1) plenoptic cameras for which we propose a novel calibration method and an improved conversion to sub-aperture views, and (2) a three-layers holographic tensor display, able to render multiple views without wearing glasses.Each piece of this work contributed to the development of photo-realistic methods; we captured and open-sourced several public datasets of high quality and precision to the research community. They are also used by MPEG to develop novel algorithms for the future of immersive television.