• Citer

Towards the Next Generation of Smart and Visual Multi-Modal Sensor

par Almasri, Feras
Président du jury Bersini, Hugues
Promoteur Debeir, Olivier ;Touhafi, Abdellah
Publication Non publié, 2022-05-02

Thèse de doctorat

• Accès en ligne
• Détails
• Contenu
• Statistiques

Résumé :

Exploiting alternative visions, such as thermography, and acoustic imaging could play a fundamental role in computer vision by enhancing the accuracy of our current standard digital vision. A machine vision built on all these modalities can see in the visible spectrum, the IR spectrum, and from visualizing sound can capture physical object details that are not visible to the human eye. With all these sensor modalities implemented in one camera, a huge amount of data is produced. Thus, it is imperative to find ways to use the information from all the sensors and fuse them to get the most benefits.Despite all the advances that have been made in the development of multi-modal spectral sensors, some obstacles hinder the fusion of their output details in order to use them as integral images. The major drawback of thermal and acoustic imaging is the lack of potential benefits by their cost effect on sensor resolution, the price of which increases exponentially with increased resolution. Their resolutions are much lower than for standard digital cameras, due to the more expensive sensor technology involved in their production.In this thesis, we aim to improve the beyond visible spectrum machine vision by integrating multi-modal spectral sensors. The emphasis is on transforming the produced images to enhance their resolutions to match the expected human perception, and to improve machine vision accuracy in its applications. We started with an extensive review of the state of the art in the image Super-resolution problem and introduced two datasets, one for thermal images and one for acoustic images. We thus analyzed, as a novel experiment, the applicability of super-resolution models developed for natural images on the thermal image super-resolution problem. The reported results outperformed the traditional up-sampling methods.Image quality is essential, as it refers to the amount of relevant information that an image conveys. Therefore, enhanced image quality can be obtained by capturing additional relevant data using a different spectrum that is less affected by the environment or at least serves as redundant information when the main source of information is affected by environmental factors. Thus, using alternative data we demonstrated that rich texture details in visible images can contribute to improving and enhancing the conducted thermal super-resolution in the previous experiment. A novel idea is presented to merge the visible images features with their corresponding thermal features to enhance the thermal super-resolution images. Also, a new perceptual image assessment was created based on social experience.Having demonstrated the success of convolutional neural networks and data multi-modality in enhancing the super-resolution of thermal images, we turned our attention to extensive experimentation to search for better model architecture and training procedures to proceed with the state-of-the-art in the development of visible and thermal images super-resolution problem. Therefore, we have done several experiments with the architecture and methods that may give better results than their current counterparts as we have presented them to several international challenges for evaluation and got good rankings.We performed generalization experiments in another imaging domain, such as acoustic imaging. To the best of our knowledge, this is the first work to conduct super-resolution models on acoustic heatmap images and provide such a large acoustic dataset. Top-rated off-the-shelf models were trained on the dataset where the proposed model outperformed them. In addition, the model proved its ability to generalize to unknown real-world captured images.Perception ability is profoundly subjective and varies from one person to another. Despite this, human perception is even better at understanding true colors than shades of gray or pseudo-colors. Since the thermography images are well suited for night vision, a transformation model is developed to enhance the thermal color representation and map it as close as possible to the human understanding of natural color. A new advancement to the problem of the night-vision thermal colorization problem based on texture transfer from real captured thermal images and predicting true color values when possible and with minimal noise. Accordingly, if the versatility of sensors is improved, a lot of interesting results can be obtained by combining different sensors for diverse goals.