Labor für Virtuelles Studio / Virtuelle Realität

Abstract: Video segmentation without specific assumptions about the background using deep learning [DE]

Automatic segmentation of videos and images has many fields of application and is constantly being researched. The present thesis deals with the design and analysis of different deep learning models that do not assume backgrounds, for the purpose of segmenting people. For the analysis there was a specific synthetic data set generated with the help of stereo camera recordings of people in front of a green screen. Color information (RGB) as well as a combination of information about color and depth (RGBD) served as input. The focus of this work lay on the comparison of RGB- and RGBD-models and the analysis of the respective impact of transfer learning within this comparison. Furthermore additional models were trained with a slightly varying data set due to data augmentation in order to verify whether these methods may lead to an optimised generalisation of the deep learning model. One of the findings was that RGB-models scored bet- ter results with the network architecture of choice, the u-net architecture, with network ResNet34 as the encoder. This leads to the assumption that for the processing of depth-information different network architectures would be necessary. Comparability between RGB- and RGBD-models with identical network architectures can therefore not be verified. Transfer learning had positive impact on learning processes and the obtained results of the respective models. However, the question whether slightly varying data sets within the learning processes may lead to an optimisation remains unanswered and requires further research.

Keywords:

Deep learning, segmentation, keying, virtual reality, augmented reality, virtual studio, stereo camera, rgb, rgbd, convolutional neural networks, transfer learning, synthetic dataset

Supervisor:

Prof. Jens Herder, Dr. Eng./Univ. of Tsukuba
Prof. Dr.-Ing. Thomas Bonse

Location:

The research took place at the Virtual Sets and Virtual Environments Laboratory.

Video

Download