 |
Abstract: Realtime Relighting of a Video Stream in an Augmented Virtuality Scene Using Markerless Motion Tracking [DE]
Till Davin, Realtime Relighting of a Video Stream in an Augmented Virtuality Scene Using Markerless Motion Tracking, Hochschule Düsseldorf, Master Thesis, 10.02.2021.
The combination of real and virtual image elements is inevitable in virtual productions
and mixed reality applications. One major problem in combining real and virtual elements
in a single image lies in the visual discrepancy between virtual and real content, whether
it is in the form of different lighting conditions, shadows or wrong object occlusion.
These differences are easily noticeable to the human eye and counteract the desired immersion.
This thesis seeks to address these challenges in augmented reality and virtual production,
and presents an implementation of realtime-relighting of a video stream with the light
setting of a virtual scene as a possible solution. The goal of this work is to develop
an application that matches the recording of an actor with that of a virtual scene, and
blurs lighting differences for the human eye. This matching will be done in realtime to
meet the needs of today’s mixed reality applications and virtual productions.
After investigating different possible approaches to solving the relighting problem,
a system is realized that utilizes image processing and modern graphics calculations
to relight the input video stream. First the two-dimensional contour is extracted from
the video image through image processing. The contour is then used to generate a
displacement texture. With this displacement texture the two-dimensional video plane
is given three-dimensional depth. The video stream then can be relit with the virtual
light in the scene and the acquired normal vectors and depth information from the generated
geometry. Further, an integrated markerless motion-tracking system creates the possibility
to correctly position the geometry in the scene. Image results, advantages, and disadvantages
of the proposed system are discussed and evaluated.
Keywords:
Real-Time Rendering, Markerless Motion Tracking, Virtual Studio, Augmented Virtuality, Relighting, Light Estimation
Supervisor:
Prof. Jens Herder, Dr. Eng./Univ. of Tsukuba
Prof. Dr. rer. nat. Christian Geiger
Location:
The research took place at the Virtual Sets and Virtual Environments Laboratory.
|
|
