 |
Real-Time Relighting of Video Streams for Augmented Virtuality Scenes
Till Davin and Jens Herder, Real-Time Relighting of Video Streams for Augmented Virtuality Scenes, GI-VRAR, Workshop Proceedings / Tagungsband: Virtuelle und Erweiterte Realität - 18. Workshop der GI-Fachgruppe VR/AR, September, 2021.
The combination of real and virtual image elements is inevitable in virtual productions and mixed reality applications.
One major problem in combining real and virtual elements lies in the visual discrepancy between virtual and real environment.
We present an implementation of real-time relighting of a video stream with the light setting of a virtual scene.
A system is realized that utilizes image processing and modern graphics calculations to relight the input video stream
with additional depth information. A two-dimensional contour is extracted from the video image through image processing.
The contour is then used to generate a displacement texture,
which gives the two-dimensional video plane three-dimensional depth.
The video stream then can be relit with the virtual light in the scene.
An integrated markerless motion-tracking system creates the possibility to correctly position
the video with geometry in the scene. Image results, advantages, and disadvantages of the
proposed system are discussed and evaluated.
Keywords:
real-time rendering, markerless motion tracking, virtual studio, augmented virtuality, relighting, light estimation
Video
The media player is loading...
Download
| Description | Length | Size |
|---|
| Real-Time Relighting of Video Streams for Augmented Virtuality Scenes 1080p50 HQ | 1:06 Min | 354 MB >> |
| Real-Time Relighting of Video Streams for Augmented Virtuality Scenes 1080p50 LQ | 1:06 Min | 81 MB >> |
| Real-Time Relighting of Video Streams for Augmented Virtuality Scenes Paper PDF | 12 Pages | 2,4 MB >> |
| Real-Time Relighting of Video Streams for Augmented Virtuality Scenes Paper Slides PDF | 15 Pages | 2 MB >> |
| Github: Contours2HeightMap | Source (Unity C#/Shader) | >> |
Credits
- Ballet dancer footage from the production "Dance on the roof"
- Tracking Hub Software from Vizrt
- Camera tracking system - Egripment/Trackmen
- Markerless person tracking - Captury Live
- Virtual Studio at Hochschule Düsseldorf - University of Applied Sciences
- Unity HDRP
Productions
- Interactive Virtual Studio - Dance on the roof
- Production at the HS Düsseldorf with the game engine Unity, 2020.
- Interactive Virtual Studio - Relighting
- Production at the HS Düsseldorf with the game engine Unity, 2020.
- Interactive Virtual Studio - Markerless talent tracking for controlling a moving head
- Production at the HS Düsseldorf with the game engine Unity, 2019.
- Interactive Virtual Studio - Indiana Jones in the land of students
- Production at the HS Düsseldorf with the game engine Unreal, 2018.
Publications
- A Lighthouse-based Camera Tracking System for Professional Virtual Studios
- Kai Vermeegen and Jens Herder, GI-VRAR, Workshop Proceedings / Tagungsband: Virtuelle und Erweiterte Realität – 15. Workshop der GI-Fachgruppe VR/AR, ISBN 978-3-8440-6215-1, Düsseldorf, pp. 19-26, Oktober, 2018.
- Mixed Reality Experience - How to Use a Virtual (TV) Studio for Demonstration of Virtual Reality Applications
- Jens Herder, Philipp Ladwig, Kai Vermeegen, Dennis Hergert, Florian Busch, Kevin Klever, Sebastian Holthausen, and Bektur Ryskeldiev, Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP, Santa Cruz, Portugal, INSTICC, DOI=10.5220/0006637502810287, Januar 27-29, 2018.
- Four Metamorphosis States in a Distributed Virtual (TV) Studio: Human, Cyborg, Avatar, and Bot - Markerless Tracking and Feedback for Realtime Animation Control
- Jens Herder, Jeff Daemen, Peter Haufs-Brusberg, Isis Abdel Aziz. Four Metamorphosis States in a Distributed Virtual (TV) Studio: Human, Cyborg, Avatar, and Bot - Markerless Tracking and Feedback for Realtime Animation Control, In Virtual Realities, pp 16-32, Springer International Publishing, DOI=10.1007/978-3-319-17043-5_2, ISBN 978-3-319-17042-8, April 1st, 2015.
- SpiderFeedback - Visual feedback for Orientation in Virtual TV Studios
- Jonathan Simsch and Jens Herder, 11th International Conference on Advances in Computer Entertainment, ACE 2014, Funchal, Portugal, November 11-14, 2014.
- Four Metamorphosis States in a Distributed Virtual (TV) Studio: Human, Cyborg, Avatar, and Bot
- Jose Burga, Jeff Daemen, Sascha Charlie Djuderija, Maren Gnehr, Lars Goossens, Sven Hartz, Peter Haufs-Brusberg, Jens Herder, Mohammed Ibrahim, Nikolas Koop, Christophe Leske, Laurid Meyer, Antje Müller, Björn Salgert, Richard Schroeder, and Simon Thiele, 10th International Conference on Visual Media Production (CVMP 2013), short paper, London, November, 2013.
- Markerless Actor Tracking for Virtual (TV) Studio Applications
- Jeff Daemen, Peter Haufs-Brusberg, and Jens Herder, International Joint Conference on Awareness Science and Technology & Ubi-Media Computing (iCAST 2013 & UMEDIA 2013), HC-2013 Track, Hamamatsu/Aizu-Wakamatsu/Duesseldorf, November, 2013.
Thesis
- Realtime Relighting of a Video Stream in an Augmented Virtuality Scene Using Markerless Motion Tracking
- Till Davin, Hochschule Düsseldorf, Master thesis, 2021
- Virtual avatars in the virtual studio using markerless tracking for body and face
- Raoul Gödel, Hochschule Düsseldorf, Bachelor thesis, 2020
|
|
