 |
Abstract: Server-based Virtual Studio [DE]
Oliver Mertens, Server-based Virtual Studio, Hochschule Düsseldorf, Master Thesis, 21.10.2021.
Virtual studios are typically based on a local interconnection of specialised
devices that perform all necessary steps to mix camera and sound recordings with
the elements of a virtual environment in real time. This thesis addresses the
questions of how far this process can be performed entirely in an IP-based network
using a central server, what latencies occur and how the synchronisation of
multiple data streams can be established. A scenario is considered in which video
and camera tracking data from mobile devices is transmitted using protocols and
technologies such as RTP and WebRTC and synchronised using clock synchronisation
and timestamps during further processing. A server application was implemented to
test the coordination of multiple clients that can be connected via WebRTC. These
clients were developed as a web app that acts as a camera source and a Unity
application to render the received video streams using GStreamer together with 3D
content. Furthermore, an AR app for Android was implemented that uses RTP to send
a video stream and corresponding tracking data directly to the Unity application.
By setting up a production pipeline in a local network, both the quality of
synchronisation and occuring latencies are determined. It is shown that the server
is able to coordinate the clients and enables a dynamic establishment and
termination of connections. The tracking data could be reliably assigned to the
associated video images by using their timestamps after a transmission and the
synchronisation of two video streams was shown to be feasible in principle, but
could not be robustly established with the present implementation and required
manual readjustment. The final output had a latency between 0.46 s and 0.54 s
when using the AR app, so it is suggested to render a feedback image directly
in the AR app, which was displayed after 0.2 s at the latest.
Keywords:
Virtual studio, video streaming, WebRTC, RTP, server-client architecture, clock synchronization, timestamps, GStreamer, ARCore, Unity
Supervisor:
Prof. Jens Herder, Dr. Eng./Univ. of Tsukuba
Prof. Dr.-Ing. Thomas Bonse
Location:
The research took place at the Virtual Sets and Virtual Environments Laboratory.
|
|
