Labor für Virtuelles Studio / Virtuelle Realität

Abstract: Automatic Camera Control and Video Switching in Live Studio Production using Deep Learning Algorithms [DE]

Robotic, track-mounted and remote-controlled camera equipment is widely used in virtual reality and news studios. Technologies and research are currently limited to predefined positions and automated person tracking. Mostly, such systems are not taking into account the image composition demands and additional layers of virtual cinematography, which only fully integrated solutions can offer. Real time control of cameras is enabled by markerless actor and camera tracking – positions of persons and parameters of equipment in the studio are continuously collected and transmitted. Based on this data, a semi-automatic prototype has been implemented and used in production. The prototype uses image composition parameters in a 3D rendering environment. Along the prototype, requirements and tasks for deep-learning supported camera and switching control are outlined, foremost the data collection. This thesis introduces an approach of robotic camera control using industry-standard communication protocols. Relying on simple interaction, methods are proposed, providing automatic control, but requiring no human intervention by default. Interaction methods and data collection approaches are provided for future implementations, that rely on the infrastructure of a virtual studio.

Keywords:

virtual cinematographer, camera control, switcher control, camera placement, camera framing, virtual environment, virtual studio, image composition, robotic camera system, AI director, deep learning, neural networks

Supervisor:

Prof. Jens Herder, Dr. Eng./Univ. of Tsukuba
Prof. Dr.-Ing. Thomas Bonse

Location:

The research took place at the Virtual Sets and Virtual Environments Laboratory.