Labor für Virtuelles Studio / Virtuelle Realität

Abstract: AR Annotations for Enhancing AI Interaction [DE]

This thesis presents the conceptual design and prototypical implementation of an augmented reality system for AI-supported generation of interactive annotations in a showroom context (ARIKAS). The objective is to explain real-world objects through context-based visual AR annotations, thereby enabling a self-guided and immersive knowledge transfer. The system architecture integrates real-time communication of video, audio, and data via WebRTC, as well as AI components and AR-based visualization on a head-mounted display. Detected objects are linked with virtual annotations that dynamically respond to user queries. A particular focus is placed on the use of locally hosted AI models to meet enterprise data protection requirements. The results demonstrate that the chosen retrieval-augmented generation (RAG) architecture enables flexible and domain-specific knowledge integration. At the same time, challenges remain in accurate spatial positioning, the selection of appropriate objects for annotation, and the consolidation of inconsistent naming conventions. Expert interviews with an augmented reality expert and a showroom tour expert confirm the potential of the approach in terms of interactivity, scalability, and innovation. However, they also highlight existing limitations compared to human presenters, particularly regarding social interaction, situational adaptability, and deep contextual understanding. Overall, the thesis shows that AI-supported AR systems represent a promising extension of traditional presentation formats and enable new application scenarios, especially in knowledge-intensive environments. Future improvements may include the integration of multiple languages, the simultaneous display of multiple annotations, a multi-user approach, and the incorporation of additional metadata beyond object detection.

Keywords:

Augmented Reality, AR Annotations, Artificial Intelligence, Human-AI Interaction

Supervisor:

Prof. Jens Herder, Dr. Eng./Univ. of Tsukuba
M.Sc. Kai Manuel Börner

Location:

The research took place at the Vodafone GmbH.