Automatic Tracking, Segmentation, Removal and Replay of Traffic in Panoramic 360-Degree Videos for Virtual Reality

Abstract

The Netherlands Aerospace Centre (NLR) has a virtual reality setup which allows users to experience an airplane fly-over with realistically simulated sound. They want to move from using static imagery for the surrounding to video. On top of this, they want to be able to have some form of control over the scenario that they have filmed. This thesis describes a method of automatic detection and segmentation of cars in equirectangular panoramic 360-degree videos by first tracking connected components during an online phase, and doing post processing to enhance the tracks. This will allow the NLR to remove cars from their videos, and replay them at specified times to create custom videos from a single source video. A user study and an objective study have been conducted to determine the quality of the method. The user study has shown that while videos with replayed vehicles are of lower quality than the originals, the custom videos are still acceptable to the majority of users. The objective study shows that the overall accuracy of the method (including replayable cars not used in the user study videos) is not acceptable for a large number of cars. However, as long as there are at least a few usable detections, custom videos can still be produced.