Delivery of omnidirectional video using saliency prediction and optimal bitrate allocation - Smolic, Aljosa (My University, 2020)
In this work, we propose and investigate a user-centric framework for the delivery of omnidirectional video (ODV) on VR systems by taking advantage of visual attention (saliency) models for bitrate allocation module. For this purpose, we formulate a new bitrate allocation algorithm that takes saliency map and nonlinear sphere-to-plane mapping into account for each ODV and solve the formulated problem using linear integer programming. For visual attention models, we use both image- and video-based saliency prediction results; moreover, we explore two types of attention model approaches: (i) salient object detection with transfer learning using pre-trained networks, (ii) saliency prediction with supervised networks trained on eye-fixation dataset. Experimental evaluations on saliency integration of models are discussed with interesting findings on transfer learning and supervised saliency approaches.
Keywords: 360°video streaming; Attention based bit- rate allocation; Saliency maps with transfer learning and supervision; Creative Arts Practice; Creative Technologies; Digital Engagement; Image Processing; Information technology in education; Multimedia & Creativity; Other