Understanding and Improving Physical Interactions in Virtual Reality

Abstract

An important challenge in AR/VR is to enable virtual interactions that look and feel natural. Our goal in this work was to identify certain failures of AR/VR interactions, understand them, and propose solutions for them so that these platforms can accommodate better and more diverse experiences. To this end, we conducted two case studies on dumbbell Lifting and ball Throwing interactions in VR, with a focus on Human Perception. We first developed a pilot system for Throwing in VR, and conducted a perceptual study to determine the importance of visual trajectory cues on throwing performance. We found that limiting visual feedback detracts from virtual throwing performance. We also ran a study to develop a better understanding of how the Point of Release (PoR) of a ball affects the perception of animated throwing motions. In this study, the participants viewed animations of a virtual human throwing a ball, in which the point of release was modified to be early or late. We found that errors in overarm throws with a late PoR are detected more easily than an early PoR, while the opposite is true for underarm throws. The viewpoint and the distance the ball travels also have an effect on perceived realism. Finally, we hypothesized that the typical experience of throwing in VR, i.e., holding a controller and using a button to release the projectile, may feel unnatural. We therefore developed a novel real-time physical interaction system, called ReTro, that allows users to throw a virtual ball without using an intermediary device such as a controller. For the implementation of the ReTro system, we developed a detection algorithm to predict the PoR of a throwing motion in real-time. This was achieved by training a PoR prediction model using motion features extracted from arm joints. The evaluation of ReTro using pre-recorded throwing motion data resulted in detection errors of less than 50 milliseconds. Another output of this analysis is a presentation of the relative importance of different joints and motion features for the PoR prediction task. Finally, qualitative results from users of ReTro in VR indicate that, although it performed better for Underarm than Overarm throws, the task of throwing without a controller felt very natural. In our second case study of Lifting, we investigated people's sensitivity to physicality errors in order to understand when they are likely to be noticeable and need to be mitigated. As a user lifts virtual objects in AR/VR, there may be dynamic inconsistencies in the motion of the virtual avatar due to a mismatch between the shapes of the user's body and their virtual avatar. There could also be a mismatch between the real and virtual objects being interacted with, such as a real controller vs. a virtual boulder. We use the term "physicality errors" to distinguish them from simple physical errors, such as footskate. Physicality errors involve plausible motions, but with dynamic inconsistencies. We used the exercise of a dumbbell lift to explore the impact of motion kinematics and varied sources of visual information, which included changing the sizes of the body and manipulated objects, and displaying muscular strain. Our results suggest that kinematic (motion) information has a dominant impact on the perception of effort, but that visual information, particularly the visual size of the lifted object, has a strong impact on perceived weight. This can lead to perceptual mismatches which reduce perceived naturalness. Small errors may not be noticeable, but large errors reduce naturalness. These results can be used to inform the development of animation algorithms