Multisensory Processing: The Formation of Object Categories

Abstract

Every day, we categorise myriad objects using information from multiple sensory modalities. For instance, when we encounter a cat, we perceive it through its visual properties (e.g., shape, size, colour), the sounds it makes (e.g., meowing), and its texture (e.g., by petting its fur). Indeed, real-world object perception is inherently multisensory (Shams & Seitz, 2008), involving concurrent visual, auditory, and tactile inputs. Although there is abundant evidence for multisensory interactions in the brain (Ghazanfar & Schroeder, 2006; Alais et al., 2010), most investigations into object categorisation to date have focused mainly on unimodal sensory processing. Therefore, the processes underlying the integration of multisensory cues in categorisation and generalisation remain poorly understood (Newell et al., 2023).

The aim of this thesis was to investigate the role of multisensory perception in the formation of object categories, and to examine whether multisensory information leads to more robust categorisation and more efficient generalisation. To this end, we explored the contribution of visual, auditory, and tactile information to the formation of multisensory perceptual categories.

In Chapter 2, we examined whether prior audio-visual exposure influenced subsequent visual categorical perception of familiar objects. The results provided no evidence that auditory object cues facilitated category representation of familiar objects. However, participants successfully categorised stimuli on the basis of visual shape alone, even in the absence of categorical perception. Chapter 3 investigated whether motion and sound cues influenced the perceived shape of unfamiliar objects, as assessed through judgements of inter-object similarity. The results suggested that visual motion cues systematically affected the perceived similarity and classification of unfamiliar objects, whereas auditory motion cues did not. In Chapter 4, we examined whether visual and tactile motion cues influenced the categorisation of unfamiliar objects, and whether the formation of these multisensory categories subsequently benefited generalisation to novel exemplars. The findings provided compelling evidence that both visual and tactile motion cues supported more efficient categorisation and generalisation of unfamiliar objects. Finally, in Chapter 5, we investigated the resolution of visuo-haptic conflict in a categorisation task conducted in virtual reality. The results revealed haptic dominance in shape-based categorisation when both visual and haptic shape information were available.

Taken together, these findings indicate differential effectiveness of distinct sensory cues (i.e., spatial versus temporal information). While visual shape cues benefited from simultaneous access to multiple spatial features (i.e., visual and haptic inputs), auditory cues, which rely primarily on temporal dynamics, did not provide sufficiently diagnostic information to support categorisation. In Chapter 6, the findings of this thesis are discussed in relation to their theoretical implications, the gaps in the literature addressed by this experimental work, and potential directions for future research.