Non Verbal Communication

The aim of this research is to investigate the role of facial displays in communication. This includes spontaneous emotion, head pose, eye gaze, etc. Past work on expression recognition has tended to focused on a narrow range of culturally independent emotions deliberately performed by actors. This may have limited applicability to every day communication, as we tend to use a wider range of subtle facial displays. This research focuses on natural spontaneous conversation recorded in a specific social setting. Relevant work within the literature exists across a broad range of fields including computer vision, psychology and discourse analysis but there is relatively little known about non-verbal displays and their relationship to verbal content.

What Features Are Used for NVC?

Given the wide range of ways that non-verbal communication can be transmitted, multiple pose and appearance factors are required to determine what non-verbal signal is occuring. These can be broadly split into three groups:

  • Motion and position of body parts
  • Appearance of the body parts
  • Non-visual signals

We only consider signals in the visual domain. Visual information needs to be converted into representation that is invariant to changes that might otherwise make recognition difficult (e.g. pose invariance) because a smile seen frontally is similiar to a smile seen at profile. Tracking is a useful method to encode the motion and position of parts of the body. Appearance based techniques (Local binary patterns) have also been used to encode appearance changes. Linear predictor flock tracking is applied to videos of natural conversation (Ong et al. 2009). Certain social signals are likely to require particular expressions involving tracking multiple points within an area of the face. Features corresponding to the shape of multiple tracker positions are generated using algorithmic and heuristic techniques. The static features generated on each video frame are:

  • LM Head Pose Estimation (Liu and Zhang 2000)
  • Affine Head Pose
  • Geometric Features (similar to mind reading features, el Kaliouby and Robinson 2005)
  • PCA and ICA of tracking
  • Local binary patterns (LBPs) of facial appearance

Press release on previous workship paper

The video below shows the strength of detected non-verbal signals for thinking, understanding, questioning and agreeing.

This is work in progress, so please check back soon for updates or contact Tim Sheerman-Chase, Richard Bowden for more details. Reference: Tim Sheerman-Chase, Eng-Jon Ong, Richard Bowden. Online Learning of Robust Facial Feature Trackers. In 3rd IEEE On-line Learning for Computer Vision Workshop, Kyoto, 2009.

Cultural and NVC

Recent work has focused on cultural differences in perception of NVC. The existence of cultural differences in expression of NVC is well documented. Differences in culture makes a single global recognition system impossible. To overcome this difficulty, we train our system on culturally specific sets of annotations. This enables our system to specialise in a particular culture and to improve recognition performance.

We have collected annotation using crowd sourced surveys and this will be made publically available in the near future.

This page was last updated on: Mar 2010