Tracking Facial Features using Flocks of Selected Linear Predictors

The aim of this research is to produce a method for efficiently and robustly tracking facial features. These include obvious parts of the face such as the eye shape, eyebrows and lip shapes (both inner and outer lips). Some of these features can be very difficult to track, for example points on the inner lip contour, due to the large amounts of visual variations that occur when a mouth opens and closes. Additionally, other features, such as the eye shape can exhibit extremely fast motions, such as blinks.

To tackle the above issues, a novel technique based on selected linear predictors was proposed. Specifically a learnt person-specific but importantly, a data-driven approach to achieve accurate and real-time tracking of facial features using only intensity information. Additional constraints such as a-priori shape models or temporal models of dynamics are neither required nor used.

To track a facial feature, we employ the method of linear predictors (Illustration 1), where linear predictor essentially provides a mapping from pixel-level information to the displacement vector of a tracked feature (e.g. Lip shape point). A single linear predictor is seldom good enough to provide robust and accurate tracking for points on the mouth. To overcome this, rigid flocks of linear predictors are used. Here, a set of linear predictors are grouped together to track a single feature point (Illustration 2). The placement and support for a linear predictor is normally randomly placed around the feature. However, given the radical texture changes that occur at certain facial features, e.g. around the inner lip during normal speech it is not obvious that this is an appropriate strategy (Illustration 3).

To address this, we propose two approaches to selecting relevant linear predictors for giving accurate and robust tracking. Interestingly, the selection procedure makes intuitive selections that are in stark contrast to random selection and results in reliable feature point tracking even for notoriously difficult aspects such as the inner lip shape (Illustration 4).

We further improve on the existing linear predictor method by introducing a new LP selection method based on probabilisitic selection. This new LP selection method proposed here is similar to the first step of particle selection used in particle filters based trackers. The method is essentially an iterative method, where at each iteration, we draw a new set of location offsets for member LPs in a flock with respect to weightings based on an inverted form of training error. Following this, it is possible to relearn the linear mappings for this new set of LPs and reiterate the whole process to progressively locate better locations for placing LPs resulting in greater tracking accuracy.

We know that increasing the support region size and the range of training displacements result in greater robustness to large displacements from a feature's location, with reduced accuracy as a trade off. In order to obtain tracking results that are both robust and accurate, a chain of LPs of decreasing sizes can be used. The largest LP is first used to predict a feature's location. The result is then passed to a less robust but more accurate LP down the chain, and repeated until we have reached the end of the LP chain, with an accurate location of a displaced feature. As a result, we employ a similar strategy for obtaining a robust and accurate feature tracking by chaining together a sequence of decreasing size LP flocks.

Tracking Results

For more details see publications or contact Dr E J Ong or Dr R Bowden