General Pose Face Recognition Using Frontal Face Model
Current face recognition systems are able to achieve very high recognition rates in the case of frontal images, however their performance has been observed to drop significantly if such ideal conditions are not satisfied. Recognition from non-frontal images remains a major challenge because of the severe changes of appearance induced by pose variation and also because part of the face usually becomes occluded in non frontal poses.
1. Methodology
One approach to the pose variation problem consists in using a pose correction technique in order to remap probe images into a frontal pose similar to that of gallery images. Our approach is based on using an Active Appearance Model (AAM) to localise the face and synthesise a frontal view which can be then fed into a conventional face recognition system. It requires only a gallery of frontal views of each subjects (e.g. mugshots) to train the recognition system, and requires only a single image of a subject in an arbitrary pose (usually non-frontal) for identification.
Face localisation
Faces and their characteristic features are localised automatically using a multi-resolution AAM initialised with the coordinates of the eye centres - such information is for example obtained from an eye detector.
Pose estimation and pose correction
The aim of these modules are firstly to estimate the pose of the face in the probe image and then to synthesise a novel view of the subject in the same pose as the gallery images, i.e. frontal in this case.
Geometric and photometric normalisation
Geometric normalisation is done by applying an affine transformation composed of a translation, rotation and scaling in order to align the eye centres with some pre-defined positions; the position of the eye centres in the original image is obtained automatically from the fitted appearance. Photometric normalisation is done by histogram equalisation.
Identification
The statistical features used for recognition are obtained by Linear Discriminant Analysis (LDA). Identification is done by comparing the projection of the probe and gallery images in the LDA subspace and selecting the gallery image which maximises the normalised correlation. The bilateral symmetry of faces can be used at this stage in order to attenuate the effect of occlusions.
2. Pose Estimation and Correction
Pose is parametrised by the pan and tilt angles - other parameters such as in-plane rotation, translation and image scaling are already encapsulated in the shape or appearance parameters. One method for modelling pose variation consists in learning the relationship between the shape or appearance parameters and these two parameters. A statistical approach similar to the one adopted by Cootes et al. can be used to learn such a relationship from a database of annotated images with ground truth pose information.
Pose Estimation
The pose of a subject in a previously unseen image is estimated by computing the pose parameters which minimise the squared norm of the difference between the actual model parameters fitted to the input image and the model parameters predicted by the statistical model representing pose variation.
Synthesising Corrected Views
Given an image of a subject with arbitrary pose which has been estimated, the statistical model can be used to predict the shape or appearance of the subject in a different pose - in this case a frontal pose (see second row of Figure 2).
Improving the Textural Content of the Corrected Views
With the previous approach, details such as moles or freckles are lost in the corrected view, because the appearance parameter representation preserves only the principal components of the image variations. In order to preserve the textural content of the original image, the previous approach can be combined with image warping techniques. In this case, the previous approach is applied only to the shape parameters, thus yielding an estimate of the position of the feature points in a frontal view. Then the texture of the corrected image is obtained by warping of the original image either by i) piece-wise affine warping or ii) thin-plate spline warping. The results obtained for both methods are illustrated in the third and fourth rows of Figure 2.
3. Use of the Bilateral Face Symmetry
One difficulty with non-frontal images is that parts of the face can become largely occluded. This can produce significant artefacts in the corrected images, that will penalise the recognition. In the case of rotations around the vertical axis, the bilateral face symmetry can been used to eliminate the occluded half of the face when needed, i.e. only the half of the face which is fully visible is considered for recognition. In this approach, three different LDA subspaces are build for full image, left half-image and right half-image respectively. Then the pose estimate for the probe images is used to select automatically the most appropriate LDA subspace to use for identification.
4. Results
Experiments carried out on a database of 570 non-frontal test images, including 148 different identities, showed that an AAM-based correction algorithm is able to improve the performance by up to 77.4% compared to a conventional approach which does not consider correction (see Table 1). It has also been observed that bilateral face symmetry can be used to alleviate the effects of occlusions by using the pose estimate to classify images into three categories for which separate LDA subspaces have been built.
Publications
1. J.-Y. Guillemaut, J. Kittler, M.T. Sadeghi, W.J. Christmas, "General Pose Face Recognition Using Frontal Face Model", submitted to 11th Iberoamerican Congress on Pattern Recognition (CIARP), (2006).
For more information, please contact Prof. Josef Kittler or Dr Jean-Yves Guillemaut.
