A facial image captured by a security camera can be an important clue in criminal investigations. However, there is a wide variety of face poses and lighting conditions, which has a significant impact on facial appearance on the image. As a result, in case of comparison to a pre-registered facial image, it is difficult to determine whether each subject is the same. This paper proposes to register 3D information of the face instead of the ordinal 2D facial image in advance. Then, it is possible to reproduce the facial image under the same conditions as the facial image in a security camera, so facial examination becomes direct comparison, and in consequence, more rapid and more reliable. In this paper, we will introduce NEC’s high-speed and high-precision face 3D measurement technology and its application to facial image comparison and facial examination.

1. Introduction

Biometrics identification employs various physiological characteristics of the human body such as fingerprints, irises, veins, and faces to identify unique individuals. Fingerprints, in particular, have long been used in criminal investigations to identify subjects based on their uniqueness and permanence. Today, with the rapid proliferation of security cameras in stores, streets, and other public spaces, a facial image captured by a security camera can often be an important clue for criminal investigations. Then, more reliable systems are expected to utilize these facial images for criminal investigations and facial examination.

For usual biometrics identification applications, a facial image captured directly from the front is more suitable. However, in most cases, security cameras are located above the scene they are recording, so faces are rarely captured directly from the front. Moreover, various lighting conditions make a huge difference in shades and shadows on the face. This makes it difficult to determine whether or not the subject in the image captured by the camera is the same as a pre-registered facial image. There is a solution to be considered for the issue. By registering 3D facial data instead of the ordinary 2D facial images, facial images under any cases of conditions can be reproduced using 3D computer graphics (3DCG). This makes us to compare and examine faces directly, in consequence, more rapidly and more accurately. To realize the solution as real applications, it is important how to measure 3D facial data and how to reproduce facial images under the same conditions present in the actual scenes.

This paper describes a couple of technologies developed by NEC. One is to measure the facial 3D shapes and the other is to apply them to facial image comparison and facial examination in forensic identification applications.

2. Measurement of 3D Facial Data

When measuring human faces, the measurement time needs to be as short as possible, because it is not so easy for the subject to keep still. At the same time, it is critical to minimize the chance of false identification of faces of people who should not be suspected. In other words, a face measurement system needs to operate reliably as a measurement device, while at the same time measuring faces with as much precision as possible.

Meeting the requirements for both the shortest measurement time and the highest possible measurement accuracy simultaneously is usually difficult and involves a trade-off between speed and accuracy. What distinguishes NEC’s 3D facial measurement system is that it successfully meets both requirements, while minimizing trade-offs and assuring balanced high performance.

2.1 3D shape measurement

NEC’s 3D shape measurement system is based on the phase shift technique with the sinusoidal wave projection, which offers excellent basic performance. The NEC system solves the issue of 2-pi uncertainty of phase values found in that method’s phase value calculation ― which is a weakness of the phase shift technique with the sinusoidal wave projection ― by using geometrical constraints among multiple cameras and multiple projectors for the first stage systems, and by combining the method with a different type of projection pattern for the second stage systems. As a result, a measurement time of 0.3 seconds or less with a depth measurement accuracy of 300 µm or less has been achieved.

2.2 3D facial measurement results

The external appearance of the NEC system is shown in the Photo below. The subject to be measured sits on the chair while pressing their head against the headrest. Our 3D shape measurement system is composed of two measurement modules. The one on the left seen from the front measures the left side of the face, while the one on the right measures the right side. The measured shapes are then combined to create the single 3D shape of the entire face.

While these measurements are being made, texture images are also shot. This allows each measured point in three dimensions to be allocated with the Red-Green-Blue (RGB) value of that point.

Fig. 1 shows an example of a measurement result. When 3D shapes as shown are available, they can be coordinated with facial images shot by security cameras to theoretically generate images with appropriate facial orientations and lighting conditions.

3. Applications of 3D Facial Data

3.1 Features of 3D facial data

The use of 3D facial data makes it possible to reproduce various facial images using 3DCG technology. The appearance of a face is determined by the specifications (image size, angle of view, etc.) of the camera in use and by the relationship between the positions and rotations of the camera and the face. The size of the face in the image can be calculated from the positional and rotational data of the face with respect to the camera in three-dimensional space. Shades and shadows on the face created by illuminations can be calculated from the relative relationship between the positions of the illuminations and the 3D shape of the face. By combining these calculations, it is possible to reproduce facial images on the computer that reflect the conditions of the facial images captured by the cameras (Fig. 2).

3.2 Application to face image matching

In face image matching, the facial image captured by the camera is compared with the 3D facial image reproduced from 3D facial data, and the degree of similarity is calculated. However, facial images captured by security cameras rarely meet the conditions required for accurate matching due to variations in face poses and lighting conditions. Direct frontal imaging is especially rare. Lighting environments can also vary considerably, and variables might include day or night, indoors or outdoors, the number and type of lighting fixtures ― all of which can radically alter the facial image by causing shades and shadows on the face to fluctuate significantly.

With 3D facial data, facial images can be reproduced in any desired position, rotation, and lighting condition using 3DCG technology. However, lighting conditions in the real world are diverse and possible combinations are numerous, so the computational cost ends up being too high.

To reproduce diverse lighting conditions at a low cost, geodesic illumination bases (GIBs)¹⁾ can be used. For calculating GIBs, facial images under various lighting conditions are reproduced in advance by using 3D facial data. Statistical processing using principal component analysis (PCA) can then be applied to those facial images to calculate representative variations of shades and shadows as GIBs (Fig. 3). Using combinations of GIBs, facial images under various lighting conditions can be generated at a low computational cost.

To achieve maximum similarity with the facial image captured by the camera, the optimal face’s positional data, rotational data and combinations of GIBs are calculated to generate facial images. This makes it possible to minimize the effects of variable face poses and lighting conditions, and to facilitate more reliable face matching (Fig. 4).

3.3 Application to facial examination

3D facial data can also be used for facial examination to examine whether or not the subject’s face matches any registered facial images.

Roughly speaking, there are three basic methods of facial examination: examination using anthropological measurement, examination using morphological characteristics, and the superimposition method. In the superimposition method, the two facial images being compared are overlapped and then the matching degrees of the positions and contours of various parts of the face are evaluated. However, because a facial image captured by a security camera vary in terms of facial pose and size, simple superimposition of facial images creates deviations, making it difficult to objectively assess any similarity between the images.

To overcome this problem, a facial image that is the same size and pose as the captured image is reproduced from 3D data. Superimposing the reproduced image on the captured image makes it possible to verify the degree of similarity between the images rapidly and reliably (Fig. 5). The facial image reproduced from 3D facial data can also be used to improve the reliability of examination using anthropological measurement and morphological characteristics.

4. Conclusion

Especially designed for face image comparison and facial examination applications, NEC’s high-speed, high-precision 3D facial shape measurement technology makes it possible to more accurately compare and match faces. Through our world leading biometric technologies, NEC contributes, and will do to realize secure, safe and fair societies all over the world.

Reference

Authors’ Profiles