4D data capture - real-time 3D

The 4D Vision project aimed to develop 3D Photometric Stereo technology to enable the capture of 3D faces in real-time.

We hoped to develop new imaging capabilities for high-speed and resolution capture of facial movements. Combined with robust multi-resolution analysis, realistic visualisation and fast interaction.

4D capture is fundamental in a number of tasks such as detection of deceptive behaviour, and realistic modelling of facial expression for gaming characters.

This project was supported by HEFCE QR funding.

Aims

• Build a system for generic 3D facial macro- and micro-movement capture.
• Real-time reconstruction of moving shapes and 3D texture information at unprecedented sub-pixel resolution.
• Realistic rendering of moving faces with real-time visualisation and interaction.
• Create and maintain open access a research database showing specific interaction sequences of facial movements.
• Design and implement novel automated feature extraction and classification methods, that exploit the spatio-temporal information of moving 3D faces and allow fast detection of macro and micro-movements.
• Generate a new facial expression taxonomy using a robust and accurate model to map facial micro and macro-movements to corresponding expressions.

Real-time capture and recovery

The capture and recovery of moving 3D faces in real-time involves the transfer and processing of high volumes of data. To address this, we used a special combination of hardware and software expertise. All reconstruction and analysis processing ran in parallel on a combined CPU and GPU processor.

In addition to enabling high-performance recognition capabilities on moving 3D data, this approach had the advantage of using development platforms that supported portable applications.

Facial expression modelling and classification

Facial expression recognition is commonly undertaken within the 2D imaging domain. Developments at CMV, such as Photoface and the 4D Vision project, allowed analysis of dense 3D surface information in both static and dynamic ways, respectively.

These systems employ a classification method which is both pose and illumination invariant, hence overcoming the limitations of 2D approaches.

Unlike other commonly used 3D capture techniques, photometric stereo provides dense high frequency spatial information. This captures of fine details, such as wrinkles and transient furrows.

This high density information also enables the extraction of curvature based features. Through statistical feature selection and SVM-based classification, we were able to classify facial expressions with accuracy.