INRIA Nancy Grand-Est (Nancy, France) - Speech Group, LORIA

Postdoctoral position

Accurate 3D Lip modeling and control in the context of animating a 3D talking head

Scientific Context

The lips play a significant role in audiovisual human communication. Several studies showed the

important contribution of the lips to the intelligibility of visual speech (Sumby & Pollack, 1954; Cohen

& Massaro 1990). In fact, it has been shown that human lips alone carry more than half the visual

information provided by the face (Benoît,1996). Since the beginning of the development of 3D virtual

talking heads, researchers showed interest to model lips (Guiard-Marigny et al., 1996, Reveret &

Benoît, 1998), as the lips increase intelligibility of the visual message. The existing models are still

considered as pure parametric and numerical models and do not take into account the dynamic

characteristic of speech. As audiovisual speech is highly dynamics, we consider that modeling this

aspect is crucial to provide a lip model that is accurately animated, and reflects the real articulatory

dynamics as observed in human vocal tract. In fact, the movement of the lips, even subtle, can

communicate relevant information to the human receiver. This is even more crucial for some

population such as hard-of-hearing people.

Missions

The goal of this work is to develop an accurate 3D lip model that can be integrated within a talking

head. A control model will also be developed. The lip model should be as accurate dynamically as

possible. When designing this model, the focus will be on the dynamics. For this reason, one can

start from a static 3D lip mesh, using a generic 3D lip model, and then we will use MRI images or 3D

scans to obtain more realistic shape of the lips. To take into account the dynamic aspect of the lip

deformation, we will use an articulograph (EMA) and motion capture technique to track sensors or

markers on the lips. The mesh will be adapted to this data. To control the lips, we will consider

allowing a skeletal animation to be controlled by the EMA sensors or motion capture markers, using

inverse kinematic technique, widely used in 3D modeling. In line with conventional skeletal animation,

an articulated armature rigged inside the mesh is mapped to vertex groups on the lip mesh by a

weight map that can be defined automatically from the envelope of the armature's shape and

manually adjusted if required, where manipulating the armature's components deforms the

surrounding mesh accordingly. The main challenge is to find the best topology of the sensors or

markers on the lips, to be able to better capture accurately its dynamics. The main outcome is to

accurately model and animate the lips based on articulatory data. It is very important to have

readable lips in that can be lip-read by hard-of-hearing people.

Bibliography

C. Benoît (1996). On the Production and the Perception of Audio-Visual Speech by Man and Machine.

Multimedia Communications and Video Coding, pp 277-284.

M. M. Cohen & D. W. Massaro (1990), Synthesis of visible speech. Behavioral Research Methods and

Instrumentation, 22, 260-263.

T. Guiard-Marigny, N. Tsingos, A. Adjoudani, C. Benoit, M.-P. Cani (1996). 3D Models of the Lips for Realistic

Speech Animation. Computer Animation 80-89

L. Reveret, C. Benoit (1998). A New 3D Lip Model for Analysis and Synthesis of Lip Motion in Speech

Production. Proc. AVSP'98, Terrigal, Australia, Dec. 4-6, 1998.

Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of Acoustic

Society of America, 26, 212-215.

Q. Summerfield (1987), 'Some preliminaries to a comprehensive account of audio-visual speech perception', In:

B. Dodd and R. Campbell, Editors, Hearing by Eye: The Psychology of Lip-Reading, Lawrence Erlbaum,

Hillsdale, NJ.

Competences

Required qualification: PhD in computer science Appropriate candidate would have good knowledge

in 3D modeling, speech processing and data analysis, as well as solid java programming skills.

Additional Information

Application deadline: 11 June 2013

Supervision and contact:

Slim Ouni ( Slim.Ouni@loria.fr ) http://www.loria.fr/~slim

Duration:

1 year (possibly extendable)

Starting date:

between Sept. 1st 2013 and Jan. 1st 2014

Salary:

2.620 euros gross monthly (about 2.135 euros net) medical insurance included.

Application Procedure

The required documents for an INRIA postdoc application are the following:

- CV, including a description of your research activities (2 pages max) and a short description of

what you consider to be your best contributions and why (1 page max and 3 contributions max); the

contributions could be theoretical, implementation, or industry transfers. Include also a brief

description of your scientific and career projects.

- The report(s) from your PhD external reviewer(s), if applicable.

- If you haven't defended yet, the list of expected members of your PhD committee (if known) and the

expected date of defense (the defense, not the manuscript submission).

- Your best publications, up to 3.

- At least one recommendation letter from your PhD advisor, and possibly up to two other letters. The

recommendation letter(s) should be sent directly by their author to the prospective postdoc advisor

All these documents should be sent

before June 11th

About INRIA

Established in 1967, Inria is the only public research body fully dedicated to computational sciences.

Combining computer sciences with mathematics, Inria’s 3,400 researchers strive to invent the digital

technologies of the future. Educated at leading international universities, they creatively integrate

basic research with applied research and dedicate themselves to solving real problems, collaborating

with the main players in public and private research in France and abroad and transferring the fruits of

their work to innovative companies. The researchers at Inria published over 4,800 articles in 2010.

They are behind over 270 active patents and 105 start-ups. The 171 project teams are distributed in

eight research centers located throughout France.

http://www.inria.fr/en/centre/nancy