Speechreading by Humans and Machines: Models, Systems, and Applications (Nato ASI Subseries F:, 150)

This book is one outcome of the NATO Advanced Studies Institute (ASI) Workshop, "Speechreading by Man and Machine," held at the Chateau de Bonas, Castera-Verduzan (near Auch, France) from August 28 to Septem­ ber 8, 1995 - the first interdisciplinary meeting devoted the subject of speechreading ("lipreading"). The forty-five attendees from twelve countries covered the gamut of speechreading research, from brain scans of humans processing bi-modal stimuli, to psychophysical experiments and illusions, to statistics of comprehension by the normal and deaf communities, to models of human perception, to computer vision and learning algorithms and hardware for automated speechreading machines. The first week focussed on speechreading by humans, the second week by machines, a general organization that is preserved in this volume. After the in­ evitable difficulties in clarifying language and terminology across disciplines as diverse as human neurophysiology, audiology, psychology, electrical en­ gineering, mathematics, and computer science, the participants engaged in lively discussion and debate. We think it is fair to say that there was an atmosphere of excitement and optimism for a field that is both fascinating and potentially lucrative. Of the many general results that can be taken from the workshop, two of the key ones are these: • The ways in which humans employ visual image for speech recogni­ tion are manifold and complex, and depend upon the talker-perceiver pair, severity and age of onset of any hearing loss, whether the topic of conversation is known or unknown, the level of noise, and so forth. Front Matter....Pages I-XV Front Matter....Pages 1-1 Psychology of Human Speechreading....Pages 3-15 Word Recognition in Speechreading....Pages 17-26 Children with Hearing Loss: Speechreading Skills....Pages 27-41 Differences in Visual Intelligibility Across Talkers....Pages 43-53 The Use of Auditory and Visual Information in Phonetic Perception....Pages 55-77 Bimodal Speech Perception: A Progress Report....Pages 79-101 Auditory-Visual Speech Perception as a Direct Process: The McGurk Effect in Infants and Across Languages....Pages 103-114 Seeing Brains Reading Speech: A Review and Speculations....Pages 115-133 Perception of Conflicting Audio-Visual Speech: an Examination across Spanish and German....Pages 135-143 Audio-Visual Speech Perception Without Speech Cues: A First Report....Pages 145-151 Perception of Synthetic Visual Speech....Pages 153-168 Homopheneity in speechreading: Effects of phonemic equivalence classes on the structure of the lexicon....Pages 169-178 Aspects of Modality in Audio-Visual Processes....Pages 179-191 Exploiting sensor fusion architectures and stimuli complementarity in AV speech recognition....Pages 193-210 Does movement on the lips mean movement in the mind?....Pages 211-219 The Dynamics of Audiovisual Behavior in Speech....Pages 221-232 Where and When are the Heard and Seen Speech Integrated: Magnetoencephalographical (MEG) Studies....Pages 233-238 Multiphasic Analysis of the Basic Nature of Speechreading....Pages 239-246 How can coarticulation models account for speech sensitivity to audio-visual desynchronization?....Pages 247-255 Working Memory and Speechreading....Pages 257-274 Front Matter....Pages 1-1 Encoding of Visual Speaker Attributes and Recognition Memory for Spoken Words....Pages 275-281 A Study of the Semantic Memory Access by Perceptual Modalities with a Semantic Priming Experiment....Pages 283-290 Lips and Jaw Movements for Vowels and Consonants: Spatio-Temporal Characteristics and Bimodal Recognition Applications....Pages 291-313 Which components of the face do humans and machines best speechread?....Pages 315-328 Front Matter....Pages 329-329 Visionary Speech: Looking Ahead to Practical Speechreading Systems....Pages 331-349 Talking Heads and Speech Recognisers That Can See: The Computer Processing of Visual Speech Signals....Pages 351-371 Automatic Speechreading using dynamic contours....Pages 373-382 Active Shape Models for Visual Speech Feature Extraction....Pages 383-390 2D Deformable Models for Visual Speech Analysis....Pages 391-398 Fast Matching of a Dynamic Lip Model to Color Video Sequences under Regular Illumination Conditions....Pages 399-407 Towards a Robust Speechreading Dialog System....Pages 409-423 Robust Face Feature Analysis for Automatic Speachreading and Character Animation....Pages 425-436 Time Delay Neural Networks for Articulatory Estimation from Speech: Suitable Subjective Evaluation Protocols....Pages 437-444 Relations of Audio and Visual Speech Signals in a Physical Feature Space: Implications for the Hearing-impaired....Pages 445-460 On the Integration of Auditory and Visual Parameters in an HMM-based ASR....Pages 461-471 Channel Separability in the Audio-Visual Integration of Speech: A Bayesian Approach....Pages 473-487 Audiovisual Sensory Integration Using Hidden Markov Models....Pages 489-496 Neural-fuzzy networks and phonetic feature recognition as a help for speechreading....Pages 497-504 Rationale for Phoneme-Viseme Mapping and Feature Selection in Visual Speech Recognition....Pages 505-515 Front Matter....Pages 517-517 Human Speechreading: Learning and Psychophysics....Pages 519-523 Front Matter....Pages 517-517 Human Speechreading: Psychology and Cognition....Pages 525-531 Sensory Integration by Humans and Machines....Pages 533-540 Databases, Standards and Comparisons....Pages 541-548 Machine Recognition and Applications....Pages 549-555 Back Matter....Pages 557-695

This is the first comprehensive book on the interdisciplinary study of speechreading (lipreading) — production, perception and learning — by both humans and machines. It is the product of the largest-ever gathering of scientists devoted to the subject, at a NATO workshop held in France in 1995. The research presented explores and promotes the incorporation of visual information into automatic speech recognizers for improved recognition accuracy (especially in noisy environments), and draws on and further elucidates knowledge of the psychology of speechreading by humans. The book is a milestone along the path to more accurate speech based interfaces, as well as to deeper understanding of fundamental mechanisms of human perception and speech recognition.

Proceedings of the NATO Advanced Study Institute on Speechreading by Man and Machine, held in Castera-Verzudan, France, August 28 - September 8, 1995