Cooperative Multimodal Communication: Second International Conference, CMC'98, Tilburg, The Netherlands, January 28-30, 1998. Selected Papers (Lecture Notes in Computer Science, 2155)

This book constitutes the thoroughly refereed post-proceedings of the Second International Conference on Cooperative Multimodal Communication, CMC'98, held in Tilburg, The Netherlands, in January 1998. The 13 revised full papers presented together with an introductory survey by the volume editors have passed through two rounds of reviewing, selection, and revision. The book offers topical sections on multimodal generation, multimodal cooperation, multimodal interpretation, and multimedia platforms and test environments. Cooperative Multimodal Communication Preface Multimodal Cooperative Communication 1 Introduction 2 Multimodal Generation, Interpretation, Collaboration, and System Design 2.1 Multimodal Presentation and Generation 2.2 Multimodal Cooperation 2.3 Multimodal Interpretation 2.4 Multimodal Platforms and Test Envirinments References Generating Textual Diagrams and Diagrammatic Texts 1 Introduction 2 Text and Diagrams 2.1 Diagrammatical Features within Texts 2.2 Text within Diagrams 3 Abstract Document Structure 3.1 Advanced Layout 4 Applications 5 Summary and Conclusions References PEDRO: Assessing Presentation Decodability on the Basis of Empirically Validated Models 1 Introduction 2 Approach 3 Example: Resolution of Pictorial Referring Expressions 4 Empirical Investigation of the Postulated Model 4.1 Hypotheses 4.2 Method 4.3 Results 4.4 Discussion 5 Conclusion References IMPROVISE: Automated Generation of Animated Graphics for Coordinated Multimedia Presentations 1 Introduction 2 Related Work 3 System Architecture 4 Examples 4.1 Presenting Information to a Nurse 4.2 Exploring a Computer Network 5 Designing Multimedia Presentations 5.1 Data Representation 5.2 Visual Techniques 5.3 Action-Based Inference Engine 5.4 Portable Visual Realizer 6 Conclusion and Future Work References Multimodal Reference to Objects: An Empirical Approach Introduction Form and Content of Referential Acts Descriptive Content Pragmatic Principles and Cognitive Concepts in Object Reference The Principle of Minimal Cooperative Effort Focus of Attention Features in the Description Reduced Information Empirical Setup Domain Properties and Definitions Inherent Salience The Focus Area Results General Observations The Influence of Changing the Focus Area Discussion Salience Redundancy of Information Descriptive Features Focus and the Principle of Connectivity Limitations Conclusions References Augmenting and Executing SharedPlans for Multimodal Communication 1 Collaborating on the Interface 2 Grosz and Sidner's Theory of Discourse Structure 2.1 The SharedPlan Theory 2.2 Modeling the Intentional Structure 3 Augmenting and Executing a SharedPlan 4 Adopting a Modified Adjacency Pairs Model 5 The Augmentation-Execution Process 5.1 The Augmentation-Execution Process I: Interpretation 5.2 The Augmentation-Execution Process II: Generation 6 The Multimedia Case 7 Conclusions References Cooperation and Flexibility in Multimodal Communication 1 Introduction 2 Cooperation in Communication 3 Multimodal Coomunication 4 Cooperation and Mutual Flexibility 5 Means of Achieving Mutual Flexibility and Preventing Conflict 5.1 Verbal Means 5.2 Nonverbal Gestural Means 6 Concluding Remarks References Communication and Manipulation Acts in a Collaborative Dialogue Model 1 Introduction 2 Dialogue Model 2.1 Underlying Theory 2.2 Architecture of the System 3 Cooperation 3.1 Cooperative Recipes 3.2 Generating the System's Utterances 4 Conclusion References Relating Imperatives to Action 1 Introduction 2 The Puzzle of Logically Complex Imperatives 3 Imperatives in Context: An Agent-Based Analysis 4 Conclusions and Further Issues References Interpretation of Gestures and Speech: A Practical Approach to Multimodal Communication 1 Introduction 2 Definition of an Interaction Style 2.1 Objectives and Requirements 2.2 General Requirements of an Interaction Style 2.3 Guidelines for the Definition of an Interaction Style 3 Characteristics of Gestures and Speech 3.1 Use of Gesture 3.2 Use of Speech 4 Integrated Interpretation of Gestures and Speech 4.1 Principles of the Model 4.2 Examples 5 Characteristics of the Dialogue System 5.1 Dialogue Mechanisms: Robustness of the System 5.2 Handling of Failures and Error Messages 6 Concluding Remarks References Why Are Multimodal Systems so Difficult to Build? - About the Difference between Deictic Gestures and Direct Manipulation 1 Introduction 2 Paradigms of Multimodal Interaction 2.1 The Direct Manipulation Based Approach to Multimodal Interaction 2.2 The NLP-Oriented Approach to Multimodal Interaction 3 The Multimodal Systems Mofa and Talky 4 Problems with the Integration of Speech and Gestures 4.1 Synchronisation of Speech and Gestures 4.2 Referring and Manipulative Functions of Gestures 5 Basic Gesture Types 5.1 Gesture Types and Gesture Forms 5.2 Type-Specific Problems 5.3 Passively Used Action Forms: Gaps and the Role of Feedback 6 Multimodal Settings for Different Gesture Forms 6.1 Click-Free Mouse Pointing 6.2 Click-Free Mose Pointing with Mouse-Wait 6.3 Pointing with Mouse Clicks 6.4 Pointing with Capacitive Touch Screen 6.5 Visual Gesture Recognition 6.6 Speech Recognition and Natural Language Processing 6.7 Dependencies between the Use of Gsetures and the Structure of Related Verbal Expressions 7 Active and Passive Gesture Forms as a Solution for the Wait Problem 8 Active and Passive Objects 8.1 Actively Used Passive Objects 8.2 Passively Used Active Objects 9 Focusing Gestures 9.1 Feedback and Focusing Gestures 9.2 Indistinguishability of Deictic Gestures and Focusing Gestures 10 Conclusion References Multimodal Cooperative Resolution of Referential Expressions in the DenK System 1 Introduction 2 Contexts in Dialogue 3 Multimodal Cooperative Reference Resolution 4 Evaluation 5 Related Research 6 Concluding Remarks References The IntelliMedia WorkBench - An Environment for Building Multimodal Systems Introduction CHAMELEON and the IntelliMedia WorkBench IntelliMedia WorkBench Sample Interaction Dialogue Architecture of CHAMELEON DACS Relation to Other Work Conclusion and Future Work References Appendix A Appendix B A Unified Framework for Constructing Multimodal Experiments and Applications 1 Introduction 2 A Fully-Automated Multimodal Map Application 2.1 Description 2.2 Implementation 3 A Hybrid Approach: The WOZZOW Experiment 4 Conclusion and Future Work References Index Author Index The Chapters In This Book Are Revised, Updated, And Edited Versions Of 13 Selected Papers From The Second International Conference On Cooperative Multimodal Communication (cmc’98), Held In Tilburg, The Netherlands, In 1998.this Wasthesecondconferencein A Series,ofwhichthe ?rstonewasheld Ineindhoven,thenetherlands,in1995.threeofthesepaperswerepresentedby Invitedspeakers;thosebydoniascott(co-authoredwithrichardpower),steven Feiner (co-authored With Michele Zhou), And Oliviero Stock (co-authored With Carlo Strapparava And Massimo Zancanaro). The Other Ten Were Among The Submitted Papers That Were Accepted By The Cmc’98 Program Committee. The Editors Contributed An Introductory Chapter To Set The Stage For The Rest Of The Book. We Thank The Programcommittee For Their Excellent And Timely Feedback To The Authors Of The Submitted Papers, And At A Later Stage For Advising On The Contents Of This Volume And For Providing Additional Suggestions For Improving Theselectedcontributions.theprogramcommittee Consistedofnicholasasher, Normannbadler,donbouwhuis,harrybunt,walthervonhahn,dieterhuber, Hans Kamp, John Lee, Joseph Mariani, Jean-claude Martin, Mark Maybury, Paulmckevitt, Robnederpelt, Keesvanoverveld,rayperrault,donia Scott, Jan Treur, Wolfgang Wahlster, Bonnie Webber, Kent Wittenburg, And Henk Zeevat. Wethanktheroyaldutchacademyofsciences(knaw)andtheorga- Zationforcooperationamonguniversitiesinbrabant(sobu)fortheirgrants That Supported The Conference. Multimodal Cooperative Communication -- Multimodal Cooperative Communication -- 1: Multimodal Generation -- Generating Textual Diagrams And Diagrammatic Texts -- Pedro: Assessing Presentation Decodability On The Basis Of Empirically Validated Models -- Improvise: Automated Generation Of Animated Graphics For Coordinated Multimedia Presentations -- Multimodal Reference To Objects: An Empirical Approach -- 2: Multimodal Cooperation -- Augmenting And Executing Sharedplans For Multimodal Communication -- Cooperation And Flexibility In Multimodal Communication -- Communication And Manipulation Acts In A Collaborative Dialogue Model -- Relating Imperatives To Action -- 3: Multimodal Interpretation -- Interpretation Of Gestures And Speech: A Practical Approach To Multimodal Communication -- Why Are Multimodal Systems So Difficult To Build? - About The Difference Between Deictic Gestures And Direct Manipulation -- Multimodal Cooperative Resolution Of Referential Expressions In The Denk System -- 4: Multimedia Platforms And Test Environments -- The Intellimedia Workbench-an Environment For Building Multimodal Systems -- A Unified Framework For Constructing Multimodal Experiments And Applications. Harry Bunt, Robbert-jan Beun (eds.). Includes Bibliographical References And Index. This work constitutes the refereed post-proceedings of the Second International Conference on Cooperative Multimodal Communication, 1998. It covers topical sections on multimodal generation, multimodal co-operation, multimodal interpretation, and multimedia platforms and test environments.