Spoken, Multilingual and Multimodal Dialogue

Dialogue systems are a very appealing technology with an extraordinary future. Spoken, Multilingual and Multimodal Dialogues Systems: Development and Assessment addresses the great demand for information about the development of advanced dialogue systems combining speech with other modalities under a multilingual framework. It aims to give a systematic overview of dialogue systems and recent advances in the practical application of spoken dialogue systems. Spoken Dialogue Systems are computer-based systems developed to provide information and carry out simple tasks using speech as the interaction mode. Examples include travel information and reservation, weather forecast information, directory information and product order. Multimodal Dialogue Systems aim to overcome the limitations of spoken dialogue systems which use speech as the only communication means, while Multilingual Systems allow interaction with users that speak different languages. Presents a clear snapshot of the structure of a standard dialogue system, by addressing its key components in the context of multilingual and multimodal interaction and the assessment of spoken, multilingual and multimodal systems In addition to the fundamentals of the technologies employed, the development and evaluation of these systems are described Highlights recent advances in the practical application of spoken dialogue systems This comprehensive overview is a must for graduate students and academics in the fields of speech recognition, speech synthesis, speech processing, language, and human-computer interaction technolgy. It will also prove to be a valuable resource to system developers working in these areas.

Ramón López-Cózar Delgado is Associate Professor in the Department of Languages and Computer Systems, Computer Science Faculty, Granada University, Spain.  He has published over 30 papers in international journals and conferences concerned with dialogue systems.  He is member of the COST Action 278 “Spoken Language Interaction in Telecommunications” Masahiro Araki is an Associate Professor at Department of Electronics and Information Science, Kyoto Institute of Technology. His current interests are spoken dialogue processing and artificial intelligence.  He is a member of ACL and ISCA. João P. Neto is Assistant Professor at Instituto Superior Técnico (IST), Technical University of Lisbon in signal theory, discrete signal processing, control systems and neural networks. His research interests focus on spoken, multimodal and multilingual dialogue systems, speech recognition and understanding, dialogue management and speech synthesis.

Preface

ix

1 Introduction to Dialogue Systems

1

(15)

1.1 Human-Computer Interaction and Speech Processing

1

(1)

1.2 Spoken Dialogue Systems

2

(2)

1.2.1 Technological Precedents

3

(1)

1.3 Multimodal Dialogue Systems

4

(3)

1.4 Multilingual Dialogue Systems

7

(1)

1.5 Dialogue Systems Referenced in This Book

7

(4)

1.6 Area Organisation and Research Directions

11

(2)

1.7 Overview of the Book

13

(2)

1.8 Further Reading

15

(1)

2 Technologies Employed to Set Up Dialogue Systems

16

(38)

2.1 Input Interface

16

(18)

2.1.1 Automatic Speech Recognition

17

(5)

2.1.2 Natural Language Processing

22

(2)

2.1.3 Face Localisation and Tracking

24

(2)

2.1.4 Gaze Tracking

26

(2)

2.1.5 Lip-reading Recognition

28

(2)

2.1.6 Gesture Recognition

30

(3)

2.1.7 Handwriting Recognition

33

(1)

2.2 Multimodal Processing

34

(10)

2.2.1 Multimodal Data Fusion

34

(2)

2.2.2 Multimodal Data Storage

36

(5)

2.2.3 Dialogue Management

41

(1)

2.2.4 Task Module

41

(1)

2.2.5 Database Module

42

(1)

2.2.6 Response Generation

43

(1)

2.3 Output Interface

44

(7)

2.3.1 Graphic Generation

44

(3)

2.3.2 Natural Language Generation

47

(1)

2.3.3 Speech Synthesis

48

(3)

2.3.4 Sound Generation

51

(1)

2.3.5 Tactile/Haptic Generation

51

(1)

2.4 Summary

51

(2)

2.5 Further Reading

53

(1)

3 Multimodal Dialogue Systems

54

(32)

3.1 Benefits of Multimodal Interaction

54

(5)

3.1.1 In Terms of System Input

54

(2)

3.1.2 In Terms of System Processing

56

(2)

3.1.3 In Terms of System Output

58

(1)

3.2 Development of Multimodal Dialogue Systems

59

(25)

3.2.1 Development Techniques

59

(4)

3.2.2 Data Fusion

63

(4)

3.2.3 Architectures of Multimodal Systems

67

(3)

3.2.4 Animated Agents

70

(9)

3.2.5 Research Trends

79

(5)

3.3 Summary

84

(1)

3.4 Further Reading

85

(1)

4 Multilingual Dialogue Systems

86

(32)

4.1 Implications of Multilinguality in the Architecture of Dialogue Systems

86

(9)

4.1.1 Consideration of Alternatives in Multilingual Dialogue Systems

86

(5)

4.1.2 Interlingua Approach

91

(1)

4.1.3 Semantic Frame Conversion Approach

92

(2)

4.1.4 Dialogue-Control Centred Approach

94

(1)

4.2 Multilingual Dialogue Systems Based on Interlingua

95

(12)

4.2.1 MIT Voyager System

95

(3)

4.2.2 MIT Jupiter System

98

(2)

4.2.3 KIT System

100

(7)

4.3 Multilingual Dialogue Systems Based on Web Applications

107

(10)

4.3.1 Requirements for Practical Multilingual Dialogue Systems

108

(1)

4.3.2 Dialogue Systems Based on Web Applications

108

(3)

4.3.3 Multilingual Dialogue Systems Based on the MVC Framework

111

(3)

4.3.4 Implementation of Multilingual Voice Portals

114

(3)

4.4 Summary

117

(1)

4.5 Further Reading

117

(1)

5 Dialogue Annotation, Modelling and Management

118

(33)

5.1 Dialogue Annotation

118

(6)

5.1.1 Annotation of Spoken Dialogue Corpora

118

(3)

5.1.2 Annotation of Multimodal Dialogue Corpora

121

(3)

5.2 Dialogue Modelling

124

(3)

5.2.1 State-Transition Networks

125

(1)

5.2.2 Plans

126

(1)

5.3 Dialogue Management

127

(4)

5.3.1 Interaction Strategies

127

(1)

5.3.2 Confirmation Strategies

128

(3)

5.4 Implications of Multimodality in the Dialogue Management

131

(10)

5.4.1 Interaction Complexity

131

(2)

5.4.2 Confirmations

133

(1)

5.4.3 Social and Emotional Dialogue

134

(1)

5.4.4 Contextual Information

135

(2)

5.4.5 User References

137

(3)

5.4.6 Response Generation

140

(1)

5.5 Implications of Multilinguality in the Dialogue Management

141

(3)

5.5.1 Reference Resolution in Multilingual Dialogue Systems

141

(1)

5.5.2 Ambiguity of Speech Acts in Multilingual Dialogue Systems

142

(1)

5.5.3 Differences in the Interactive Behaviour of Multilingual Dialogue Systems

143

(1)

5.6 Implications of Task Independency in the Dialogue Management

144

(5)

5.6.1 Dialogue Task Classification

144

(2)

5.6.2 Task Modification in Each Task Class

146

(3)

5.7 Summary

149

(1)

5.8 Further Reading

150

(1)

6 Development Tools

151

(38)

6.1 Tools for Spoken and Multilingual Dialogue Systems

151

(25)

6.1.1 Tools to Develop System Modules

151

(8)

6.1.2 Web-Oriented Standards and Tools for Spoken Dialogue Systems

159

(11)

6.1.3 Internet Portals

170

(6)

6.2 Standards and Tools for Multimodal Dialogue Systems

176

(11)

6.2.1 Web-Oriented Multimodal Dialogue

176

(3)

6.2.2 Face and Body Animation

179

(2)

6.2.3 System Development Tools

181

(4)

6.2.4 Multimodal Annotation Tools

185

(2)

6.3 Summary

187

(1)

6.4 Further Reading

187

(2)

7 Assessment

189

(30)

7.1 Overview of Evaluation Techniques

189

(3)

7.1.1 Classification of Evaluation Techniques

190

(2)

7.2 Evaluation of Spoken and Multilingual Dialogue Systems

192

(10)

7.2.1 Subsystem-Level Evaluation

192

(4)

7.2.2 End-to-End Evaluation

196

(1)

7.2.3 Dialogue Processing Evaluation

197

(2)

7.2.4 System-to-System Automatic Evaluation

199

(3)

7.3 Evaluation of Multimodal Dialogue Systems

202

(15)

7.3.1 System-Level Evaluation

203

(5)

7.3.2 Subsystem-Level Evaluation

208

(2)

7.3.3 Evaluation of Multimodal Data Fusion

210

(2)

7.3.4 Evaluation of Animated Agents

212

(5)

7.4 Summary

217

(1)

7.5 Further Reading

218

(1)

Appendix A Basic Tutorial on VoiceXML

219

(10)

Appendix B Multimodal Databases

229

(4)

Appendix C Coding Schemes for Multimodal Resources

233

(2)

Appendix D URLs of Interest

235

(2)

Appendix E List of Abbreviations

237

(2)

References

239

(14)

Index

253

What is included with this book?

The New copy of this book will include any supplemental materials advertised. Please check the title of the book to determine if it should include any access cards, study guides, lab manuals, CDs, etc.

The Used, Rental and eBook copies of this book are not guaranteed to include any supplemental materials. Typically, only the book itself is included. This is true even if the title states it includes any access cards, study guides, lab manuals, CDs, etc.

FREE SHIPPING

Amazon no longer offers textbook rentals. We do!

Spoken, Multilingual and Multimodal Dialogue Systems Development and Assessment

Summary

Author Biography

Table of Contents

Supplemental Materials