An agent-based architecture for multimodal interaction

Author： Jonker C.M. Treur J. Wijngaards W.C.A.

ISSN： 1071-5819

Source： International Journal of Human-Computer Studies, Vol.54, Iss.3, 2001-03, pp. : 351-405

Disclaimer: Any content in publications that violate the sovereignty, the constitution or regulations of the PRC is not accepted or approved by CNPIEC.

Previous Menu Next

Abstract

In this paper, an executable generic process model is proposed for combined verbal and non-verbal communication processes and their interaction. The agent-based architecture can be used to create multimodal interaction. The generic process model has been designed, implemented and used to simulate different types of interaction between verbal and non-verbal communication processes: a non-verbal communication process can add and modify content to a verbal communication process, but can also provide a protocol for the (control of the) verbal communication process. With respect to the communication protocol both stimulus–response behaviour and deliberative behaviour have been modelled and simulated. The semantics of the model has been formalized by three-levelled partial temporal models, covering both the material and mental proceses and their relations.