Project Multimodal Interaction (N2)
In communication both the content and the way in which the information is exchanged are important. We don’t just respond to the words people say but also, sometimes especially, to the way they are said and the gestures and faces used. Moreover, the attitude of the receiver determines if and how we address each other. But how does this work in human-machine interaction? Can a computer determine the mood we are in today?
Excellent user experience
This observation and application of non-verbal communication by multimedia applications is the central research topic within the Multimodal Interaction project. Its goal is to realize an excellent user experience during the interaction between a person and a multimedia machine. Such an experience includes the performances (effectiveness, efficiency) as well as the perception (emotion, trust) while using multimedia.
New methods
The project develops new methods to attune this interaction to the user’s intentions and emotions. Firstly, these are methods that help determine the user’s frame of mind and intentions by looking at his speech, body posture, movement, facial expressions, and context of use. Secondly, they serve to generate feedback, in order to maintain or create the desired user condition and user actions. An example is addressing a bored person in such a way that he or she reacts in an alert and adequate way to important new information.
Applications
These methods are applied in both the cultural and professional field. Examples are VJ’s that process image and sound data during a cultural happening or operators that process this kind of data in professional control chambers. The challenge the MultimediaN project team keeps on facing is to adequately detect meaningful states and actions in various domains.
Participants
The Multimodal Interaction project worktable is organized differently from the other MultimediaN worktables. Within MultimediaN it is common for scientists, students, and market parties to work together on one work package, using several worktables for each project. The Multimodal Interaction project works with one big worktable on a project level, with several work packages lying on it. In this way, user questions can be combined into integrated solutions and all kinds of diagonal connections can be made. Two knowledge institutions take seat at the worktable, namely TU Delft and TNO. Four market parties are involved in the project: Waag Society, V2_, Vicar Vision, and LogicaCMG.
To contribute to software development, V2 brings in her MUSH-interface, with which gestures can be recognized. Waag society brings in her Keyworx and Scratchworx platforms, VicarVision provides her FaceReader software as a sensor for basic emotions and user identity on the basis of a camera image. Research by LogicaCMG on technology that is used to define locations provides the necessary input to perfect the human-machine interaction.
Learning Features (N1)
Multimodal Interaction (N2)
Ambient Multimedia Databases (N3)
Semantic Multimedia Access (N5)
Professional's Dashboard (N6)
Video At Your Fingertips (N7)
E-Culture (N9C)
PERsonal Information Services (N9MI)