Multimodal Dialogue in Human-Robot-Interaction - Einzelansicht

"Alexa, can you help me assemble this shelf?" Interactions like these 
are still far from the scientific reality and that's not merely because 
robotic hardware development is still lacking behind. The amount of
reasoning and human-robot cooperation that is required to solve this 
task is complex: The robot needs to understand what a shelf is, what 
needs to be done to assemble one, observe what the human is doing and 
infer what actions could best support the human in the given task. The 
aim of this project is to get an understanding of the complexity of such 
human-agent collaboration from a practical and theoretical perspective 
by discussing the relevant literature as well as implementing a 
well-constrained part of the general assembly-task described above.

During the project, you will develop a multimodal and incremental 
dialogue system for an industrial robot arm. The task for the robot arm 
is to help a human with solving a Pentomino puzzle. Doing so requires 
the robot to observe a visual scene, identify puzzle pieces by shape, 
color and location in the scene, and hand over the correct puzzle piece 
verbally described by the human interaction partner. The project work 
will be performed in teams and the focus of each project will be 
determined in the beginning of the course. While the teams can work 
individually, the final goal is to integrate the pieces and prepare a 
full dialogue pipeline that allows a human to play the Pentomino-game 
together with our physical robot arm. The practical project work will be 
enhanced by a series of seminar talks and paper discussions to get a 
better understanding of the fundamentals of embodied interaction.