Towards multimodal neural robot learning

Stefan Wermter, Cornelius Weber, Mark Elshaw, Christo Panchev, Harry R. Erwin, Friedemann Pulvermüller

Abstract: Learning by multimodal observation of vision and language offers a potentially powerful paradigm for robot learning. Recent experiments have shown that ‘mirror’ neurons are activated when an action is being performed, perceived, or verbally referred to. Different input modalities are processed by distributed cortical neuron ensembles for leg, arm and head actions. In this overview paper we consider this evidence from mirror neurons by integrating motor, vision and language representations in a learning robot.