Robust Instruction Following via Deep Reinforcement Learning

Project Abstract/Statement of Work:

Humans will interact with home robots at least in part through natural language instructions. At present robust instruction following by robots is not achievable. We propose innovations in DeepRL (combination of Deep Learning and Reinforcement Learning) to address the following challenges: 1) generalization of training on verb object-location instructions, e.g, pick up box, bring me a pencil from the living room, etc., to unseen combinations of verb-object-location pairings in test instructions, 2) generalization of training on tasks composed of sub-task sequences to tasks composed of unseen subtasks, 3) automatic hierarchical decomposition of high-level and complex task instruction into previously trained and untrained subtasks with verbal explanation of subtask goals to user for confirmation and feedback, 4) dialog with user for (sub)task clarification when needed/useful.