SIGdial 2016

17th Annual SIGdial Meeting on Discourse and Dialogue

Training an adaptive dialogue policy for interactive learning of visually grounded word meanings

Yanchao Yu, Arash Eshghi and Oliver Lemon
We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/generation frame-work - Dynamic Syntax and Type Theory with Records (DS-TTR) - with a set of visual classifiers that are learned through-out the interaction and which ground the meaning representations that it produces. We use this system in interaction with a simulated human tutor to study the effects of different dialogue policies and capabilities on accuracy of learned meanings, learning rates, and efforts/costs to the tutor. We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues; (2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical and incrementally constructed dialogue turns. Ultimately, we train an adaptive dialogue policy which optimises the trade-off between classifier accuracy and tutoring costs.