Speech Transcript - A data-driven model of explanations for a chatbot that helps to practice conversation in a foreign language

0:00:16	hello everybody again and welcome to class tokenization
0:00:23	i would like to say
0:00:25	talk about the data driven model of explanation for chat about that helps to practise
0:00:31	conversation in a foreign language
0:00:32	this work has been done known as i was a at the university of maximum
0:00:36	that's why see here to those but
0:00:40	and no one with the different this situation
0:00:43	this is a different kind of data driven it differs a lot from those data
0:00:48	driven approaches that
0:00:50	the first keynote speaker at this conference presented a novel but there's still
0:00:55	level c l what we can do with the data
0:00:59	besides that was the statistical one is just approaches
0:01:03	but first let me
0:01:05	outline what
0:01:06	tends to happen
0:01:08	in the next twenty minutes at first i would like to give you a little
0:01:11	bit more background about to start itself the nets with was written in the people
0:01:15	are just that the it it's a extra lights a premium service for the participants
0:01:20	of the conference
0:01:21	and then i would just there
0:01:23	explain what y dot and the weighted that this way
0:01:27	i will present the data a more of a piece of the data and
0:01:32	just explain all the empirical findings and then we will go to the maybe more
0:01:37	interesting part
0:01:38	for you to the computational modelling to a all the race generalization of these empirical
0:01:44	findings and to the
0:01:48	to the case study psychology implementation case study i will explain why it's like a
0:01:53	and
0:01:55	then we will finish what the overview of the results of the huge to the
0:02:00	big battle field
0:02:02	where it was a time
0:02:04	is started with the
0:02:06	artificial companions would be ideal to the machine that interacts with language learners
0:02:12	just you know as it is an artificial for and to have a kind of
0:02:16	france in your instant messenger
0:02:19	it was two thousand eleven it was before the chequebook around
0:02:24	and then you just and this check whether the context into account at least and
0:02:28	just
0:02:29	right it's start talking
0:02:31	practise for language and this
0:02:33	in this and
0:02:35	the way
0:02:36	but then later i found out okay the wrong on the people what to simplify
0:02:40	things and they work in the area
0:02:42	cold computers i intelligent computer assisted language learning and so these two things are somehow
0:02:48	overlap on the intersection between those fields and
0:02:54	we you can imagine how many people from different disciplines already were very natural language
0:02:59	processing for language teaching
0:03:01	second language acquisition computer science
0:03:04	journal corpus research computational linguistics in general
0:03:08	and don't
0:03:10	on the other hand they so many publications in conversation analysis
0:03:14	which exactly focus on the learner
0:03:18	interactions between language learners one that non-native speakers nineteen speakers
0:03:23	for between only two speakers
0:03:25	and no idea you just look what the query
0:03:28	at one or
0:03:30	one and conversation analysis these buttons to than they require domain we see what
0:03:35	can what within
0:03:37	okay model
0:03:40	because
0:03:41	i had initially this idea of
0:03:44	having a machine that
0:03:45	i don't i
0:03:48	behaves like a language experts in the channel buttons it is not a teacher
0:03:53	because i
0:03:54	do you have a clue you can do about what is not exactly i was
0:03:58	not in table two
0:04:00	so top of what we loaded experiment for data collection because it didn't have any
0:04:04	idea
0:04:05	about
0:04:06	what exactly this person these operators there was a lot that's to me to behave
0:04:10	like a language expert in an informal chat
0:04:13	and i for like the dataset
0:04:15	examples of future work
0:04:17	and text
0:04:18	you can take it for free it's on the language resource a repository it's in
0:04:23	germany
0:04:24	a dataset of truman evident needed only two speaker conversations it's seventy two dialogues it's
0:04:29	about
0:04:30	for now that wasn't
0:04:31	turns
0:04:33	and
0:04:33	that was my treasure
0:04:36	so i to this data and
0:04:38	a lot okay what is the
0:04:40	i met that's of conversation analysis because i didn't have anybody this all
0:04:45	what to look for and that's what they call unmotivated looking
0:04:49	it just look at a guy without any idea what
0:04:51	will you will find
0:04:53	and then you may collections of
0:04:55	interesting sequences of typical
0:04:58	sequencers and then you try to generalize describe prototypical structure of this
0:05:04	sequences
0:05:05	and then
0:05:07	is a computer scientist
0:05:09	i then looked at these prototypes and transform them into grammars and roles
0:05:15	and
0:05:16	sometimes it was even possible to do very simple machine learning
0:05:22	and then i set up this implementation case that i is a call that case
0:05:26	study because you can take a dialogue system
0:05:29	any complexity but i two the simplest one
0:05:32	i took and ai ml-based chat but
0:05:36	that and based language understanding
0:05:37	and so how far can go
0:05:41	just to give an overview of what have found
0:05:44	there are different and directional practice
0:05:46	of how
0:05:48	participants of an interaction can orient to the air
0:05:53	linguistic identities all
0:05:56	language learners or language experts in chat
0:05:59	it includes a different forms of face working of negation where language learners six q
0:06:06	is made matrix you made excuses for they are insufficient knowledge for errors timit health
0:06:12	assessment but that was not real self assessment it was on the very beginning of
0:06:16	the interaction that was more like
0:06:17	you know fishing for compliments
0:06:19	or
0:06:21	they got brace for excellent language learners for their
0:06:24	talk one
0:06:27	during the during this data collection and then what you are far different types like
0:06:31	me to talk about language lorna learning and collaborative learning the people
0:06:35	practice
0:06:37	like in the role playing
0:06:39	i data x m situations for instance or
0:06:43	they compared grammatical systems of their native language used
0:06:48	so it was talk about the language
0:06:51	and then we have this
0:06:53	very prominent type of
0:06:55	a positioning
0:06:57	hum is not obvious are expecting some bins and somebody writes
0:07:02	a different kind of creepy a in this case it was rupiah would linguistic troubles
0:07:06	of still that this also problems
0:07:07	in all grace repair sequences was
0:07:11	because what was what's due to
0:07:14	insufficient knowledge of the foreign language
0:07:17	and the focus of this talk is marked the rat
0:07:22	both their explanations upon request is only one type of text while i'm
0:07:27	one subtype of this one type of all the
0:07:31	possible
0:07:33	incarnations of a language expert
0:07:36	and this is
0:07:37	this is the research objective of this paper so i wanted to create computational models
0:07:42	of interactional practice
0:07:44	where only two speakers of truman in chat the of what troubles in comprehension in
0:07:50	a chat but conversation for learning would native speakers
0:07:53	why conversation for learning because it was an informal chat but it wasn't this
0:07:58	yes the bit the participants met because of the they are status of native and
0:08:04	non-native speakers of the with rubber but together because they have these different statuses that's
0:08:09	why it was a conversation for learning it was not just the naughty a conversation
0:08:13	in this sense
0:08:14	why is that challenging i said in the beginning
0:08:17	i had forty five thousand about forty five thousand turns
0:08:21	and maybe you remember all of elements that in the in his you know talk
0:08:25	a eleven reappears
0:08:28	that there are challenging for speech recognition or core approximately every two and half torrance
0:08:36	i had only thirty
0:08:38	so i can i can forget all the machine learning
0:08:42	and
0:08:43	ideally an example of
0:08:46	these
0:08:47	what i five sequences
0:08:49	so that the data original data are in german let there are a translations
0:08:56	he did not need not
0:08:58	non-native speaker has the difficulty to understand or not the i do magic expression and
0:09:04	how can
0:09:06	request
0:09:07	a clarification how this clarification is formatted it's just repeat
0:09:12	all these
0:09:13	probably might think it's not what
0:09:15	there is no
0:09:16	did you mean how what is a it's just a repetition in the question mark
0:09:21	off the dock
0:09:22	and
0:09:24	this is only one
0:09:26	format of a repair initiation but there are many others
0:09:29	and then
0:09:33	after to really be initiation
0:09:36	the unknown speaker
0:09:38	provides the explanation so we it carries out the repair that the but this is
0:09:42	the prototypical structure
0:09:44	of repair sequence we have what troubles source
0:09:47	which can be everything
0:09:50	it never know what will corset
0:09:52	problem and in comprehension then there is a rip initiation which can theoretically your occur
0:09:58	everywhere even have to silence it has been shown already
0:10:02	and then it can be followed by a repair carry out but it doesn't have
0:10:05	to
0:10:07	and
0:10:08	okay the empirical part
0:10:10	would be
0:10:13	finished in this place
0:10:14	what the what i found was
0:10:17	questioning is the praxis but it was not really my finding i just conform to
0:10:21	what has been found before for oral interaction but it
0:10:25	what the same in chat
0:10:26	and
0:10:27	the right different
0:10:29	devices
0:10:31	specific the in the interaction resources
0:10:35	that we have a unit chat to signal that we have trouble
0:10:39	and there are also a specific interaction of resources
0:10:44	that well we half an hour these pet a disposal
0:10:49	two point to the trouble source also every pair initiation contains
0:10:56	kind of signal and the kind of
0:10:59	reference to the trouble source
0:11:03	only repeat initiations the time talking about l corresponding
0:11:06	to the second position
0:11:08	repair initiations
0:11:10	so it's the first structurally defined place where the other speaker can initiate but they
0:11:16	can still
0:11:18	immediate or delay because it and this is because of the of the specific
0:11:23	structure of chat because we can
0:11:25	just you know have mount multiple threads or
0:11:28	in certain things in between and but
0:11:30	that is they steal the su of all the same type of second position
0:11:35	and
0:11:35	but some of them come directly after trouble source or and some of them a
0:11:39	little bit later
0:11:40	and the this has an influence on the resources that need to be
0:11:45	employed for the area a pointing to this trouble source
0:11:51	then
0:11:52	i am used on this example
0:11:55	there was a repeat as a as a as a as a reference to the
0:11:58	trouble source used
0:11:59	but their own
0:12:01	because we have to deal with non-native speakers
0:12:03	but cannot say that
0:12:04	only
0:12:06	as syntactic i syntactic you can be repeated
0:12:09	i it regardless of unit boundaries so a piece of
0:12:14	trouble source to an can be copied and pasted
0:12:17	so we i do you cannot rely on the on the completeness of the second
0:12:22	syntactic structure
0:12:24	and then
0:12:26	what is very common for all interaction
0:12:29	but it's
0:12:31	i can find it in chat
0:12:33	that when you didn't understand something completely i just acoustically but because it's difficult to
0:12:39	follow the overall talk native speakers of mandarin native speaker sometimes
0:12:43	and
0:12:45	then i is the repair
0:12:47	and just the representation of the troubles source
0:12:50	is it is okay is acceptable you don't find it in chat case you can
0:12:53	just really the everything
0:12:56	but still i have was surprised at
0:12:59	some people really
0:13:00	i read it in the wrong way but it becomes usable not through the rip
0:13:05	in each iteration
0:13:06	but there through
0:13:08	i don't things where people try to repeat so that every time the that things
0:13:13	and you see from these retyping that they we applied
0:13:16	role labeling
0:13:19	and there are there are also
0:13:22	things that are typical for money non-native speakers
0:13:25	and if we have very much from the native speaker talk it's to the design
0:13:29	of the repair itself so it's
0:13:32	it's more about the sense of the word that it explains the meaning of the
0:13:36	word order the meaning of the of the of the use this
0:13:41	yes semantic unit
0:13:42	and their it's less it's less about it
0:13:46	something like functional or a foreign
0:13:51	the intention or something like that it not an intention but with the meaning of
0:13:54	the word was
0:13:55	repeat or explained
0:13:58	for their repair
0:14:00	carry out
0:14:01	the of the
0:14:03	participants you was used a different direction results again
0:14:07	like it just looking synonym so paraphrases
0:14:11	but sometimes they also just
0:14:13	you know use google translate
0:14:15	and translated everything in the native language of the l two learners
0:14:19	not to be added one going out to be funny or something they translated that
0:14:24	really with machine translation and that not explanation
0:14:28	and
0:14:30	or they just the arm
0:14:32	because it was difficult explain some of the phenomenon a like what is that what
0:14:37	is a
0:14:38	lapsed we it was difficult explain than words and they just
0:14:42	pasted linked one example
0:14:44	and then it was clear somehow and
0:14:51	again to the same as a rip initiation survey carry out can be delayed or
0:14:55	immediate but the same reasons
0:14:57	and we have a distinct is
0:14:59	so that it was type of repeat
0:15:02	very pi carry out here is a
0:15:05	and i so i called it's speech reap here if
0:15:08	l where
0:15:11	utterance is unclear
0:15:13	or a longer part of a longer utterance is unclear the and not
0:15:17	every word is explained somehow but
0:15:20	only something that is supposed to be difficult
0:15:24	so with that it is clear that didn't
0:15:26	units in each difficult unit is explained but not everything is rephrased all par for
0:15:33	a store
0:15:37	elaborated somehow
0:15:39	so what we need to know for the chat what's your
0:15:42	and first
0:15:43	what does the chat but
0:15:46	need to be able to
0:15:49	do the same joke was a native speaker do you hear the first to chatbot
0:15:53	needs to recognise we can initiate and then detect what is to extract a trouble
0:15:58	source and then generate a repair proper because you cannot predict
0:16:02	what it will be you cannot just used
0:16:04	scripts for ep is forty packing it needs to be generated from what linguistic database
0:16:10	maybe
0:16:10	and is what i've done so why i just used
0:16:13	dictionary
0:16:15	as the linguistic resources and a field templates with the knowledge from the dictionary
0:16:19	and the interactional resources at which my machine looked where
0:16:24	all these signals that are found in any
0:16:29	corpus and with question marks dishes and
0:16:31	a quotation marks and then lexical and things like unclear or i don't understand
0:16:37	the directional resources not allowed to print the trouble sources include repeats but also just
0:16:43	the adjacent addition because lp initiation may consist of only three question marks and then
0:16:48	only the position of this trip initiation points to the trouble source target it's exactly
0:16:53	the previous turn so these
0:16:55	but for instance this type of
0:16:59	pointing cannot be used in the delay position
0:17:04	for the implementation case study that said i used
0:17:07	and i ml based chat about it was
0:17:10	the program d its name a limb interpreter for german and their use the as
0:17:14	a baseline this german the emails that
0:17:16	we take standard by several categories allow that element is to render the
0:17:22	rip here
0:17:23	carry out
0:17:24	based on the island imaginary
0:17:26	now let's and i added to processors the processors in the in problems
0:17:31	process different tasks and i added to different tax that the law to do with
0:17:36	three pairs that was down explanation and meaning tag why this three because
0:17:44	we have a
0:17:47	two different types of questions
0:17:49	that there are kind of baseline questions to which all the rip initiations can be
0:17:54	mapped it's
0:17:55	apple are questions requiring a yes or no
0:17:59	hence there were it's a content question and out that requires an explanation like synonyms
0:18:05	of paraphrases
0:18:06	which translation and then
0:18:09	i need to distinguish between
0:18:10	two of down
0:18:12	i automatically and that's why all the all the request were mapped only two
0:18:16	to this functions and there's white
0:18:18	i had only these two processors
0:18:21	what does that mean for the linguistic knowledge that we need for not
0:18:25	it to recognize repair initiations it might be sufficient just to have this pattern based
0:18:29	language understanding
0:18:30	and
0:18:32	and determine formats that o can be used to initially a creepy a
0:18:37	can be described as patterns
0:18:41	but
0:18:42	we have still real related nlp problem sets are really hard for either princess referring
0:18:46	expression generations because our pointers to the to the trouble source
0:18:51	are referring expressions
0:18:54	but only the domain is a different one we have don't have the whole conversation
0:18:57	only in this a local rupiah domain low in the local bps sequences what we
0:19:03	need your
0:19:04	and in contrast to
0:19:07	to the other two d or overall problem of they're referring expression referring expression generation
0:19:13	there we are normally nouns and
0:19:17	pronouns i seen as the main the main results for that here we can see
0:19:22	also
0:19:23	entire sentence or sentences or phrases or works because a repetition of labor
0:19:30	points to the trouble source of them were
0:19:34	and
0:19:35	then for the
0:19:37	repair carry out
0:19:38	we can use
0:19:40	as a set their definitions paraphrasing synonyms translations and demonstrations and you know probably that
0:19:46	paraphrasing is a hard problem
0:19:49	synonyms is hard problem
0:19:50	finding it automatically
0:19:52	it's also hard to say if the if you're
0:19:56	to in a
0:19:59	confirmation in a in
0:20:03	i mean exact situation but to use it things are expressed
0:20:05	that's this one mean the same as this one it's hard to say
0:20:09	yes or no just
0:20:11	without specific resources
0:20:14	but not worse
0:20:15	low numbers
0:20:16	is not the only challenge other challenges contingency
0:20:20	so
0:20:22	utterances form as rip initiations can have also different accents on their functions like jokes
0:20:28	or error correction
0:20:30	or rejection of surprise are many others
0:20:33	and that's why
0:20:34	it remains still challenging because i don't have a solution
0:20:38	and
0:20:42	it is so i have i don't only one minute but maybe the time is
0:20:45	over again
0:20:47	so i i'm just i just a finishing we have different
0:20:53	results forget regarding the complexity of rip initiations and their repair carry out i compared
0:21:00	with literature that i us in before with work well by david
0:21:04	that's line and
0:21:06	work from conversation analysis like documents and it by the way we are this for
0:21:11	like described
0:21:13	rip initiation formants
0:21:15	across languages and their own
0:21:18	i think that it's quite
0:21:20	language-independent
0:21:21	and that's why
0:21:23	for me to the most of the most
0:21:27	and
0:21:28	positive outcome of this work was that they can use this model
0:21:32	first the cover other languages and second to cover other domains because definition talk works
0:21:38	in the same way in engineering and model and in every other domains what i
0:21:42	need to explain something
0:21:44	and then
0:21:45	a so i can go beyond duty cycle
0:21:49	application case
0:21:51	just to zooming out ic not
0:21:53	conversational this
0:21:55	method helps understand what's going on in human interaction and help to
0:21:59	ground
0:22:00	our conditional models and them into built on a
0:22:03	but we need datasets good data set of
0:22:06	good quality is really large
0:22:08	but of a specific quality
0:22:10	not
0:22:11	we take a to speech sixteen systems that we want to simulate in the and
0:22:15	so we i want to simulate a
0:22:18	dialogue between line i learner and an artificial friend i want to see first how
0:22:24	it works in a similar thing i cannot take
0:22:28	an interview for that
0:22:30	as an days
0:22:31	and the
0:22:32	maybe we can have
0:22:33	just simple chat bots is amenable waibel product in this case but
0:22:38	if you want to cover everything it's but it becomes
0:22:41	very quickly and a complete on we need all the end of knowledge that
0:22:45	well that the that people had produced you know
0:22:48	to
0:22:50	cover all the phenomena that interest
0:23:03	okay we have the two and half minutes for questions
0:23:26	so i'm also interested in computer mediated human interaction
0:23:31	and i wonder if did you serve in these interactions some kind of the interleaving
0:23:36	of comments
0:23:37	"'cause" i imagine that would be the problem with two humans having a conversation over
0:23:41	messenger rather than a human robot because they're we would be more interleaving
0:23:45	in like the manner that people do in
0:23:48	spoken conversation
0:23:50	a about fitting
0:23:52	how much interleaving is there between the utterances of your computer mediated dialogs and rt
0:24:00	similar to spoken
0:24:02	it was between can be that can be eliminated in spoken
0:24:06	is there a lot of interleaving of
0:24:10	i didn't compare datasets i only compared what i found to go to define it
0:24:14	will findings that are described in literature
0:24:17	okay
0:24:18	and the
0:24:20	the right
0:24:24	there are things that i the same
0:24:28	like
0:24:34	formats all replay initiations
0:24:38	some of them are the same as an oral interaction
0:24:40	but
0:24:42	the because we have different directional resources available in chat
0:24:48	we don't have the prosody for instance we don't have data the phase we don't
0:24:51	have the voice
0:24:52	i am that they are they are somehow replaced internet by other things like a
0:24:58	motion a multi consider and instead of laugh
0:25:03	or
0:25:04	when determining when you want to twenty participants wanted to emphasise something
0:25:09	they made uppercase a word stretches
0:25:12	or i had
0:25:14	one example
0:25:16	the data collection that took place in two thousand twelve a what is it
0:25:20	european some cocoa
0:25:22	football cup and the at this time and sometimes participants that just typed at the
0:25:26	same time or in front of their t
0:25:28	and watched again
0:25:31	incremented
0:25:32	and that's how i don't know the word german work goal for what
0:25:37	sixty two holes in the high
0:25:39	and this is really what you say well what the what a reply data and
0:25:44	then to relative to these oral while and when they screen
0:25:48	and so it's
0:25:49	i would say
0:25:52	there are the same things but the expressed by different directional resources
0:25:57	that's the first thing and the other thing is some of the things
0:26:00	cannot be replaced because they become
0:26:03	irrelevant because
0:26:05	but don't we don't have the voice for instance because that's why i didn't find
0:26:09	any repair initiation that require the repetition after that because it's not necessary you can
0:26:14	read everything but these are two differences that i would describe
0:26:20	okay so what to do this
0:26:29	one of things you just informed about the database is that it's montague that you
0:26:34	mentioned supervector doodle for one straight or something but effective some just curious with mobility
0:26:40	longitude minimum assessment luminance
0:26:43	perhaps not increase the learning used to do something like that because but also potentially
0:26:48	useful project work because you want so what kind of increasing importance density distribution of
0:26:53	the material is thanks so that evaluating was not the focus that just adding but
0:26:57	that
0:27:00	whistle normally when you talk about a talk about learning or at least
0:27:04	i'll with this is second language acquisition theory to your in the background
0:27:09	well normally people look at error corrections as a sign for learning
0:27:16	or any kind of a meeting negotiation a call it may negotiations all these repair
0:27:20	sequences that it or are we explain to date technical it meaning negotiations you know
0:27:25	what
0:27:26	and then
0:27:29	this may be costly also obvious
0:27:33	normally only these two things are an online but
0:27:38	i so also
0:27:39	the learning for all
0:27:47	i'm sorry at
0:27:49	i forgot the word
0:27:51	in this
0:27:54	but they wouldn't but there is just the
0:27:56	the null something or didn't use a structure and then
0:27:59	based on example of that and repeated that without any rate wer so that you
0:28:04	want to say that but
0:28:07	you know but
0:28:08	not observation but
0:28:10	making likely making a native speakers
0:28:13	and then and then no i found also that
0:28:17	they learn from implicit corrections which are really hard to capture which are normally not
0:28:23	use the bathroom research
0:28:25	or not the not no not the that they are not use the use of
0:28:28	the wrong word but normally people don't pay attention to that because it's not evident
0:28:33	enough it's not
0:28:35	a node in there is no evidence that people butlers notice these corrections
0:28:40	but i have evidence that
0:28:42	in the data
0:28:44	because they've repeated things that have been corrected through implicit embedded corrections later in later
0:28:50	sections for instance then repeated that's an incorrect wait for it
0:28:53	it's more than just
0:28:57	i'm afraid that drifted a little different direction
0:29:03	anyway changes over time so that the why i explain this thing with artificial companions
0:29:08	in the beginning i guess that posterior have these artificial friendship knows the user and
0:29:13	userspreferences and everything and
0:29:16	and that's why i set up to study the data collection in this way that's
0:29:19	why i'm talking about a specific speech actually systems every participant of the study was
0:29:23	put out every load it was would wherein appear within a speaker and they directed
0:29:28	in pairs for a longer time
0:29:30	and i wanted to see the development
0:29:34	and i can say
0:29:36	the development in learning was not only because they interact longer but because some of
0:29:40	them engage in these corrections and in this evident obvious selling sequences in the beginning
0:29:47	and that's why it developed somehow more intensively later and in either appears it was
0:29:53	not relevant
0:29:54	they just
0:29:56	so i don't have a so we can continue offline five minutes once it uses
0:30:01	this isn't the speaker

A data-driven model of explanations for a chatbot that helps to practice conversation in a foreign language

Second WOCHAT Special Session on Chatbots and Conversational Agents (WOCHAT-SS)

Sviatlana Höhn