0:00:18and the last talk of the social beyond characterizing the response place of questions
0:00:23a corpus of us english and polish
0:00:48in some sense
0:00:49all in the sense of a competition coherence
0:00:53which is amplified here in this example one
0:00:57the on the on the first line of answers are kind of relevant
0:01:01is that chain you yes it's a louis fourteen replica you
0:01:06and second line is i
0:01:08the ones that are not appropriate
0:01:10so this notion is the cornerstone of fears of dialog the same way that say
0:01:15grammatic allergies to syntax
0:01:18and you could argue that basically what the during test is about is exactly relating
0:01:25to relevance and whether that the that that's a good test for
0:01:29you know when we managed to simulate a human intelligence
0:01:34so i'm gonna a restrict attention to data corpus study relevance relating to queries
0:01:41possible responses to queries
0:01:44a bit surprising they perhaps
0:01:47even if you restricted in this way the been actually very few
0:01:50comprehensive attempts to characterize it
0:01:53the some references in a paper and also in some early work that we done
0:01:56the that a talk about moment
0:01:59so the some
0:02:04discussion of this in the language acquisition a literature
0:02:07and some discussion this in some conversational analysis
0:02:13which is primarily that it to show that there's a at a difference between
0:02:18three classes on so as not know announces a non responses
0:02:24so in this study were based i've as it looked at n different languages and
0:02:29show incident gone distributions and
0:02:31we also flying i'm not the paper they won't korean the child quite different distributions
0:02:36between say you the results that style but i don't english and the results and
0:02:39korean but mainly about this
0:02:41basically about this these three classes on says non ounces and a
0:02:46a non responses
0:02:49so today i'm gonna talk about starting by taxonomy that we developed for just characterising
0:02:55query responses appear responses to queries
0:02:59something that's sporty characteristic to certain race
0:03:06we then will mention of a basic hypothesis that we used to just scale up
0:03:10to the general case
0:03:13talk a bit about annotation scheme in the results
0:03:15and very briefly
0:03:17talk about how one might model relevance and what the ready complications then hence
0:03:24so the starting point of this work in the typology that we developed in the
0:03:29top level of work of skin myself developed in some work that was published in
0:03:36in two thousand sixteen in the journal of language modeling
0:03:40and this is a wide-coverage taxonomy for question-question a sequence is was tested on
0:03:46the bnc child's
0:03:50the be corpus mammoths corpus
0:03:52and there was also their formal modelling of the resulting classes in the framework of
0:03:59our costs t l
0:04:04this study consisted of about fifteen hundred slightly less than computing how to query a
0:04:10response to as
0:04:12and what are merged with seven classes of questions what are called elegy classes for
0:04:16all not all corinne sponsor but for the two people those one
0:04:21in this study
0:04:22so we have yet clarification requests
0:04:25things like hamlet as a response to what time it about
0:04:29depending questions
0:04:31so these are things like does anybody want to one m spread a given the
0:04:35where you can do the inference that one question depends on the other whether anybody
0:04:40wants to ban and strive depends on whether you gonna give it away
0:04:45a classical motive which questions about underlying motivation what's the matter why
0:04:52for class
0:04:54whether responses and the changing the topic
0:04:56well as you on so always yours
0:04:59a fixed cost questions the duck a wet whether the you're trying to understand what
0:05:05which way you're supposed on sit you know what makes black coffee is
0:05:09which country
0:05:13and the final to one is questions with the presuppose on so whether
0:05:17question-response is somehow
0:05:19indirectly in indicating on the to the first question
0:05:22and the seven cases where
0:05:25response ignores initial question but still addresses the same situations of things like
0:05:31do you go you wanna go down have a look at that now what is
0:05:33what when there and the response is why haven't they finished yet white with of
0:05:37a is about the workmen so it still about the same situation but it's not
0:05:40at all responding about the to the question
0:05:44so that was that those of the seven classes we found the i need to
0:05:51cool question response to questions which is about twenty percent of all at least at
0:05:56the time we found was about twenty percent of all
0:05:58responses to questions
0:06:00and a main above hypothesis for this study
0:06:03e is
0:06:05that responses drawn from all concerning these class of questions
0:06:10plus direct indirect on the food
0:06:13that's going to exhaust the response space of a query
0:06:19so basically
0:06:20you get the following kind of scheme
0:06:22so a response to question can either be a non se and here you have
0:06:28to subclasses direct on as an indirect on says
0:06:30and ultimately in the paper also discusses that these actually needs some extras the process
0:06:34within them
0:06:36and if it's not i don't on so then it can either be a question
0:06:38response like we've
0:06:41already discussed with these seven classes
0:06:43or it could be a noun so it can be the kind of gone response
0:06:47so a kind of an acknowledgement
0:06:50two classes that i'll give an example a second this the i don't know class
0:06:55and this is difficult to provide response glass
0:06:57and then declarative responses the about
0:07:00the these issues that also already all rows in the in the question kind of
0:07:05so the i don't know is this kind of
0:07:08very not uncommon kind of response where and equally the this is a difficult provider
0:07:15announce the case and acknowledgement of course you all the very familiar with those guys
0:07:22so the data for english comes from the bnc
0:07:28the be corpus and not on the map task corpus
0:07:31so as you
0:07:32you probably most of you familiar with these corpora the bnc is a
0:07:37ta p honestly conversations
0:07:39be contain speech or dialogues from the class courses
0:07:42and map task consists of donald the code for direction providing task
0:07:47so we took about five hundred past and b and c two and fifty from
0:07:53b and about
0:07:55slightly less and five wonderful map task
0:08:00the way this a good was a random ten selection of turn units ending with
0:08:04a question mark
0:08:06where we also eliminated type questions and turns with missing text and tens of missing
0:08:14the polish data was taken from the scruggs corpus which is basically the spoken part
0:08:19of the polish at national corpus
0:08:22and that consists of that corpus consists of about two and fifty thousand utterances
0:08:27and for this we chose about two hundred yes
0:08:33okay so the basic results
0:08:35all that for english the
0:08:40the other classes is it is less than three percent so we have
0:08:44more or less close to ninety something ninety seven percent coverage with this taxonomy
0:08:51perhaps not hugely surprising
0:08:54the most we can cluster responses in all three corpora in english
0:08:57and approach i direct on says
0:08:59in the bnc the biggest next biggest classes clarification requests
0:09:03so be the next biggest classes indirect announces the map task the second biggest is
0:09:08you know actually
0:09:12ignore the case where you respond with another utterance which is about the same situation
0:09:16but it's not respond to the question
0:09:19so you can already see that is fair amount of variability across corpora
0:09:23for polish the two most frequent last response is a on says so direct ones
0:09:28and indirect ones
0:09:31and then the next to a frequent classes or the i don't know class
0:09:35and the ignore class
0:09:39so this is roughly the results and obviously it'll be a bit hard for you
0:09:42this is all in the paper so you can if you in the resulting in
0:09:46detail you can you can see it there but you can see at the top
0:09:49you have the of course the most of the masses taken by the
0:09:54the direct on says
0:09:57but with
0:09:58the task oriented of course getting much more direct then
0:10:02something like to be and see that the and open corpora like b and c
0:10:06and spokes
0:10:07and there you see and then you can see
0:10:10that's there's a fair amount of variability
0:10:14across corpora fulfilled you different kind of classes telling you that you know you're not
0:10:18gonna get a good on you can't there's no chance of getting good characterisation of
0:10:22this problem just by looking one corpus
0:10:25and as we found in the in the question study at is quite a large
0:10:30variability across corpora in terms of these kind of distributions so the nature of the
0:10:37corpus really
0:10:38again that's not very surprisingly influence is very much the kind distributions you get
0:10:45as far as real reliability goes
0:10:49so we did a in a i just speak about the english part of reasons
0:10:54the time but the polish is discussed in table two so we did an intent
0:10:59eight is a study we had would had to my main annotators were also
0:11:07and a work try to students in object linguistics
0:11:12l two speakers of english and then to when assemble training sessions with the me
0:11:19both annotated around five hundred paths and from this we extracted five hundred calmly bad
0:11:28we got a cap of for our about one sixty five a group and of
0:11:32about one sixty six
0:11:38ninety four cases where the annotation to the disagreements where annotations agreements a occurred
0:11:45the main disagreements concerned direct on says this is indirect on says so weak that's
0:11:51about a third able to disagreements
0:11:54it could no versus
0:11:56change the topic acknowledgement a direct depending question and a direct answer
0:12:02and acknowledgement this is the
0:12:06so direct indirect disagreements mostly occurred with why questions how questions and what is x
0:12:12doing questions
0:12:14and visa cases where on says all by a lot sentential
0:12:19and for which has been significant can promising theoretical literature on how to characterize onset
0:12:24so just to give a couple of examples
0:12:27so we have here case with the why question why deep tan'll to know that
0:12:34well as the new guy
0:12:35so the annotators disappear i was a direct or indirect and eventually was a resolve
0:12:41to indirect
0:12:45and is another example a web
0:12:48this is a four to one again to why question i thought very nice is
0:12:53it no it isn't what is why isn't it "'cause" it isn't
0:12:56and this with again to go clean direct on statistical model
0:13:01and eventually resolved to an indirect on sit since it indirectly indicated is actually no
0:13:07okay so this is just we just to give you a sort of flavourful for
0:13:13the nature of kind of disagreements and
0:13:16the fact that probably
0:13:19this is a kind of task
0:13:21a notion of annotate a more sophisticated notion annotation we wait which doesn't necessarily
0:13:26lead to a resolution but leads to actual different kinds of judgements having to maintain
0:13:32it is probably needed
0:13:35okay so the final thing i'll just mention is
0:13:39that sort of formal analysis that it
0:13:42that is needed in order to solve this to
0:13:46two can describe this problem formally
0:13:50so in our original paper we provided rules within the cost is the follows them
0:13:59how the coherence or of
0:14:02these seven class of questions that can the kind of "'cause" response to questions
0:14:06and to the extent that what we've
0:14:09what the study is shown is that basically
0:14:13the class all of
0:14:17basically on says
0:14:21direct indirect bounces plus
0:14:25things that are address these basic issues
0:14:28then we already have essentially a complete characterization of the response space
0:14:34which i in again potentially in the in implement able form in the sense that
0:14:38this is the cost easier formalism is i is it is a sort of information
0:14:41state type formal is them so it's but actually giving you a
0:14:47has potential for implementing a kind of
0:14:54a for dialogue manager
0:14:56so just to make a few a few comments in that respect the most basic
0:14:59a notion of answered i you might we might say is
0:15:03something one has been a cool simple answer would
0:15:08so if you think of what a question is for mathematical if you essentially a
0:15:11some kind of a allowed abstract
0:15:14where does for broke white ball questions it's a i'm abstraction of a empty set
0:15:20of variables and for the rich questions over a set from of one or more
0:15:27a simple utterances are of course for polar questions just the two polar opposites
0:15:36for all other research questions they are on the instantiations and then negation is
0:15:42and this is actually a system plots the hood if you're the corpus has pretty
0:15:46good coverage as
0:15:49we know from this is of course and a way of they're pretty a direct
0:15:55way of talking about slot filling
0:15:58but that the ultimate notion of on subword which a goal here about nist had
0:16:02encode about a similar in the real lecture
0:16:06have to be
0:16:08actually ultimately
0:16:10if you want really wide got a coverage have to include things the go beyond
0:16:14simple onset would so it has to accommodate conditional
0:16:19we demoralised and quantification on says
0:16:22so this addresses some of these kind of questions that the silicone is been asking
0:16:26what all these poor people who are just a filling slots
0:16:32and so
0:16:34that was so
0:16:36again i'm not of the i don't have time hated to see how to say
0:16:38how you can have formally deal these opinion the discussed in the recollection questions
0:16:46but at the same time even though that there has been discussion of how to
0:16:49accommodate these kind of
0:16:51on says to
0:16:54so that the that also direct on says
0:16:57with still lacking a comprehensive empirically based experiment extracted tested account for
0:17:04of right a wh words okay so the all of this the reading lectures based
0:17:08on based on very small number of a examples just for a small number wh
0:17:16and of course additional notion of their questions needs is some if an exhaustive knows
0:17:21which has to prevent wrap traumatised
0:17:26whether responses exhaustive well
0:17:29can determine whether response of except the required for a query so this leads to
0:17:33what we i mentioned before that we need to find a great sub division of
0:17:36the honest categories
0:17:40and therefore on the base of about this and some notion of a source the
0:17:44best one can define question dependence
0:17:47and that's that the basis for instance for kind of a rules that you can
0:17:52give the dialogue manager like if a question some discussion respond with an utterance which
0:17:57is a few specific another with this either provides an answer a whole a dependent
0:18:02question-response so that an example of the kind of way of
0:18:07characterizing the coherence of
0:18:09various some classes of a responses
0:18:12the fine across all mention which is the another very big class and has again
0:18:17fit as a important
0:18:22implications for the kind of information that you need annual
0:18:25representations is clarification requests
0:18:30sold in work by again there's been a quite all of the reckon work on
0:18:34that going back to what by a matthew purver myself
0:18:38where we showed how to account for the main class of clarification requests
0:18:42users using rules that enable clarification questions to be relevant a given utterance
0:18:48so the basic idea
0:18:50we are going at any for details is that involves accommodating to context certain kinds
0:18:56of clarification questions
0:18:58with rules of this basic format
0:19:01so that the input is so much as you given so much as you want
0:19:06something state you would the constituent of this is actions on that application
0:19:11then you can accommodate any of these kind of a this class of questions what
0:19:15a mean by you one what would today is that you one or a kind
0:19:19of a confirmation kind of questions
0:19:21but knows to do this you could not do this just on the basis of
0:19:27content based that the content of the question as input you need the whole sign
0:19:34that's associated with an utterance
0:19:41conclusions so i presented here and initial study for the for
0:19:46what we've as possible we can see the first detailed form in depend characterisation of
0:19:50response basic queries
0:19:53and k s
0:19:55a lot of things that need to be done
0:19:57so one thing is cross question type in comparison so as to set the that
0:20:02the question-response pairs that we looked at
0:20:04was selected randomly and obviously it's interesting to consider distribution responses relative to fix parts
0:20:09of questions
0:20:11so different foster wh questions polar questions and so on and again we can be
0:20:14facial that they'll be different distributions different fit for different parts of questions
0:20:20we need to apply machine learning to acquire the response classification scheme of course so
0:20:25there's been some work on this severance than the men ability of nonsentential utterances
0:20:29so that that's a
0:20:32subclass of the kind of was a response exist
0:20:34so that gives hope for the non the bit of some of the sound classes
0:20:38we anticipate that it's a some of these classically pretty difficult to learn for instance
0:20:44the ones that a
0:20:47heavily based on inference like indirect ounces and a more will change the topic
0:20:56as everybody here is interested in down to just implementation so we'd like to test
0:21:01these in a in a dialogue system with a fairly sophisticated management the of that
0:21:05for instance of the goat is class
0:21:09so there's been some initial experiments and work by arrive at allen and gotten but
0:21:14and of course another here we gave you a
0:21:18bit of work on english and polish
0:21:20but of course get which only show you some differences
0:21:24so is a signal us a given challenge we think is of see how you
0:21:27test this classification with languages that
0:21:32speech corpora such as
0:21:35about ninety five percent ninety percent of aligned is on have
0:21:39so we we're starting doing some work on this respectively we go and we and
0:21:45just by using online games
0:21:47online games the proposed
0:21:56we have a few minutes of questions
0:22:03hi first of all thank you for your talk this is really interesting
0:22:07so the question that i had was that if you could go back to slide
0:22:10seven please
0:22:16so you're example here for the changing of the topic a it seems to me
0:22:21that this is not exactly a changing of the topic because you're staying on the
0:22:25same topic they asked you what your answer was in us to what there's at
0:22:28the same general topic but rather more of an indirect refusal to insert
0:22:32so i was wondering it seems like changing the topic is always an indirect refusal
0:22:37to answer and we consider a refusal to answer as part of your ontology
0:22:43so i mean this is a kind of indirect basically thing that you might is
0:22:47an implicature all providing this kind of response is that you know you're trying to
0:22:53i mean you certainly not addressing this issue right
0:22:57so we i
0:22:58in our original work we actually suggested that these kind of it commented that when
0:23:04you provide is kind of response
0:23:07maybe for posting reasons that the most common way to do this is by taking
0:23:12have a question which is kind unify able with the with the original one with
0:23:16more general one you know we should talk about what or whatever is well though
0:23:21this is you know
0:23:23so this work some for many cases but
0:23:25i mean the more general thing is just to provide i mean you could you
0:23:28can you know you can sort of do this kind of changes topic
0:23:32in a way that it can be less smooth of course but you know by
0:23:35throwing something that is quite different and these things also happen so this
0:23:39this is this the smoothest way of doing it just from in proposing point of
0:23:43but it's not have a well that's gonna be
0:23:46in this way
0:23:47so you the basic dynamics
0:23:51for this coherence have to ultimately allow you to that
0:23:55as a consequence potentially
0:23:57to get one of these questions eliminated
0:24:00so that works in the in the setup did you did you see any instances
0:24:05of like direct refusals to answer a question in your corpora
0:24:08where somebody asks a question somebody just as i don't i don't one answer that
0:24:11are i refuse to install i mean they're the coming the not very calm the
0:24:15this another common but and how would your skin like what class would you with
0:24:21that many well that's the
0:24:27so that's here so these of the character ones that a about the issue that
0:24:31is down the underlying issue of changing the topic i see thank you
0:24:37we have time for another question
0:24:45thanks a i wanted to follow up on your future work about the relating to
0:24:50quite a types of questions
0:24:53and i guess you have and only about but i wonder how much of your
0:24:58differences in the corpora might be due to different distribution of questions that are in
0:25:04those corpora versus distribution of types of answers to those types of questions
0:25:10then also maybe could use quickly comment on how you define question "'cause" i think
0:25:15you so there's just a question mark them a corpus so you're
0:25:18you can probably one reason you're not including declarative for direct a no so basically
0:25:24we in terms of a pharmaceutical questions we just
0:25:28doing this kind of family
0:25:32so i mean that's kind of building on the fact that
0:25:35you know transcription has decided this is a question
0:25:40and so that basically means that it's typically going to be i'd average questions all-pole
0:25:46which could be either you know they could also be so the declarative the data
0:25:50that have a question mark the end so but they usually have the same basic
0:25:53function as well as a sort of draw people what
0:26:00so you're set your i mean i guess because since we have done this we
0:26:04don't know
0:26:06and we i think it's all the cnn interesting question also exactly you know
0:26:12i again i'm not aware of what the street address this all that is look
0:26:16at you know what the difference we have actually in a as the c l
0:26:20paper that we had a two thousand seven of oracle financial mapping and me we
0:26:25actually did have some tables of the different distributions of different kinds of wh questions
0:26:32clauses and lexical ones so there is some work on this actually but you know
0:26:41that was just for one of hope that i think of some b and c
0:26:44so it's
0:26:50i don't think the speaker one moment