0:00:06um
0:00:07today
0:00:08uh i'm going to share with you some uh results for that human assisted
0:00:12speaker recognition
0:00:13nist sre ten
0:00:15uh but
0:00:15first
0:00:16uh i want to point out that
0:00:17this was
0:00:18a possible only through the efforts of hard work
0:00:22uh of a lot of different people
0:00:24and so uh credit should go to
0:00:27uh jack godfrey and george got into
0:00:30as well as joe campbell uh our friends of the L D C
0:00:34uh and then
0:00:35um i can't think of it more as well
0:00:39so
0:00:40there's a question
0:00:42how how can human experts effectively utilise automatic speaker recognition technology
0:00:47um so this was
0:00:49um
0:00:50the perspective
0:00:51uh that
0:00:53uh we are coming from trying to answer this question
0:00:55uh when considering
0:00:56uh has a
0:00:57for sre ten
0:00:59um it's important to know this was a pilot
0:01:02um and also important to note
0:01:05that it was not
0:01:06uh what we hope
0:01:07it is uh uh informative and of interest
0:01:10add to the forensic community it was not
0:01:13uh design
0:01:14uh to be a forensic tests
0:01:16uh we
0:01:17did not use forensic data we
0:01:19uh leverage data that we're already using
0:01:21for the
0:01:22sorry
0:01:23ten evaluation
0:01:25so the hazard task
0:01:26uh was given to different speech segments determine whether
0:01:30uh they're both spoken by the same speaker
0:01:33uh has or was a subset
0:01:35of the sre core test
0:01:37uh there were fifteen trouser has a one hundred and fifty thousand hazard to
0:01:42a hazard wise human listening
0:01:44uh to assist in decision making and system descriptions
0:01:47um
0:01:49describe
0:01:50how much in this day
0:01:51uh exactly was involved
0:01:53uh participation was open to all who might be interested as it is
0:01:57uh recognition uh
0:01:58nist evaluations
0:01:59and we're just wouldn't strange
0:02:01uh from experts to
0:02:05so
0:02:06um
0:02:08when
0:02:09considering how much
0:02:11um
0:02:12data
0:02:13humans are able to process it
0:02:15fixed length of time
0:02:17um
0:02:18we determined that it would be necessary to try to select
0:02:22a few really difficult data
0:02:24uh in order to
0:02:26uh make the experiment
0:02:28uh challenging enough to be interesting
0:02:30and so the first question that came up was
0:02:32or even able to select
0:02:35data
0:02:36so uh we ran a preliminary experiment
0:02:39uh and this also
0:02:41a lot of success the protocol
0:02:43for the
0:02:43proposed as or study
0:02:45uh so what we did was we identified confusable speaker pairs from that's really
0:02:49for what
0:02:50um
0:02:51we use the automatic
0:02:52system
0:02:53uh i'm sorry a a set of automatic cyst
0:02:57uh from the follow up
0:02:58and
0:02:58for some voting system
0:03:00selected out forty seven pairs
0:03:02multiple cyst
0:03:03first
0:03:04and we listen to all the interviews for each speaker in each pair
0:03:09um
0:03:09they're these portions of the
0:03:11and selected ten pairs
0:03:12that's the most difficult to
0:03:14which
0:03:15uh from those we selected eight
0:03:17nontarget trials
0:03:18and then from the same speakers we also chose
0:03:21uh
0:03:22target rows
0:03:23uh something actually don't know now
0:03:25um
0:03:26that there's a little bit of
0:03:28but challenging
0:03:29terminology
0:03:30um
0:03:33so
0:03:33when we say uh
0:03:36target trials
0:03:37for model
0:03:38uh for
0:03:39test segment
0:03:40uh what we really mean in the case of target trials
0:03:43same speaker
0:03:44nontarget trials a different speaker
0:03:46and then for the model you might uh call this the known
0:03:50uh for the
0:03:51um
0:03:53uh
0:03:57suspect maybe
0:03:59uh and
0:04:00a case of the test
0:04:02um
0:04:03maybe the question
0:04:06uh let's see so
0:04:08um
0:04:09this every level here
0:04:10was used
0:04:11uh
0:04:12both training test
0:04:13uh
0:04:14four
0:04:16all of the
0:04:18uh
0:04:18has are probably expenditure
0:04:22the symbol fourteen human evaluators
0:04:24uh the uh volunteers were
0:04:26uh
0:04:27people involved in the project
0:04:28uh either
0:04:29um
0:04:29working for the government or
0:04:32they are
0:04:32uh as a data provider
0:04:34uh and people were permitted to unrestricted mostly
0:04:37table three
0:04:38test
0:04:39um
0:04:40evaluators provide a
0:04:41actual decision
0:04:43uh and optionally a coffee
0:04:44it's cool
0:04:45so
0:04:46i hear the results from experiment
0:04:50um
0:04:51see the overall miss rate was
0:04:53well
0:04:53percent
0:04:54the overall false alarm rate from four to six percent
0:04:57uh
0:04:58you see that on the left
0:05:00uh on the right
0:05:01you see the number of errors per trial
0:05:04uh you can see from the some trials are very challenging
0:05:07for example uh fifty
0:05:09five eight had nine nine
0:05:11eight years respectively
0:05:13two
0:05:14table
0:05:15uh
0:05:16three trials had
0:05:18where is that correct decision
0:05:20additional three have
0:05:21more than one third errors
0:05:22uh and one target one month or control
0:05:25there's
0:05:26no
0:05:28and this
0:05:29uh
0:05:30why shows
0:05:32uh
0:05:32number of errors by value
0:05:34a solid red or
0:05:36false alarm
0:05:37the outline
0:05:38yes
0:05:39uh
0:05:39you see all evaluators had
0:05:41false alarm errors
0:05:43i have had to
0:05:45yeah
0:05:47uh so
0:05:48the trials that probably be quite challenging
0:05:50and so this
0:05:51a preliminary experiment supported the idea
0:05:53that it would be possible to have a meaningful has evaluation
0:05:57uh with this use fifteen trial
0:05:59um
0:06:00when uh initially discussing
0:06:02potential for
0:06:04uh retina has experiment
0:06:05uh
0:06:06several
0:06:07potential participants were
0:06:09reluctant to do more than fifteen trials
0:06:11uh but do the concern for such systems
0:06:14if we can
0:06:14uh we decided to have a fifteen trial
0:06:17having fifty
0:06:18uh shock hazard
0:06:20um
0:06:20uh all the nodes
0:06:22they're still in
0:06:25he
0:06:25sure
0:06:28so let's talk about
0:06:29the
0:06:30has
0:06:31which in itself
0:06:33uh trans consisted of training and test speech segments
0:06:36as i said
0:06:37they have different
0:06:38the terminology
0:06:40uh
0:06:40trials
0:06:41where to be processed separately and independently
0:06:44um
0:06:46so uh what we did to help accommodate this
0:06:49was create a automated email system
0:06:52uh where
0:06:53um
0:06:53participants were
0:06:54so that the results for a given trial
0:06:56and then would receive a
0:06:58automated
0:06:59response
0:07:00um
0:07:02hopefully shortly after submitting their job
0:07:04uh
0:07:05with the lake
0:07:06to the neck
0:07:07yeah
0:07:08an extra
0:07:09uh unlimited amounts
0:07:10time permitting
0:07:11uh
0:07:13this thing was
0:07:14um was
0:07:15hello
0:07:17uh human listeners could in fact were
0:07:19in some cases other one person or a panel
0:07:22uh and uh decision and a likelihood score
0:07:25were acquired for each role
0:07:27uh_huh decisions could be made either from a combination of automatic
0:07:30uh processing and human expertise
0:07:33uh were made solely
0:07:34based on human history
0:07:36uh and we did not define a cost function
0:07:38four
0:07:39this evaluation
0:07:40uh and so
0:07:41this morning consisted solely of
0:07:43having this is
0:07:43false
0:07:45uh as i said earlier we us all difficult
0:07:49uh
0:07:50uh trials
0:07:51due to the number of journals
0:07:54and we did so from the mixture six corpus which was to be
0:07:58uh the training data was from interviews including various remarks
0:08:02and the test data was from phone calls
0:08:04uh
0:08:04put in some time
0:08:07uh to select the nontarget speaker pairs we're and he
0:08:12a speaker recognition system over four majors
0:08:14oh possible interview
0:08:16training of you test
0:08:18sex
0:08:19um i think an opportunity now
0:08:21two
0:08:21fig power
0:08:22uh how atlantic sea
0:08:24uh for helping us
0:08:25set up a
0:08:26uh speaker recognition system
0:08:28uh for the spring
0:08:30um
0:08:31after running the system over
0:08:33uh the four majors we identified
0:08:34speaker pairs
0:08:36uh that had a large number
0:08:37um
0:08:38uh
0:08:39very
0:08:40um
0:08:41hi
0:08:42scores
0:08:43uh and
0:08:44uh
0:08:45thus
0:08:45select the thirties and speaker pairs
0:08:47uh and
0:08:48from them as we listen to
0:08:50uh interview and
0:08:52uh
0:08:53phone call
0:08:53trials
0:08:54and selected
0:08:55nine
0:08:56uh based on perception
0:08:58uh that seem to be
0:08:59recent
0:09:01uh for the target rolls ran a full matrix
0:09:03of uh interview
0:09:05train telephone test
0:09:07uh and selected the actual
0:09:08trials that way
0:09:10uh
0:09:10thirty from a score
0:09:12and
0:09:13uh likewise by perception selected six
0:09:16just to be
0:09:17most assume
0:09:19after hazard to
0:09:20uh the first fifteen trials were
0:09:22uh
0:09:23the same
0:09:24as for has one for many another thirty five
0:09:27um
0:09:27we change some threshold
0:09:29to allow for additional speaker pairs
0:09:32uh we're trials but otherwise selected
0:09:34uh
0:09:35the data randomly
0:09:37are chosen
0:09:38oh we did listen for anomalous egg
0:09:41one
0:09:41this is
0:09:42speech or some
0:09:43uh
0:09:44uh
0:09:44and
0:09:45um
0:09:48uh so let's play a game
0:09:50uh
0:09:52yeah wisconsin
0:09:53same speaker different speaker
0:09:55um
0:09:56so
0:09:57uh
0:09:58here we're going to play
0:09:59that you segment
0:10:01and uh
0:10:02try to determine whether
0:10:03same speaker
0:10:04more
0:10:11oh
0:10:12oh
0:10:21oh
0:10:33yeah
0:10:34oh
0:10:35but
0:10:44should note
0:10:45no
0:10:50yeah
0:10:51oh
0:10:52oh
0:10:53i
0:10:54oh
0:10:55okay
0:10:57oh
0:10:58yeah
0:11:00yeah
0:11:02i
0:11:02i
0:11:04oh
0:11:04oh
0:11:05oh
0:11:05oh
0:11:06oh
0:11:07oh
0:11:07okay
0:11:08so
0:11:09how many people thought same speaker
0:11:12oh
0:11:13five different
0:11:14speaker
0:11:16okay
0:11:17yeah
0:11:18that we have
0:11:20sorry
0:11:23oh well
0:11:24yeah
0:11:25they rely zero
0:11:26okay and then for a child to
0:11:29oh yeah
0:11:31oh
0:11:31yeah
0:11:32yeah
0:11:35yeah
0:11:36oh
0:11:37oh
0:11:39oh
0:11:39oh
0:11:42yeah
0:11:44yeah
0:11:45oh
0:11:46one
0:11:47yeah
0:11:48oh
0:11:50oh
0:11:51yeah
0:11:52oh
0:11:54right
0:11:55oh
0:11:56oh
0:11:56where
0:11:58oh
0:11:59oh
0:12:00oh
0:12:18oh
0:12:21yeah
0:12:23okay
0:12:24well
0:12:26uh same speaker
0:12:29speaker
0:12:31yeah we have
0:12:31yeah
0:12:32uh so
0:12:34the uh first one was
0:12:35same
0:12:39yeah
0:12:42ah
0:12:42so uh
0:12:44twenty
0:12:45twenty systems from fifteen brave souls
0:12:48uh
0:12:50willing to
0:12:51uh
0:12:52dissipate a complete the fifteen trouser has a one
0:12:55uh and it's systems for six sites
0:12:58uh
0:12:58yeah
0:13:00i needed additional hundred thirty five trials
0:13:03answer to
0:13:04um oh sites that participated in has are also participated in the
0:13:09ministry ten about
0:13:11uh and the other side recruitment
0:13:13uh represented academic and government organisations from six different countries
0:13:18uh so for has or one
0:13:20uh here's the story
0:13:22um
0:13:22blue or
0:13:24uh target trials and read or nontarget trials
0:13:27uh and the darker colours uh represent their
0:13:30uh so at the bottom you can see sort of number of errors per trial
0:13:35and then to the columns
0:13:36you can see
0:13:37and this is
0:13:38false alarms
0:13:39for each
0:13:41each side
0:13:46okay
0:13:48uh missus corresponding information for hazard to
0:13:51uh just
0:13:52the information from the colours at the end
0:13:55um
0:13:55this is
0:13:56ooh
0:13:57also
0:14:04so
0:14:04um
0:14:05one question we are interested in was whether the human listening uh was
0:14:10uh worthwhile little selecting
0:14:12uh
0:14:12trials for the uh hazards power
0:14:15uh so uh are
0:14:17first look at this was
0:14:19um that the
0:14:21uh has one travels with the twenty systems had about
0:14:24thirty eight percent
0:14:25miss rate forty seven percent false alarm language wasn't too different
0:14:29uh from the eight systems or
0:14:31a hundred and thirty five trial so without L
0:14:33uh
0:14:35uh
0:14:36probably not worth the human life
0:14:38uh
0:14:39which was
0:14:39quite a bit uh to select the
0:14:42uh has or one draws weapons like
0:14:43same
0:14:44are sensitive
0:14:45uh we recently
0:14:46yeah maybe last year or so
0:14:48uh
0:14:48brought this out
0:14:49and do the same
0:14:51it's systems
0:14:52here we see the miss rate is about
0:14:54saying
0:14:55uh but the false alarm rate
0:14:57exactly signal
0:14:58hi
0:14:59uh so it seems as though maybe that has a one trials were a little more difficult but
0:15:03uh hopefully the hasn't you channels
0:15:05uh we're difficult enough
0:15:08uh this is
0:15:09yeah another view of the
0:15:11has a one has or two comparison
0:15:13uh the blue dots are
0:15:14uh has a while the
0:15:16red circles are
0:15:17or has or two
0:15:18uh and the lines connecting them
0:15:20uh sure there's the same system
0:15:22and all cases
0:15:23exception um
0:15:26i'm not sure
0:15:28um
0:15:29with the exception of
0:15:30this one
0:15:31system here
0:15:32uh
0:15:33and all cases has a two
0:15:35uh
0:15:35uh was less challenging
0:15:37i was or what
0:15:43and now this is a real obviously so uh
0:15:47they probably wouldn't
0:15:51and
0:15:53sure
0:15:53so uh
0:15:56with
0:15:57um
0:15:58has one
0:15:59uh and has or two we had some interest and
0:16:02uh how how to estimate also how systems did
0:16:05uh all the systems we were actually a little concerned that we're bias in this
0:16:08again systems
0:16:09since we use the system to
0:16:11uh select difficult trials
0:16:14um
0:16:15what we see here uh each red dot represents
0:16:17uh
0:16:18aside
0:16:19uh performance
0:16:20or
0:16:21uh has a one
0:16:23something
0:16:24to know someone brought up
0:16:25um
0:16:26and the last presentation that
0:16:28uh
0:16:29the dots on the axes
0:16:30uh actually shouldn't be on the axes
0:16:32and there is zero percent
0:16:34uh but
0:16:35that would otherwise
0:16:36be
0:16:37i'm not be visible so we
0:16:39put them there just a
0:16:40i see
0:16:40uh and uh so is is it the red dots represent the
0:16:44uh has or sites
0:16:46uh
0:16:46the lines
0:16:48uh represent
0:16:49top systems
0:16:51from
0:16:51uh the sre ten evaluation
0:16:54and
0:16:54thing
0:16:55uh
0:16:55is notable here
0:16:56uh is that in all cases the dots or or outside
0:17:00uh
0:17:01uh
0:17:02donna yellow line
0:17:07um
0:17:07so
0:17:09uh we do something similar what has or two
0:17:12uh and
0:17:13uh here we have all the same plot
0:17:16automatic systems
0:17:17uh and uh has or two systems
0:17:20uh the thick lines
0:17:22or
0:17:23uh automatic system
0:17:24and
0:17:25the lines
0:17:26or has a two system
0:17:27uh and we note uh
0:17:30a fair amount of separation
0:17:31uh between the two
0:17:34uh
0:17:35though these are the top
0:17:36sre
0:17:37uh
0:17:38automatic system
0:17:40uh and the slide
0:17:42we see
0:17:43um
0:17:45systems that
0:17:47uh or sites that participated both in house or two
0:17:50uh and in the
0:17:51uh sorry tell about
0:17:53uh and corresponding colours
0:17:56i represent the same site once again like lines
0:17:59uh are
0:18:00automatic systems
0:18:01the lines
0:18:02uh or has or system
0:18:04and the thing to note here is
0:18:05uh within colour
0:18:07uh the thick lines
0:18:08uh oh is closer to the origin
0:18:11then
0:18:12um
0:18:16uh so to summarise
0:18:17uh effort to select challenging data uh for has or was successful we feel
0:18:23uh we
0:18:24very pleased to that we were able to do that
0:18:27uh
0:18:28has or two trials uh only somewhat less challenging than has or what
0:18:32uh we can probably revises and say they were more challenging
0:18:36enhancer one though once again perhaps answer
0:18:39uh to was challenging enough
0:18:41uh for purpose
0:18:42this time
0:18:43uh performance of the hazard systems did not compare favourably with that of automatic systems uh on the other channels
0:18:51uh although i i feel obliged to remind everyone
0:18:54uh is dominant
0:18:55that
0:18:55there's
0:18:56uh there are so contrast
0:18:58that's
0:18:59um
0:19:00it's difficult to find any signal
0:19:01again
0:19:02uh
0:19:03in the
0:19:04uh statistical significance
0:19:06results
0:19:07um
0:19:08and it
0:19:09then there's a question uh if
0:19:10has our evaluations invaluable
0:19:13uh how should this be extended
0:19:14uh
0:19:15should the tense particle be changed or wasn't reasonable uh
0:19:19uh is there a good way to uh uh after the trial selection
0:19:24uh or wasn't uh also like was reasonable and
0:19:27uh what we keep on
0:19:29uh bumping up against
0:19:31this will signify
0:19:32so is there
0:19:33away
0:19:34uh to be able to get
0:19:36uh
0:19:36additional statistical significance
0:19:39uh without
0:19:40um
0:19:40burning the site
0:19:43too much
0:19:44and
0:19:45that is all
0:19:53we have time for
0:19:54some questions
0:19:55come on
0:19:59thank you craig
0:20:00um
0:20:01can you
0:20:02briefly
0:20:03clean
0:20:04oh
0:20:04what
0:20:05protocols
0:20:06oh
0:20:07i would be
0:20:07transmit
0:20:09uh what people actually do
0:20:11right
0:20:12hmmm
0:20:13sure so what i can say is that it it varied from side to side
0:20:16um and insights and uh system description
0:20:20um
0:20:20that was available to all the uh
0:20:22has approaches
0:20:24um
0:20:25in many cases
0:20:26that they were not even
0:20:28as with
0:20:29earlier in some cases
0:20:30and they um
0:20:32do you uh are or maybe more for the process of
0:20:36uh
0:20:37some uh
0:20:38uh process be online
0:20:46the question
0:20:54oh
0:20:55oh
0:20:55right
0:20:59i'm sorry
0:21:03did you produce
0:21:05they were just giving you binary
0:21:06so
0:21:07sure so um
0:21:09each uh has a participant was required in addition to giving the decision to give a
0:21:14uh
0:21:15score
0:21:17no that was i'm sorry yes but that was in the preliminary experiment
0:21:21yeah it was also on that and uh has or
0:21:23uh evaluation
0:21:24is
0:21:33we
0:21:34exp risk
0:21:35we we
0:21:36we
0:21:38making
0:21:38virtually impossible
0:21:40where do not exist
0:21:43bias
0:21:44it is
0:21:45or
0:21:45on a priori
0:21:48two
0:21:50humans are more
0:21:53three
0:21:54oh
0:21:55but
0:21:56where
0:21:58with
0:21:59fig
0:22:00Q
0:22:01well this is our concern
0:22:03um
0:22:04not a node in the case of the
0:22:06uh
0:22:07has or one trials
0:22:08i kiwi perception was also involved
0:22:11uh to select
0:22:12right
0:22:13so uh maybe it wasn't
0:22:15yeah
0:22:16i
0:22:17my suggestion
0:22:19two
0:22:20right
0:22:21right
0:22:22right
0:22:22wow
0:22:24her
0:22:25to use
0:22:26you know
0:22:27speaker I D
0:22:28yeah
0:22:30two
0:22:31oh
0:22:31right
0:22:32thanks
0:22:34yes
0:22:35fig
0:22:35a system
0:22:37two
0:22:38picks
0:22:38right
0:22:39sure
0:22:40i
0:22:41something
0:22:42was
0:22:43we have to
0:22:46i like to be
0:22:53you you said that one of the reasons of doing this is to see if the combination of human
0:22:58a machine
0:22:58would help
0:22:59yeah
0:22:59she just
0:23:00for a considerable
0:23:01oh faster
0:23:02did you see
0:23:03uh persists for people have a system
0:23:07the normal needs to you know
0:23:09and then if they want to answer
0:23:11yeah
0:23:11a hundred fifty trials
0:23:13they could listen
0:23:14so you can decide
0:23:15which ones
0:23:16oh
0:23:16fig
0:23:17be helpful to listen to
0:23:19i
0:23:20and then
0:23:21um
0:23:22you can listen to whatever we
0:23:24just a decision is just a score
0:23:27uh
0:23:28of course then you wouldn't have seen from sri system but you have
0:23:31you combine
0:23:33decision
0:23:34where
0:23:35something which you have
0:23:37buying so
0:23:38yes i think that would be interesting to think oh no
0:23:40uh however is that some sites did
0:23:42their goals
0:23:43you know human machine fusion
0:23:45uh
0:23:46so
0:23:46i guess we
0:23:47um
0:23:48are able to get
0:23:49uh
0:23:50some very small glimpse of that
0:23:52uh but i like yours
0:23:53efficient
0:23:55also
0:23:56you
0:23:57cool
0:23:58so attribute description
0:24:00during the walkthrough we look from you
0:24:02you too
0:24:03you you man
0:24:04be
0:24:04system
0:24:05you know the true system
0:24:07and we are trying not to answer two questions
0:24:10try to evaluate you just a few men people human expert
0:24:14and try to
0:24:16this time
0:24:17you selection process
0:24:19you
0:24:20maybe not
0:24:22you do
0:24:24if we try to to see if we are to
0:24:27system
0:24:28it's good to evaluate remember
0:24:30no
0:24:31you
0:24:31we must maximum
0:24:34yeah
0:24:39um so i just one and
0:24:42that um of course are
0:24:45it's
0:24:45i
0:24:45yeah
0:24:46table
0:24:46what
0:24:47we
0:24:48we're more interest
0:24:49see
0:24:50she
0:24:51no
0:24:51you
0:24:54um
0:24:55so
0:24:56in mind that
0:24:59we're going to be establishing um
0:25:02in additional mechanism for
0:25:04communicating about
0:25:06uh
0:25:07future evaluations along this line
0:25:10how
0:25:11um
0:25:12for a couple
0:25:13or
0:25:14in
0:25:14right
0:25:15um
0:25:16what might
0:25:17no
0:25:18in future english
0:25:20to help mode discussions and whatnot
0:25:23just as
0:25:24there was just one
0:25:26group for
0:25:27story continuation
0:25:29uh will you
0:25:31well thing
0:25:32where
0:25:32he's or
0:25:33oh
0:25:33yeah
0:25:34see dover downs
0:25:35and
0:25:36suggestions for
0:25:37how to make this
0:25:38oh
0:25:39an interesting um
0:25:41evaluation in the future uh
0:25:44and
0:25:44many very interesting
0:25:46suggestions including multi
0:25:48we
0:25:50and
0:25:51related activities like you can
0:25:54yeah
0:25:56where do we
0:25:57like
0:25:58well
0:25:59it's
0:25:59and uh
0:26:00um and you
0:26:01contact me your
0:26:02nist people
0:26:06because
0:26:08that's cute
0:26:08a fusion between human beings
0:26:11and the automatic
0:26:12to to see
0:26:12but
0:26:13well the human part
0:26:14it's something
0:26:15commissioner
0:26:16didn't
0:26:18yeah
0:26:18i didn't have one
0:26:19sure yes uh we have not
0:26:21uh
0:26:23we're not really looked at this
0:26:24uh
0:26:25no i find it very interesting
0:26:27part of
0:26:28uh
0:26:29part of it
0:26:29challenge is
0:26:31um
0:26:32that in some cases
0:26:34uh the human and
0:26:36um
0:26:37uh machine
0:26:38uh results were
0:26:39use prior to coming to us
0:26:41so would be difficult those cases that is out
0:26:43what
0:26:44part of that
0:26:46percent
0:26:48thank you
0:26:49Q from