Přepis řeči - PCA-based Feature Extraction for Phonotactic Language Recognition

0:00:06	okay i would start
0:00:09	my name is
0:00:09	my skin colour and i will present
0:00:11	uh
0:00:12	my work that was done to delete my code for
0:00:15	the typical but
0:00:16	on taken back problematic outcry
0:00:19	we would get then
0:00:20	and did not ski
0:00:22	uh the
0:00:23	this
0:00:24	presentation will be about
0:00:25	feature extraction for phonotactic language recognition
0:00:29	it should be done
0:00:30	by
0:00:30	pca
0:00:33	and
0:00:34	this is the overview of the whole cable first
0:00:37	pick a little bit
0:00:37	about motivation of this work
0:00:39	by
0:00:40	we want to do it
0:00:42	and
0:00:43	then i will describe
0:00:44	uh the results on the nist
0:00:46	uh
0:00:47	uh language recognition
0:00:49	evolution that was mine
0:00:56	so basically for the introduction
0:00:59	oh
0:01:00	if we want to
0:01:01	uh recognise languages
0:01:03	um by phonotactic model C
0:01:05	basically can
0:01:06	i do we use uh anger models like language models very
0:01:09	compute
0:01:10	likelihood
0:01:11	oh well
0:01:12	sometimes there's a given
0:01:13	specific uninterrupted
0:01:15	models of
0:01:16	languages
0:01:17	uh or we can actually tried to use
0:01:20	discriminative models like
0:01:21	as the M based models
0:01:23	that are
0:01:24	usually performing better
0:01:25	and this is
0:01:26	what we will
0:01:27	talking about
0:01:28	this presentation
0:01:31	usually
0:01:32	uh for this as the N models
0:01:34	a linear kernel and soft margin are used
0:01:38	means
0:01:39	basically that
0:01:40	uh we L O somehow flyer
0:01:44	uh so the problem with
0:01:46	with uh this as the M approach is that
0:01:48	we need to really
0:01:49	very large
0:01:50	feature vectors
0:01:51	if we use uh let's say trigrams for
0:01:53	five
0:01:54	uh foreground
0:01:56	uh
0:01:56	going for higher are orders it's
0:01:59	computationally
0:02:00	uh yeah
0:02:01	almost impossible because
0:02:03	uh the growth of the
0:02:04	uh of the
0:02:05	uh features so feature set
0:02:07	uh
0:02:08	like
0:02:08	financial
0:02:10	uh i like his work
0:02:11	here
0:02:12	slide
0:02:13	and
0:02:13	we can easily compute that for some
0:02:15	second all use like
0:02:17	if the
0:02:18	set of the phonemes
0:02:19	large like
0:02:20	it
0:02:20	phonemes
0:02:21	it will be using four grams then
0:02:23	easily they can deal with much more than
0:02:26	million of possible features of course
0:02:28	always all these features
0:02:30	um
0:02:31	are
0:02:31	present
0:02:32	in the data about
0:02:33	this
0:02:34	this is like two article
0:02:35	a limitation
0:02:42	so
0:02:43	uh we need to somehow we meet this space
0:02:46	and usually
0:02:47	we can either discard
0:02:48	features like that will perform some selection
0:02:51	sure
0:02:52	or we can do a combination of the features
0:02:54	which
0:02:55	we'll call here
0:02:56	feature extraction
0:02:58	that is what i'll be describing later
0:03:01	so basically for the feature selection we can either
0:03:04	choose the the features that
0:03:05	but you're frequently in the data like
0:03:08	it is useless to have
0:03:09	uh
0:03:10	uh in this feature vector
0:03:12	combinations of uh
0:03:13	of phonemes that form anger on that
0:03:15	never a cure the
0:03:17	well training set which can
0:03:18	easily
0:03:19	uh a cure like
0:03:20	some some combinations of phonemes are
0:03:23	uh
0:03:23	very unlikely to happen and we maybe
0:03:26	uh some reasonable pruning then
0:03:28	uh such thing of another
0:03:30	a cure
0:03:31	and of course there
0:03:33	there are also other combinations of power in grounds that actually
0:03:36	that can happen
0:03:37	joker sometimes but
0:03:39	uh it is not very meaningful to them
0:03:41	in in that features that
0:03:43	uh and we can uh
0:03:45	usually discard them
0:03:46	a base
0:03:47	on a on a some threshold value so
0:03:49	basically
0:03:50	although although
0:03:52	thank around
0:03:53	or cure
0:03:54	well then sample you can be
0:03:55	discarded
0:03:56	other approach
0:03:57	two
0:03:58	use this kinetic information and
0:04:01	uh that
0:04:01	means that
0:04:02	we will try to
0:04:03	keep all the
0:04:04	and grammars that are actually good for the classification of languages
0:04:08	it is slightly different because
0:04:10	you can imagine that
0:04:12	someone
0:04:12	a low frequency in grounds that are quite rare
0:04:15	uh in general across languages
0:04:17	might be
0:04:18	uh quite uh
0:04:19	why this can wrap it or
0:04:21	quite informative
0:04:22	for discrimination
0:04:24	or some such a language
0:04:25	uh so
0:04:26	uh this can be like a better way to
0:04:29	this card
0:04:30	feature
0:04:34	uh
0:04:35	oh here i would like to
0:04:36	show that idea
0:04:38	why
0:04:38	we tried to
0:04:39	use feature extraction
0:04:41	because
0:04:42	uh
0:04:43	we can easily see that for example someone
0:04:46	uh combinations of phonemes will
0:04:48	have well
0:04:49	various like zero values we can discard them
0:04:52	is it a like i
0:04:53	that on the bruise
0:04:54	uh slide but on the other hand
0:04:56	there can be combinations
0:04:58	and grounds like
0:04:59	i have written
0:05:00	then here like
0:05:01	being being being anything which
0:05:04	it just something that sounds are computed the same but in some cases like
0:05:08	is
0:05:09	right
0:05:09	for example but it can only happen that
0:05:11	some
0:05:12	a phoneme combinations will have very similar pronunciation variant and then
0:05:16	maybe i'll
0:05:17	uh frequently come here and uh
0:05:19	in the lattices
0:05:21	and uh of course
0:05:23	even if a frequency of these i think around
0:05:25	quite high
0:05:26	it would be a good idea at least class then together somehow so
0:05:31	that we would need
0:05:32	not to deal with this
0:05:34	uh with this uh
0:05:35	oh
0:05:36	uh with this the amount of features that is like
0:05:39	you use less
0:05:39	goes
0:05:40	it can be seen that
0:05:42	you don't need to like four
0:05:43	choose here but it would be
0:05:44	you know
0:05:45	have just one so this is like motivation example
0:05:48	what we try to do
0:05:55	and uh
0:05:56	uh we we have tried to
0:05:58	to use a simple pca
0:06:00	average
0:06:01	like a dinner
0:06:02	projection
0:06:03	which can be used to compute some of the same some matrix two
0:06:06	or from some linear projection of the
0:06:08	but original feature space to some lower dimensional feature space
0:06:12	and uh
0:06:13	it's
0:06:13	seems like it is uh a good way how to
0:06:16	five
0:06:16	curse of dimensionality that is caused by the
0:06:19	financial
0:06:20	increasing numbers
0:06:21	number of parameters than me
0:06:23	increase the size of the context like
0:06:25	and we go from trigram
0:06:26	program and so on
0:06:28	and actually quite similar idea works in
0:06:30	um normal bigram language modelling
0:06:33	uh so
0:06:34	it sounds like yeah
0:06:35	some reasonable way
0:06:37	to go
0:06:38	and uh that of course
0:06:40	the other is
0:06:41	like
0:06:41	uh are that we don't need to tune many parameters
0:06:45	to try
0:06:46	that's all
0:06:47	it's very fast simple and there's still plenty of
0:06:50	tools that can be used to compute pca
0:06:53	so so simplicity is
0:06:54	one of the reasons like
0:06:56	using
0:06:57	technique
0:06:59	and now i will
0:07:00	discuss the results that we have
0:07:01	thing
0:07:02	on a nice
0:07:03	uh language recognition
0:07:04	thousand
0:07:05	nine uh
0:07:06	on the coast
0:07:07	that condition
0:07:09	or all the durations
0:07:10	will be in the
0:07:12	the law
0:07:13	line
0:07:14	so
0:07:15	uh for our development set uh
0:07:17	but
0:07:18	used for business
0:07:19	uh
0:07:20	thousand nine
0:07:21	uh
0:07:22	we have tried to
0:07:23	first
0:07:24	best again
0:07:25	what happens then we discard
0:07:27	features by their frequency
0:07:29	so we are keeping only
0:07:31	uh the most frequently appearing five thousand features in that and that was and so on and
0:07:36	you can actually see
0:07:38	that uh that the accuracy of the system
0:07:41	rows of an interview at the more features and it seems like
0:07:44	natural that it would be good to people all the features and
0:07:48	what
0:07:48	the as the ends to
0:07:49	cool and what are they use the useful features and
0:07:53	and not the discard any of them
0:07:55	but of course this is impossible we don't have a result
0:07:58	in in this table what would happen if
0:08:01	yeah
0:08:01	if we would
0:08:02	use all the features
0:08:04	because that will feature space
0:08:05	like over
0:08:06	one minute of combinations of course be
0:08:09	uh not all these combinations they really happened to appear in the
0:08:12	the training set but
0:08:13	uh the amount of four combinations that actually happened
0:08:16	it's like out several hundreds of thousands
0:08:19	uh and
0:08:20	this is
0:08:21	simply impossible to
0:08:22	to compute
0:08:23	uh in a reasonable time
0:08:25	so
0:08:26	so uh
0:08:27	this is like
0:08:28	the the
0:08:29	result that can be
0:08:31	interpreted like yeah that
0:08:33	we cannot go further
0:08:38	and uh then we have to use the
0:08:41	you see a
0:08:42	actually this is shown on the
0:08:44	and the trigram because uh as i have said
0:08:46	previously on the program
0:08:48	uh
0:08:49	steam
0:08:49	of it longer and
0:08:51	a recogniser
0:08:52	you are not able to compute the the full
0:08:54	uh for feature space a base
0:08:57	so
0:08:57	uh yeah
0:08:58	so
0:08:59	uh this is
0:09:00	on trigram
0:09:01	we are
0:09:01	can
0:09:02	seen that the
0:09:03	for system is around two point three
0:09:05	C average and that's that
0:09:07	and that
0:09:08	that's the last line
0:09:09	and the previous lines are
0:09:11	when we
0:09:12	previews
0:09:12	this
0:09:13	this feature space from this
0:09:14	thirty six thousand features like
0:09:16	one hundred
0:09:17	five hundred and so on the
0:09:19	you can see actually that
0:09:20	when we go to something like five or five hundred or one thousand features
0:09:24	which is like
0:09:25	uh okay
0:09:26	six times less
0:09:27	and the original your space
0:09:29	we can uh
0:09:30	uh get almost the same performance then
0:09:33	then the speed of that is described
0:09:35	in more detail in the paper
0:09:36	can be had a large
0:09:38	oh
0:09:38	for training that's
0:09:39	stan
0:09:40	and testing
0:09:41	uh actually the the
0:09:42	you don't have the
0:09:43	think basis even
0:09:44	faster than all the training phase because in the training phase
0:09:47	basically need to estimate first
0:09:49	pca one
0:09:51	while
0:09:51	in the testing phase we don't need to do this
0:09:54	the only project the data
0:09:56	so
0:09:57	it can be seen from from this like that
0:10:00	actually
0:10:00	seems to work reasonable
0:10:06	uh yes and
0:10:07	maybe i can add it did we actually tried to use
0:10:10	um more
0:10:11	uh more toolkits
0:10:13	that are freely available to compute these svms models and like
0:10:17	uh we have tried to tune
0:10:19	all of these to obtain the best performance then
0:10:22	like um
0:10:24	my
0:10:24	very cool experience is that
0:10:26	and it as a gmm svm search
0:10:28	but quite good results
0:10:30	and pleading there is
0:10:31	like
0:10:32	ten times faster but
0:10:33	about five percent worse in accuracy
0:10:40	and now for the
0:10:42	for the result with multiple systems because we have trained
0:10:45	uh a hungarian
0:10:47	anger in phoneme recogniser
0:10:49	english phoneme recogniser impression phoneme recogniser
0:10:53	then in the end we use all the results together
0:10:55	we'll see that later
0:10:57	and we can see on this table of a basically happened
0:11:00	like if you would focus on the
0:11:02	i greens
0:11:03	payment
0:11:05	you can see that actually five hundred
0:11:06	features
0:11:07	were quite well but
0:11:08	uh when we go two thousand bits
0:11:10	actually better and then
0:11:12	uh we don't observe any real time
0:11:14	man
0:11:15	from going to four thousand
0:11:16	features
0:11:17	so it seems like the the value around one thousand
0:11:20	features uh
0:11:21	seems to be quite good
0:11:22	uh then the interesting thing is that
0:11:24	actually foreground
0:11:26	work uh
0:11:27	um horsemen trigrams
0:11:29	of course
0:11:30	space
0:11:31	at the feature space of foreground
0:11:33	yeah it's not full
0:11:34	we need it apart from some feature selection there
0:11:37	because otherwise in
0:11:38	the estimation of pca would be difficult
0:11:40	do
0:11:41	uh so so basically it seems a reasonable to use just trigrams
0:11:45	and it should work okay
0:11:49	yeah
0:11:50	that the data
0:11:51	that results in more they
0:11:52	detail later
0:11:53	now for the english system we can see that
0:11:55	so i does basically the same thing
0:11:58	as uh for the hungarians
0:11:59	stan
0:12:00	even it seems that
0:12:01	it would be enough to keep it just
0:12:03	five hundred
0:12:03	matures
0:12:04	of course
0:12:05	uh
0:12:06	the optimal size of of this
0:12:08	a reduced uh
0:12:09	space
0:12:10	uh
0:12:10	depends also on density of values in the lattices and these things so
0:12:14	so it's not like sound
0:12:16	some singularly about
0:12:17	it should be
0:12:18	uh somehow tuned for every system
0:12:20	but of course using bigger
0:12:22	uh bigger features
0:12:24	some of the problem
0:12:25	so the competition fine goes up
0:12:28	and then twenty four directions system it was the last one
0:12:31	and the the largest uh phoneme set
0:12:34	was quite difficult to train
0:12:36	one
0:12:37	uh we have uh actually tried to
0:12:39	use some more
0:12:40	uh training data
0:12:42	as uh in all the to use uh systems we have used
0:12:45	uh our ten thousand to
0:12:47	uh training samples
0:12:49	to train the sustains about for the last
0:12:51	then we have
0:12:51	actually
0:12:52	uh used to almost fifty thousand is already
0:12:55	very large and the original feature space those
0:12:58	uh
0:12:59	more than one hundred thousand
0:13:00	sure so
0:13:01	there's also a very age sixteen men
0:13:04	uh
0:13:05	quite belong to a given training that
0:13:06	pca
0:13:07	but
0:13:08	can be seen that in the end it works
0:13:10	the buttons
0:13:11	it would be definitely good to
0:13:13	train the systems
0:13:14	all this
0:13:14	stints on all the
0:13:15	available data
0:13:17	it can be seen from
0:13:18	this result
0:13:20	in the end we did not
0:13:21	apart from that
0:13:26	uh here we have
0:13:27	final result
0:13:28	actually happens when we
0:13:29	use all the trigrams
0:13:31	stance
0:13:31	from
0:13:32	of the previous slide
0:13:33	and all the forums
0:13:35	since
0:13:35	can be seen that
0:13:36	but the trigrams
0:13:37	are they performing rather than foreground
0:13:41	and from the from the combination of
0:13:43	no three grams
0:13:44	for foreground
0:13:46	uh we can get some small improvement
0:13:48	that goes across all the conditions but it's
0:13:51	very small
0:13:52	uh what
0:13:53	what this more useful
0:13:54	is so i think a system that was trained on more data
0:13:57	that
0:13:58	russian trigram all
0:14:00	stan
0:14:01	and so that gives us
0:14:02	actually
0:14:03	uh better improvement than using foreground
0:14:06	and
0:14:07	that in the end of an
0:14:08	when uh we were able to fix
0:14:10	so the development set
0:14:12	it was described by
0:14:13	all double quote in
0:14:14	in this presentation in the morning
0:14:16	uh it
0:14:17	possible to to get even much better result
0:14:20	and but it's
0:14:21	around one point eight
0:14:22	on the original thought
0:14:24	second switch
0:14:25	like
0:14:25	right
0:14:26	number
0:14:27	and
0:14:28	uh
0:14:29	well
0:14:30	i don't have the results for the fusion but
0:14:32	uh that is in the paper
0:14:34	at all levels
0:14:35	presenting
0:14:39	so for the conclusion we can say that
0:14:41	we can achieve uh very high speed up
0:14:44	as was
0:14:44	in the previous tables like
0:14:46	we can uh
0:14:47	uh a system that is trained
0:14:49	um much faster than a hundred times
0:14:52	and that we don't lose almost any performance like on the accuracy
0:14:57	and uh of course
0:14:58	uh this
0:14:59	uh this uh technique and it can be used to
0:15:01	radius of the parameter space
0:15:03	for the future space
0:15:05	uh that would allow us to
0:15:07	to use um a more complicated techniques like
0:15:10	as the ends with nonlinear kernels then
0:15:12	very need to tune the more parameters which is
0:15:15	quite difficult to do
0:15:16	uh when we were operating
0:15:18	false
0:15:18	just by
0:15:20	and
0:15:21	some ideas for
0:15:23	for some future work
0:15:24	of course
0:15:25	we can think about some more
0:15:27	complicate it a feature reduction technique that would be
0:15:30	like
0:15:30	something nonlinear maybe some neural net
0:15:33	and can be
0:15:34	oh it
0:15:35	estimated that this kind of
0:15:37	oh okay
0:15:38	even bigger
0:15:39	uh feature space reductions
0:15:41	uh it
0:15:42	it's similar performance
0:15:43	and of course uh
0:15:45	uh in this uh in this
0:15:46	examples um for the
0:15:48	speed up the results of that
0:15:49	uh
0:15:50	i have shown here
0:15:51	we have been estimated you see all the data
0:15:54	which is not really need it and
0:15:56	we have given some other results
0:15:58	from which we know that we can estimate P C
0:16:00	yeah
0:16:01	you say just on the subset of the data and then
0:16:04	you can get even faster
0:16:05	yeah
0:16:06	so that would be like all
0:16:09	thanks for attention
0:16:17	questions
0:16:22	thanks
0:16:22	um
0:16:23	i mean that's very surprising
0:16:26	um
0:16:27	one hundred dimensions to the
0:16:30	and that's
0:16:31	two
0:16:32	capture
0:16:33	sonnets
0:16:34	yeah
0:16:37	um
0:16:38	usually
0:16:39	when you do principal components and um
0:16:43	some sort of room
0:16:44	um like you
0:16:46	estimated to to learn as well
0:16:50	and then use
0:16:52	how many are not used in to account for
0:16:55	ninety percent
0:16:57	my
0:16:57	five percent
0:16:58	oh
0:16:59	hmmm
0:17:01	i wondered
0:17:02	do you know
0:17:03	uh how much of the variability is captured
0:17:07	um
0:17:08	well um
0:17:09	actually i didn't try to compute this i was just looking at them and the final results link
0:17:13	the accuracy of the system and i was reduced
0:17:15	the data
0:17:16	so i'm
0:17:17	would not
0:17:18	i am not able
0:17:19	for this
0:17:20	uh
0:17:21	i think it might be interesting and i think it might be interesting to do the calculation when you huh
0:17:27	the
0:17:27	right
0:17:28	description
0:17:29	hmmm
0:17:29	or not
0:17:32	uh
0:17:32	see
0:17:33	the variability here
0:17:35	really
0:17:37	um
0:17:37	due to the
0:17:38	redundancy in in the in the something that
0:17:41	hmmm i know the right
0:17:43	mine's are
0:17:44	goings
0:17:45	we choose
0:17:46	many
0:17:47	very similar yes yeah
0:17:50	so it's a
0:17:51	that you can project to when you're not there
0:17:54	right
0:17:55	not
0:17:57	oh
0:17:57	if you just have the correct transcription
0:18:01	but you don't have that
0:18:02	yeah
0:18:03	my intuition would be
0:18:05	the
0:18:07	phonotactic very
0:18:09	um
0:18:11	thirty second utterance
0:18:12	still usable
0:18:13	oh
0:18:14	so
0:18:14	and
0:18:16	hmmm
0:18:16	would be much time
0:18:18	um
0:18:19	dimension
0:18:20	well
0:18:23	my impression
0:18:23	hmmm
0:18:24	i
0:18:25	uh_huh
0:18:25	i
0:18:26	i think it would be nice
0:18:28	if we could see some like that
0:18:30	hmmm
0:18:31	okay
0:18:41	yeah
0:18:43	i assume that
0:18:44	right
0:18:45	if you just you you know
0:18:47	yeah
0:18:48	so we set and case
0:18:50	you might
0:18:51	well you
0:18:52	some of the you you you get some nice um
0:18:55	i'm sorry
0:18:55	you need
0:18:57	you can't use an
0:18:58	no need to be
0:18:59	weeks and i these
0:19:01	well i mean it's not a nice
0:19:04	yeah
0:19:04	yeah so um yeah i i guess that's
0:19:07	table it's like the most
0:19:08	the simple technique to use them
0:19:10	uh that was like the the reason why we have used it here
0:19:13	just to see that the idea of a work or not
0:19:15	but of course i'm not saying that because the optimal thing
0:19:19	thank you
0:19:23	questions
0:19:26	yeah
0:19:27	yeah this is
0:19:28	based on ignorance but you see you see
0:19:31	my question reason in surrey
0:19:33	oh
0:19:33	use it is
0:19:34	pca
0:19:35	it's easy easy
0:19:37	yeah
0:19:38	where
0:19:39	dimension to the million
0:19:41	hmmm
0:19:43	so we choose
0:19:44	specifically you do they
0:19:46	as far as i
0:19:47	do you see a you need it
0:19:49	covariance matrix
0:19:50	we should i think i mean you know
0:19:52	in our paper we
0:19:54	we use site uh another paper of our there is
0:19:57	right
0:19:58	a technicality
0:19:59	maybe
0:20:00	a on large amounts of data
0:20:01	right efficiently
0:20:03	and uh
0:20:03	really
0:20:04	uh you can find even code
0:20:06	from a lot
0:20:07	uh for this
0:20:08	estimation of
0:20:09	pca
0:20:09	where you don't need to compute to all covariance matrix
0:20:12	but uh it is uh it is based on iterative algorithm
0:20:16	uh that uh
0:20:17	arounds like
0:20:18	uh that doesn't mean that the full covariance matrix
0:20:25	yeah well you don't if you have more questions
0:20:29	anything else
0:20:31	you

PCA-based Feature Extraction for Phonotactic Language Recognition

SESSION 10: Language recognition – phonotactics

Přidáno: 14. 7. 2010 11:08, Autor: Tomáš Mikolov, Oldřich Plchot, Ondřej Glembek, Lukáš Burget, Jan Černocký (Brno University of Technology), Délka: 0:20:38