Speech Transcript - SCORE INFORMED AUDIO SOURCE SEPARATION USING A PARAMETRIC MODEL OF NON-NEGATIVE SPECTROGRAM

0:00:13	a
0:00:21	okay good the uh would have to on it everyone
0:00:24	i i uh or main come from the a part that can i will present you my work
0:00:29	a lot scoring
0:00:30	or you know
0:00:31	operation
0:00:32	which of them we one but no and up on that
0:00:37	so uh but buddy
0:00:39	basically this plot
0:00:41	uh deals we monaural source of separation
0:00:44	in uh music as uh spectrogram
0:00:46	so we just try to super rights
0:00:49	the the signal of
0:00:50	inch
0:00:51	each uh instrument in uh the mixture
0:00:54	but we don't do it's blindly we try to add some extra information
0:01:01	which is extracted from the score of the feast
0:01:04	and this
0:01:05	uh in an extra information is used to get the separation process
0:01:10	and
0:01:11	yeah the score is a me file which is a line
0:01:15	on which is you to be at a line is an yeah that and we do not deal
0:01:19	we uh
0:01:20	alignment matters so there is a lot of
0:01:23	it our job of this and
0:01:25	uh will walking system should uh in Q and a preprocessing pre-processing step
0:01:31	uh that
0:01:32	uh
0:01:33	we do is that months between
0:01:35	the score and the signal
0:01:38	um and more a we only deal with a harmonic instruments
0:01:41	so
0:01:42	uh we X to them uh modeling up
0:01:45	uh back to see instruments in this model
0:01:49	so um basically or system is based on a parametric spectrogram model which is derived right
0:01:56	from non-negative matrix factorisation
0:01:58	and we use
0:02:00	these uh parametric metric spectrogram the to decompose the mixture
0:02:03	a spectrogram
0:02:05	and
0:02:06	uh it consists in uh
0:02:08	but a tree time-frequency mask which are computed for
0:02:12	uh each instruments
0:02:14	and which are initialised that
0:02:16	and constraints that we as the information in the score and then finally estimated in
0:02:22	a very us as a similar way as landing the
0:02:26	a that as a uh and then F
0:02:28	uh and and so it's
0:02:29	to me to feed in the mixture your spectrogram
0:02:33	and is then uh there's are metric mask
0:02:36	are used to support to
0:02:38	this in out of
0:02:39	instruments uh a using a no filtering
0:02:43	so uh before for uh beginning uh the torque
0:02:47	uh i will the try to as well as the question why use a scroll me uh because maybe some
0:02:53	people uh will argues that's
0:02:55	it is cheating because actually
0:02:57	we the signal you don't at the score
0:03:00	but actually
0:03:01	there is a
0:03:03	that's a base
0:03:04	of
0:03:05	uh me files on Z and ten it and basically you can find
0:03:08	almost
0:03:09	um and me a you wants a about
0:03:13	i ni
0:03:14	uh not any but of a lot
0:03:17	of uh
0:03:18	of of uh a is of music on the net
0:03:22	and it's it's a very compact
0:03:24	uh description of so you
0:03:26	so if you all uh it's much more compact that's the audio it set so if you can store somewhere
0:03:33	uh the audio you will be able to store also it is very little extra information
0:03:38	and more of a i i would say that's in some cases
0:03:42	blind the separation
0:03:44	or remains very difficult S and sometimes
0:03:47	or place because if you want to separate
0:03:49	uh for example
0:03:51	uh two voices is of
0:03:53	the same instruments
0:03:54	you won't be able to uh do we blindly
0:03:59	so you're is the outline of my at
0:04:02	so first i will remain you the basic principle of
0:04:05	and then they get to matrix factorisation
0:04:07	and then a a uh i will present use apartment tick spectrogram the L that we do right from
0:04:13	uh and then F
0:04:15	and a last i we present you or uh wall score informed source separation system
0:04:21	and i will present you some results
0:04:23	off
0:04:24	this system
0:04:25	so first let's talk a bit about
0:04:27	and the map
0:04:28	so and M F is a very powerful row a wrong prediction a C is um
0:04:34	are the reason
0:04:35	uh as that's a lows
0:04:36	to extract
0:04:38	read and don't patterns in a negative that a
0:04:41	so yeah uh or a nonnegative that that is
0:04:45	the um
0:04:47	uh
0:04:48	the
0:04:50	amplitude spectrogram amplitude of or or spectrogram V
0:04:53	here it can see
0:04:55	on uh a first mode which is play alone and then
0:04:59	a second one and then the both play to get a
0:05:02	and so if you try to decompose the spectrogram
0:05:05	uh we've and non-negative matrix factorisation
0:05:09	you when you get to a matrices which is
0:05:12	the first one is the at so matrices and you we extract
0:05:17	that's some plate of one note
0:05:19	so is don't eight which is very read and in this
0:05:22	uh uh that yeah
0:05:23	and the don't plate of your are nodes and
0:05:27	the other my trees yeah as a the matrix
0:05:29	H
0:05:30	contains the information formation the temporal information so where but it is the notes are
0:05:36	and there is um negativity constraints on both
0:05:40	this matrices
0:05:42	and there is a second constraints
0:05:44	uh uh which is uh
0:05:46	the rank of the product should be uh
0:05:50	uh a lower is and
0:05:51	so wrong got the original that data
0:05:56	so
0:05:57	and F is very powerful to extract them on buttons from the data such as a nodes as a a
0:06:03	i are shown
0:06:05	and the fundamental probability
0:06:07	uh R
0:06:07	this technique is the non negativity constraint
0:06:10	which was shown to uh provide
0:06:13	uh very best it fit up it chill description of
0:06:17	the data
0:06:18	so we will use this
0:06:20	a a non negativity constraints in a or problem a tree yes make the role model
0:06:25	uh to a this plastic job uh D mentioned in uh our algorithm
0:06:30	so there are some limitation
0:06:32	uh we've
0:06:33	and then F
0:06:34	the first one is
0:06:36	that
0:06:36	it does not mean
0:06:38	to deal uh
0:06:39	efficiently we've
0:06:41	time-frequency evaluation uh
0:06:44	a a as an example when you are speech by edition over time it's very difficult to
0:06:49	um of that it's actually right she with and F and so you cannot
0:06:53	efficiently model uh
0:06:55	phenomena as
0:06:56	uh T though
0:06:58	and
0:06:59	as a on problem for
0:07:01	our our approach is uh is that
0:07:04	to do some
0:07:05	um
0:07:07	a score in phone uh
0:07:09	a source separation
0:07:12	we need a a representation which is more to the
0:07:15	uh a the parameters
0:07:16	of of
0:07:17	interest which are yeah the fundamental frequency of
0:07:21	the note
0:07:24	so we decided to um
0:07:27	develop a a parametric spectrogram more then
0:07:30	so
0:07:31	our parametric spectrogram of that is based on a pretty use one that we presented
0:07:36	in this paper
0:07:38	and which is a parameter E spectrogram of that for only one instruments also so for single instrument
0:07:45	and
0:07:46	to two but
0:07:46	this model we just as
0:07:48	why does and that's on uh look like and musical spectrogram when you are when most of the amounts
0:07:54	or uh
0:07:55	instruments notes
0:07:57	and
0:07:57	well i uh you don't have
0:07:59	uh to to back "'cause" use the
0:08:02	don't
0:08:03	so
0:08:04	most of
0:08:05	this elements are are money H
0:08:08	so
0:08:09	the part and at you
0:08:10	the buttons that you we dig extract eve and then F will be also so a money so we decided
0:08:16	just to put
0:08:17	ah
0:08:18	a one each atoms directly in the negative much factorisation
0:08:22	and
0:08:24	a to to meant to rise
0:08:25	then
0:08:26	to uh i have an excess too
0:08:28	uh the parameters of ins there are i of interest which are the for them at that frequency of
0:08:34	that um
0:08:35	and
0:08:36	as a global about uh
0:08:38	but uh all block
0:08:41	of uh that too
0:08:45	so
0:08:45	well made or that is a parametric model of spectrogram we've
0:08:49	sends at each harmonic atoms and we
0:08:52	does
0:08:54	to
0:08:55	the question is uh of this down on and then at
0:08:58	and i i
0:08:59	in a
0:09:01	the uh at my trees
0:09:03	um
0:09:05	uh dependency dependent C with respect to the parameter a
0:09:08	yeah which is a zero and which would be is the fundamental frequency
0:09:12	all that "'em"
0:09:13	and we also a and i a and then C
0:09:16	uh
0:09:17	we respect to
0:09:19	um time so basically
0:09:21	uh in or model
0:09:23	i
0:09:24	jam vary over time and
0:09:26	is
0:09:27	the makes it possible to model L
0:09:29	uh something i mean a as
0:09:31	a vibrato
0:09:33	so here we is a or a very simple that's that's models so basically we
0:09:37	a sent size
0:09:38	or
0:09:39	uh i am
0:09:40	by
0:09:41	taking the for you transform of the analysis window we use to compute the spectrogram
0:09:46	and we just sound it's on the phone them at that frequency it and on the frequency of
0:09:51	the difference
0:09:52	a money
0:09:53	and we just multiplied this
0:09:55	uh forty eight to uh this window
0:09:58	uh we've
0:09:59	and not P to the of
0:10:01	the all money so we get
0:10:04	these are meant you get some uh there
0:10:07	and
0:10:08	we thus yet
0:10:10	these parametric metric spectrogram for uh single instrument we've
0:10:15	uh the parameter
0:10:16	uh uh K which is
0:10:19	the amplitude of each harmonic
0:10:21	yeah you you of the fundamental frequency
0:10:23	or
0:10:24	each at two on
0:10:26	for each time so this fundamental frequency can vary of of time and here
0:10:30	you're are uh the activation which is very similar to the activation
0:10:34	in in in that
0:10:35	and tell you
0:10:36	uh where
0:10:38	yeah if a notes is active or a a
0:10:42	so uh when you try to
0:10:44	estimate to make these
0:10:45	a a bomb us in red
0:10:47	uh
0:10:48	so we use the um
0:10:50	bit that that since cost function
0:10:53	but
0:10:54	in fortunately
0:10:56	uh this cost function out uh
0:10:58	as a lot of
0:10:59	local minima we respect to the for them at that frequency
0:11:03	uh of course you a local minima
0:11:06	i um
0:11:08	at the position of the right from "'em" of that frequency but also at the up to have
0:11:12	and double up to uh and feast
0:11:15	if
0:11:15	the notes are very similar so
0:11:18	we channel do a global optimization we were like to the can of that frequency
0:11:22	so we decided to
0:11:24	introduce one at so on
0:11:25	for each meeting so for each not
0:11:29	off
0:11:29	the from i to scale
0:11:31	and then the optimization
0:11:33	is down
0:11:34	uh a locally so we will are
0:11:37	uh a fine estimate
0:11:39	of so from the most at frequency for uh
0:11:42	uh i
0:11:42	each atom and each time
0:11:47	so here is uh an example of suit the composition that we can get
0:11:52	we've uh all model so you're is the spectral
0:11:55	of the first bass
0:11:57	of uh
0:11:59	uh the bar
0:12:00	uh first braided which is played by a synthesiser
0:12:04	and if we try to decompose
0:12:05	it we've all or a i'm we will get this profile of activation so yeah
0:12:10	uh i just prison the activation
0:12:13	so
0:12:13	you don't out
0:12:15	a a complete you'd of the when it's and the fundamental frequency estimates
0:12:20	but as you can see
0:12:22	uh we can recognise uh the
0:12:25	note
0:12:26	in red yeah
0:12:27	which play the
0:12:28	uh the first braided by uh uh
0:12:32	but
0:12:33	so
0:12:33	as you can see there are some problems are on you know which is uh
0:12:38	uh
0:12:41	which are leading to similar to the as
0:12:44	uh we've uh or stuff and twelve send double up that but it's not really a problem
0:12:49	as we will see later in uh
0:12:52	in um
0:12:53	you know system of source separation
0:12:56	so
0:12:56	as you can see yeah
0:12:58	uh you have a
0:12:59	a a a value of activation for
0:13:02	each meeting notes so basic V these
0:13:05	looks like
0:13:06	a channel role and it would be very important in know
0:13:09	you know system
0:13:10	that it
0:13:11	these representation is
0:13:13	linked
0:13:14	to uh what's you can get we've me
0:13:18	so
0:13:19	no we have a
0:13:20	a spectrogram model for for a single instruments sets
0:13:23	was a present in yeah
0:13:25	is that i just present it you
0:13:27	and
0:13:28	we need a "'em" each model because
0:13:30	we want to separate
0:13:31	instruments so we i'll
0:13:33	a mixture juror with several instruments so the mixture model is very easy
0:13:38	uh you just a
0:13:40	the single
0:13:41	instrument for that for each instrument
0:13:43	and
0:13:44	some then
0:13:45	that's right yeah and you get
0:13:48	the mixture your spectral model
0:13:50	and
0:13:50	we we'll have to
0:13:53	estimate
0:13:55	for each instrument so for each source K
0:13:58	the fundamental frequency at
0:14:00	if for each atom at each time
0:14:03	the amplitudes
0:14:04	or or money for each source and
0:14:07	uh they're profile of activation for each source
0:14:12	and the they competition is a we've a but if you get evolve vulgar reason
0:14:16	uh which uh N sets
0:14:19	minimizing a a bit at the since
0:14:21	between
0:14:22	uh our original uh mixture spectrogram and uh
0:14:26	or power meter each steal your spectrogram
0:14:29	and this i varies them is very similar to
0:14:32	uh as the one you have
0:14:33	um
0:14:35	for in the math
0:14:36	we've met to to get to got that for
0:14:39	so
0:14:41	no we have a model for the mixture spectrogram and i will show you all we can use it
0:14:47	to um
0:14:49	and to do some score informed source separation
0:14:52	so we have
0:14:53	all mixture spectrogram and
0:14:56	or or our score
0:14:58	and from the score or uh you chan
0:15:01	you can know where the nodes all
0:15:03	for each instrument so you can very easy
0:15:06	um build
0:15:07	a the channel role
0:15:09	binary general wall each uh uh just tell you
0:15:13	where the nodes
0:15:14	are
0:15:16	and as they say that
0:15:17	uh the activation matrix
0:15:19	um of
0:15:21	each
0:15:22	uh instruments is very linked to this general and we will just
0:15:27	use
0:15:28	this general the days binary
0:15:30	a have channel roles
0:15:32	as in each sterilisation for or activation in know model
0:15:37	but as we use
0:15:38	um but if you could you at the true
0:15:42	if you put a a zero i was zero somewhere
0:15:45	uh in the activation uh
0:15:48	uh
0:15:49	matt tree
0:15:50	as N it will uh it will uh uh
0:15:54	it will remain zero all along the iteration
0:15:57	so it's a very ah constraints
0:16:01	so
0:16:02	once we
0:16:03	um
0:16:05	uh in each yeah eyes
0:16:06	our or a parametric spectrogram we get some very coarse parametric expect rounds which are represented here
0:16:13	and then we use
0:16:15	uh our our goal re
0:16:17	oh our algorithm
0:16:19	oh the mixture spectrogram
0:16:21	to finally
0:16:22	uh at that
0:16:23	uh this
0:16:24	this parametric spectral run
0:16:26	to the meat sure six spectrogram and basic iffy
0:16:29	the song of
0:16:31	is
0:16:32	three spectral spectrograms
0:16:33	should be uh
0:16:35	very similar to the mixture spectrogram that we are
0:16:39	and
0:16:40	so we get
0:16:42	this a time-frequency mask
0:16:44	and we can separate
0:16:46	as a tracks
0:16:47	a using a a of filtering
0:16:50	so if is
0:16:51	uh on example
0:16:52	so it's based and
0:16:54	sense it that's a
0:16:55	uh because it is the the ground is that
0:16:59	um
0:17:01	is this the the
0:17:02	the the signal is perfectly aligned we've
0:17:05	so me defined is that we L
0:17:07	so here is a mixture signal
0:17:13	i
0:17:14	a
0:17:16	so you
0:17:16	three instruments and using or not a reason a score
0:17:21	we get
0:17:23	oh
0:17:24	i
0:17:26	oh is uh
0:17:27	the bass
0:17:28	the base
0:17:29	i
0:17:30	it is a two
0:17:32	uh_huh
0:17:34	hmmm
0:17:35	you
0:17:36	and he has that's in the again
0:17:38	you can
0:17:39	yeah the race
0:17:40	also i mean it's i
0:17:41	i don't care about
0:17:42	oh yeah yeah
0:17:44	a more than for that which is
0:17:46	uh uh which is like noise and we only only of a model for
0:17:50	the harmonic many parts
0:17:51	and finally
0:17:52	the very night
0:17:55	i
0:18:02	so your it is uh we can probably
0:18:05	our algorithm the reason we've uh
0:18:08	another one reason which is based on probability to uh
0:18:12	probabilistic that some component than a disease
0:18:15	uh and which is
0:18:17	a somewhat different because you need to send the size a midi tracks
0:18:21	first so you need
0:18:23	um
0:18:24	you need to send to use it and you need to know
0:18:28	the instruments of
0:18:30	each uh of each trucks
0:18:33	so basically we used to it as a uh that that's sets which consists in the same uh um files
0:18:39	which are
0:18:40	uh plastic classical
0:18:41	hmmm it's
0:18:42	use each
0:18:43	but uh we sent a size uh each we've differ on sound bounce so it's
0:18:50	uh
0:18:52	so our signals are sent aside from the media because uh we need it to be perfectly online to
0:18:59	they would you and
0:19:01	yeah uh out the rays in the that we obtain
0:19:04	so in red is
0:19:06	uh something which is very similar to on all right L
0:19:09	so it's like and uh probably me
0:19:12	uh for
0:19:13	um
0:19:14	if you now by based them
0:19:16	suppression of them
0:19:18	and
0:19:19	as you can see
0:19:20	our uh algorithm
0:19:22	but from quite
0:19:23	where am and better on this
0:19:26	first the that that's set
0:19:27	then the P L C based of a reason
0:19:29	and it's
0:19:31	are the most
0:19:32	um
0:19:33	on most the it performs almost
0:19:35	saying saying
0:19:36	in uh
0:19:37	we've a the signal
0:19:39	some long
0:19:40	and
0:19:41	the main difference
0:19:43	in this
0:19:44	seconds on monk i think it's
0:19:46	uh it contains a lot
0:19:47	more all uh uh a a lot of uh review of iteration and
0:19:51	or or or a reason is very sensitive to that
0:19:55	so
0:19:56	to conclude we presented and a very efficient you a
0:19:59	for uh uh score in informed source separation
0:20:02	which is based on the parametric model
0:20:05	uh which makes it possible uh
0:20:08	to access directly um the nodes and
0:20:11	so to uh on the fine he's the sound
0:20:16	uh i think that no we should focus
0:20:19	on
0:20:20	there of instruments bit and
0:20:22	all
0:20:23	or
0:20:23	of uh
0:20:25	oh of the on which are not a monique actually because
0:20:28	a a or it
0:20:29	that is only designed to deal with
0:20:32	uh a money sounds
0:20:34	and maybe uh included
0:20:36	um more complex um
0:20:39	uh model they'll for the team or of
0:20:42	the elements
0:20:43	and
0:20:44	also um
0:20:47	a a in order to make
0:20:48	the more than
0:20:49	more robust
0:20:51	to uh
0:20:52	the real all uh
0:20:54	re signal actually
0:20:56	and
0:20:57	maybe we can also tried to use extra information as
0:21:01	uh as the P S C A based on call reason
0:21:04	no ween
0:21:05	the chamber of
0:21:07	the
0:21:08	um
0:21:09	of the instrument and using supervised learning of
0:21:12	that some plates
0:21:13	so thank you for uh you
0:21:15	attention
0:21:31	i
0:21:32	and did you make a is so you need compare compare in the gives you is the algorithm for this
0:21:37	source separation and you
0:21:39	and the you don't from that this we're and uh net may question is what does the the the i'm
0:21:44	take
0:21:44	model ring
0:21:46	with respect to the fact that you put zeros in H
0:21:49	the activation matrix
0:21:51	so have you tried running the algorithm just like putting zeros in H non in the dictionary and and H
0:21:57	and compare it with you know
0:21:58	but on a tree can uh
0:22:00	model
0:22:01	uh i i
0:22:03	i don't know this to you crush so that the to that signal to go
0:22:06	is the the bottom think there yeah and putting zeros in H
0:22:10	yeah and it was a a nice edition which is a constraint to yeah
0:22:15	so it's to
0:22:16	can uh since and zero not being yeah
0:22:19	so you sure you there isn't them longer than we the to building but
0:22:23	uh a and then would like to know if you tried uh only by putting a the zero in age
0:22:28	and what it to
0:22:30	oh using basically is a very coarse estimate of some mask yeah
0:22:34	yeah you the it's what you mean okay but it is the interest of
0:22:39	apart from being a fine estimation
0:22:41	easy job creation yes
0:22:43	okay and in so a result
0:22:45	actually if
0:22:46	you
0:22:47	uh look
0:22:49	closely uh the spectrogram there are some
0:22:52	a evaluation for clarinet and for the base
0:22:57	and so if
0:22:58	you don't
0:22:59	uh
0:23:00	you don't to take
0:23:02	this
0:23:03	valuations into account
0:23:05	you read
0:23:06	a a very bad as the person results
0:23:09	and moreover or maybe you can out some problem of
0:23:12	tuning tuning
0:23:13	and
0:23:14	a the fundamental frequency
0:23:16	even if it
0:23:17	not moving of a of time
0:23:19	uh can be
0:23:20	slightly different on from
0:23:22	the equal to a month so if you're a yeah this studies
0:23:26	i shall i Z to the you but some parameter
0:23:28	we
0:23:29	a provide
0:23:30	bad suppression rise as i think
0:23:33	to you
0:23:36	anything else
0:23:39	use also a quite it's so thank you

SCORE INFORMED AUDIO SOURCE SEPARATION USING A PARAMETRIC MODEL OF NON-NEGATIVE SPECTROGRAM

Music Signal Processing

Presented by: Romain Hennequin, Author(s): Romain Hennequin, Bertrand David, Roland Badeau, Institut TELECOM / TELECOM ParisTech, France