Speech Transcript - AUTOMATIC MUSICAL THUMBNAILING BASED ON AUDIO OBJECT LOCALIZATION AND ITS EVALUATION

0:00:13	so much
0:00:13	um yeah
0:00:15	a my name is here so but are from now that of science and technology chip and that today i
0:00:20	talk about the automatic music
0:00:22	some naming based and the or the object will write issue
0:00:27	so this is a it's while but this talk the press
0:00:30	i
0:00:30	it's sprain them are our motivation in the back um
0:00:33	and the next the i X for in the spatial information C it's ms of based on the of i'm
0:00:39	best
0:00:40	i k-means clustering and uh after the experiment result
0:00:44	i will can
0:00:47	so this is that that one or or or or or our research
0:00:50	the image "'cause" common eh
0:00:52	a a feature is a the scraps
0:00:55	of of the uh music to a disk chan uh show the abstract information of the musical tune
0:01:03	a a a a for example
0:01:05	that we can hear if we can cure the some easy call some no
0:01:09	and uh we can easily and this ten and that i was strapped all the you formation of the tune
0:01:15	and the
0:01:15	we can easily i judge
0:01:18	to buy it
0:01:19	we not
0:01:20	uh i a pretty far about one
0:01:23	so
0:01:23	the as a music because some no
0:01:25	is a very important
0:01:27	but the problem is that common tree the music some nodes um mainly made about nine thirty
0:01:32	so that that very big problem because the uh
0:01:35	uh there is so many do D in
0:01:37	uh doing a is this so many times a so many music tunes
0:01:42	exist
0:01:43	and uh for example the
0:01:44	so many music uh
0:01:46	uh produce
0:01:47	a by the current to be shows as we as the uh all these
0:01:52	so
0:01:52	the one difficulty the is uh uh a how to deal use
0:01:57	the means go tuned
0:01:59	a i and the make the some name
0:02:02	in into a
0:02:05	by
0:02:05	but to not a ways
0:02:07	and the and seven
0:02:08	problem is the back so have now provide that's that very files
0:02:13	and the bound to the time segment
0:02:15	so for example random pick up a have the scraps
0:02:18	are the two is the bit about way
0:02:21	so this is a very random ready scrapped and uh
0:02:25	um abstract information is not so good
0:02:28	oh
0:02:29	the
0:02:30	i i go or is that to a uh a the technology for and the thing the weight can strapped
0:02:36	and of the musical tune
0:02:41	that relate it to what is
0:02:42	the
0:02:44	a this one and this one
0:02:46	and uh this one is based on the construction or is is by the beach tracking
0:02:51	and uh also the uh this one is uh construction and standing by the main male would be and Y
0:02:57	so the but mister
0:02:59	a a you deal with the uh more or signal
0:03:03	and also both met sort
0:03:05	ah
0:03:07	try to extract this hum middle D what be
0:03:11	if
0:03:13	uh
0:03:13	but uh
0:03:15	uh
0:03:16	in this talk
0:03:17	i is it
0:03:19	i propose
0:03:19	to use them under of the alternative information
0:03:23	the for example
0:03:24	the most three a a a very but music to and is this still we'll we're a channel for lot
0:03:31	so we can easily we extract the spatial information is that the are here such at time temporal information
0:03:38	oh that the motivation
0:03:42	so the the uh this the research and is
0:03:46	the we provide
0:03:47	we try to provide
0:03:49	the are kind tape
0:03:51	a Q
0:03:52	for making them is you call us some now reading
0:03:56	the is that are the uh
0:03:58	temporal information that for cheese used in a conventional missile
0:04:03	the fast
0:04:03	i propose the sum and noise this mess
0:04:06	for of the uh for peak picking at the
0:04:09	spatial information of the two
0:04:11	and the the neck
0:04:13	uh we we a have the uh some uh investigation are be some difference
0:04:18	with the L a proposed ms so and this some temporal based ms so
0:04:22	and then we propose the might
0:04:25	approach
0:04:28	so i
0:04:29	but going to be uh estimation
0:04:31	a a all the uh all the obvious yeah
0:04:34	so we estimate
0:04:36	that's spatial information of the uh included in the audio tunes
0:04:42	the for example
0:04:43	that dramas and some uh questions and uh
0:04:46	well clothes
0:04:47	a are it at the send
0:04:49	normally money
0:04:50	but sometimes
0:04:52	and the uh
0:04:54	and the P are known and uh uh side a get or something
0:04:57	uh located at the uh site
0:05:00	so
0:05:01	the to go to an have D S a structure of the spatial direction
0:05:06	i a as the temporal direction
0:05:08	so that we
0:05:10	it want to extract that this spatial information but using the sum of quantization technique
0:05:19	this is uh a but we all the estimation process of the of the object
0:05:24	well as we up right time fourier analysis right at like the uh commission method
0:05:30	and uh
0:05:32	we do the some k-means means right clustering technique
0:05:35	um but and that i will explain the
0:05:38	and uh we pick up
0:05:40	thus of quantization but the that
0:05:43	expresses is that direction
0:05:45	oh the each
0:05:46	or the was yeah each instrument
0:05:49	and the then
0:05:50	the uh we
0:05:52	but also the extract the sum activation to be patient function
0:05:56	all the uh each
0:05:58	uh object
0:05:59	then
0:06:00	and the weak classifier
0:06:03	and the we
0:06:04	uh we detect
0:06:05	that changing time all would be uh some structure yeah or of the uh uh some
0:06:11	um music two
0:06:14	so this is uh a model all the a time-frequency input signal to it at the
0:06:19	so so this is the general form but the no nodding most three we use
0:06:23	only them
0:06:24	uh to channel signal as the still form a so and equal to
0:06:30	and uh this is a quantization but that we want to
0:06:33	put it into the this
0:06:35	signal
0:06:36	and that this is yeah the uh uh a sorry for B is the in the
0:06:41	um maybe the in
0:06:43	so this is a mistake
0:06:44	um that this is that any given engine now can tie this sum bit to
0:06:48	but that this and can be that the mean by our so
0:06:54	and that is is a figure yeah are be uh from
0:06:57	a
0:06:58	spectral read
0:07:00	uh exist in their right
0:07:02	hand side and the left hand side channel
0:07:04	so
0:07:05	in a
0:07:06	frequency and the time
0:07:08	uh a means that
0:07:09	signal now that's still signal i have the such a configuration
0:07:14	then
0:07:15	we you one to
0:07:17	a a put the sum and the system better the right this
0:07:20	the one the express is that one direction X be a some
0:07:25	so sees
0:07:26	the and the for example this direction mean
0:07:29	uh a present that almost the sent the rock at eight
0:07:33	uh instruments right the dramas and the vocal
0:07:36	and that this
0:07:37	uh left side
0:07:39	a a a bit to mean the left on side instruments reich that's side you tell what something
0:07:46	so this is uh clustering string
0:07:49	uh mathematics in the press ring
0:07:52	the we want to
0:07:54	classify
0:07:55	the each object
0:07:57	but from not i'm of the each cell
0:08:00	but the only the direction
0:08:02	R be uh is this is a a a a difference from the a conventional and the normal approach to
0:08:08	the for example that we
0:08:10	we are i don't want to take yeah the i was only to about all that each component at the
0:08:17	left and the right hand
0:08:18	sound they would domain
0:08:19	but that we want to extract that
0:08:22	and cool
0:08:23	uh from the uh quantization back to
0:08:26	so oh
0:08:29	so this is that input put at the one three can see under one time
0:08:33	and that this is that one
0:08:35	chan data of that call quantization back to
0:08:38	yeah but that this point i the some back that is normal
0:08:42	as a as you need
0:08:44	for example
0:08:45	and the we calculate not that you a difference between the this
0:08:50	that there is
0:08:51	but uh we want to cry to eight the in a from the um
0:08:55	all the from the upon point this but on whole
0:08:58	and that is input signal bit on
0:09:02	so
0:09:03	the k-means clustering
0:09:05	a a is uh some
0:09:07	time the
0:09:08	something though
0:09:09	where before like
0:09:10	the first we set the in for us
0:09:13	and uh some it's sensor it that this see the initial right
0:09:17	and and that
0:09:17	we update the centroid
0:09:19	uh right B S
0:09:21	the we cut greek but such a a a a or sign ever
0:09:24	cosine signed distance and of a a a a with we round us
0:09:28	signals
0:09:30	and the but that this scroll saying error a is simplified the this so this problem
0:09:36	is very simplified
0:09:37	uh a right the uh this solution is given by the finding the maximum eigenvalue problem
0:09:43	or that this correlation function of the input signal we thing that one for us
0:09:49	then that we solve this the maximization problem uh using the sum it's B D or something
0:09:55	then we
0:09:57	a uh you define the
0:09:59	centroid new centre right
0:10:01	then
0:10:02	uh we define the new
0:10:04	you a and all the a price
0:10:06	then
0:10:07	go back to a
0:10:08	so and that we calculate that this
0:10:10	a hmmm
0:10:13	a of to buy station for each class
0:10:15	each in four
0:10:20	the factory
0:10:21	no we obtain the optimal quantization back to at each and H T close that such a a uh since
0:10:27	centroid the C
0:10:28	and that that press flight each component to in an old you all and that's all process index i
0:10:35	yeah the this
0:10:36	class index i is the important because the a it just right
0:10:40	the representation of the activation are be a round direction
0:10:44	at the time and if we can see
0:10:47	so that we or would be it
0:10:49	a
0:10:50	uh
0:10:51	trust index function is or audio object localization
0:10:55	so that mean the at that one
0:10:58	at uh
0:11:00	frequency and can time point
0:11:02	the if the i function equals and
0:11:05	and for example the one and two and three
0:11:07	and that there
0:11:08	um the this very equal one um that the wide do so this
0:11:13	mean the
0:11:14	some up to be should so
0:11:16	right see the uh example of the P yeah or the set yeah i function
0:11:22	the
0:11:22	this is that we all music tune
0:11:24	uh the praying the trump bit but prone tramp bad bass and drums and that's that's form
0:11:31	and the entire bow a
0:11:33	is is the that the or all right
0:11:35	and the entire bar be E the saxophone solo pot
0:11:39	so i think you can see the uh in the one
0:11:42	class is very active at that in borrow a be the a trump it is very active
0:11:48	and then we see the very short
0:11:51	it re all or a T V active area or a for channels of that this the mean that row
0:11:57	few we
0:11:58	the the purely
0:11:59	but it short there three
0:12:01	and then
0:12:01	sift
0:12:02	to D uh in the L be at that
0:12:05	sets once so so this is the sex
0:12:07	so this is just like the sum
0:12:09	separation results of the aspects down
0:12:16	and that there that we
0:12:18	well do you want to pick up that changing time point
0:12:21	so that we more simplify the information
0:12:25	that we march or the such that the a you for make up to be shown information
0:12:31	uh a a all with that the see we march
0:12:34	and then we define a this is the a based P at the time
0:12:38	so that a we give the some frequency weighting
0:12:42	and that this is the sum example
0:12:44	or the us
0:12:45	block this um do then T
0:12:47	so
0:12:48	a is the changing time be L structure change point
0:12:52	so the first and this back and
0:12:54	crass and is that
0:12:56	that maybe that one
0:12:57	it's a we up
0:12:59	and that then of the the changing point
0:13:02	the and and that's that
0:13:04	S S is a
0:13:06	so
0:13:06	this mean that the
0:13:08	and the instrument
0:13:10	uh is
0:13:11	a the a yeah be of and then save to the uh press one so this is just like this
0:13:17	um
0:13:17	um
0:13:18	i to be shown
0:13:20	sequence
0:13:21	uh i run with the spatial
0:13:23	direction
0:13:26	so
0:13:27	we define
0:13:28	and that we want to pick up
0:13:31	that that i mean that is
0:13:33	so
0:13:33	being point
0:13:35	changing point right it
0:13:39	the sometimes
0:13:40	there are the sound fine at fat to write this some vibration
0:13:45	so uh and the we
0:13:46	a a a a some losing the very simple simple supposing technique technical and with the time
0:13:52	and also also do we do the we ring
0:13:55	R be at this a requisition things P
0:13:58	and uh into the ability to get the number of classes
0:14:02	as the for example that we a assumed a for a little
0:14:06	stay it the first date
0:14:08	that one
0:14:09	re on side is that sent i that B
0:14:13	only the right hand side is that the
0:14:16	and
0:14:17	or
0:14:17	doubt where this
0:14:19	signal is
0:14:21	a
0:14:22	so we classified the full state
0:14:25	and that this
0:14:26	a a close is that very robust to result
0:14:30	okay
0:14:31	rats go to the evaluation
0:14:34	the we do the experiment using that out that was C P
0:14:38	popular pure music database that we you got the twenty five
0:14:42	a a popular music signal that the see the john
0:14:45	that he pop rock are up they can goes the pops
0:14:49	the uh is on bruce or metal
0:14:52	and the we money you white put uh a a two hundred sixty seven structure changed time
0:14:57	by man you're right
0:14:59	in the database that which are regarded as the correct
0:15:02	so so in this our experiment
0:15:05	the number of was is that to so that this is a
0:15:08	the real marked
0:15:09	and uh we
0:15:11	set the quantization but the the number of sent on this some but the
0:15:15	and
0:15:15	E so we
0:15:20	so this is uh a
0:15:21	a a result using the will propose a men so
0:15:25	the number of or or the quite cells is that two hundred sixty seven
0:15:29	and uh
0:15:31	we pick gap though one hundred to ninety three
0:15:34	and
0:15:35	or
0:15:35	all the uh correct on so the with the D is uh not do that it that this is the
0:15:40	fire detection
0:15:41	so that precision that the record is a most the uh a point seven seventy percent more than seven me
0:15:47	person
0:15:47	and the a major
0:15:49	is there a
0:15:51	the point seven
0:15:52	for all
0:15:53	so the more than seventy percent
0:15:56	uh a detection are correct
0:16:01	so
0:16:02	the somebody
0:16:04	a a a a a have a some uh interest
0:16:08	in the comparison with the this
0:16:10	a a a spatial information base ms so
0:16:13	and the we've D S some conventional temporal based missile
0:16:17	so that we compare
0:16:19	the we uh do the experiment with the apple bit tape method it's that P L C A proposed by
0:16:26	white
0:16:27	uh to sell them ten that this is that and map based ms
0:16:31	and automatic detect detector to duration
0:16:34	and that yeah do you know so this is the temporal based missile
0:16:39	the as of like the
0:16:40	a a and then stuff can be a conventional missile
0:16:44	so
0:16:45	but that
0:16:45	a is not that i propose a is the to talk talking
0:16:49	and that's spatial
0:16:50	base
0:16:51	mission
0:16:52	so that they say that that's the comprise and we
0:16:55	uh i think priest showed the uh
0:16:57	precision and the recall and if me
0:16:59	so at the you can see
0:17:01	the uh in a if a major
0:17:03	then not so much
0:17:05	a a a a a different
0:17:08	alright is
0:17:09	in in in that if it's itself
0:17:11	but
0:17:13	the contents
0:17:14	the re are detection
0:17:16	behave year
0:17:17	is before and so this is a some investigation the part the difference
0:17:22	or be a propose the spatial based ms so we've the it's some temporal basements missile
0:17:26	yeah
0:17:27	that we
0:17:29	how to wait the some you relation P here all the uh conventional ms and uh proposed mess
0:17:36	so the this
0:17:38	um the
0:17:40	one hundred twenty seven detection
0:17:43	uh uh
0:17:45	by whole
0:17:46	ms
0:17:47	but uh
0:17:48	fifty is three detection
0:17:51	only only at that
0:17:52	he detected it only by the proposed mess
0:17:55	so this is a detected by the spatial information
0:17:59	and and the side
0:18:00	the forty nine detections
0:18:02	only by probable at the
0:18:04	detected by the if that P yeah a so the see the temporal
0:18:08	result
0:18:09	so
0:18:09	at the you can see
0:18:11	the yeah yeah is very similar or local you relation
0:18:15	um
0:18:16	or or they uh if a major is the one the same but the a from the desired is a
0:18:22	very different
0:18:23	so
0:18:24	and uh also so this is a very complimentary
0:18:28	so um then next that is that we apply this sum
0:18:32	a margin technique
0:18:34	yeah
0:18:35	so
0:18:36	a maybe may you you you do in the uh
0:18:38	we have this so many idea of the matching technique but that in this
0:18:44	and i
0:18:45	a right a very simple one
0:18:47	but row or operation the very simple one but the very effective
0:18:52	so this is the result of the uh
0:18:54	marched up two
0:18:56	so that we can see the very good
0:18:59	if a major
0:19:00	so the the baseline the proposed a mess so that's a show on B
0:19:05	so
0:19:06	ah a
0:19:08	the if major is at the point a seven for what the this my suit
0:19:12	technique
0:19:13	a a deep the uh more than eighty percent
0:19:16	accuracy the the so
0:19:18	the in control region the spatial and temporal information
0:19:23	yeah be a very good
0:19:24	information and the very complimentary
0:19:27	so that if we
0:19:29	gives a
0:19:29	the bells
0:19:30	that we caff a the a better the result
0:19:34	so this is uh can jumps that i propose the new alternative mess
0:19:40	to detect the changing point all the M is you to post music some naming
0:19:45	the conventional a method is based on the temporal structure extract
0:19:49	it's tracking ms
0:19:50	but the uh will probable the ms not lead the spatial
0:19:54	uh information based ms
0:19:56	that we detect
0:19:57	that changing the time of the dsp shows structure
0:20:01	or be a music amount chan beauty to
0:20:04	so that they the
0:20:06	using the i miss that the seventy per and at U S C
0:20:10	uh we here
0:20:11	but uh um where are are Q a three we here have we can get
0:20:16	if we use the
0:20:18	mod
0:20:18	approaches is with the uh
0:20:20	uh temporal information or temporal based miss out
0:20:24	the thank you so much
0:20:30	for
0:20:31	i i have the time the i i want to show that the more all the uh some narrow
0:20:36	okay
0:20:46	for
0:20:51	yeah
0:20:52	i mean um the
0:20:55	that are so that's so
0:20:58	yeah
0:21:02	yeah yeah
0:21:03	i
0:21:08	that
0:21:09	this is the second pass
0:21:18	i
0:21:18	yeah than
0:21:20	i
0:21:22	i
0:21:22	i
0:21:25	and
0:21:26	i
0:21:29	yeah
0:21:30	a i
0:21:33	john
0:21:38	the high that i had a
0:21:40	or not
0:21:41	i
0:21:44	a
0:21:44	a
0:21:45	i
0:21:47	and
0:21:48	a
0:21:49	a
0:21:51	this is a
0:21:52	i
0:21:54	i
0:21:55	in a set
0:21:56	that you can and that's standard that is music
0:21:58	yeah
0:21:59	okay thanks somewhat
0:22:01	i'm not sure that i'd by the music
0:22:02	that might be a my tear is not to
0:22:04	i
0:22:07	i i do have one question i haven't thought about audio summary
0:22:10	but
0:22:11	it be some video summaries of movies the that real are often does not look at all like that was
0:22:15	normal be
0:22:17	and a morning with the white answer is for music do you want
0:22:20	just excerpts service you want
0:22:21	summing style are used
0:22:23	what's
0:22:24	what's gonna be best the mail
0:22:27	and if you if you are disk could design or on from know will the use
0:22:31	we only do with the uh music
0:22:34	B six them out
0:22:35	a by the much on the L format
0:22:37	yeah
0:22:39	what what would be the ideal summary
0:22:42	some are we you know or from you know i mean if you could do you want summary what we
0:22:45	look like
0:22:46	no no things you know what user can
0:22:49	um but from some some the mean this some the house say not
0:22:54	like the you know
0:22:55	some no
0:22:57	the what what would be the ideal of male for piece of music
0:23:01	and the
0:23:02	this one
0:23:04	no in general would be just pieces of music would be a beginning a middle of you and would it
0:23:09	be
0:23:10	something completely different
0:23:12	um
0:23:14	so
0:23:15	so in there
0:23:16	put a music that very easy to detect the speech whom
0:23:20	information and the we can easily make this some now
0:23:23	but the for general
0:23:25	and uh some time uh
0:23:27	some use a is that very difficult
0:23:30	to deal with
0:23:31	a a for example of this something funny music
0:23:34	uh is difficult but the
0:23:37	well though the this is not so just you you are
0:23:40	uh
0:23:41	and sell your president but that
0:23:43	on the
0:23:44	the one
0:23:46	in a general
0:23:47	to deal with that know music
0:23:50	the oh one big problem is uh
0:23:53	many
0:23:54	instrument music
0:23:56	right the seem twenty and a is such a case that we
0:23:59	a um
0:24:00	thus so many
0:24:02	quantization vector
0:24:04	yeah
0:24:05	that
0:24:05	the after that we can
0:24:07	it it the sum
0:24:08	on changing point but the
0:24:10	some classification are with them is a very
0:24:13	complex
0:24:15	okay
0:24:19	but not turn solutions of colleagues minor doing is actually putting it up on the a web single people click
0:24:23	on
0:24:24	so we doing for image from males
0:24:26	and so then we can actually measure
0:24:28	click to performance a way to right
0:24:31	for males
0:24:31	hmmm
0:24:32	which might be points of view at some point
0:24:36	you know questions
0:24:39	make much
0:24:40	thank you

AUTOMATIC MUSICAL THUMBNAILING BASED ON AUDIO OBJECT LOCALIZATION AND ITS EVALUATION

Music Signal Processing

Presented by: Hiroshi Saruwatari, Author(s): Hiroyuki Nawata, Noriyoshi Kamado, Hiroshi Saruwatari, Kiyohiro Shikano, Nara Institute of Science and Technology, Japan