Speech Transcript - A SPECTRAL APPROACH TO RECURSIVE END-TO-END DISTORTION ESTIMATION FOR SUB-PIXEL MOTION-COMPENSATED VIDEO CODING

0:00:13	i
0:00:13	so in in from university of a company as some babble
0:00:17	and and and present a selection
0:00:20	a a close to a recursive and and distortion estimation for
0:00:24	subpixel pixel motion compensated video coding
0:00:26	is what was down
0:00:28	a a a a batteries min an hour a let's that's it you know
0:00:32	so
0:00:32	what we do call there's
0:00:34	a employee to
0:00:35	a a temporal of the motion compensated prediction
0:00:38	uh to exploit the temporal a redundancy and that choose
0:00:43	where a high operating efficiency
0:00:45	so a a if that is
0:00:47	a when a and at uh to channel distortion
0:00:50	like the eigen use
0:00:52	is it are updated
0:00:53	due to that so or or or truncation
0:00:56	so there are many in already rescinded
0:00:58	techniques
0:00:59	like a
0:01:01	multiple description uh skin what we do coding
0:01:03	or or a need for protection extension
0:01:06	ah
0:01:07	are introduced
0:01:08	to mitigate this
0:01:09	you fight of or or propagation
0:01:12	and the basic principle of all of them is
0:01:14	that's really
0:01:15	a a screen
0:01:16	some
0:01:17	coding efficiency for the transmission us next
0:01:20	and
0:01:21	so
0:01:23	i lots and quickly you mean by on the other and it is to estimate
0:01:27	the actual and and distortion which is that
0:01:29	incorporating in the rate distortion
0:01:31	a more to optimize that's done
0:01:34	so we to use the comes all and and distortion
0:01:38	but the formal definition of are written at the uh
0:01:42	uh
0:01:43	this this
0:01:43	so the basic idea is
0:01:45	that's say we use
0:01:47	this i
0:01:47	a a and super square i
0:01:50	to denote that a a a a fixed i and frame and
0:01:53	and this is the original and two of this pixel
0:01:56	and and we use that high
0:01:58	to denote know that
0:01:59	the encoders reconstruction
0:02:01	which is only a a a a a third of to compression
0:02:04	and then we use
0:02:05	to that
0:02:06	to denote the decoders we construct a
0:02:08	which is a
0:02:09	subject to the packet loss and or assume and a higher
0:02:14	a a or a tradition
0:02:15	so
0:02:16	that
0:02:17	the end-to-end distortion is really you by the difference
0:02:20	from this
0:02:21	decoder reconstruction
0:02:23	uh
0:02:24	between ah a from this
0:02:26	to this all original binary
0:02:28	so it i that can work
0:02:30	like that
0:02:31	a a lossy compression of the channel
0:02:33	and
0:02:34	consume and what is
0:02:35	uh
0:02:36	that's space
0:02:37	but the problem is
0:02:39	oh we we do the encoder a we do the encoding process at the encoder side
0:02:44	and this
0:02:45	we to it is really and known to the encoder do two
0:02:50	the fact that
0:02:50	a a packet loss is
0:02:52	is really a i can loss is really of anything but
0:02:55	with respect to the encoder
0:02:57	oh the we to optimal per pixel estimate is
0:03:01	a a pretty well known approach to X T makes such
0:03:04	a a and and distortion
0:03:06	the basic idea is
0:03:07	uh the end and distortion
0:03:09	oh of pixel
0:03:10	can be formulated as that's
0:03:12	and
0:03:13	but decomposing composing these
0:03:14	in two or three turn
0:03:16	we we see that is really a
0:03:18	linear combination of
0:03:19	some the first and second moments of
0:03:22	we you call there's reconstruction
0:03:25	and a trees the decoder reconstruction of each pixel as seven of arrival
0:03:29	then recursively compute
0:03:31	a to the second moments of the recurrence are reconstructed pixels
0:03:34	and then
0:03:35	in in first and second moments of the the comes of a decoder reconstruction
0:03:40	a we i that the the expected end-to-end distortion i think of their side
0:03:44	and which is then incorporated it into
0:03:46	uh
0:03:47	the we use a simple a of optimization frame more
0:03:51	or the at the recursion explicitly accounts for
0:03:54	a all of the operations is and like
0:03:57	uh
0:03:58	coding and and consume and and
0:04:00	also the channel still cast it
0:04:03	um a have an extension and he's
0:04:05	whether it's these tensions have been that's that we great i
0:04:08	in into it is
0:04:09	a a method for we're raising
0:04:11	coding
0:04:12	so here are some
0:04:13	a a update equation is not no
0:04:16	the basic idea is that a
0:04:17	do the first and second moments of the reference pixel
0:04:21	we can
0:04:22	a i was compute the first and second moments of the current pixel
0:04:26	and
0:04:27	which
0:04:27	then is that the uh uh i as i expected it and and distortion
0:04:33	so
0:04:34	everything recent fine
0:04:36	but
0:04:36	there is a limitation that it
0:04:38	the come on but i it to me computes the end-to-end distortion of each pixel
0:04:43	that is pretty good from what
0:04:44	a single
0:04:45	pixel in the previously reconstructed frame
0:04:48	but many we do a regions night
0:04:50	of
0:04:51	a a set it's so motion compensation
0:04:53	but if more people reconstructed pixels in the higher for
0:04:57	a to pretty that signal
0:04:58	to all at such prediction usually in that are model uh in the format of like
0:05:03	linear combination
0:05:05	in the so called cross correlation issue
0:05:08	and
0:05:09	i was
0:05:10	a quickly you real at that time that and accurate estimation of are
0:05:15	a score nation terms
0:05:17	will you better and
0:05:19	are there are and square no
0:05:21	compute and memory units
0:05:23	where this is and where
0:05:24	you a we use these big then
0:05:26	to do you know the total
0:05:28	number of pixels in for a
0:05:30	so this is really out
0:05:32	computation computation of
0:05:33	such
0:05:34	optimal and a distortion
0:05:35	estimate
0:05:37	a i think we're
0:05:38	example to show
0:05:40	calls that's "'cause" condition term you emerges
0:05:43	in two
0:05:44	a a our update recursion
0:05:46	some that's consider and
0:05:47	by near and
0:05:48	prediction
0:05:49	the
0:05:50	and it is
0:05:51	we could just the average of two
0:05:53	a reconstructed pixel X and Y
0:05:56	so now we need to first and second moment of the
0:05:59	interpolated pixel
0:06:01	for the first moment is fine
0:06:02	mean you just
0:06:03	a and linear combination of the first
0:06:06	no of the
0:06:07	of of the reconstructed pixel
0:06:09	but for the second moment yeah the two
0:06:12	second
0:06:12	well most of the
0:06:14	reconstructed a pixel
0:06:15	there's that
0:06:16	start uh additional term
0:06:18	yeah of X Y which is the cross-correlation correlation
0:06:21	and we don't have a in our for
0:06:24	uh
0:06:25	many higher are a is
0:06:27	uh have in
0:06:28	the two
0:06:30	so a overcome these
0:06:32	each you
0:06:33	ah
0:06:34	while idea is
0:06:35	that's just a
0:06:37	a a the this cross correlation
0:06:39	but its maximum and which is provided by
0:06:42	the second moment of two marginal
0:06:44	all
0:06:46	second moment
0:06:48	however we can narrow yourself doing
0:06:51	the cross condition directly
0:06:52	we come up with a correlation coefficient
0:06:55	as a function of the distance
0:06:57	the the use of X Y is really
0:06:59	the spatial distance between two pixel
0:07:01	oh two pixels X and Y
0:07:04	and this class conditional model as
0:07:06	oh and this is uh exponentially decreasing
0:07:10	with this
0:07:11	this data
0:07:14	oh
0:07:14	in this work
0:07:15	we performance
0:07:16	an alternative or of perspective
0:07:19	in the transform domain
0:07:20	so we know that the means square or or it's really comes and they're
0:07:23	the unitary transformation
0:07:25	and that we propose
0:07:27	a some how a
0:07:29	spectrum coefficient vision wise
0:07:30	optimal recursive because if that's image
0:07:32	recall score
0:07:33	a a to it this and distortion the two in the transformed my
0:07:37	it provides a per transform coefficient
0:07:40	i made of the end-to-end and distortion
0:07:42	and things like that that's per pixel
0:07:45	but uh and i have to we you most the recursion recursive what all computation
0:07:50	a first and second moments of
0:07:51	oh
0:07:52	so some coefficients of
0:07:54	oh we well
0:07:55	yeah right
0:07:56	oh as a set of a mention here that it is really a good about
0:08:00	a can for what is
0:08:02	a the best coding operations that are perceived in the transform domain
0:08:06	but since were
0:08:07	what in this work or and exclusively focusing on
0:08:10	but and more accurate and and distortion for subpixel motion come the decoding
0:08:14	ah
0:08:16	but just put it
0:08:17	but it and that's i
0:08:19	a a basic idea or
0:08:20	we have
0:08:21	the original band you of the transmit coefficient vision
0:08:24	and you know K
0:08:26	in frame and
0:08:28	and
0:08:29	in there to construct a once again next hi
0:08:32	decoder the reconstructed X
0:08:33	a to to
0:08:34	but uh X to there is a random variable with respect to the T encoder
0:08:40	and the and is don't expect it and and distortion
0:08:42	can be from as set
0:08:44	and we C this and and not change really
0:08:48	a you can and a first and second moments of the transform coefficient
0:08:53	or for in time and
0:08:55	uh
0:08:55	the uh the is pretty much say
0:08:58	the same is now
0:08:59	so what would be days
0:09:00	with probability one man P
0:09:03	the packet that contents
0:09:05	this transform coefficient
0:09:06	what is that are right at the at the decoder
0:09:09	and was a so right
0:09:12	the decoder time a reproduced
0:09:14	it's that this the time
0:09:16	a reconstruction as the encoder so that's why we're using
0:09:19	i which is
0:09:20	in there's reconstruction
0:09:22	and with probability P this it will be about
0:09:26	and then
0:09:27	the order consume and will be calm
0:09:29	and we know that the console
0:09:31	pixel uh
0:09:32	sort to some coefficient stuff is really are than the right
0:09:35	so that's why we are using
0:09:36	X
0:09:37	a here
0:09:38	two
0:09:39	uh generate
0:09:40	a first moment of
0:09:42	but some feature
0:09:43	and same idea of ice two second or one
0:09:47	the a a a main computational right is
0:09:51	in the in
0:09:52	uh
0:09:53	a big recursion
0:09:55	where
0:09:56	T the most a reference
0:09:57	and
0:09:58	it's okay
0:09:59	it's not necessarily a great in fact uh i
0:10:03	really possible that these guys
0:10:05	all right
0:10:06	and i that we only
0:10:08	have that
0:10:09	first and second moment of transform coefficients
0:10:11	oh we well
0:10:12	so
0:10:14	we really need to generate
0:10:16	a
0:10:16	that's why what is and were using you to
0:10:19	oh
0:10:20	you K
0:10:21	but
0:10:22	a a great i
0:10:23	so we really need to generally first and second moments of a piece of work will i the
0:10:27	well
0:10:28	from those i'm great
0:10:30	what
0:10:31	a a that's just assume that we now
0:10:33	the moment all this
0:10:35	right but now the make a run
0:10:37	our update recursion
0:10:39	that's that's
0:10:39	with probably he one as P
0:10:41	we have the it which contents
0:10:43	the we but the if you
0:10:45	and with the residual and the motion vector
0:10:47	so what have the rate is you which use exactly the same as the encoder
0:10:50	but
0:10:51	the reference is star
0:10:52	it's
0:10:53	and no "'cause"
0:10:54	due to the higher or uh i can also it
0:10:56	and or or or or or a publication
0:10:59	and that probably be happy
0:11:01	assume and
0:11:02	the same thing for second moment
0:11:04	now
0:11:05	that's can see that
0:11:06	how to generate this
0:11:07	a a first and second moments of the
0:11:09	right for a
0:11:10	the
0:11:11	we can go
0:11:12	a
0:11:14	but is a can are in general
0:11:16	a a reference that you which is the right one
0:11:19	which is a a a great
0:11:21	and all the thing to do that are only but
0:11:25	so
0:11:26	a i'm not that of generate a for by four
0:11:28	reference well
0:11:30	encoder nice too
0:11:31	reference up to my and we go to to
0:11:35	a six kind uh i a filter you
0:11:38	ah
0:11:38	used for interpolation
0:11:40	i each of sixty four
0:11:43	so
0:11:44	the transformation is
0:11:45	which is typically D C is simple linear transformation
0:11:49	there exists is a constant a set of constants that right
0:11:52	we have a
0:11:53	to chosen coefficients of the a green well
0:11:56	a a a a a a a a a way and now it's really a linear combination
0:12:00	all that was
0:12:01	one great but
0:12:02	and then and we have to
0:12:03	for uh first moment
0:12:05	you
0:12:06	two young being a combination of a known
0:12:09	a hundred which is
0:12:11	exactly
0:12:12	oh
0:12:13	which is a tractable
0:12:15	but that's a commandment
0:12:16	it seems that we were getting
0:12:18	owing to this commission you choose okay
0:12:22	we need to generate this
0:12:23	cross correlation
0:12:24	which we don't know
0:12:26	by the major advantage that
0:12:27	what do everything the transform domain is that
0:12:30	are the spatial transform style as already removed
0:12:34	i i remove the correlation code uh
0:12:36	the correlation between
0:12:38	transform coefficient
0:12:40	so
0:12:41	ah
0:12:42	specifically
0:12:43	where tree
0:12:44	this is a cost on the into two categories
0:12:47	the first one
0:12:49	a
0:12:50	to transform coefficients
0:12:51	a different frequency but in the same now
0:12:54	that content i was in the same packet
0:12:57	so that by either simultaneously
0:12:59	or or or that the clutter
0:13:01	so we use X that are to uh
0:13:04	to represent
0:13:05	the you call we the packet is received
0:13:09	and X F E
0:13:10	then
0:13:10	decoder can assume in the packet is not
0:13:13	so
0:13:14	first
0:13:15	moment of these
0:13:16	a reconstruction and consume
0:13:19	are really
0:13:20	accurate and a tractable
0:13:22	and it it "'cause" that cross correlation can be
0:13:25	from the data sets
0:13:26	with probability one month P
0:13:28	input uh the packet right
0:13:30	and then
0:13:31	uh at the decoder will we produce
0:13:34	to to constructions
0:13:35	and that list that the P
0:13:37	i can last
0:13:38	because that will produce
0:13:40	so a will generate to consume and
0:13:43	so this is really the exact cross correlation
0:13:46	but that's a that we do know this
0:13:48	two
0:13:49	a a cross correlation
0:13:50	we seem to
0:13:51	a a a a a a a to make this two
0:13:54	i
0:13:55	the product of
0:13:56	to marginal of first form
0:13:59	ah
0:14:00	appealing to
0:14:01	the are going to be an is in the transform but my
0:14:05	and and that the second category
0:14:07	it's for
0:14:08	in temporal correlation
0:14:10	so
0:14:10	a recall the energy come a complex and the of dct
0:14:14	with say
0:14:15	the the uh i don't have a correlation is really dominated by
0:14:19	that be in the P C
0:14:21	components
0:14:22	and which is
0:14:23	which is that's you meant to be unity
0:14:25	and for all other easy C coefficients which
0:14:28	we we compute as that are uncorrelated
0:14:31	so we note that some more re fine
0:14:33	models are possible
0:14:35	but our experiments show that such treatment provides firmly
0:14:39	uh i accurate estimate for most nature of the
0:14:43	a here's seven
0:14:44	uh overview of the procedures of this up recursion
0:14:48	so given
0:14:49	first and second moments of
0:14:51	oh we uh are transform
0:14:53	coefficients of we well in from a my one
0:14:57	a a condition are the packet a ride or not
0:15:00	with then
0:15:01	update
0:15:02	the moments for frame and uh you frame and
0:15:05	so a first at five where the reference block base
0:15:08	and then
0:15:09	a a like this
0:15:10	first and second moments of the reference point
0:15:13	using a linear combination of the known
0:15:15	a moments
0:15:17	and the we compute
0:15:18	the
0:15:19	a a and
0:15:20	moments
0:15:21	a a helping to
0:15:22	in or inter mode
0:15:23	so either
0:15:24	oh holding model
0:15:27	so here are some
0:15:28	a a image accuracy
0:15:30	so we first try
0:15:32	uh in the slide in
0:15:33	all also all
0:15:35	where a low is known to provide the
0:15:37	a Q more and and distortion
0:15:39	so in this
0:15:40	peak sure the black i one
0:15:43	is our our a simulation where
0:15:45	we have a or where like
0:15:47	fifty two a hundred
0:15:49	uh
0:15:49	i
0:15:50	and a packet loss or realisation issue
0:15:52	and
0:15:53	the point one
0:15:54	is the and end distortion provide be by low
0:15:57	which is known to be the optimal
0:15:59	and the
0:16:00	the right one
0:16:01	which
0:16:02	uh
0:16:03	well
0:16:04	be seen
0:16:05	this
0:16:06	is the estimate provided by score
0:16:08	is C
0:16:09	uh that
0:16:10	and in the second row
0:16:11	for pixels square really
0:16:14	i from a practical a soon
0:16:16	two
0:16:17	the law which is the optimal a
0:16:20	and the in want to the subpixel pixel siding
0:16:23	where do this
0:16:24	a a what we're miss some modifications to accommodate such probably nature
0:16:30	so
0:16:30	we use a
0:16:32	this close these wires approximation
0:16:35	and
0:16:36	the pixel so model respect
0:16:38	to generate the variance of right
0:16:40	also or
0:16:41	and the two blue curves are the estimated
0:16:44	uh i estimates provided by those to my third
0:16:47	i one
0:16:48	is the simulation a house
0:16:50	a mining
0:16:51	a a and and packet loss realisation issues
0:16:54	and the the right one is
0:16:55	the uh and when distortion one by score
0:16:59	i see that's
0:16:59	score are really what war
0:17:01	i
0:17:02	and distortions
0:17:03	the second off
0:17:04	the pixel motion compensated video coding
0:17:06	so
0:17:08	in conclusion
0:17:09	i we proposed
0:17:10	a spectrum coefficient once optimal recursive
0:17:13	as as may
0:17:14	to estimate the end-to-end distortion
0:17:17	the spatial transform my
0:17:19	as as well as
0:17:21	the correlation property of the special
0:17:22	for
0:17:23	as well as energy compact set a property
0:17:27	two close and be checked the cross correlation
0:17:29	and provides a more accurate estimate of
0:17:31	uh and when distortion in the deciding of sub pixel
0:17:34	and we also know that
0:17:36	square
0:17:37	and then be shown to sub soon within in is our original no function that
0:17:43	yeah
0:17:49	so we have to one question
0:18:06	you mean of a job um as
0:18:08	yeah
0:18:09	a
0:18:10	yes so this is a a question
0:18:12	so they you is that a a a a a really person
0:18:15	where
0:18:16	a computational
0:18:17	calls compared to know
0:18:19	but the thing is
0:18:20	this
0:18:21	i Z
0:18:22	this
0:18:22	ah
0:18:24	back here
0:18:24	so this
0:18:25	set of a constant coefficient which is
0:18:28	this small i
0:18:29	which is really
0:18:30	a a a a a lot
0:18:31	spatial location dependent parameter
0:18:34	uh
0:18:34	not all of them are you can be important
0:18:36	usually the usual case is that
0:18:38	but most of them that are really
0:18:40	new vegetable able the value of them are really that
0:18:47	yeah that the
0:18:48	that's so so that there it's eight
0:18:50	uh is that are like fast algorithm within two
0:18:54	simply by these computational complexity
0:18:56	but in this work were we focusing on
0:18:59	there are uh
0:19:01	optimal he
0:19:03	okay is you
0:19:04	and

A SPECTRAL APPROACH TO RECURSIVE END-TO-END DISTORTION ESTIMATION FOR SUB-PIXEL MOTION-COMPENSATED VIDEO CODING

Video Coding

Presented by: Jingning Han, Author(s): Jingning Han, University of California Santa Barbara, United States; Vinay Melkote, Dolby Laboratories Inc., United States; Kenneth Rose, University of California Santa Barbara, United States