Přepis řeči - INCOHERENT COLOR FRAMES FOR COMPRESSIVE DEMOSAICING

0:00:16	okay
0:00:16	a
0:00:17	the next oh is my talk
0:00:19	and and this work was done in collaboration with the by to graduate students
0:00:24	you moment and then and from and the goods there
0:00:27	and motor G calm more from an eastman kodak research flipped
0:00:32	um
0:00:33	that time for the top basically all the to be simple i'm gonna start uh by giving good a fairly
0:00:38	high level uh overview of the problem of you as can just sink case there are people may not be
0:00:42	familiar with a
0:00:44	and then i'll talk little bit about the this compressive demosaicing framework that the we have it to to do
0:00:50	reduce recently
0:00:51	and then not talk about colour frames for compressing
0:00:54	compressed you missing
0:00:58	so uh again a bit introduction to the problem of demosaicing mosaic
0:01:03	um
0:01:04	we are familiar with the fact that uh images
0:01:07	at is the way human visual system perceive then uh requires three colour plane
0:01:12	uh traditionally additionally red green and blue
0:01:15	and
0:01:16	fact if you wanna a capture the such images uh all these three color planes you let's roll need three
0:01:22	uh C C D sensors in your come
0:01:25	i the problem with this of course is that there is tremendous amount of course uh and also that size
0:01:30	issues
0:01:31	so i to the all the come as that uh uh with
0:01:34	and a multiple C C D sensors
0:01:37	so it turns out that um
0:01:39	the vast majority maybe almost every single kind of that
0:01:42	people have including the ones maybe in your i i've phone or i part of rubber
0:01:47	a i actually uses only a single uh C C D
0:01:51	uh
0:01:52	says and the way this is uh don is basically a they put what's known is a colour filter already
0:01:58	which literally it admits all the single
0:02:01	colour per pixel in instead of
0:02:03	capturing in three
0:02:04	uh pixels
0:02:06	so um
0:02:08	traditionally additionally if you look actually if you are a lot to look inside the got of if you're a
0:02:12	which we usually cannot not
0:02:15	uh you'll see actually the and may each on the right
0:02:18	uh uh is the one that is captured why the camera
0:02:22	and uh you know this is a original image so let's really we do a you do not have access
0:02:27	to all the colours that exist in the original image
0:02:30	uh obviously the image that's kept it is very much dictated by and the nature of the
0:02:37	colour for tell in colour pattern of that C F eight
0:02:40	i the most popular or uh C F A it used or which was developed uh by X actually
0:02:46	uh uh is known as the bayer filter out uh which should basically uses to green colours
0:02:52	for every you read then a uh and a green but uh i'm sorry
0:02:55	to green colours for a really right and blue pixel
0:02:59	and in any to by two uh a block of range
0:03:03	fact if you lose now uh you know all as in here looking at the and is just looks kind
0:03:07	of green but you only doing to put a population
0:03:10	how well but if you actually zoom and fact that the
0:03:12	my presentation which was a powerpoint
0:03:14	unfortunately for had some distortion
0:03:17	if you zoom in actually at the zoom more could see the red and blue pixels in addition to the
0:03:21	green
0:03:23	okay so how do you actually do you mosaic the image i mean the problem due mosaic basically recovering the
0:03:28	original red green and blue uh
0:03:31	signal
0:03:32	uh from what you have capture which just a single pixel a single colour per pixel
0:03:37	well that are basically you know to type of
0:03:39	redundancies and dependencies that exist in your uh to signal but you could exploit
0:03:45	uh the first one which is the obvious one is to the spatial or the pixel
0:03:49	i dependencies in here if you look at the
0:03:52	but you capture in of the right
0:03:54	the green and W blue obvious you have a lot of missing pixels and lot false
0:03:58	a let's really could use any form of interpolation or some kind of smart
0:04:02	frequency domain um
0:04:04	prices to really fill in the gap
0:04:07	uh that that kind of the is see that you can exploit is uh let's really the um
0:04:12	depends is that exist among the
0:04:15	the colours them self you know there is a fair amount of
0:04:18	but done the see the so if you look at the red green and
0:04:21	uh and blue channels uh in a tradition of people you go to a colour difference step of think will
0:04:27	from there with that from a been compression standards
0:04:29	and then to be able to also exploit that and uh i you get a just some form
0:04:34	fact they've and uh of sparsity by doing that
0:04:37	so you know i i don't know that are probably about ten maybe you know close to hundreds of
0:04:42	you know do you more taking order them is that actually tried to solve this problem and try to recover
0:04:47	the original red green is you know with different the variations on this
0:04:51	you know theme
0:04:53	um um just to give you a flavour about you know the challenges associate to this problem uh
0:04:58	in a list take a look at
0:04:59	what's really merging as
0:05:01	we could call what the the economic canonical test image for that you music can problem
0:05:06	uh this is known as the lighthouse image
0:05:08	and particle that is this block which is the fancy sarah because has a very high spatial frequency
0:05:14	as quite challenging actually to cover all the three colours
0:05:17	uh you know from any new kind of colour of white or black or great
0:05:22	"'cause" all close colours actually the are are present
0:05:25	so this is an example like you
0:05:27	uh some of the leading approach is already at approach is that are highly
0:05:31	um
0:05:32	um um a the and the literature
0:05:35	and you could really see that our a fair amount of you know artifacts actually few trying to do that
0:05:41	uh and the
0:05:43	you know net really trying to criticise these particle to two approaches in how a just wanna give you uh
0:05:47	flavour
0:05:48	uh for the you know the challenges that you really could phase despite the fact that you know these approaches
0:05:53	are the based on very sound
0:05:55	uh i know that skull and theoretical a frame or
0:05:58	uh the are actually approach approaches which do uh uh uh better uh are good be much better
0:06:03	and again i just you want to to highlight like you know some of the challenges you could see and
0:06:07	these are examples
0:06:09	a a of uh some of that affects because
0:06:12	so how could we map the problem of you mosaicing to confront sensing well um it's relatively easy actually from
0:06:18	in principle uh i
0:06:20	you know them uh the problem doom as a king you you basically doing
0:06:24	compressed sensing in the sense that you are
0:06:27	do compress a same by factor of three
0:06:29	and uh you could look at your account not a C F A and that to present or sensing metrics
0:06:35	in
0:06:36	and the compressed sensing setting
0:06:37	and uh you are made you could you know uh choose any sparsifying kind of dictionary
0:06:43	uh that you'd like an try to recover the signal uh you know based on that
0:06:47	um
0:06:48	the that be an but if you work actually apply compressed sensing to this problem
0:06:52	uh
0:06:53	nevertheless a you know especially if you focus on the fact that
0:06:57	the kind see if a something you cannot not do much about
0:07:00	and most of the work really been done in the context of using
0:07:04	the um
0:07:05	uh the they are packed them
0:07:07	uh by the way uh actually really emphasise you know the
0:07:10	this project or
0:07:11	uh has
0:07:13	looked really horrible
0:07:14	colour
0:07:15	artifacts so
0:07:16	i we have a lip college as i'm you could tell this is not an image processing conference so
0:07:21	um at anyway
0:07:23	maybe we'll
0:07:24	one be the same token i are people
0:07:26	and i i any right so um
0:07:28	so the
0:07:31	in here these supposed to be yeah you know green but they look like a at any anyway
0:07:35	so given that the C F is actually is given you that is not what you could do about the
0:07:39	of the focus is on
0:07:40	you know what is really the sparse of a sparsifying dictionary right here
0:07:44	so uh the lead and work and the say it really been don uh at is my opinion by uh
0:07:49	more well uh jewel and or L and uh some of his colleagues
0:07:53	well the actually the a a a whole bunch of all learning
0:07:56	a a line i'm sorry learning link dictionary all go them uh that actually tried to uh figure out what
0:08:02	is the optimal
0:08:03	diction there's the sparse one of the could use in order to recover the colours
0:08:07	and they have a whole bunch of uh you know techniques uh a some of them uh uh the most
0:08:12	actually problem the one and the like this one
0:08:14	the coloured learned simple to a sparse coding a less
0:08:17	i C
0:08:18	and this is some of the results this is some of the are it was also actually to see some
0:08:22	artifact and got improve significantly
0:08:24	through a last
0:08:25	i C
0:08:27	so um this approach even this learning approach actually is
0:08:31	um a i a is a quite promising still has a whole bunch of problem
0:08:36	uh in fact a this particle image and i'm not sure of men a built to zoom but
0:08:40	uh in here uh yeah to look at the snow at and i believe i have an image yes
0:08:45	and that if you can see it but this is the original actually snow
0:08:48	and this is the a cover of through this out them on you know sparsifying dictionary and
0:08:52	hopefully "'cause" see that our uh some fair amount of facts actually the convolution and white
0:08:58	uh you know so the the two colours rule not be recovered
0:09:03	so um so what we have developed act is an alternative framework which we call compressive demosaicing K and it's
0:09:09	it's fairly simple actually
0:09:10	uh again but i apologise this is supposed to be green you know maybe it's it's is gonna some be
0:09:15	people's eyes but
0:09:16	so in here uh basically you could be the image or presented through three matches is that a green and
0:09:22	blue
0:09:22	and these forty three images are are being uh multiplied through this uh as in a simple point wise multiplication
0:09:30	uh had the are the top a multiplication by three different uh mattress mattresses
0:09:35	and we have a linear combination uh to present the measurement that we actually have
0:09:40	so um if you put everything together in terms of metrics for you could the vector or an image that
0:09:45	the vector right the are red green "'em" we'll that you're trying to recover
0:09:49	and this uh
0:09:51	uh
0:09:52	multiple uh multiple just as an here they are present really the different for presentation of your sense
0:09:58	and didn't of what you capturing
0:09:59	this represent present of course a little bit the more general
0:10:02	you know a a a a framework contains of it does not have to work it with the bayer pat
0:10:06	and but you you could capture any kind of pat then
0:10:09	the the same time you're actually i i i'd hearing to the constraint which is very important for the problem
0:10:14	you was aching
0:10:14	which is capturing all a single
0:10:17	colour or pixel
0:10:18	and a but important distinction there that colour that you capture lack you has to belong to that article pixel
0:10:24	you cannot do and any linear combination anything
0:10:27	that's why you C Ds match this is actually they are gonna
0:10:30	so this is a very important constraint that is not much you could do a what
0:10:33	and that there was you cannot not generalise
0:10:36	this matrix anymore
0:10:37	uh that's the most general could have
0:10:40	okay so um
0:10:42	so basically in a few uh
0:10:44	uh a you know
0:10:45	a to this kind of framework which is a a of simple now uh the idea is to go to
0:10:50	the rgb vector eyes an agent tried to replace a basically with the sparse representation as we have done before
0:10:56	but is no or you not much new they are so now we could represent
0:10:59	some kind of frequency or presentation all the different
0:11:03	coloured the R G and B
0:11:05	and now we could actually use different uh dictionary so if you put everything together again
0:11:09	um uh uh now we have again you C cfa image or C F i'm sorry uh at tricks which
0:11:15	is sense the metrics and then you have
0:11:17	you're a sparsifying dictionary and now we have the flexibility of using different dictionaries actually for different colour planes
0:11:23	uh if you wish to
0:11:25	um
0:11:26	now uh
0:11:27	up to this point actually um you know that is really not much significant improvement if you try this kind
0:11:33	of framework which is really very simple and you could of course go and try to
0:11:37	uh find the sparse error back to sell
0:11:40	the biggest problem of this approach actually and in general are in fact it's if you still
0:11:44	uh operate and a three dimensional colour space and does out of its our G V or Y V
0:11:51	then you really not exploiting the
0:11:54	the uh
0:11:55	core correlation among the different colour planes and more specifically you really cannot get much sparsity
0:12:01	so this this uh back to to self it's actually sparse is not sparse in a
0:12:05	so what we have done actually was started to expand these uh you know the are uh
0:12:10	atoms if you all
0:12:12	and too much larger a you know a a dictionary what we could act start to look at colour that
0:12:16	you know sometimes i in colours that that yeah uh oh what i do not see
0:12:20	so this an example you what we have used in our uh in know a little work
0:12:24	where we have used the you know
0:12:26	more than three colours and this colours actually you could
0:12:30	uh design and uh using a a you know classical uh
0:12:34	at a for main a frame or try to achieve you know maximum uh in
0:12:40	uh now this is kind of a little uh you know like the go for um
0:12:45	the more general framework that uh you know we are uh focusing on in this particular
0:12:51	you know um uh paper in here
0:12:53	uh this just to show that a cook results about two what happened when you start use compressive and demosaicing
0:12:58	again components some of the traditional approaches
0:13:00	and a see actually the of a fair amount to the artifacts actually got a a a a reduce or
0:13:05	in fact eliminated
0:13:06	this still like just some problems here that a point total bit later and the talk
0:13:10	so um what you have that actually can generalising this compress you mosaic can by uh a working got it
0:13:15	uh a little bit more of broad or framework
0:13:19	and what you're proposing is really to have
0:13:21	this clear distinction between two type of sparse fine
0:13:25	dictionary is one of them for
0:13:27	uh the spatial or than the C another one for for the colour then that's
0:13:31	so this is the overall all kind of frame can the question here
0:13:34	if you are given the counter a cfa
0:13:37	and that's assume you could use any spatial sparse to find the channel and here really you could use anything
0:13:41	including
0:13:42	a dictionary that you could learn uh
0:13:45	uh a line i real time or off
0:13:48	so the question is uh you know what can you do with the colour sparsifying dictionary
0:13:53	so with that uh and place and of for going back to the you know kind of the more general
0:13:57	phone we started it
0:13:59	so uh what we could do actually we could start to look at the different
0:14:04	spatial frequencies
0:14:05	with the rgb uh you know vector that we have a uh a uh a a i
0:14:11	and
0:14:13	if we pick any uh either frequency component spatial frequency component of these uh are rgb colours that we trying
0:14:19	to model present
0:14:21	or or could pick any actually frequency band doesn't have to be a only a single frequency component
0:14:26	then uh this for a spatial frequency you could actually tried to sparse if fight
0:14:31	by using you know uh uh as many colours looks really as you like
0:14:35	and hopefully that will give you more sparse solution and that will also help you
0:14:39	compressed sensing solve to actually find the sparse solution
0:14:42	uh you know will bit better
0:14:44	so uh now this particle or uh
0:14:47	you know a a few and here the this is for one or what a particle or spatial frequency that
0:14:53	um
0:14:54	a that we have
0:14:55	if you put everything together you could actually have
0:14:58	uh all your rgb top let's
0:15:01	or all different to spatial frequencies
0:15:03	and just the number of spatial frequencies you could have could be as many as a list what is uh
0:15:08	uh as you want to it's a function all of the spatial frequency that you have used
0:15:13	uh just parts why or i'm and that's could be D you could be way of what could be of
0:15:16	an over complete
0:15:17	and for each of them what's really could have you on colour dictionary
0:15:20	so could have different colour different colour presentation here
0:15:23	so uh uh these different to color frames stand represent your uh a new set of the chin that you
0:15:29	could use
0:15:30	and they could be combined of course uh with the sparse representation that
0:15:34	each of them will be view uh different
0:15:37	spatial frequency representation presentation of sparse representation
0:15:40	so uh uh for put everything together and here actually uh you had this uh uh a to six which
0:15:46	include all the frames
0:15:47	and then and this is need to be combined actually with the permutation metrics in fact just to give you
0:15:52	a matching between or spatial
0:15:54	frequency or meant and you're frequency yeah a colour uh
0:15:57	frequency range me
0:15:59	so um
0:16:00	now if we put the again everything together uh so what we have
0:16:04	uh for for a given a spatial frequency uh uh a sparsifying dictionary dictionary's
0:16:09	and giving also see C F A
0:16:11	we could represent present or now uh colours sparsifying find dictionary as combination of open imitation metrics
0:16:17	and as a a a a and a colour frame uh and the colour frame is actually as i showed
0:16:23	this would kind of a able uh with
0:16:26	uh diagonal type of metrics
0:16:28	and of that of this is now your overall the sparsifying dictionary
0:16:32	is really combination of three mattresses as one of them is the spatial
0:16:36	uh as sparsifying dictionary that other one is the permutation metrics and the third one is you colour frames
0:16:41	and metrics
0:16:42	uh are more familiar perspective for compressed sensing is your projection matrix is really consist of four different mattresses
0:16:49	the sensing matrix
0:16:50	spatial
0:16:51	ugh permutation and then the colour frames
0:16:54	so now basically this reduce the problem simply
0:16:57	once you model this way
0:16:59	you have your P and all you do is basically look for your sparse a solution already could use a
0:17:03	one or a minimization of course you know a lot so basis pursuit
0:17:07	one of our uh you prefer
0:17:09	so uh the cook think about the simulation results so what we did actually of course you know as i
0:17:13	mentioned
0:17:14	with the colour frames you could actually designed a color frame for every single spatial frequency
0:17:20	uh for example if using dct or only have
0:17:23	you know let's say in this example sixty four possible
0:17:26	color dictionaries you could use
0:17:28	each dictionary could be of different number of of atoms if you want
0:17:32	or basis vectors
0:17:33	uh what you have done actually we all the design of three bands
0:17:37	uh called the for three band so the for the first and most important one what is the D C
0:17:41	one
0:17:42	and one interesting characteristic of the D C one uh is the fact that is is always positive
0:17:47	so that is no uh uh uh
0:17:49	you know a good reason to really have your or uh
0:17:53	you a basis vectors and that dictionary called diction or to go beyond just the the positive or
0:17:58	so that's what we focus on there
0:18:00	and also a a of course very important that include the luminance
0:18:03	and in jail or the more colours the maria
0:18:06	a this is you know of an extreme examples of all back you know colour atoms that you could use
0:18:11	and such dictionary
0:18:12	uh you know we want to uh up to like sixty four different colours in fact but um you know
0:18:17	you we do not need that
0:18:19	and what we end up
0:18:20	well using good uh something like this which is always seven and or nine maybe up to twelve
0:18:25	to be you know practical and also um you know what you find don't actually works
0:18:29	you know quite well
0:18:30	now the second band which we call it a band or sometimes you "'cause" you know arguably call will medium
0:18:35	band
0:18:36	and here of course now we have positive and negative
0:18:39	in general depending on what your spatial frequencies are but if you use T
0:18:43	then you have positive or negative values
0:18:45	then you only need to expand the whole uh you know uh
0:18:49	a three dimensional space
0:18:50	and you could basically use really anything get from P T a for a normalization of free T and this
0:18:55	is really an example of
0:18:57	a colour frame that you could use
0:18:59	uh and again you
0:19:01	but could from media an optimize it using in a uh in a different technique
0:19:05	and last but not least is the high frequency uh bad
0:19:09	and here actually uh in fact you don't need much
0:19:12	uh
0:19:13	uh colours to use uh uh and a fact if used too much colour could get some colour artifacts
0:19:18	so only uh
0:19:20	you know few colours in fact some of the our experiments we all use a single back that would just
0:19:24	a low men and to you know quite fine
0:19:27	so this is kind of an example for the high frequency one
0:19:30	so if you put all the three colours band this a basically but you get
0:19:33	uh this is some of uh uh the simulation results are we're getting this the original image
0:19:37	"'cause" see this at which is kind of a challenge one for some of the leading approaches
0:19:42	"'cause" you some call aspect here
0:19:44	uh that will while guy actually you know does very just job but in fact if to look at a
0:19:48	little bit in here very hard to see that are some artifacts
0:19:51	in our case uh you know seems like we do some of those out the to since some of techniques
0:19:56	uh this is again that's no uh region
0:19:59	uh uh you can see those are track that type point that they are actually
0:20:03	maybe again pretty hard to see "'em" sorry here
0:20:05	but uh if you look at a show no oh no real monitor um most of this out to face
0:20:10	got to eliminate
0:20:11	uh this is there a again then it or S uh like house image
0:20:15	seems to reconstruct it fairly well
0:20:17	um i don't to deceive actually and the sense that you know the this problem still quite all there is
0:20:22	still a lot of problems actually in here i was supposed to assume but that can do it
0:20:26	"'cause" it don't have the power point
0:20:27	and if you look at the fancy at actually there's a fair amount of artifact in our case
0:20:32	and also on the on line running
0:20:34	so um uh in conclusion basically um what we have uh is
0:20:39	this a new for mark where we capture out making that clear distinction between
0:20:43	spatial sparsifying dictionaries and colour sparsifying find dictionaries
0:20:47	seems that uh were able and most of the techniques actually that use compressed sensing
0:20:52	uh to the denoising problem are able to recover most of the colours not all of the colours
0:20:57	nevertheless we believe that is still to the someone of actual challenges this is really by for uh and on
0:21:02	so problem
0:21:04	and are many good reasons for that and
0:21:06	with that all stop and
0:21:10	oh open for questions
0:21:26	there are no questions
0:21:27	ooh
0:21:30	one question
0:21:31	and i i i found you
0:21:34	uh the um
0:21:36	the the you you be or a new uh
0:21:38	reference thing this endangering dictionary four connor
0:21:42	for car and i think is not
0:21:44	but
0:21:45	there were or an or dimension of connery used three
0:21:48	i G B three
0:21:50	and you would uh i'm not that the mission of the of space
0:21:54	oh no it's not
0:21:55	but not
0:21:56	a you use a um more than three uh uh
0:21:59	con uh vector is to represent the connor
0:22:02	kind space
0:22:03	but
0:22:04	uh no the problem is
0:22:06	uh if you we now uh uh
0:22:09	increase the about there's the number of the vectors
0:22:11	and the car space
0:22:13	you also something uh
0:22:15	i nice and are the the the lance of the vector the back to as in this sparse
0:22:19	it's in that
0:22:20	the last of this past are
0:22:22	uh and that maybe are
0:22:25	make the optimization
0:22:27	a a problem a house C
0:22:30	more complex
0:22:31	yeah a more complex
0:22:32	no that's very true that's why actually wanna just
0:22:35	select than a the right number of of colour frames and he do not when over do it
0:22:39	and fact this is one of the problems we kind of work all right now which is trying to figure
0:22:43	out what is the optimal number
0:22:45	and many of the result i just presented this uh is a you know we can of gain by experience
0:22:50	and somebody body will charge of problem to figure out a mean definitely for example for the high frequencies you
0:22:54	do not twenty use
0:22:56	to many colours we could you see artifacts right the way
0:22:58	aside from the fact that it's complexity so you
0:23:01	uh you lots of the like to do not twenty use
0:23:03	you know
0:23:03	to many colours and the something
0:23:06	stinks
0:23:11	okay are no more questions were move to
0:23:14	the last but not least talk by
0:23:16	professor about work can see

INCOHERENT COLOR FRAMES FOR COMPRESSIVE DEMOSAICING

Compressed Sensing and Sparse Representation of Signals

Přednášející: Hayder Radha, Autoři: Abdolreza Abdolhosseini Moghadam, Mohammad Aghagolzadeh, Michigan State University, United States; Mrityunjay Kumar, Eastman Kodak Company, United States; Hayder Radha, Michigan State University, United States