0:00:17the next oh is my talk
0:00:19and and this work was done in collaboration with the by to graduate students
0:00:24you moment and then and from and the goods there
0:00:27and motor G calm more from an eastman kodak research flipped
0:00:33that time for the top basically all the to be simple i'm gonna start uh by giving good a fairly
0:00:38high level uh overview of the problem of you as can just sink case there are people may not be
0:00:42familiar with a
0:00:44and then i'll talk little bit about the this compressive demosaicing framework that the we have it to to do
0:00:50reduce recently
0:00:51and then not talk about colour frames for compressing
0:00:54compressed you missing
0:00:58so uh again a bit introduction to the problem of demosaicing mosaic
0:01:04we are familiar with the fact that uh images
0:01:07at is the way human visual system perceive then uh requires three colour plane
0:01:12uh traditionally additionally red green and blue
0:01:16fact if you wanna a capture the such images uh all these three color planes you let's roll need three
0:01:22uh C C D sensors in your come
0:01:25i the problem with this of course is that there is tremendous amount of course uh and also that size
0:01:31so i to the all the come as that uh uh with
0:01:34and a multiple C C D sensors
0:01:37so it turns out that um
0:01:39the vast majority maybe almost every single kind of that
0:01:42people have including the ones maybe in your i i've phone or i part of rubber
0:01:47a i actually uses only a single uh C C D
0:01:52says and the way this is uh don is basically a they put what's known is a colour filter already
0:01:58which literally it admits all the single
0:02:01colour per pixel in instead of
0:02:03capturing in three
0:02:04uh pixels
0:02:06so um
0:02:08traditionally additionally if you look actually if you are a lot to look inside the got of if you're a
0:02:12which we usually cannot not
0:02:15uh you'll see actually the and may each on the right
0:02:18uh uh is the one that is captured why the camera
0:02:22and uh you know this is a original image so let's really we do a you do not have access
0:02:27to all the colours that exist in the original image
0:02:30uh obviously the image that's kept it is very much dictated by and the nature of the
0:02:37colour for tell in colour pattern of that C F eight
0:02:40i the most popular or uh C F A it used or which was developed uh by X actually
0:02:46uh uh is known as the bayer filter out uh which should basically uses to green colours
0:02:52for every you read then a uh and a green but uh i'm sorry
0:02:55to green colours for a really right and blue pixel
0:02:59and in any to by two uh a block of range
0:03:03fact if you lose now uh you know all as in here looking at the and is just looks kind
0:03:07of green but you only doing to put a population
0:03:10how well but if you actually zoom and fact that the
0:03:12my presentation which was a powerpoint
0:03:14unfortunately for had some distortion
0:03:17if you zoom in actually at the zoom more could see the red and blue pixels in addition to the
0:03:23okay so how do you actually do you mosaic the image i mean the problem due mosaic basically recovering the
0:03:28original red green and blue uh
0:03:32uh from what you have capture which just a single pixel a single colour per pixel
0:03:37well that are basically you know to type of
0:03:39redundancies and dependencies that exist in your uh to signal but you could exploit
0:03:45uh the first one which is the obvious one is to the spatial or the pixel
0:03:49i dependencies in here if you look at the
0:03:52but you capture in of the right
0:03:54the green and W blue obvious you have a lot of missing pixels and lot false
0:03:58a let's really could use any form of interpolation or some kind of smart
0:04:02frequency domain um
0:04:04prices to really fill in the gap
0:04:07uh that that kind of the is see that you can exploit is uh let's really the um
0:04:12depends is that exist among the
0:04:15the colours them self you know there is a fair amount of
0:04:18but done the see the so if you look at the red green and
0:04:21uh and blue channels uh in a tradition of people you go to a colour difference step of think will
0:04:27from there with that from a been compression standards
0:04:29and then to be able to also exploit that and uh i you get a just some form
0:04:34fact they've and uh of sparsity by doing that
0:04:37so you know i i don't know that are probably about ten maybe you know close to hundreds of
0:04:42you know do you more taking order them is that actually tried to solve this problem and try to recover
0:04:47the original red green is you know with different the variations on this
0:04:51you know theme
0:04:53um um just to give you a flavour about you know the challenges associate to this problem uh
0:04:58in a list take a look at
0:04:59what's really merging as
0:05:01we could call what the the economic canonical test image for that you music can problem
0:05:06uh this is known as the lighthouse image
0:05:08and particle that is this block which is the fancy sarah because has a very high spatial frequency
0:05:14as quite challenging actually to cover all the three colours
0:05:17uh you know from any new kind of colour of white or black or great
0:05:22"'cause" all close colours actually the are are present
0:05:25so this is an example like you
0:05:27uh some of the leading approach is already at approach is that are highly
0:05:32um um a the and the literature
0:05:35and you could really see that our a fair amount of you know artifacts actually few trying to do that
0:05:41uh and the
0:05:43you know net really trying to criticise these particle to two approaches in how a just wanna give you uh
0:05:48uh for the you know the challenges that you really could phase despite the fact that you know these approaches
0:05:53are the based on very sound
0:05:55uh i know that skull and theoretical a frame or
0:05:58uh the are actually approach approaches which do uh uh uh better uh are good be much better
0:06:03and again i just you want to to highlight like you know some of the challenges you could see and
0:06:07these are examples
0:06:09a a of uh some of that affects because
0:06:12so how could we map the problem of you mosaicing to confront sensing well um it's relatively easy actually from
0:06:18in principle uh i
0:06:20you know them uh the problem doom as a king you you basically doing
0:06:24compressed sensing in the sense that you are
0:06:27do compress a same by factor of three
0:06:29and uh you could look at your account not a C F A and that to present or sensing metrics
0:06:36and the compressed sensing setting
0:06:37and uh you are made you could you know uh choose any sparsifying kind of dictionary
0:06:43uh that you'd like an try to recover the signal uh you know based on that
0:06:48the that be an but if you work actually apply compressed sensing to this problem
0:06:53nevertheless a you know especially if you focus on the fact that
0:06:57the kind see if a something you cannot not do much about
0:07:00and most of the work really been done in the context of using
0:07:04the um
0:07:05uh the they are packed them
0:07:07uh by the way uh actually really emphasise you know the
0:07:10this project or
0:07:11uh has
0:07:13looked really horrible
0:07:15artifacts so
0:07:16i we have a lip college as i'm you could tell this is not an image processing conference so
0:07:21um at anyway
0:07:23maybe we'll
0:07:24one be the same token i are people
0:07:26and i i any right so um
0:07:28so the
0:07:31in here these supposed to be yeah you know green but they look like a at any anyway
0:07:35so given that the C F is actually is given you that is not what you could do about the
0:07:39of the focus is on
0:07:40you know what is really the sparse of a sparsifying dictionary right here
0:07:44so uh the lead and work and the say it really been don uh at is my opinion by uh
0:07:49more well uh jewel and or L and uh some of his colleagues
0:07:53well the actually the a a a whole bunch of all learning
0:07:56a a line i'm sorry learning link dictionary all go them uh that actually tried to uh figure out what
0:08:02is the optimal
0:08:03diction there's the sparse one of the could use in order to recover the colours
0:08:07and they have a whole bunch of uh you know techniques uh a some of them uh uh the most
0:08:12actually problem the one and the like this one
0:08:14the coloured learned simple to a sparse coding a less
0:08:17i C
0:08:18and this is some of the results this is some of the are it was also actually to see some
0:08:22artifact and got improve significantly
0:08:24through a last
0:08:25i C
0:08:27so um this approach even this learning approach actually is
0:08:31um a i a is a quite promising still has a whole bunch of problem
0:08:36uh in fact a this particle image and i'm not sure of men a built to zoom but
0:08:40uh in here uh yeah to look at the snow at and i believe i have an image yes
0:08:45and that if you can see it but this is the original actually snow
0:08:48and this is the a cover of through this out them on you know sparsifying dictionary and
0:08:52hopefully "'cause" see that our uh some fair amount of facts actually the convolution and white
0:08:58uh you know so the the two colours rule not be recovered
0:09:03so um so what we have developed act is an alternative framework which we call compressive demosaicing K and it's
0:09:09it's fairly simple actually
0:09:10uh again but i apologise this is supposed to be green you know maybe it's it's is gonna some be
0:09:15people's eyes but
0:09:16so in here uh basically you could be the image or presented through three matches is that a green and
0:09:22and these forty three images are are being uh multiplied through this uh as in a simple point wise multiplication
0:09:30uh had the are the top a multiplication by three different uh mattress mattresses
0:09:35and we have a linear combination uh to present the measurement that we actually have
0:09:40so um if you put everything together in terms of metrics for you could the vector or an image that
0:09:45the vector right the are red green "'em" we'll that you're trying to recover
0:09:49and this uh
0:09:52multiple uh multiple just as an here they are present really the different for presentation of your sense
0:09:58and didn't of what you capturing
0:09:59this represent present of course a little bit the more general
0:10:02you know a a a a framework contains of it does not have to work it with the bayer pat
0:10:06and but you you could capture any kind of pat then
0:10:09the the same time you're actually i i i'd hearing to the constraint which is very important for the problem
0:10:14you was aching
0:10:14which is capturing all a single
0:10:17colour or pixel
0:10:18and a but important distinction there that colour that you capture lack you has to belong to that article pixel
0:10:24you cannot do and any linear combination anything
0:10:27that's why you C Ds match this is actually they are gonna
0:10:30so this is a very important constraint that is not much you could do a what
0:10:33and that there was you cannot not generalise
0:10:36this matrix anymore
0:10:37uh that's the most general could have
0:10:40okay so um
0:10:42so basically in a few uh
0:10:44uh a you know
0:10:45a to this kind of framework which is a a of simple now uh the idea is to go to
0:10:50the rgb vector eyes an agent tried to replace a basically with the sparse representation as we have done before
0:10:56but is no or you not much new they are so now we could represent
0:10:59some kind of frequency or presentation all the different
0:11:03coloured the R G and B
0:11:05and now we could actually use different uh dictionary so if you put everything together again
0:11:09um uh uh now we have again you C cfa image or C F i'm sorry uh at tricks which
0:11:15is sense the metrics and then you have
0:11:17you're a sparsifying dictionary and now we have the flexibility of using different dictionaries actually for different colour planes
0:11:23uh if you wish to
0:11:26now uh
0:11:27up to this point actually um you know that is really not much significant improvement if you try this kind
0:11:33of framework which is really very simple and you could of course go and try to
0:11:37uh find the sparse error back to sell
0:11:40the biggest problem of this approach actually and in general are in fact it's if you still
0:11:44uh operate and a three dimensional colour space and does out of its our G V or Y V
0:11:51then you really not exploiting the
0:11:54the uh
0:11:55core correlation among the different colour planes and more specifically you really cannot get much sparsity
0:12:01so this this uh back to to self it's actually sparse is not sparse in a
0:12:05so what we have done actually was started to expand these uh you know the are uh
0:12:10atoms if you all
0:12:12and too much larger a you know a a dictionary what we could act start to look at colour that
0:12:16you know sometimes i in colours that that yeah uh oh what i do not see
0:12:20so this an example you what we have used in our uh in know a little work
0:12:24where we have used the you know
0:12:26more than three colours and this colours actually you could
0:12:30uh design and uh using a a you know classical uh
0:12:34at a for main a frame or try to achieve you know maximum uh in
0:12:40uh now this is kind of a little uh you know like the go for um
0:12:45the more general framework that uh you know we are uh focusing on in this particular
0:12:51you know um uh paper in here
0:12:53uh this just to show that a cook results about two what happened when you start use compressive and demosaicing
0:12:58again components some of the traditional approaches
0:13:00and a see actually the of a fair amount to the artifacts actually got a a a a reduce or
0:13:05in fact eliminated
0:13:06this still like just some problems here that a point total bit later and the talk
0:13:10so um what you have that actually can generalising this compress you mosaic can by uh a working got it
0:13:15uh a little bit more of broad or framework
0:13:19and what you're proposing is really to have
0:13:21this clear distinction between two type of sparse fine
0:13:25dictionary is one of them for
0:13:27uh the spatial or than the C another one for for the colour then that's
0:13:31so this is the overall all kind of frame can the question here
0:13:34if you are given the counter a cfa
0:13:37and that's assume you could use any spatial sparse to find the channel and here really you could use anything
0:13:42a dictionary that you could learn uh
0:13:45uh a line i real time or off
0:13:48so the question is uh you know what can you do with the colour sparsifying dictionary
0:13:53so with that uh and place and of for going back to the you know kind of the more general
0:13:57phone we started it
0:13:59so uh what we could do actually we could start to look at the different
0:14:04spatial frequencies
0:14:05with the rgb uh you know vector that we have a uh a uh a a i
0:14:13if we pick any uh either frequency component spatial frequency component of these uh are rgb colours that we trying
0:14:19to model present
0:14:21or or could pick any actually frequency band doesn't have to be a only a single frequency component
0:14:26then uh this for a spatial frequency you could actually tried to sparse if fight
0:14:31by using you know uh uh as many colours looks really as you like
0:14:35and hopefully that will give you more sparse solution and that will also help you
0:14:39compressed sensing solve to actually find the sparse solution
0:14:42uh you know will bit better
0:14:44so uh now this particle or uh
0:14:47you know a a few and here the this is for one or what a particle or spatial frequency that
0:14:54a that we have
0:14:55if you put everything together you could actually have
0:14:58uh all your rgb top let's
0:15:01or all different to spatial frequencies
0:15:03and just the number of spatial frequencies you could have could be as many as a list what is uh
0:15:08uh as you want to it's a function all of the spatial frequency that you have used
0:15:13uh just parts why or i'm and that's could be D you could be way of what could be of
0:15:16an over complete
0:15:17and for each of them what's really could have you on colour dictionary
0:15:20so could have different colour different colour presentation here
0:15:23so uh uh these different to color frames stand represent your uh a new set of the chin that you
0:15:29could use
0:15:30and they could be combined of course uh with the sparse representation that
0:15:34each of them will be view uh different
0:15:37spatial frequency representation presentation of sparse representation
0:15:40so uh uh for put everything together and here actually uh you had this uh uh a to six which
0:15:46include all the frames
0:15:47and then and this is need to be combined actually with the permutation metrics in fact just to give you
0:15:52a matching between or spatial
0:15:54frequency or meant and you're frequency yeah a colour uh
0:15:57frequency range me
0:15:59so um
0:16:00now if we put the again everything together uh so what we have
0:16:04uh for for a given a spatial frequency uh uh a sparsifying dictionary dictionary's
0:16:09and giving also see C F A
0:16:11we could represent present or now uh colours sparsifying find dictionary as combination of open imitation metrics
0:16:17and as a a a a and a colour frame uh and the colour frame is actually as i showed
0:16:23this would kind of a able uh with
0:16:26uh diagonal type of metrics
0:16:28and of that of this is now your overall the sparsifying dictionary
0:16:32is really combination of three mattresses as one of them is the spatial
0:16:36uh as sparsifying dictionary that other one is the permutation metrics and the third one is you colour frames
0:16:41and metrics
0:16:42uh are more familiar perspective for compressed sensing is your projection matrix is really consist of four different mattresses
0:16:49the sensing matrix
0:16:51ugh permutation and then the colour frames
0:16:54so now basically this reduce the problem simply
0:16:57once you model this way
0:16:59you have your P and all you do is basically look for your sparse a solution already could use a
0:17:03one or a minimization of course you know a lot so basis pursuit
0:17:07one of our uh you prefer
0:17:09so uh the cook think about the simulation results so what we did actually of course you know as i
0:17:14with the colour frames you could actually designed a color frame for every single spatial frequency
0:17:20uh for example if using dct or only have
0:17:23you know let's say in this example sixty four possible
0:17:26color dictionaries you could use
0:17:28each dictionary could be of different number of of atoms if you want
0:17:32or basis vectors
0:17:33uh what you have done actually we all the design of three bands
0:17:37uh called the for three band so the for the first and most important one what is the D C
0:17:42and one interesting characteristic of the D C one uh is the fact that is is always positive
0:17:47so that is no uh uh uh
0:17:49you know a good reason to really have your or uh
0:17:53you a basis vectors and that dictionary called diction or to go beyond just the the positive or
0:17:58so that's what we focus on there
0:18:00and also a a of course very important that include the luminance
0:18:03and in jail or the more colours the maria
0:18:06a this is you know of an extreme examples of all back you know colour atoms that you could use
0:18:11and such dictionary
0:18:12uh you know we want to uh up to like sixty four different colours in fact but um you know
0:18:17you we do not need that
0:18:19and what we end up
0:18:20well using good uh something like this which is always seven and or nine maybe up to twelve
0:18:25to be you know practical and also um you know what you find don't actually works
0:18:29you know quite well
0:18:30now the second band which we call it a band or sometimes you "'cause" you know arguably call will medium
0:18:36and here of course now we have positive and negative
0:18:39in general depending on what your spatial frequencies are but if you use T
0:18:43then you have positive or negative values
0:18:45then you only need to expand the whole uh you know uh
0:18:49a three dimensional space
0:18:50and you could basically use really anything get from P T a for a normalization of free T and this
0:18:55is really an example of
0:18:57a colour frame that you could use
0:18:59uh and again you
0:19:01but could from media an optimize it using in a uh in a different technique
0:19:05and last but not least is the high frequency uh bad
0:19:09and here actually uh in fact you don't need much
0:19:13uh colours to use uh uh and a fact if used too much colour could get some colour artifacts
0:19:18so only uh
0:19:20you know few colours in fact some of the our experiments we all use a single back that would just
0:19:24a low men and to you know quite fine
0:19:27so this is kind of an example for the high frequency one
0:19:30so if you put all the three colours band this a basically but you get
0:19:33uh this is some of uh uh the simulation results are we're getting this the original image
0:19:37"'cause" see this at which is kind of a challenge one for some of the leading approaches
0:19:42"'cause" you some call aspect here
0:19:44uh that will while guy actually you know does very just job but in fact if to look at a
0:19:48little bit in here very hard to see that are some artifacts
0:19:51in our case uh you know seems like we do some of those out the to since some of techniques
0:19:56uh this is again that's no uh region
0:19:59uh uh you can see those are track that type point that they are actually
0:20:03maybe again pretty hard to see "'em" sorry here
0:20:05but uh if you look at a show no oh no real monitor um most of this out to face
0:20:10got to eliminate
0:20:11uh this is there a again then it or S uh like house image
0:20:15seems to reconstruct it fairly well
0:20:17um i don't to deceive actually and the sense that you know the this problem still quite all there is
0:20:22still a lot of problems actually in here i was supposed to assume but that can do it
0:20:26"'cause" it don't have the power point
0:20:27and if you look at the fancy at actually there's a fair amount of artifact in our case
0:20:32and also on the on line running
0:20:34so um uh in conclusion basically um what we have uh is
0:20:39this a new for mark where we capture out making that clear distinction between
0:20:43spatial sparsifying dictionaries and colour sparsifying find dictionaries
0:20:47seems that uh were able and most of the techniques actually that use compressed sensing
0:20:52uh to the denoising problem are able to recover most of the colours not all of the colours
0:20:57nevertheless we believe that is still to the someone of actual challenges this is really by for uh and on
0:21:02so problem
0:21:04and are many good reasons for that and
0:21:06with that all stop and
0:21:10oh open for questions
0:21:26there are no questions
0:21:30one question
0:21:31and i i i found you
0:21:34uh the um
0:21:36the the you you be or a new uh
0:21:38reference thing this endangering dictionary four connor
0:21:42for car and i think is not
0:21:45there were or an or dimension of connery used three
0:21:48i G B three
0:21:50and you would uh i'm not that the mission of the of space
0:21:54oh no it's not
0:21:55but not
0:21:56a you use a um more than three uh uh
0:21:59con uh vector is to represent the connor
0:22:02kind space
0:22:04uh no the problem is
0:22:06uh if you we now uh uh
0:22:09increase the about there's the number of the vectors
0:22:11and the car space
0:22:13you also something uh
0:22:15i nice and are the the the lance of the vector the back to as in this sparse
0:22:19it's in that
0:22:20the last of this past are
0:22:22uh and that maybe are
0:22:25make the optimization
0:22:27a a problem a house C
0:22:30more complex
0:22:31yeah a more complex
0:22:32no that's very true that's why actually wanna just
0:22:35select than a the right number of of colour frames and he do not when over do it
0:22:39and fact this is one of the problems we kind of work all right now which is trying to figure
0:22:43out what is the optimal number
0:22:45and many of the result i just presented this uh is a you know we can of gain by experience
0:22:50and somebody body will charge of problem to figure out a mean definitely for example for the high frequencies you
0:22:54do not twenty use
0:22:56to many colours we could you see artifacts right the way
0:22:58aside from the fact that it's complexity so you
0:23:01uh you lots of the like to do not twenty use
0:23:03you know
0:23:03to many colours and the something
0:23:11okay are no more questions were move to
0:23:14the last but not least talk by
0:23:16professor about work can see