okay a the next oh is my talk and and this work was done in collaboration with the by to graduate students you moment and then and from and the goods there and motor G calm more from an eastman kodak research flipped um that time for the top basically all the to be simple i'm gonna start uh by giving good a fairly high level uh overview of the problem of you as can just sink case there are people may not be familiar with a and then i'll talk little bit about the this compressive demosaicing framework that the we have it to to do reduce recently and then not talk about colour frames for compressing compressed you missing so uh again a bit introduction to the problem of demosaicing mosaic um we are familiar with the fact that uh images at is the way human visual system perceive then uh requires three colour plane uh traditionally additionally red green and blue and fact if you wanna a capture the such images uh all these three color planes you let's roll need three uh C C D sensors in your come i the problem with this of course is that there is tremendous amount of course uh and also that size issues so i to the all the come as that uh uh with and a multiple C C D sensors so it turns out that um the vast majority maybe almost every single kind of that people have including the ones maybe in your i i've phone or i part of rubber a i actually uses only a single uh C C D uh says and the way this is uh don is basically a they put what's known is a colour filter already which literally it admits all the single colour per pixel in instead of capturing in three uh pixels so um traditionally additionally if you look actually if you are a lot to look inside the got of if you're a which we usually cannot not uh you'll see actually the and may each on the right uh uh is the one that is captured why the camera and uh you know this is a original image so let's really we do a you do not have access to all the colours that exist in the original image uh obviously the image that's kept it is very much dictated by and the nature of the colour for tell in colour pattern of that C F eight i the most popular or uh C F A it used or which was developed uh by X actually uh uh is known as the bayer filter out uh which should basically uses to green colours for every you read then a uh and a green but uh i'm sorry to green colours for a really right and blue pixel and in any to by two uh a block of range fact if you lose now uh you know all as in here looking at the and is just looks kind of green but you only doing to put a population how well but if you actually zoom and fact that the my presentation which was a powerpoint unfortunately for had some distortion if you zoom in actually at the zoom more could see the red and blue pixels in addition to the green okay so how do you actually do you mosaic the image i mean the problem due mosaic basically recovering the original red green and blue uh signal uh from what you have capture which just a single pixel a single colour per pixel well that are basically you know to type of redundancies and dependencies that exist in your uh to signal but you could exploit uh the first one which is the obvious one is to the spatial or the pixel i dependencies in here if you look at the but you capture in of the right the green and W blue obvious you have a lot of missing pixels and lot false a let's really could use any form of interpolation or some kind of smart frequency domain um prices to really fill in the gap uh that that kind of the is see that you can exploit is uh let's really the um depends is that exist among the the colours them self you know there is a fair amount of but done the see the so if you look at the red green and uh and blue channels uh in a tradition of people you go to a colour difference step of think will from there with that from a been compression standards and then to be able to also exploit that and uh i you get a just some form fact they've and uh of sparsity by doing that so you know i i don't know that are probably about ten maybe you know close to hundreds of you know do you more taking order them is that actually tried to solve this problem and try to recover the original red green is you know with different the variations on this you know theme um um just to give you a flavour about you know the challenges associate to this problem uh in a list take a look at what's really merging as we could call what the the economic canonical test image for that you music can problem uh this is known as the lighthouse image and particle that is this block which is the fancy sarah because has a very high spatial frequency as quite challenging actually to cover all the three colours uh you know from any new kind of colour of white or black or great "'cause" all close colours actually the are are present so this is an example like you uh some of the leading approach is already at approach is that are highly um um um a the and the literature and you could really see that our a fair amount of you know artifacts actually few trying to do that uh and the you know net really trying to criticise these particle to two approaches in how a just wanna give you uh flavour uh for the you know the challenges that you really could phase despite the fact that you know these approaches are the based on very sound uh i know that skull and theoretical a frame or uh the are actually approach approaches which do uh uh uh better uh are good be much better and again i just you want to to highlight like you know some of the challenges you could see and these are examples a a of uh some of that affects because so how could we map the problem of you mosaicing to confront sensing well um it's relatively easy actually from in principle uh i you know them uh the problem doom as a king you you basically doing compressed sensing in the sense that you are do compress a same by factor of three and uh you could look at your account not a C F A and that to present or sensing metrics in and the compressed sensing setting and uh you are made you could you know uh choose any sparsifying kind of dictionary uh that you'd like an try to recover the signal uh you know based on that um the that be an but if you work actually apply compressed sensing to this problem uh nevertheless a you know especially if you focus on the fact that the kind see if a something you cannot not do much about and most of the work really been done in the context of using the um uh the they are packed them uh by the way uh actually really emphasise you know the this project or uh has looked really horrible colour artifacts so i we have a lip college as i'm you could tell this is not an image processing conference so um at anyway maybe we'll one be the same token i are people and i i any right so um so the in here these supposed to be yeah you know green but they look like a at any anyway so given that the C F is actually is given you that is not what you could do about the of the focus is on you know what is really the sparse of a sparsifying dictionary right here so uh the lead and work and the say it really been don uh at is my opinion by uh more well uh jewel and or L and uh some of his colleagues well the actually the a a a whole bunch of all learning a a line i'm sorry learning link dictionary all go them uh that actually tried to uh figure out what is the optimal diction there's the sparse one of the could use in order to recover the colours and they have a whole bunch of uh you know techniques uh a some of them uh uh the most actually problem the one and the like this one the coloured learned simple to a sparse coding a less i C and this is some of the results this is some of the are it was also actually to see some artifact and got improve significantly through a last i C so um this approach even this learning approach actually is um a i a is a quite promising still has a whole bunch of problem uh in fact a this particle image and i'm not sure of men a built to zoom but uh in here uh yeah to look at the snow at and i believe i have an image yes and that if you can see it but this is the original actually snow and this is the a cover of through this out them on you know sparsifying dictionary and hopefully "'cause" see that our uh some fair amount of facts actually the convolution and white uh you know so the the two colours rule not be recovered so um so what we have developed act is an alternative framework which we call compressive demosaicing K and it's it's fairly simple actually uh again but i apologise this is supposed to be green you know maybe it's it's is gonna some be people's eyes but so in here uh basically you could be the image or presented through three matches is that a green and blue and these forty three images are are being uh multiplied through this uh as in a simple point wise multiplication uh had the are the top a multiplication by three different uh mattress mattresses and we have a linear combination uh to present the measurement that we actually have so um if you put everything together in terms of metrics for you could the vector or an image that the vector right the are red green "'em" we'll that you're trying to recover and this uh uh multiple uh multiple just as an here they are present really the different for presentation of your sense and didn't of what you capturing this represent present of course a little bit the more general you know a a a a framework contains of it does not have to work it with the bayer pat and but you you could capture any kind of pat then the the same time you're actually i i i'd hearing to the constraint which is very important for the problem you was aching which is capturing all a single colour or pixel and a but important distinction there that colour that you capture lack you has to belong to that article pixel you cannot do and any linear combination anything that's why you C Ds match this is actually they are gonna so this is a very important constraint that is not much you could do a what and that there was you cannot not generalise this matrix anymore uh that's the most general could have okay so um so basically in a few uh uh a you know a to this kind of framework which is a a of simple now uh the idea is to go to the rgb vector eyes an agent tried to replace a basically with the sparse representation as we have done before but is no or you not much new they are so now we could represent some kind of frequency or presentation all the different coloured the R G and B and now we could actually use different uh dictionary so if you put everything together again um uh uh now we have again you C cfa image or C F i'm sorry uh at tricks which is sense the metrics and then you have you're a sparsifying dictionary and now we have the flexibility of using different dictionaries actually for different colour planes uh if you wish to um now uh up to this point actually um you know that is really not much significant improvement if you try this kind of framework which is really very simple and you could of course go and try to uh find the sparse error back to sell the biggest problem of this approach actually and in general are in fact it's if you still uh operate and a three dimensional colour space and does out of its our G V or Y V then you really not exploiting the the uh core correlation among the different colour planes and more specifically you really cannot get much sparsity so this this uh back to to self it's actually sparse is not sparse in a so what we have done actually was started to expand these uh you know the are uh atoms if you all and too much larger a you know a a dictionary what we could act start to look at colour that you know sometimes i in colours that that yeah uh oh what i do not see so this an example you what we have used in our uh in know a little work where we have used the you know more than three colours and this colours actually you could uh design and uh using a a you know classical uh at a for main a frame or try to achieve you know maximum uh in uh now this is kind of a little uh you know like the go for um the more general framework that uh you know we are uh focusing on in this particular you know um uh paper in here uh this just to show that a cook results about two what happened when you start use compressive and demosaicing again components some of the traditional approaches and a see actually the of a fair amount to the artifacts actually got a a a a reduce or in fact eliminated this still like just some problems here that a point total bit later and the talk so um what you have that actually can generalising this compress you mosaic can by uh a working got it uh a little bit more of broad or framework and what you're proposing is really to have this clear distinction between two type of sparse fine dictionary is one of them for uh the spatial or than the C another one for for the colour then that's so this is the overall all kind of frame can the question here if you are given the counter a cfa and that's assume you could use any spatial sparse to find the channel and here really you could use anything including a dictionary that you could learn uh uh a line i real time or off so the question is uh you know what can you do with the colour sparsifying dictionary so with that uh and place and of for going back to the you know kind of the more general phone we started it so uh what we could do actually we could start to look at the different spatial frequencies with the rgb uh you know vector that we have a uh a uh a a i and if we pick any uh either frequency component spatial frequency component of these uh are rgb colours that we trying to model present or or could pick any actually frequency band doesn't have to be a only a single frequency component then uh this for a spatial frequency you could actually tried to sparse if fight by using you know uh uh as many colours looks really as you like and hopefully that will give you more sparse solution and that will also help you compressed sensing solve to actually find the sparse solution uh you know will bit better so uh now this particle or uh you know a a few and here the this is for one or what a particle or spatial frequency that um a that we have if you put everything together you could actually have uh all your rgb top let's or all different to spatial frequencies and just the number of spatial frequencies you could have could be as many as a list what is uh uh as you want to it's a function all of the spatial frequency that you have used uh just parts why or i'm and that's could be D you could be way of what could be of an over complete and for each of them what's really could have you on colour dictionary so could have different colour different colour presentation here so uh uh these different to color frames stand represent your uh a new set of the chin that you could use and they could be combined of course uh with the sparse representation that each of them will be view uh different spatial frequency representation presentation of sparse representation so uh uh for put everything together and here actually uh you had this uh uh a to six which include all the frames and then and this is need to be combined actually with the permutation metrics in fact just to give you a matching between or spatial frequency or meant and you're frequency yeah a colour uh frequency range me so um now if we put the again everything together uh so what we have uh for for a given a spatial frequency uh uh a sparsifying dictionary dictionary's and giving also see C F A we could represent present or now uh colours sparsifying find dictionary as combination of open imitation metrics and as a a a a and a colour frame uh and the colour frame is actually as i showed this would kind of a able uh with uh diagonal type of metrics and of that of this is now your overall the sparsifying dictionary is really combination of three mattresses as one of them is the spatial uh as sparsifying dictionary that other one is the permutation metrics and the third one is you colour frames and metrics uh are more familiar perspective for compressed sensing is your projection matrix is really consist of four different mattresses the sensing matrix spatial ugh permutation and then the colour frames so now basically this reduce the problem simply once you model this way you have your P and all you do is basically look for your sparse a solution already could use a one or a minimization of course you know a lot so basis pursuit one of our uh you prefer so uh the cook think about the simulation results so what we did actually of course you know as i mentioned with the colour frames you could actually designed a color frame for every single spatial frequency uh for example if using dct or only have you know let's say in this example sixty four possible color dictionaries you could use each dictionary could be of different number of of atoms if you want or basis vectors uh what you have done actually we all the design of three bands uh called the for three band so the for the first and most important one what is the D C one and one interesting characteristic of the D C one uh is the fact that is is always positive so that is no uh uh uh you know a good reason to really have your or uh you a basis vectors and that dictionary called diction or to go beyond just the the positive or so that's what we focus on there and also a a of course very important that include the luminance and in jail or the more colours the maria a this is you know of an extreme examples of all back you know colour atoms that you could use and such dictionary uh you know we want to uh up to like sixty four different colours in fact but um you know you we do not need that and what we end up well using good uh something like this which is always seven and or nine maybe up to twelve to be you know practical and also um you know what you find don't actually works you know quite well now the second band which we call it a band or sometimes you "'cause" you know arguably call will medium band and here of course now we have positive and negative in general depending on what your spatial frequencies are but if you use T then you have positive or negative values then you only need to expand the whole uh you know uh a three dimensional space and you could basically use really anything get from P T a for a normalization of free T and this is really an example of a colour frame that you could use uh and again you but could from media an optimize it using in a uh in a different technique and last but not least is the high frequency uh bad and here actually uh in fact you don't need much uh uh colours to use uh uh and a fact if used too much colour could get some colour artifacts so only uh you know few colours in fact some of the our experiments we all use a single back that would just a low men and to you know quite fine so this is kind of an example for the high frequency one so if you put all the three colours band this a basically but you get uh this is some of uh uh the simulation results are we're getting this the original image "'cause" see this at which is kind of a challenge one for some of the leading approaches "'cause" you some call aspect here uh that will while guy actually you know does very just job but in fact if to look at a little bit in here very hard to see that are some artifacts in our case uh you know seems like we do some of those out the to since some of techniques uh this is again that's no uh region uh uh you can see those are track that type point that they are actually maybe again pretty hard to see "'em" sorry here but uh if you look at a show no oh no real monitor um most of this out to face got to eliminate uh this is there a again then it or S uh like house image seems to reconstruct it fairly well um i don't to deceive actually and the sense that you know the this problem still quite all there is still a lot of problems actually in here i was supposed to assume but that can do it "'cause" it don't have the power point and if you look at the fancy at actually there's a fair amount of artifact in our case and also on the on line running so um uh in conclusion basically um what we have uh is this a new for mark where we capture out making that clear distinction between spatial sparsifying dictionaries and colour sparsifying find dictionaries seems that uh were able and most of the techniques actually that use compressed sensing uh to the denoising problem are able to recover most of the colours not all of the colours nevertheless we believe that is still to the someone of actual challenges this is really by for uh and on so problem and are many good reasons for that and with that all stop and oh open for questions there are no questions ooh one question and i i i found you uh the um the the you you be or a new uh reference thing this endangering dictionary four connor for car and i think is not but there were or an or dimension of connery used three i G B three and you would uh i'm not that the mission of the of space oh no it's not but not a you use a um more than three uh uh con uh vector is to represent the connor kind space but uh no the problem is uh if you we now uh uh increase the about there's the number of the vectors and the car space you also something uh i nice and are the the the lance of the vector the back to as in this sparse it's in that the last of this past are uh and that maybe are make the optimization a a problem a house C more complex yeah a more complex no that's very true that's why actually wanna just select than a the right number of of colour frames and he do not when over do it and fact this is one of the problems we kind of work all right now which is trying to figure out what is the optimal number and many of the result i just presented this uh is a you know we can of gain by experience and somebody body will charge of problem to figure out a mean definitely for example for the high frequencies you do not twenty use to many colours we could you see artifacts right the way aside from the fact that it's complexity so you uh you lots of the like to do not twenty use you know to many colours and the something stinks okay are no more questions were move to the last but not least talk by professor about work can see