okay

a

the next oh is my talk

and and this work was done in collaboration with the by to graduate students

you moment and then and from and the goods there

and motor G calm more from an eastman kodak research flipped

um

that time for the top basically all the to be simple i'm gonna start uh by giving good a fairly

high level uh overview of the problem of you as can just sink case there are people may not be

familiar with a

and then i'll talk little bit about the this compressive demosaicing framework that the we have it to to do

reduce recently

and then not talk about colour frames for compressing

compressed you missing

so uh again a bit introduction to the problem of demosaicing mosaic

um

we are familiar with the fact that uh images

at is the way human visual system perceive then uh requires three colour plane

uh traditionally additionally red green and blue

and

fact if you wanna a capture the such images uh all these three color planes you let's roll need three

uh C C D sensors in your come

i the problem with this of course is that there is tremendous amount of course uh and also that size

issues

so i to the all the come as that uh uh with

and a multiple C C D sensors

so it turns out that um

the vast majority maybe almost every single kind of that

people have including the ones maybe in your i i've phone or i part of rubber

a i actually uses only a single uh C C D

uh

says and the way this is uh don is basically a they put what's known is a colour filter already

which literally it admits all the single

colour per pixel in instead of

capturing in three

uh pixels

so um

traditionally additionally if you look actually if you are a lot to look inside the got of if you're a

which we usually cannot not

uh you'll see actually the and may each on the right

uh uh is the one that is captured why the camera

and uh you know this is a original image so let's really we do a you do not have access

to all the colours that exist in the original image

uh obviously the image that's kept it is very much dictated by and the nature of the

colour for tell in colour pattern of that C F eight

i the most popular or uh C F A it used or which was developed uh by X actually

uh uh is known as the bayer filter out uh which should basically uses to green colours

for every you read then a uh and a green but uh i'm sorry

to green colours for a really right and blue pixel

and in any to by two uh a block of range

fact if you lose now uh you know all as in here looking at the and is just looks kind

of green but you only doing to put a population

how well but if you actually zoom and fact that the

my presentation which was a powerpoint

unfortunately for had some distortion

if you zoom in actually at the zoom more could see the red and blue pixels in addition to the

green

okay so how do you actually do you mosaic the image i mean the problem due mosaic basically recovering the

original red green and blue uh

signal

uh from what you have capture which just a single pixel a single colour per pixel

well that are basically you know to type of

redundancies and dependencies that exist in your uh to signal but you could exploit

uh the first one which is the obvious one is to the spatial or the pixel

i dependencies in here if you look at the

but you capture in of the right

the green and W blue obvious you have a lot of missing pixels and lot false

a let's really could use any form of interpolation or some kind of smart

frequency domain um

prices to really fill in the gap

uh that that kind of the is see that you can exploit is uh let's really the um

depends is that exist among the

the colours them self you know there is a fair amount of

but done the see the so if you look at the red green and

uh and blue channels uh in a tradition of people you go to a colour difference step of think will

from there with that from a been compression standards

and then to be able to also exploit that and uh i you get a just some form

fact they've and uh of sparsity by doing that

so you know i i don't know that are probably about ten maybe you know close to hundreds of

you know do you more taking order them is that actually tried to solve this problem and try to recover

the original red green is you know with different the variations on this

you know theme

um um just to give you a flavour about you know the challenges associate to this problem uh

in a list take a look at

what's really merging as

we could call what the the economic canonical test image for that you music can problem

uh this is known as the lighthouse image

and particle that is this block which is the fancy sarah because has a very high spatial frequency

as quite challenging actually to cover all the three colours

uh you know from any new kind of colour of white or black or great

"'cause" all close colours actually the are are present

so this is an example like you

uh some of the leading approach is already at approach is that are highly

um

um um a the and the literature

and you could really see that our a fair amount of you know artifacts actually few trying to do that

uh and the

you know net really trying to criticise these particle to two approaches in how a just wanna give you uh

flavour

uh for the you know the challenges that you really could phase despite the fact that you know these approaches

are the based on very sound

uh i know that skull and theoretical a frame or

uh the are actually approach approaches which do uh uh uh better uh are good be much better

and again i just you want to to highlight like you know some of the challenges you could see and

these are examples

a a of uh some of that affects because

so how could we map the problem of you mosaicing to confront sensing well um it's relatively easy actually from

in principle uh i

you know them uh the problem doom as a king you you basically doing

compressed sensing in the sense that you are

do compress a same by factor of three

and uh you could look at your account not a C F A and that to present or sensing metrics

in

and the compressed sensing setting

and uh you are made you could you know uh choose any sparsifying kind of dictionary

uh that you'd like an try to recover the signal uh you know based on that

um

the that be an but if you work actually apply compressed sensing to this problem

uh

nevertheless a you know especially if you focus on the fact that

the kind see if a something you cannot not do much about

and most of the work really been done in the context of using

the um

uh the they are packed them

uh by the way uh actually really emphasise you know the

this project or

uh has

looked really horrible

colour

artifacts so

i we have a lip college as i'm you could tell this is not an image processing conference so

um at anyway

maybe we'll

one be the same token i are people

and i i any right so um

so the

in here these supposed to be yeah you know green but they look like a at any anyway

so given that the C F is actually is given you that is not what you could do about the

of the focus is on

you know what is really the sparse of a sparsifying dictionary right here

so uh the lead and work and the say it really been don uh at is my opinion by uh

more well uh jewel and or L and uh some of his colleagues

well the actually the a a a whole bunch of all learning

a a line i'm sorry learning link dictionary all go them uh that actually tried to uh figure out what

is the optimal

diction there's the sparse one of the could use in order to recover the colours

and they have a whole bunch of uh you know techniques uh a some of them uh uh the most

actually problem the one and the like this one

the coloured learned simple to a sparse coding a less

i C

and this is some of the results this is some of the are it was also actually to see some

artifact and got improve significantly

through a last

i C

so um this approach even this learning approach actually is

um a i a is a quite promising still has a whole bunch of problem

uh in fact a this particle image and i'm not sure of men a built to zoom but

uh in here uh yeah to look at the snow at and i believe i have an image yes

and that if you can see it but this is the original actually snow

and this is the a cover of through this out them on you know sparsifying dictionary and

hopefully "'cause" see that our uh some fair amount of facts actually the convolution and white

uh you know so the the two colours rule not be recovered

so um so what we have developed act is an alternative framework which we call compressive demosaicing K and it's

it's fairly simple actually

uh again but i apologise this is supposed to be green you know maybe it's it's is gonna some be

people's eyes but

so in here uh basically you could be the image or presented through three matches is that a green and

blue

and these forty three images are are being uh multiplied through this uh as in a simple point wise multiplication

uh had the are the top a multiplication by three different uh mattress mattresses

and we have a linear combination uh to present the measurement that we actually have

so um if you put everything together in terms of metrics for you could the vector or an image that

the vector right the are red green "'em" we'll that you're trying to recover

and this uh

uh

multiple uh multiple just as an here they are present really the different for presentation of your sense

and didn't of what you capturing

this represent present of course a little bit the more general

you know a a a a framework contains of it does not have to work it with the bayer pat

and but you you could capture any kind of pat then

the the same time you're actually i i i'd hearing to the constraint which is very important for the problem

you was aching

which is capturing all a single

colour or pixel

and a but important distinction there that colour that you capture lack you has to belong to that article pixel

you cannot do and any linear combination anything

that's why you C Ds match this is actually they are gonna

so this is a very important constraint that is not much you could do a what

and that there was you cannot not generalise

this matrix anymore

uh that's the most general could have

okay so um

so basically in a few uh

uh a you know

a to this kind of framework which is a a of simple now uh the idea is to go to

the rgb vector eyes an agent tried to replace a basically with the sparse representation as we have done before

but is no or you not much new they are so now we could represent

some kind of frequency or presentation all the different

coloured the R G and B

and now we could actually use different uh dictionary so if you put everything together again

um uh uh now we have again you C cfa image or C F i'm sorry uh at tricks which

is sense the metrics and then you have

you're a sparsifying dictionary and now we have the flexibility of using different dictionaries actually for different colour planes

uh if you wish to

um

now uh

up to this point actually um you know that is really not much significant improvement if you try this kind

of framework which is really very simple and you could of course go and try to

uh find the sparse error back to sell

the biggest problem of this approach actually and in general are in fact it's if you still

uh operate and a three dimensional colour space and does out of its our G V or Y V

then you really not exploiting the

the uh

core correlation among the different colour planes and more specifically you really cannot get much sparsity

so this this uh back to to self it's actually sparse is not sparse in a

so what we have done actually was started to expand these uh you know the are uh

atoms if you all

and too much larger a you know a a dictionary what we could act start to look at colour that

you know sometimes i in colours that that yeah uh oh what i do not see

so this an example you what we have used in our uh in know a little work

where we have used the you know

more than three colours and this colours actually you could

uh design and uh using a a you know classical uh

at a for main a frame or try to achieve you know maximum uh in

uh now this is kind of a little uh you know like the go for um

the more general framework that uh you know we are uh focusing on in this particular

you know um uh paper in here

uh this just to show that a cook results about two what happened when you start use compressive and demosaicing

again components some of the traditional approaches

and a see actually the of a fair amount to the artifacts actually got a a a a reduce or

in fact eliminated

this still like just some problems here that a point total bit later and the talk

so um what you have that actually can generalising this compress you mosaic can by uh a working got it

uh a little bit more of broad or framework

and what you're proposing is really to have

this clear distinction between two type of sparse fine

dictionary is one of them for

uh the spatial or than the C another one for for the colour then that's

so this is the overall all kind of frame can the question here

if you are given the counter a cfa

and that's assume you could use any spatial sparse to find the channel and here really you could use anything

including

a dictionary that you could learn uh

uh a line i real time or off

so the question is uh you know what can you do with the colour sparsifying dictionary

so with that uh and place and of for going back to the you know kind of the more general

phone we started it

so uh what we could do actually we could start to look at the different

spatial frequencies

with the rgb uh you know vector that we have a uh a uh a a i

and

if we pick any uh either frequency component spatial frequency component of these uh are rgb colours that we trying

to model present

or or could pick any actually frequency band doesn't have to be a only a single frequency component

then uh this for a spatial frequency you could actually tried to sparse if fight

by using you know uh uh as many colours looks really as you like

and hopefully that will give you more sparse solution and that will also help you

compressed sensing solve to actually find the sparse solution

uh you know will bit better

so uh now this particle or uh

you know a a few and here the this is for one or what a particle or spatial frequency that

um

a that we have

if you put everything together you could actually have

uh all your rgb top let's

or all different to spatial frequencies

and just the number of spatial frequencies you could have could be as many as a list what is uh

uh as you want to it's a function all of the spatial frequency that you have used

uh just parts why or i'm and that's could be D you could be way of what could be of

an over complete

and for each of them what's really could have you on colour dictionary

so could have different colour different colour presentation here

so uh uh these different to color frames stand represent your uh a new set of the chin that you

could use

and they could be combined of course uh with the sparse representation that

each of them will be view uh different

spatial frequency representation presentation of sparse representation

so uh uh for put everything together and here actually uh you had this uh uh a to six which

include all the frames

and then and this is need to be combined actually with the permutation metrics in fact just to give you

a matching between or spatial

frequency or meant and you're frequency yeah a colour uh

frequency range me

so um

now if we put the again everything together uh so what we have

uh for for a given a spatial frequency uh uh a sparsifying dictionary dictionary's

and giving also see C F A

we could represent present or now uh colours sparsifying find dictionary as combination of open imitation metrics

and as a a a a and a colour frame uh and the colour frame is actually as i showed

this would kind of a able uh with

uh diagonal type of metrics

and of that of this is now your overall the sparsifying dictionary

is really combination of three mattresses as one of them is the spatial

uh as sparsifying dictionary that other one is the permutation metrics and the third one is you colour frames

and metrics

uh are more familiar perspective for compressed sensing is your projection matrix is really consist of four different mattresses

the sensing matrix

spatial

ugh permutation and then the colour frames

so now basically this reduce the problem simply

once you model this way

you have your P and all you do is basically look for your sparse a solution already could use a

one or a minimization of course you know a lot so basis pursuit

one of our uh you prefer

so uh the cook think about the simulation results so what we did actually of course you know as i

mentioned

with the colour frames you could actually designed a color frame for every single spatial frequency

uh for example if using dct or only have

you know let's say in this example sixty four possible

color dictionaries you could use

each dictionary could be of different number of of atoms if you want

or basis vectors

uh what you have done actually we all the design of three bands

uh called the for three band so the for the first and most important one what is the D C

one

and one interesting characteristic of the D C one uh is the fact that is is always positive

so that is no uh uh uh

you know a good reason to really have your or uh

you a basis vectors and that dictionary called diction or to go beyond just the the positive or

so that's what we focus on there

and also a a of course very important that include the luminance

and in jail or the more colours the maria

a this is you know of an extreme examples of all back you know colour atoms that you could use

and such dictionary

uh you know we want to uh up to like sixty four different colours in fact but um you know

you we do not need that

and what we end up

well using good uh something like this which is always seven and or nine maybe up to twelve

to be you know practical and also um you know what you find don't actually works

you know quite well

now the second band which we call it a band or sometimes you "'cause" you know arguably call will medium

band

and here of course now we have positive and negative

in general depending on what your spatial frequencies are but if you use T

then you have positive or negative values

then you only need to expand the whole uh you know uh

a three dimensional space

and you could basically use really anything get from P T a for a normalization of free T and this

is really an example of

a colour frame that you could use

uh and again you

but could from media an optimize it using in a uh in a different technique

and last but not least is the high frequency uh bad

and here actually uh in fact you don't need much

uh

uh colours to use uh uh and a fact if used too much colour could get some colour artifacts

so only uh

you know few colours in fact some of the our experiments we all use a single back that would just

a low men and to you know quite fine

so this is kind of an example for the high frequency one

so if you put all the three colours band this a basically but you get

uh this is some of uh uh the simulation results are we're getting this the original image

"'cause" see this at which is kind of a challenge one for some of the leading approaches

"'cause" you some call aspect here

uh that will while guy actually you know does very just job but in fact if to look at a

little bit in here very hard to see that are some artifacts

in our case uh you know seems like we do some of those out the to since some of techniques

uh this is again that's no uh region

uh uh you can see those are track that type point that they are actually

maybe again pretty hard to see "'em" sorry here

but uh if you look at a show no oh no real monitor um most of this out to face

got to eliminate

uh this is there a again then it or S uh like house image

seems to reconstruct it fairly well

um i don't to deceive actually and the sense that you know the this problem still quite all there is

still a lot of problems actually in here i was supposed to assume but that can do it

"'cause" it don't have the power point

and if you look at the fancy at actually there's a fair amount of artifact in our case

and also on the on line running

so um uh in conclusion basically um what we have uh is

this a new for mark where we capture out making that clear distinction between

spatial sparsifying dictionaries and colour sparsifying find dictionaries

seems that uh were able and most of the techniques actually that use compressed sensing

uh to the denoising problem are able to recover most of the colours not all of the colours

nevertheless we believe that is still to the someone of actual challenges this is really by for uh and on

so problem

and are many good reasons for that and

with that all stop and

oh open for questions

there are no questions

ooh

one question

and i i i found you

uh the um

the the you you be or a new uh

reference thing this endangering dictionary four connor

for car and i think is not

but

there were or an or dimension of connery used three

i G B three

and you would uh i'm not that the mission of the of space

oh no it's not

but not

a you use a um more than three uh uh

con uh vector is to represent the connor

kind space

but

uh no the problem is

uh if you we now uh uh

increase the about there's the number of the vectors

and the car space

you also something uh

i nice and are the the the lance of the vector the back to as in this sparse

it's in that

the last of this past are

uh and that maybe are

make the optimization

a a problem a house C

more complex

yeah a more complex

no that's very true that's why actually wanna just

select than a the right number of of colour frames and he do not when over do it

and fact this is one of the problems we kind of work all right now which is trying to figure

out what is the optimal number

and many of the result i just presented this uh is a you know we can of gain by experience

and somebody body will charge of problem to figure out a mean definitely for example for the high frequencies you

do not twenty use

to many colours we could you see artifacts right the way

aside from the fact that it's complexity so you

uh you lots of the like to do not twenty use

you know

to many colours and the something

stinks

okay are no more questions were move to

the last but not least talk by

professor about work can see