okay
a
the next oh is my talk
and and this work was done in collaboration with the by to graduate students
you moment and then and from and the goods there
and motor G calm more from an eastman kodak research flipped
um
that time for the top basically all the to be simple i'm gonna start uh by giving good a fairly
high level uh overview of the problem of you as can just sink case there are people may not be
familiar with a
and then i'll talk little bit about the this compressive demosaicing framework that the we have it to to do
reduce recently
and then not talk about colour frames for compressing
compressed you missing
so uh again a bit introduction to the problem of demosaicing mosaic
um
we are familiar with the fact that uh images
at is the way human visual system perceive then uh requires three colour plane
uh traditionally additionally red green and blue
and
fact if you wanna a capture the such images uh all these three color planes you let's roll need three
uh C C D sensors in your come
i the problem with this of course is that there is tremendous amount of course uh and also that size
issues
so i to the all the come as that uh uh with
and a multiple C C D sensors
so it turns out that um
the vast majority maybe almost every single kind of that
people have including the ones maybe in your i i've phone or i part of rubber
a i actually uses only a single uh C C D
uh
says and the way this is uh don is basically a they put what's known is a colour filter already
which literally it admits all the single
colour per pixel in instead of
capturing in three
uh pixels
so um
traditionally additionally if you look actually if you are a lot to look inside the got of if you're a
which we usually cannot not
uh you'll see actually the and may each on the right
uh uh is the one that is captured why the camera
and uh you know this is a original image so let's really we do a you do not have access
to all the colours that exist in the original image
uh obviously the image that's kept it is very much dictated by and the nature of the
colour for tell in colour pattern of that C F eight
i the most popular or uh C F A it used or which was developed uh by X actually
uh uh is known as the bayer filter out uh which should basically uses to green colours
for every you read then a uh and a green but uh i'm sorry
to green colours for a really right and blue pixel
and in any to by two uh a block of range
fact if you lose now uh you know all as in here looking at the and is just looks kind
of green but you only doing to put a population
how well but if you actually zoom and fact that the
my presentation which was a powerpoint
unfortunately for had some distortion
if you zoom in actually at the zoom more could see the red and blue pixels in addition to the
green
okay so how do you actually do you mosaic the image i mean the problem due mosaic basically recovering the
original red green and blue uh
signal
uh from what you have capture which just a single pixel a single colour per pixel
well that are basically you know to type of
redundancies and dependencies that exist in your uh to signal but you could exploit
uh the first one which is the obvious one is to the spatial or the pixel
i dependencies in here if you look at the
but you capture in of the right
the green and W blue obvious you have a lot of missing pixels and lot false
a let's really could use any form of interpolation or some kind of smart
frequency domain um
prices to really fill in the gap
uh that that kind of the is see that you can exploit is uh let's really the um
depends is that exist among the
the colours them self you know there is a fair amount of
but done the see the so if you look at the red green and
uh and blue channels uh in a tradition of people you go to a colour difference step of think will
from there with that from a been compression standards
and then to be able to also exploit that and uh i you get a just some form
fact they've and uh of sparsity by doing that
so you know i i don't know that are probably about ten maybe you know close to hundreds of
you know do you more taking order them is that actually tried to solve this problem and try to recover
the original red green is you know with different the variations on this
you know theme
um um just to give you a flavour about you know the challenges associate to this problem uh
in a list take a look at
what's really merging as
we could call what the the economic canonical test image for that you music can problem
uh this is known as the lighthouse image
and particle that is this block which is the fancy sarah because has a very high spatial frequency
as quite challenging actually to cover all the three colours
uh you know from any new kind of colour of white or black or great
"'cause" all close colours actually the are are present
so this is an example like you
uh some of the leading approach is already at approach is that are highly
um
um um a the and the literature
and you could really see that our a fair amount of you know artifacts actually few trying to do that
uh and the
you know net really trying to criticise these particle to two approaches in how a just wanna give you uh
flavour
uh for the you know the challenges that you really could phase despite the fact that you know these approaches
are the based on very sound
uh i know that skull and theoretical a frame or
uh the are actually approach approaches which do uh uh uh better uh are good be much better
and again i just you want to to highlight like you know some of the challenges you could see and
these are examples
a a of uh some of that affects because
so how could we map the problem of you mosaicing to confront sensing well um it's relatively easy actually from
in principle uh i
you know them uh the problem doom as a king you you basically doing
compressed sensing in the sense that you are
do compress a same by factor of three
and uh you could look at your account not a C F A and that to present or sensing metrics
in
and the compressed sensing setting
and uh you are made you could you know uh choose any sparsifying kind of dictionary
uh that you'd like an try to recover the signal uh you know based on that
um
the that be an but if you work actually apply compressed sensing to this problem
uh
nevertheless a you know especially if you focus on the fact that
the kind see if a something you cannot not do much about
and most of the work really been done in the context of using
the um
uh the they are packed them
uh by the way uh actually really emphasise you know the
this project or
uh has
looked really horrible
colour
artifacts so
i we have a lip college as i'm you could tell this is not an image processing conference so
um at anyway
maybe we'll
one be the same token i are people
and i i any right so um
so the
in here these supposed to be yeah you know green but they look like a at any anyway
so given that the C F is actually is given you that is not what you could do about the
of the focus is on
you know what is really the sparse of a sparsifying dictionary right here
so uh the lead and work and the say it really been don uh at is my opinion by uh
more well uh jewel and or L and uh some of his colleagues
well the actually the a a a whole bunch of all learning
a a line i'm sorry learning link dictionary all go them uh that actually tried to uh figure out what
is the optimal
diction there's the sparse one of the could use in order to recover the colours
and they have a whole bunch of uh you know techniques uh a some of them uh uh the most
actually problem the one and the like this one
the coloured learned simple to a sparse coding a less
i C
and this is some of the results this is some of the are it was also actually to see some
artifact and got improve significantly
through a last
i C
so um this approach even this learning approach actually is
um a i a is a quite promising still has a whole bunch of problem
uh in fact a this particle image and i'm not sure of men a built to zoom but
uh in here uh yeah to look at the snow at and i believe i have an image yes
and that if you can see it but this is the original actually snow
and this is the a cover of through this out them on you know sparsifying dictionary and
hopefully "'cause" see that our uh some fair amount of facts actually the convolution and white
uh you know so the the two colours rule not be recovered
so um so what we have developed act is an alternative framework which we call compressive demosaicing K and it's
it's fairly simple actually
uh again but i apologise this is supposed to be green you know maybe it's it's is gonna some be
people's eyes but
so in here uh basically you could be the image or presented through three matches is that a green and
blue
and these forty three images are are being uh multiplied through this uh as in a simple point wise multiplication
uh had the are the top a multiplication by three different uh mattress mattresses
and we have a linear combination uh to present the measurement that we actually have
so um if you put everything together in terms of metrics for you could the vector or an image that
the vector right the are red green "'em" we'll that you're trying to recover
and this uh
uh
multiple uh multiple just as an here they are present really the different for presentation of your sense
and didn't of what you capturing
this represent present of course a little bit the more general
you know a a a a framework contains of it does not have to work it with the bayer pat
and but you you could capture any kind of pat then
the the same time you're actually i i i'd hearing to the constraint which is very important for the problem
you was aching
which is capturing all a single
colour or pixel
and a but important distinction there that colour that you capture lack you has to belong to that article pixel
you cannot do and any linear combination anything
that's why you C Ds match this is actually they are gonna
so this is a very important constraint that is not much you could do a what
and that there was you cannot not generalise
this matrix anymore
uh that's the most general could have
okay so um
so basically in a few uh
uh a you know
a to this kind of framework which is a a of simple now uh the idea is to go to
the rgb vector eyes an agent tried to replace a basically with the sparse representation as we have done before
but is no or you not much new they are so now we could represent
some kind of frequency or presentation all the different
coloured the R G and B
and now we could actually use different uh dictionary so if you put everything together again
um uh uh now we have again you C cfa image or C F i'm sorry uh at tricks which
is sense the metrics and then you have
you're a sparsifying dictionary and now we have the flexibility of using different dictionaries actually for different colour planes
uh if you wish to
um
now uh
up to this point actually um you know that is really not much significant improvement if you try this kind
of framework which is really very simple and you could of course go and try to
uh find the sparse error back to sell
the biggest problem of this approach actually and in general are in fact it's if you still
uh operate and a three dimensional colour space and does out of its our G V or Y V
then you really not exploiting the
the uh
core correlation among the different colour planes and more specifically you really cannot get much sparsity
so this this uh back to to self it's actually sparse is not sparse in a
so what we have done actually was started to expand these uh you know the are uh
atoms if you all
and too much larger a you know a a dictionary what we could act start to look at colour that
you know sometimes i in colours that that yeah uh oh what i do not see
so this an example you what we have used in our uh in know a little work
where we have used the you know
more than three colours and this colours actually you could
uh design and uh using a a you know classical uh
at a for main a frame or try to achieve you know maximum uh in
uh now this is kind of a little uh you know like the go for um
the more general framework that uh you know we are uh focusing on in this particular
you know um uh paper in here
uh this just to show that a cook results about two what happened when you start use compressive and demosaicing
again components some of the traditional approaches
and a see actually the of a fair amount to the artifacts actually got a a a a reduce or
in fact eliminated
this still like just some problems here that a point total bit later and the talk
so um what you have that actually can generalising this compress you mosaic can by uh a working got it
uh a little bit more of broad or framework
and what you're proposing is really to have
this clear distinction between two type of sparse fine
dictionary is one of them for
uh the spatial or than the C another one for for the colour then that's
so this is the overall all kind of frame can the question here
if you are given the counter a cfa
and that's assume you could use any spatial sparse to find the channel and here really you could use anything
including
a dictionary that you could learn uh
uh a line i real time or off
so the question is uh you know what can you do with the colour sparsifying dictionary
so with that uh and place and of for going back to the you know kind of the more general
phone we started it
so uh what we could do actually we could start to look at the different
spatial frequencies
with the rgb uh you know vector that we have a uh a uh a a i
and
if we pick any uh either frequency component spatial frequency component of these uh are rgb colours that we trying
to model present
or or could pick any actually frequency band doesn't have to be a only a single frequency component
then uh this for a spatial frequency you could actually tried to sparse if fight
by using you know uh uh as many colours looks really as you like
and hopefully that will give you more sparse solution and that will also help you
compressed sensing solve to actually find the sparse solution
uh you know will bit better
so uh now this particle or uh
you know a a few and here the this is for one or what a particle or spatial frequency that
um
a that we have
if you put everything together you could actually have
uh all your rgb top let's
or all different to spatial frequencies
and just the number of spatial frequencies you could have could be as many as a list what is uh
uh as you want to it's a function all of the spatial frequency that you have used
uh just parts why or i'm and that's could be D you could be way of what could be of
an over complete
and for each of them what's really could have you on colour dictionary
so could have different colour different colour presentation here
so uh uh these different to color frames stand represent your uh a new set of the chin that you
could use
and they could be combined of course uh with the sparse representation that
each of them will be view uh different
spatial frequency representation presentation of sparse representation
so uh uh for put everything together and here actually uh you had this uh uh a to six which
include all the frames
and then and this is need to be combined actually with the permutation metrics in fact just to give you
a matching between or spatial
frequency or meant and you're frequency yeah a colour uh
frequency range me
so um
now if we put the again everything together uh so what we have
uh for for a given a spatial frequency uh uh a sparsifying dictionary dictionary's
and giving also see C F A
we could represent present or now uh colours sparsifying find dictionary as combination of open imitation metrics
and as a a a a and a colour frame uh and the colour frame is actually as i showed
this would kind of a able uh with
uh diagonal type of metrics
and of that of this is now your overall the sparsifying dictionary
is really combination of three mattresses as one of them is the spatial
uh as sparsifying dictionary that other one is the permutation metrics and the third one is you colour frames
and metrics
uh are more familiar perspective for compressed sensing is your projection matrix is really consist of four different mattresses
the sensing matrix
spatial
ugh permutation and then the colour frames
so now basically this reduce the problem simply
once you model this way
you have your P and all you do is basically look for your sparse a solution already could use a
one or a minimization of course you know a lot so basis pursuit
one of our uh you prefer
so uh the cook think about the simulation results so what we did actually of course you know as i
mentioned
with the colour frames you could actually designed a color frame for every single spatial frequency
uh for example if using dct or only have
you know let's say in this example sixty four possible
color dictionaries you could use
each dictionary could be of different number of of atoms if you want
or basis vectors
uh what you have done actually we all the design of three bands
uh called the for three band so the for the first and most important one what is the D C
one
and one interesting characteristic of the D C one uh is the fact that is is always positive
so that is no uh uh uh
you know a good reason to really have your or uh
you a basis vectors and that dictionary called diction or to go beyond just the the positive or
so that's what we focus on there
and also a a of course very important that include the luminance
and in jail or the more colours the maria
a this is you know of an extreme examples of all back you know colour atoms that you could use
and such dictionary
uh you know we want to uh up to like sixty four different colours in fact but um you know
you we do not need that
and what we end up
well using good uh something like this which is always seven and or nine maybe up to twelve
to be you know practical and also um you know what you find don't actually works
you know quite well
now the second band which we call it a band or sometimes you "'cause" you know arguably call will medium
band
and here of course now we have positive and negative
in general depending on what your spatial frequencies are but if you use T
then you have positive or negative values
then you only need to expand the whole uh you know uh
a three dimensional space
and you could basically use really anything get from P T a for a normalization of free T and this
is really an example of
a colour frame that you could use
uh and again you
but could from media an optimize it using in a uh in a different technique
and last but not least is the high frequency uh bad
and here actually uh in fact you don't need much
uh
uh colours to use uh uh and a fact if used too much colour could get some colour artifacts
so only uh
you know few colours in fact some of the our experiments we all use a single back that would just
a low men and to you know quite fine
so this is kind of an example for the high frequency one
so if you put all the three colours band this a basically but you get
uh this is some of uh uh the simulation results are we're getting this the original image
"'cause" see this at which is kind of a challenge one for some of the leading approaches
"'cause" you some call aspect here
uh that will while guy actually you know does very just job but in fact if to look at a
little bit in here very hard to see that are some artifacts
in our case uh you know seems like we do some of those out the to since some of techniques
uh this is again that's no uh region
uh uh you can see those are track that type point that they are actually
maybe again pretty hard to see "'em" sorry here
but uh if you look at a show no oh no real monitor um most of this out to face
got to eliminate
uh this is there a again then it or S uh like house image
seems to reconstruct it fairly well
um i don't to deceive actually and the sense that you know the this problem still quite all there is
still a lot of problems actually in here i was supposed to assume but that can do it
"'cause" it don't have the power point
and if you look at the fancy at actually there's a fair amount of artifact in our case
and also on the on line running
so um uh in conclusion basically um what we have uh is
this a new for mark where we capture out making that clear distinction between
spatial sparsifying dictionaries and colour sparsifying find dictionaries
seems that uh were able and most of the techniques actually that use compressed sensing
uh to the denoising problem are able to recover most of the colours not all of the colours
nevertheless we believe that is still to the someone of actual challenges this is really by for uh and on
so problem
and are many good reasons for that and
with that all stop and
oh open for questions
there are no questions
ooh
one question
and i i i found you
uh the um
the the you you be or a new uh
reference thing this endangering dictionary four connor
for car and i think is not
but
there were or an or dimension of connery used three
i G B three
and you would uh i'm not that the mission of the of space
oh no it's not
but not
a you use a um more than three uh uh
con uh vector is to represent the connor
kind space
but
uh no the problem is
uh if you we now uh uh
increase the about there's the number of the vectors
and the car space
you also something uh
i nice and are the the the lance of the vector the back to as in this sparse
it's in that
the last of this past are
uh and that maybe are
make the optimization
a a problem a house C
more complex
yeah a more complex
no that's very true that's why actually wanna just
select than a the right number of of colour frames and he do not when over do it
and fact this is one of the problems we kind of work all right now which is trying to figure
out what is the optimal number
and many of the result i just presented this uh is a you know we can of gain by experience
and somebody body will charge of problem to figure out a mean definitely for example for the high frequencies you
do not twenty use
to many colours we could you see artifacts right the way
aside from the fact that it's complexity so you
uh you lots of the like to do not twenty use
you know
to many colours and the something
stinks
okay are no more questions were move to
the last but not least talk by
professor about work can see