0:00:14and you
0:00:15hello
0:00:16and
0:00:17going to talk about the signal then an estimation of stereo party
0:00:21and optical flow
0:00:24we have a
0:00:25mean a curve video set up it means we have to calibrate it and synchronise came
0:00:30observing a scene and producing a stream of
0:00:33is is is there are images
0:00:35and the task is to compute
0:00:37uh
0:00:39this this party map
0:00:42can
0:00:47is there are sir
0:00:48the middle
0:00:50i i can show
0:00:51just
0:00:53uh the computed disparity map between
0:00:56uh between they're a pair
0:00:58and
0:01:02thank you and the optical flow maps
0:01:05between a
0:01:06a consecutive images
0:01:08and
0:01:09together with the calibration this kicks that's that's we D scene so
0:01:13so you you flow it's emotion you of it means that for
0:01:15each reconstruct point we have that to it that it's but C
0:01:21so how does it work with a
0:01:23a simple geometrical
0:01:25uh
0:01:25we have a you can cameras
0:01:27and to a point in three D space
0:01:29this point project to the images
0:01:32and
0:01:33having a the corresponding
0:01:34all these two images
0:01:36we can you can reconstruct
0:01:38C point white a relation
0:01:41that and
0:01:41this this point moves
0:01:44in time T plus one
0:01:45when not or of creation
0:01:46to gain project you me
0:01:49and the from the corresponding from these part we can compute we can reconstruct it
0:01:53one in three
0:01:54but we need to know that
0:01:56points belong want to get or
0:01:57that
0:01:58uh
0:01:59these course
0:02:00and
0:02:01therefore we need to compute also the correspondence
0:02:04i in D
0:02:05in no uh it of the images
0:02:07which is
0:02:08in fact you can flow
0:02:12uh
0:02:13so that
0:02:14that the the task is the plot we we
0:02:17we are given
0:02:18we are and points
0:02:20yeah the the point
0:02:22X time that's time one
0:02:25for
0:02:26and
0:02:27we should compute
0:02:28the
0:02:29one at time T possible one in the next frame
0:02:32where is really point
0:02:34means we we have to compute
0:02:37uh
0:02:39this optical flow
0:02:40and
0:02:42uh this topic of flow in right image in class
0:02:44or or we can compute
0:02:46yeah to of no into a in that image and the just in and in the in the second frame
0:02:51or or we can compute as is per uh optical of in the right frame
0:02:54and the disparity
0:02:56i
0:02:57in the in the second frame
0:02:58which means that these problems are couple
0:03:01that we can
0:03:02a that they usually to each other
0:03:05a you because we have more constraint
0:03:07that the un use task because he's user
0:03:11uh before or
0:03:12well explained and the output hmmm
0:03:14uh show you the results that that you can have tuition
0:03:18well well
0:03:19but you can expect
0:03:21uh so
0:03:22uh this is an input image let image each or is also right image but i don't show it
0:03:27it looks it looks similar this is the disparity map
0:03:30or or to it
0:03:31where my course are close to the camera
0:03:33and the black is
0:03:35and that are on mixed pixel
0:03:38and
0:03:39uh
0:03:40this this is this is that
0:03:42and this is the motion map this is a horizontal component of the optical flow and
0:03:46vertical component of of the call for what gain or coding
0:03:50uh new means that
0:03:51right and down motion
0:03:53yellow or right means
0:03:55left and a motion
0:03:57i and like the video
0:04:02uh you can see D uh how to this party and motion yeah walls
0:04:06well
0:04:08so this basis
0:04:09the in fact this
0:04:10this team of
0:04:11to presentation of the see the
0:04:14so everything single i see now
0:04:16or one on a correspondence problem the correspondence problem is one of the fundamental problems in computer vision and approach
0:04:22leads
0:04:23stop button
0:04:24"'cause" we have a lot of ambiguity but
0:04:26what what else
0:04:28uh
0:04:29i for constraint of course
0:04:31and the
0:04:32new you we use a constraint that each there and optical flows
0:04:35some
0:04:36spatial smoothness
0:04:37because
0:04:38never break is to have similar parties
0:04:41it's two
0:04:41but not
0:04:42at occlusions or object and function boundaries and then process
0:04:45because
0:04:46and of the disparity and optical flow changes abruptly
0:04:49except for or or your quick and about to pound
0:04:53there some solution in the age sorry
0:04:55is to use an explicit regularization
0:04:57uh
0:04:58this
0:05:00this
0:05:01so it's typically it to M R F and
0:05:04partition addition relation
0:05:06uh which is
0:05:07and to because it very computationally in in dense
0:05:11and
0:05:11moreover it produces
0:05:13it's very
0:05:14uh a it it might
0:05:16produce
0:05:17the artifacts
0:05:18which were cost them that were prior model so over the we data
0:05:23and
0:05:23this is
0:05:24not suitable for some applications
0:05:27i was to be we can
0:05:29there are other approaches which are discriminative but it's
0:05:32which just keeps us
0:05:34gives up but the
0:05:36i make use part of the solution and finds only the on a bit used to use far
0:05:40and
0:05:40we we we for to do to in this way
0:05:43lee back
0:05:45or for this purpose we you know that it test seed growing technique
0:05:49uh the basic idea of this going to technique is that we have a set
0:05:53initial correspondences
0:05:55so called
0:05:56and
0:05:58uh uh the these
0:05:59the other correspondences are are found
0:06:02in a small neighborhood around
0:06:04these C
0:06:06and then these
0:06:07you correspondences are you seats end
0:06:10the that this way the
0:06:11the
0:06:12the growing process continue
0:06:15and
0:06:16we have a recently developed a you growing stereo or not not a not but only mister
0:06:22do you
0:06:23some of these that is was published in but mean
0:06:25and a we view be scrolling not work my such way of between
0:06:29uh a of energy minimization that centre
0:06:32complete complete D local metal
0:06:34because the
0:06:35neighbouring structure of the solution is not leak more completely
0:06:39it's uh
0:06:40the the growing process
0:06:41rick is is the solution implicitly
0:06:44a the other uh_huh dissolves
0:06:46a robust against the initial seeds
0:06:49and it's very fast
0:06:50do to to search space reduction action it's
0:06:52and power three
0:06:54goes to and about to where
0:06:56and power to is the
0:06:57the
0:06:58and and and it's quite is that is the size of the images
0:07:01so that that the
0:07:02exhaustive this part space
0:07:05is of this i
0:07:07and the the the work of the uh and that it doesn't produce
0:07:11fully
0:07:11then so results
0:07:13we don't match all pixels in the scene but only
0:07:16set uh
0:07:17some subset
0:07:19which is
0:07:20for for for many applications that is factor
0:07:23so before i'll explain the seen how how we grow to see the why we
0:07:28you if how the
0:07:30uh stammer or row wink our work
0:07:34uh so is there and we have only two images
0:07:37left and right and
0:07:38but say this is the correspondence seat which are
0:07:41the the the C can be obtained by matching a distinct
0:07:44uh have for distinctive if you image features
0:07:48and
0:07:49uh the growing process
0:07:50finds the correspondences
0:07:52in the in the neighbourhood of the C so would it's a four to the right
0:07:56uh it
0:07:57performs the local optimization of the image correlation
0:08:02i still
0:08:03because
0:08:03the pixel which matches the best
0:08:05and if the the correlation is about a trace row then this is accepted
0:08:09you match
0:08:11and uh and uh uh where same
0:08:13and
0:08:14okay the same
0:08:16and these are you match found these matches because seat ten
0:08:19okay
0:08:20a process with pete
0:08:22uh in
0:08:23you can see it for
0:08:24this is as there are there are with the seat
0:08:27and the disparity map
0:08:28scroll
0:08:29so from a single seat you can grow to be
0:08:32a large so
0:08:35uh to up to
0:08:38uh
0:08:39with with this with the
0:08:40scene flow it's
0:08:42it's
0:08:43pretty much yeah
0:08:44similar
0:08:45we have to grow simultaneously disparity map and optical flow
0:08:50so we have
0:08:51well us there oh
0:08:53to there there is that
0:08:54time one and uh
0:08:55time T plus one
0:08:57and
0:08:58the seat
0:08:59it's not
0:09:00a pair of images at at a pair
0:09:02points
0:09:03a course for it the
0:09:04see
0:09:05is a correspondence of for or
0:09:09so it
0:09:10fully determine
0:09:11the
0:09:12that the
0:09:12the local scene flow
0:09:15and
0:09:16uh
0:09:16we have given it is point to map at time T one from
0:09:20from stereo matching i ching for from a previous frame
0:09:23and then the seats
0:09:25the seats are they and here we used in our implementation by matching harris
0:09:29or
0:09:30and
0:09:31low look look as gonna tracker
0:09:33of harris point
0:09:35to obtain
0:09:36a local of the "'cause"
0:09:39and then we we the growing process is the same so it
0:09:43a a it looks in the neighbourhood of the of the
0:09:46initial C
0:09:47the it is like a local bic you can five speed in the paper
0:09:50but the
0:09:51it locally fines
0:09:52locally maximise the correlation the correlation measures
0:09:56uh
0:09:57the similar to
0:09:58all all
0:09:59all
0:10:00all three
0:10:02correspondences
0:10:03and if the correlation is about a racial
0:10:07exceeds X threshold and
0:10:09and you match just found
0:10:11before
0:10:12a a a a a a it covers this what
0:10:14and that's it it's
0:10:15this is that
0:10:16pretty straightforward extension all of these
0:10:19we
0:10:20a of this uh
0:10:22there are going to work but
0:10:24it it it works quite nice
0:10:27so well uh
0:10:29for for the results we perform a think that the ground two experiments want to have a way to
0:10:34uh the performance of the outboard in
0:10:36so we seem to ties the
0:10:39a or playing with
0:10:40the tech
0:10:40texture play
0:10:42which works moving and we
0:10:44we can at the
0:10:46a what with with
0:10:47with noise
0:10:48and we also
0:10:50the D
0:10:51what
0:10:51the seem like we texture
0:10:53and that
0:10:55the the the conclusion of this expert experiment is that
0:10:58the
0:10:58similar in is
0:10:59estimation of joint formulation of optical flow and disparity
0:11:03uh a house a lot and how
0:11:05and uh uh the other work is better than
0:11:08independence dependence on top to cool well
0:11:11you can find it doesn't the paper is i'm not cool
0:11:13in
0:11:15uh
0:11:16some
0:11:17some under real results
0:11:20uh i again the same left image disparity map
0:11:23and to a motion the horizontal and vertical components of the of
0:11:27though
0:11:28you can
0:11:29no the is that the was between objects are what
0:11:33and and they are no
0:11:34that no smoothing got up there are no
0:11:37uh
0:11:38even for a for for you objects which are
0:11:41close
0:11:42to each other different that the different motion that the uh the are is are not confusing
0:11:47there are some and that we that the solution is not fully that that
0:11:51but the
0:11:52i
0:11:53i believe that for many applications is
0:11:55now
0:11:56another example
0:11:58uh the
0:11:59i was clapping can
0:12:01and C D
0:12:02uh
0:12:04colours so
0:12:06and that are
0:12:08and under example is uh
0:12:11moving camera that was predisposed was
0:12:12is that the camera
0:12:14and that the rooms
0:12:15moving we was mounted on a and that may we can arrange and strolled a tab
0:12:20street
0:12:21so we can
0:12:22a
0:12:24the result of the T O what you can find here
0:12:29so the the cameras with be quite right
0:12:31and that that the C
0:12:32C is complex they are many
0:12:34uh a menu objects in three D the various motions
0:12:38but has to hence the
0:12:39cars and
0:12:41the
0:12:43and in the
0:12:44they are shot boundaries
0:12:47a is just a a you objects are
0:12:49and
0:12:50nicely lead
0:12:51these thing which from from the background
0:12:54you with
0:12:55in in that and motion
0:12:59so
0:13:00uh the conclude a uh i would summarise the problem so the proposed that work
0:13:05it
0:13:06a
0:13:06a large displacement between frames
0:13:08more we have more than
0:13:09for a certain pixel in the last
0:13:12a stick for forty
0:13:13which is
0:13:14which is a disaster because it four
0:13:15uh a or was which you for of the variational
0:13:19optical flow and of
0:13:20because the S you in and is not motion
0:13:23uh that or
0:13:26bandages is that
0:13:27i the the which in boundaries are what reserve
0:13:30uh
0:13:31as a problem for
0:13:33hmmm i wouldn't with strong relation they tend to smooth out
0:13:36clues about every
0:13:38is but to maps are
0:13:40for late like what we are the the
0:13:43but are we have a better result than three by friends there one
0:13:46for based thing optical flow
0:13:47yeah you could you could see some flickering is true but the it's a we much better than computing frame
0:13:52by frame
0:13:54and in that the other i think is that
0:13:56uh the other
0:13:57lost
0:13:57it to our implementation is
0:14:00how to sell to team
0:14:01still are like matlab
0:14:03it's france
0:14:04in this or a solution about five or what i have a five seconds it up with that normal
0:14:09what to P C
0:14:10and
0:14:11it means that there was not a significant extra post with respect to a a single step um
0:14:17uh
0:14:18uh uh the reason is that a low low covered make a as a low of with a complexity
0:14:24the the search space
0:14:25which is to be but the correspondences are sort
0:14:29is
0:14:30and how to five for each pixel you have to find that this partly or something vertical to control
0:14:35three uses two
0:14:36and squared
0:14:37the quite the size of the image
0:14:40and
0:14:40that well because that's results are not lead that spot seven
0:14:44we had a nest and version of this song one disciple
0:14:46a point in that
0:14:48a a in the by
0:14:50i wine
0:14:51uh
0:14:52for for for processing the store
0:14:53with with also motion prediction and we have a
0:14:56where in this if you got a paper we have uh
0:14:59experiments a person with other state-of-the-art
0:15:01my thoughts in a in a scene flow special the for the topic of
0:15:07okay thank you
0:15:13can have some questions
0:15:21yes
0:15:32uh i just one thing did you have bet against uh we did but it it has it's and are
0:15:36if you so that you would i come undone god
0:15:40now in this case we didn't comparing in a better T as it because uh a the our our or
0:15:45move does not provide fully dense results and
0:15:48it's not comparable a few
0:15:51it you compare only some part of the solution and
0:15:54to the this party
0:15:56everywhere
0:15:56so it's
0:15:57it wouldn't be fair compare
0:16:04and other question
0:16:10the man
0:16:10a question
0:16:11you so
0:16:12for for is uh
0:16:15the the results are not then so but the if somebody needs to be to have some and we don't
0:16:21to have a any idea of to make it a
0:16:23a more dense
0:16:25okay
0:16:26the
0:16:27it that the uh miss some sub or it's how to control the trade-off between density and or right
0:16:32uh we break her in in for our applications that the
0:16:36that there are
0:16:38less errors and
0:16:40oh it's also has density
0:16:41which is natural
0:16:43i have but you can you can slide control
0:16:45if you relax the artwork and to match then
0:16:48uh the
0:16:49uh
0:16:50then you are getting more more and more illusions send
0:16:54then in a
0:16:55do you want to knit some
0:16:57okay
0:16:57course you can do some processing
0:16:59a some
0:17:00some are are a lot of use
0:17:03you if you know what you are looking at you can interpolate place some some it you can learn some
0:17:08so
0:17:09two for one
0:17:10uh
0:17:12this is another way and T
0:17:14another another option is of course two
0:17:17to
0:17:17incorporate the primary goal you mice
0:17:20similar or to the the
0:17:22you to my station that
0:17:28okay there is no question ms
0:17:30a course again
0:17:34i