0:00:13but i well they
0:00:14first off thirty is my to than would to silver of a
0:00:17and the other authors have from johns hopkins jobs at kings into more S
0:00:21and so the idea is that we use hearing and that's to convey a graphical and pictorial information and so
0:00:28i'm
0:00:28so the in that X and here is
0:00:30i it's more of a translation between a a video and an audio
0:00:34and and what using that is the pointing device you'll see
0:00:38so the idea is to to be able to do the display just display information or maybe eventually use it
0:00:44for navigation
0:00:45a a and and a key thing is that the user that explores with the finger
0:00:50a if usually a
0:00:51and of course i i mean if you want to to give visual information to people were visually impaired our
0:00:57main target is visually impaired
0:00:59i mean there are really a invasive procedures like visual usual T is you you in since F the cheap
0:01:04the in that a team at that
0:01:06no quite quite invasive
0:01:08that have less invasive but still
0:01:10not not too pleasant approach is like the time display view for example
0:01:15where you in of um it on the on the time can and that
0:01:18stimulates the time according some visual input
0:01:21okay and and there have the remotes for doing this like a line drawings for example
0:01:26a a and and the that been some side is on those
0:01:29of course i mean a a our results can also benefits i the people in cases where
0:01:33that is not like i mean file fighter operations and smoke military missions at night and so
0:01:39so a little bit of you study how we arrived that these
0:01:43i i i i one of our goals "'cause" been to i mean he's been to press load visual to
0:01:47tactilely me silly
0:01:49a so we have any mud uh we segment
0:01:57we it we have any much which to segment
0:02:00and and then we present segment by different fact that X two
0:02:03and and it's important to have perceptually distinct act i'll text
0:02:07so one thought wise and several people have suggested this
0:02:10that to enhance the differences between the tactile textures you may want to large sums so different sounds
0:02:16okay
0:02:17a a and that would work very well if you have a display that can both modify the self us
0:02:22and and and
0:02:23a it i i acoustic feedback is is out of that in the set
0:02:28but but existing displays is the you cannot do that i mean we have a i pods and and and
0:02:32i thought is and also of
0:02:34devices
0:02:35and
0:02:35it that it that let okay
0:02:37a a that they can at it's point with the to that's but they can change the sense for so
0:02:42we thought it would be interesting to do or the fact that display
0:02:46used that's
0:02:47just as the pointing device
0:02:49and and and this and use the i was the display
0:02:52so the idea is a very simple idea
0:02:54we have a i mean we simplified very simple shapes
0:02:57okay and again as you explore with your finger
0:03:00you hear a different sounds is you is you moving to different it saves and you can that you can
0:03:05be you can try to to to to to identify the saves as you pray
0:03:09okay
0:03:10so
0:03:11a and of course uh it's you're moving or fink get and you hearing the audio your and how you
0:03:15moving a finger is what really gives you the information so that's they can can a static feed
0:03:21so i'm just to give you an idea
0:03:24it's or i one one in one of our configurations
0:03:27you have a sound for the background so
0:03:30wait a you
0:03:32getting
0:03:35yeah and and they have a complete the it
0:03:37this thing it
0:03:38a a sound for for the
0:03:40inside you
0:03:41so as you move your get you hear the different sounds and you form an opinion of quite are the
0:03:46object
0:03:47now
0:03:48a a
0:03:49we we chose to to have one sound inside one sound outside
0:03:53because uh there is evidence in the literature that it's easier to
0:03:57i mean i can from from i from uh back that perception
0:04:01that a days a raised line drawing is more difficult to perceive then you've you have a strong
0:04:06object
0:04:08so we tried to do that
0:04:09the same in sound
0:04:11but what we found easy it's important for people to identify the object when that inside the object but they
0:04:16also tried to trace the board
0:04:18so probably we need
0:04:19we conclude that the probably we need a combination of both
0:04:22so then we tried another configuration what do we have one sound inside one outside
0:04:27and the distinct sound for that for the
0:04:29for the ball
0:04:31is the same sounds here
0:04:33okay
0:04:34but uh and N is an inside the the other
0:04:37but
0:04:38but hope
0:04:39well i have to
0:04:40but then you have a very different sound for the background
0:04:44for the
0:04:45for the board
0:04:46and then the
0:04:47sound sound that you have before four
0:04:50in P
0:04:50so we have three
0:04:52this thing cost on
0:04:55no
0:04:55that is a problem with the racing the the the the bound E
0:04:59in that if you make the bound that it to not your fink it can sleep outside and it's it's
0:05:03kind of annoying
0:05:04going on and off the bound E
0:05:06if you make it to broad
0:05:07then then you don't necessarily for the so you might fall another project and you may get that own perception
0:05:13about that the pair that the the bond it looks like
0:05:16so we thought that we need that some proximity feedback
0:05:19inside the board there
0:05:20okay and for that we try to use them as melissa
0:05:24so that i'm only describe
0:05:25is a low frequency a it this guy by that and rate but i at that and we use those
0:05:30to to define the object
0:05:32and then we say the carrier frequency
0:05:34to so it
0:05:36how half we out of a proximity hundred a new out of the bar
0:05:39so just so you with the picture
0:05:41okay so
0:05:42before a like the more detail
0:05:44again that at
0:05:46two major segments but it's segment "'cause" the board is done and yellow is the is the board that between
0:05:51the inside and the outside
0:05:52and and
0:05:54uh the
0:05:55the F R it is constantly image segment but increases as the fink that our approach is the bank E
0:06:00and here is how it very
0:06:02so we have
0:06:03one sound
0:06:04for the inside
0:06:07okay
0:06:07and another sound
0:06:09for the outside
0:06:11can he this
0:06:13and then we have a different sound at the border there
0:06:16for example like these and they five play then type thing
0:06:20it's it's constant the the F R and then ink is this one you are on the board there
0:06:26and then
0:06:27decreases and then eventually been one and big now there was a problem with
0:06:35the problem is in principle it's fine
0:06:38but in practice the inside and outside
0:06:41sounds are not very easy to define say are not is perceptually distinct is the other signals it i
0:06:46and also that i a lot eight depending on how you move your finger made sense to fast
0:06:51in the small area to be not simple so this didn't quite work very well
0:06:56well
0:06:57the next step was to use the X and also
0:07:00so we wanted to use a a directional acoustic feedback in the back for example when your finger is down
0:07:05here
0:07:06the sound you can hear the sound coming from a and in as with that direction
0:07:11and also when you train is the but is the bound T this sound guys you to move in a
0:07:16in a direction to for of the the bond but in a wise five
0:07:21now to do these okay we had again three segments
0:07:25a a background and
0:07:27foreground and
0:07:28and the board there
0:07:30but now um
0:07:31the inside the still constant but the outside is the one that was the tax from now
0:07:36so
0:07:37i don't know if you should be these from one side
0:07:40i O
0:07:42i
0:07:43right
0:07:43yeah
0:07:44and and then when you are on the board the you he of the board the sound but again in
0:07:49the direction that you're supposed to move
0:07:51okay and and
0:07:53we
0:07:54to
0:07:55in in a
0:07:57practical suggesting you if if phones
0:07:59and the sound comes from that a
0:08:02right side
0:08:03and we i wanted to a realistically simulate more late
0:08:06the idea of the thing of if your it is over there
0:08:09to have the sound coming from the object
0:08:12or if you are here to be coming from the object then is you approach the object the sound becomes
0:08:17a loud that and we wanted to simulate at least E
0:08:20gays well
0:08:21there is a sound source in maybe in the coordinate of the object
0:08:24and there's your finger approaches
0:08:25you hear a quite you would have to you were standing there and listening to be lot
0:08:31okay case of for this
0:08:32the mathematical formulation that you need for this is what is called the head related transfer function
0:08:36to to of the directionality and then you play back was that okay
0:08:41okay now again a is a said
0:08:44uh the list and there is assumed to be the pose just on your fingers so so you out of
0:08:48it early that is
0:08:50on on the screen
0:08:51okay and then you hear the sounds coming from that property direction
0:08:55okay again if you in the background from close just point the object
0:08:59and when you're on the board from the direction in which you need to move
0:09:04finally we i i i mean this i i and define an object eight
0:09:08but are also interested in an overall layout
0:09:11and and if you if you to test this idea
0:09:14is i mean if we have a very simple layout but you can imagine you are standing at the window
0:09:19and you are looking at the buildings across the state
0:09:22and and two
0:09:23but was from the from the blind
0:09:25but examples
0:09:26i mean typically a blind person like have a a and and and try to explore than about "'em" in
0:09:31front of them
0:09:32but might think these being the features stick that you that at the objects on on on the
0:09:36you know a across the study
0:09:38and and and you you explore what's there
0:09:40okay
0:09:41so if this is a three let's say in a very simply far you make have sound
0:09:46of wood
0:09:49and and this may be a glass house
0:09:53a so you may have another sound
0:09:55a a a and this maybe
0:09:58oh sorry this was
0:10:00a one is met that and is defined as a
0:10:03yeah a few times
0:10:04okay so so so now the sound here
0:10:07conveys is the something about the object
0:10:10and also you get an idea of the layout so we not
0:10:13in in that in the same here if you need to explore a shape maybe you have another remote what
0:10:17used mean and and you are like what we discussed before
0:10:20but this can give you a nice idea of what the layout days
0:10:23okay and you don't have to be realistic with the sounds even though you are a scanning with your fingers
0:10:30i mean we can
0:10:31the the egg
0:10:32pretend that you are tapping with with a with a steak
0:10:35and and and you can have very testing so it's quite is if you have a different object to you
0:10:39think it maybe you'll get
0:10:40almost the same sound or no sound the all so we don't have to be really
0:10:45okay
0:10:46now a few words about about that experiments
0:10:49we use the subjects in the five configurations
0:10:52and i i mean i have the details if your of and to a that in the paper if you
0:10:55want to
0:10:56and to to look them map we use the and and i but that screen
0:11:00and for our for out of the most a sounds and i have one with me
0:11:04if you want to play with it after the the session
0:11:07and and use the is that or headphones
0:11:10and
0:11:12now it was very important
0:11:14how would and the experiment
0:11:16so first to gave taxes to the subjects but we give them no information about what objects that are going
0:11:21to be looking at
0:11:22and that actually even during the experiment
0:11:25if they were guessing one object to really did not make it clear that the same object my a appeal
0:11:29again so the at you did not have any idea of what that exploring the only thing maybe be that
0:11:33that are let at a relatively simple
0:11:36and we use the training example to to make them
0:11:38am
0:11:39can for the these
0:11:41a we did not want them to be blindfolded okay well actually you would say i have a screen that
0:11:46"'cause" nothing when need but even if you watch your finger that's visual feedback so one to eliminate the visual
0:11:51feedback because out assuming a case what you can not see
0:11:55so so what do you D we put them
0:11:57that that screen and that of box so they could not see that that screen or the thing
0:12:02but you didn't want to i mean typical i guess in in in this type of experiment
0:12:06you don't want to blind for people because that's not a very comfortable thing so that the most not sort
0:12:10of thing is to to have the the
0:12:12the ipod pod and the and that of box and then you have eye contact the experiment there his i
0:12:17contact with with a subject
0:12:19and then they try to explore
0:12:20but to explain to them what's happening first they tried it by looking at the shape
0:12:25and and and X and the fink get an exploding
0:12:28then we turn off the the the visual
0:12:31signal day
0:12:32the triangle or very
0:12:34the cost
0:12:35this case
0:12:35okay and and
0:12:37they could see only the thing get and finally they went to that is situation for they could not see
0:12:42the finger or or or or
0:12:45okay of course we use the different objects for training then for for the testing
0:12:50actually use the cost for the
0:12:52for the training and and it's quite a set in the triangle for
0:12:57for the testing
0:12:59okay and
0:13:00and and you know some details about the fourth configuration basically what kind of a that the you have we
0:13:05use
0:13:06uh and and we had
0:13:09i mean a date that you have it's important to much to that to the specific characteristics of of the
0:13:14user
0:13:15and we had the very simple configuration of whether are two plus is and you could calibrate one of the
0:13:20other but of course in in a critical situation you might have an individual it's to you have
0:13:25for the particular user and that is expected to work but
0:13:31so
0:13:31now the end of the experiment we did not immediately ask them to tell us what's a these
0:13:36but again lead from that from the literature
0:13:39a a i and and and the work of winces is that the that of a uh in a well
0:13:44i i actually
0:13:46is is done it do to his work
0:13:47and
0:13:49it
0:13:50we asked them to first the this shape
0:13:53okay with with a pen on on on the on and
0:13:56one on paper
0:13:57and then ask them what do you think this a piece
0:14:00and that
0:14:01that what but there are much better for them okay because it it helps you a realise what you have
0:14:07five
0:14:08okay
0:14:09so again we did not give that the subject any feedback
0:14:13i i i do you all the experiments
0:14:16i Q know that is out
0:14:18okay in the first configuration the simple of fig iteration with
0:14:21a object and background we got
0:14:23sixty seven percent i
0:14:25in the second configuration of we had that the ball there we got seventy seven percent so there is significant
0:14:30improvement
0:14:31that i a low
0:14:33really didn't help we what that the back to
0:14:36to low where a rate and finally detect X sound good the big impact
0:14:40in can is to to if three press
0:14:42the cats of course sees
0:14:43he i it wasn't
0:14:45if very immediate affect it took like three to four me needs
0:14:49to be able to identify say
0:14:51but our whole please
0:14:52that these these subjects were not playing
0:14:55and and is you train sound it's especially visually impaired subjects who depend critically on on doing these
0:15:00that that will become much much five
0:15:03okay again i well as
0:15:05what can you a sound only
0:15:06all obviously you can you can combine with with that
0:15:10a that that signals when the device that are available and then you can improve more
0:15:15but but but i think it's of very it "'em" see what that of the means of
0:15:18i one sound is
0:15:21finally for they seem layout
0:15:24these these what to use this a training seen of this as the actual testing scene
0:15:27and we have the subjects six indicate that a this was used sounds
0:15:31this sage but the couldn't tell much about the as you "'cause" there not this
0:15:35in know detailed is the individual ones
0:15:37and the material that there are made
0:15:40and that were quite well
0:15:42they were able to identify the materials
0:15:44quite quite uh
0:15:46reliably there was some confusion between glass and metal bell and met the link last
0:15:50okay
0:15:51and and
0:15:52and of course is as is like a first i if you do you record of the sounds more get
0:15:56for your big different sound you can do even but
0:16:00so
0:16:01we present an approach for conveying graphical a pictorial information by hearing and that
0:16:06and the subjects were able to identify
0:16:09basic geometric shapes
0:16:11okay
0:16:12and and and compare to other approaches that didn't go into details of existing approaches a had to so significantly
0:16:19better performance
0:16:20and we also tried if its all came approach square
0:16:23it's a basic simple steps and then subjects that it able to locate and identify all object
0:16:28and again but that's sell applications out of my map perception and navigation
0:16:33and of course
0:16:34yeah at goal is
0:16:36to combine with that's
0:16:37and and find and and use the best of its modality
0:16:40to to do the best and conveying the visual information
0:16:43thank you very much
0:16:51i from X or to start with a weird rubber and the now we can have some questions for houses
0:16:58hmmm you are are and the where some former audience
0:17:01stop