but i well they first off thirty is my to than would to silver of a and the other authors have from johns hopkins jobs at kings into more S and so the idea is that we use hearing and that's to convey a graphical and pictorial information and so i'm so the in that X and here is i it's more of a translation between a a video and an audio and and what using that is the pointing device you'll see so the idea is to to be able to do the display just display information or maybe eventually use it for navigation a a and and a key thing is that the user that explores with the finger a if usually a and of course i i mean if you want to to give visual information to people were visually impaired our main target is visually impaired i mean there are really a invasive procedures like visual usual T is you you in since F the cheap the in that a team at that no quite quite invasive that have less invasive but still not not too pleasant approach is like the time display view for example where you in of um it on the on the time can and that stimulates the time according some visual input okay and and there have the remotes for doing this like a line drawings for example a a and and the that been some side is on those of course i mean a a our results can also benefits i the people in cases where that is not like i mean file fighter operations and smoke military missions at night and so so a little bit of you study how we arrived that these i i i i one of our goals "'cause" been to i mean he's been to press load visual to tactilely me silly a so we have any mud uh we segment we it we have any much which to segment and and then we present segment by different fact that X two and and it's important to have perceptually distinct act i'll text so one thought wise and several people have suggested this that to enhance the differences between the tactile textures you may want to large sums so different sounds okay a a and that would work very well if you have a display that can both modify the self us and and and a it i i acoustic feedback is is out of that in the set but but existing displays is the you cannot do that i mean we have a i pods and and and i thought is and also of devices and it that it that let okay a a that they can at it's point with the to that's but they can change the sense for so we thought it would be interesting to do or the fact that display used that's just as the pointing device and and and this and use the i was the display so the idea is a very simple idea we have a i mean we simplified very simple shapes okay and again as you explore with your finger you hear a different sounds is you is you moving to different it saves and you can that you can be you can try to to to to to identify the saves as you pray okay so a and of course uh it's you're moving or fink get and you hearing the audio your and how you moving a finger is what really gives you the information so that's they can can a static feed so i'm just to give you an idea it's or i one one in one of our configurations you have a sound for the background so wait a you getting yeah and and they have a complete the it this thing it a a sound for for the inside you so as you move your get you hear the different sounds and you form an opinion of quite are the object now a a we we chose to to have one sound inside one sound outside because uh there is evidence in the literature that it's easier to i mean i can from from i from uh back that perception that a days a raised line drawing is more difficult to perceive then you've you have a strong object so we tried to do that the same in sound but what we found easy it's important for people to identify the object when that inside the object but they also tried to trace the board so probably we need we conclude that the probably we need a combination of both so then we tried another configuration what do we have one sound inside one outside and the distinct sound for that for the for the ball is the same sounds here okay but uh and N is an inside the the other but but hope well i have to but then you have a very different sound for the background for the for the board and then the sound sound that you have before four in P so we have three this thing cost on no that is a problem with the racing the the the the bound E in that if you make the bound that it to not your fink it can sleep outside and it's it's kind of annoying going on and off the bound E if you make it to broad then then you don't necessarily for the so you might fall another project and you may get that own perception about that the pair that the the bond it looks like so we thought that we need that some proximity feedback inside the board there okay and for that we try to use them as melissa so that i'm only describe is a low frequency a it this guy by that and rate but i at that and we use those to to define the object and then we say the carrier frequency to so it how half we out of a proximity hundred a new out of the bar so just so you with the picture okay so before a like the more detail again that at two major segments but it's segment "'cause" the board is done and yellow is the is the board that between the inside and the outside and and uh the the F R it is constantly image segment but increases as the fink that our approach is the bank E and here is how it very so we have one sound for the inside okay and another sound for the outside can he this and then we have a different sound at the border there for example like these and they five play then type thing it's it's constant the the F R and then ink is this one you are on the board there and then decreases and then eventually been one and big now there was a problem with the problem is in principle it's fine but in practice the inside and outside sounds are not very easy to define say are not is perceptually distinct is the other signals it i and also that i a lot eight depending on how you move your finger made sense to fast in the small area to be not simple so this didn't quite work very well well the next step was to use the X and also so we wanted to use a a directional acoustic feedback in the back for example when your finger is down here the sound you can hear the sound coming from a and in as with that direction and also when you train is the but is the bound T this sound guys you to move in a in a direction to for of the the bond but in a wise five now to do these okay we had again three segments a a background and foreground and and the board there but now um the inside the still constant but the outside is the one that was the tax from now so i don't know if you should be these from one side i O i right yeah and and then when you are on the board the you he of the board the sound but again in the direction that you're supposed to move okay and and we to in in a practical suggesting you if if phones and the sound comes from that a right side and we i wanted to a realistically simulate more late the idea of the thing of if your it is over there to have the sound coming from the object or if you are here to be coming from the object then is you approach the object the sound becomes a loud that and we wanted to simulate at least E gays well there is a sound source in maybe in the coordinate of the object and there's your finger approaches you hear a quite you would have to you were standing there and listening to be lot okay case of for this the mathematical formulation that you need for this is what is called the head related transfer function to to of the directionality and then you play back was that okay okay now again a is a said uh the list and there is assumed to be the pose just on your fingers so so you out of it early that is on on the screen okay and then you hear the sounds coming from that property direction okay again if you in the background from close just point the object and when you're on the board from the direction in which you need to move finally we i i i mean this i i and define an object eight but are also interested in an overall layout and and if you if you to test this idea is i mean if we have a very simple layout but you can imagine you are standing at the window and you are looking at the buildings across the state and and two but was from the from the blind but examples i mean typically a blind person like have a a and and and try to explore than about "'em" in front of them but might think these being the features stick that you that at the objects on on on the you know a across the study and and and you you explore what's there okay so if this is a three let's say in a very simply far you make have sound of wood and and this may be a glass house a so you may have another sound a a a and this maybe oh sorry this was a one is met that and is defined as a yeah a few times okay so so so now the sound here conveys is the something about the object and also you get an idea of the layout so we not in in that in the same here if you need to explore a shape maybe you have another remote what used mean and and you are like what we discussed before but this can give you a nice idea of what the layout days okay and you don't have to be realistic with the sounds even though you are a scanning with your fingers i mean we can the the egg pretend that you are tapping with with a with a steak and and and you can have very testing so it's quite is if you have a different object to you think it maybe you'll get almost the same sound or no sound the all so we don't have to be really okay now a few words about about that experiments we use the subjects in the five configurations and i i mean i have the details if your of and to a that in the paper if you want to and to to look them map we use the and and i but that screen and for our for out of the most a sounds and i have one with me if you want to play with it after the the session and and use the is that or headphones and now it was very important how would and the experiment so first to gave taxes to the subjects but we give them no information about what objects that are going to be looking at and that actually even during the experiment if they were guessing one object to really did not make it clear that the same object my a appeal again so the at you did not have any idea of what that exploring the only thing maybe be that that are let at a relatively simple and we use the training example to to make them am can for the these a we did not want them to be blindfolded okay well actually you would say i have a screen that "'cause" nothing when need but even if you watch your finger that's visual feedback so one to eliminate the visual feedback because out assuming a case what you can not see so so what do you D we put them that that screen and that of box so they could not see that that screen or the thing but you didn't want to i mean typical i guess in in in this type of experiment you don't want to blind for people because that's not a very comfortable thing so that the most not sort of thing is to to have the the the ipod pod and the and that of box and then you have eye contact the experiment there his i contact with with a subject and then they try to explore but to explain to them what's happening first they tried it by looking at the shape and and and X and the fink get an exploding then we turn off the the the visual signal day the triangle or very the cost this case okay and and they could see only the thing get and finally they went to that is situation for they could not see the finger or or or or okay of course we use the different objects for training then for for the testing actually use the cost for the for the training and and it's quite a set in the triangle for for the testing okay and and and you know some details about the fourth configuration basically what kind of a that the you have we use uh and and we had i mean a date that you have it's important to much to that to the specific characteristics of of the user and we had the very simple configuration of whether are two plus is and you could calibrate one of the other but of course in in a critical situation you might have an individual it's to you have for the particular user and that is expected to work but so now the end of the experiment we did not immediately ask them to tell us what's a these but again lead from that from the literature a a i and and and the work of winces is that the that of a uh in a well i i actually is is done it do to his work and it we asked them to first the this shape okay with with a pen on on on the on and one on paper and then ask them what do you think this a piece and that that what but there are much better for them okay because it it helps you a realise what you have five okay so again we did not give that the subject any feedback i i i do you all the experiments i Q know that is out okay in the first configuration the simple of fig iteration with a object and background we got sixty seven percent i in the second configuration of we had that the ball there we got seventy seven percent so there is significant improvement that i a low really didn't help we what that the back to to low where a rate and finally detect X sound good the big impact in can is to to if three press the cats of course sees he i it wasn't if very immediate affect it took like three to four me needs to be able to identify say but our whole please that these these subjects were not playing and and is you train sound it's especially visually impaired subjects who depend critically on on doing these that that will become much much five okay again i well as what can you a sound only all obviously you can you can combine with with that a that that signals when the device that are available and then you can improve more but but but i think it's of very it "'em" see what that of the means of i one sound is finally for they seem layout these these what to use this a training seen of this as the actual testing scene and we have the subjects six indicate that a this was used sounds this sage but the couldn't tell much about the as you "'cause" there not this in know detailed is the individual ones and the material that there are made and that were quite well they were able to identify the materials quite quite uh reliably there was some confusion between glass and metal bell and met the link last okay and and and of course is as is like a first i if you do you record of the sounds more get for your big different sound you can do even but so we present an approach for conveying graphical a pictorial information by hearing and that and the subjects were able to identify basic geometric shapes okay and and and compare to other approaches that didn't go into details of existing approaches a had to so significantly better performance and we also tried if its all came approach square it's a basic simple steps and then subjects that it able to locate and identify all object and again but that's sell applications out of my map perception and navigation and of course yeah at goal is to combine with that's and and find and and use the best of its modality to to do the best and conveying the visual information thank you very much i from X or to start with a weird rubber and the now we can have some questions for houses hmmm you are are and the where some former audience stop