Conditions for Conjuring Reality

MC : I've just come from experiencing the VR in the CABIN (Computer Augmented Booth for Image Navigation), a telepresence research environment which you developed. There were scenes of computer graphic simulations of racing through urban environments, yet I was very surprised to feel myself as a mystic flying through space. I guess I'm still a bit excited from the experience, but I felt that this kind of reality really goes to the heart of what VR is all about.

Surrounded left, right and above, seeming reality on all sides. Of course, computer calculation speed is one of the keys to this experience, but suppose that for an instant the computers were not able to keep up with my actions, and that when I looked up I found myself in a temporarily suspended space--caught in a kind of gap because some of the elements which I require to constitute my reality had vanished. Could you please describe the elements of your invention or the essential point to be considered to constitute the reality which I was just experiencing?

HM: That is an extremely difficult question.... What we can say, however, is that recently it seems we're coming to the realization that there are many realities. For example, the reality that you experienced in the CABIN is probably better described as a visually induced sense of presence. Because the CABIN features a display field mapped across five planes (left-right-up-down-forward), the CABIN user is offered an extremely comprehensive field of view. This is perhaps one of the "points" which you referred to. Unlike, for example with the TV, you are no longer outside of the medium looking in.

You are immersed in the medium--immersive telepresence. Being within the imagery in this way, being assaulted by imagery from all sides, allows you a "first person" experience. If we describe the television as a third party medium, then we can describe VR as a first person medium.

Being able to see in 3-D is also an important factor in creating a sense of "reality." Having a sense of depth in the imagery makes things appear to leap out at the user. In other words, it is important to have a world before your eyes in which 3-D depth plays an important part. The level of detail is another "point." High-definition television seems more real than normal TV. It is also important to think of interactivity in creating a sense of reality. Moving your head left and right, you see objects nearby moving more dramatically than objects in the distance. In the CABIN this is also replicated, so that you're not only seeing things in 3-D with a sense of depth, but this interactivity makes your sense of "reality" seem to multiply. What most people praise about the CABIN system is the way in which it utilizes these elements of computer graphic technologies.

The same effects can be achieved theoretically using HMD (Head Mounted Displays), but with HMDs, the user often experiences a kind of motion sickness, because you're trying to make the images move in response to the head's every motion. This problem is solved in the CABIN.

MC: Because there is a delay in the HMDs....

HM: Precisely. Raising one's head in an HMD, the images don't follow in real-time. This is an important difference between the CABIN and the HMD. Because the imagery is already being displayed on the surrounding screens, there is no need to change the imagery with each movement of the head.

With the HMD, you always need to be concerned with accurately updating the display field. With the CABIN, you only need to update the fields when the head changes its translational position, so the burden of representation is halved.

With the HMD, you need to always assure that the head position and data are continually in real-time synch. Computers are generally not fast enough to equal the task. This dormant half allows for quite a perceived difference.

MC: When I first entered the CABIN, I noticed that the four-cornered room was completely different than the world of the HMD. If one follows the history of VR, I think that its logical predecessor was the 360-degree domination of image found in 19th century panoramas.

From the ceilings of baroque, South Germanic and Northern Italian chapels, each are 360-degree panoramas of sort. Whether containing the faithful or not, anyone raising their eyes "towards heaven"was surrounded by these images. In this sense, being contained in one room seems to be another extraordinarily important point.

The condition of a body-- surrounded in a synchronicity with the image--seems to create a key element to such "realities."

HM: When beginning to study VR, I considered what elements my system would be required to have. The first was presence, the second interaction and the third was simulation. The connection between simulation and reality may be difficult to understand, but say, for example, that you were to throw a rock in VR. It is clear that the rock's trajectory must either seem accurate, or be shown as a lie. It must behave as reality would behave.

MC: It must behave according to the physical laws as we have experienced them, or we will cease to believe it as reality.

HM: As such, there are many instances of things which must behave within acceptable reality tolerances. Of course there are many instances where elements of reality must be letter perfect, and then there are elements of reality which can meld into the broad visual field. Other elements of reality will only become apparent when actually engaged. This is why when you ask the question "what is reality?" I can only offer that there are many.

MC: In the 1980s, when VR first appeared, there were a number of new reality-specific research fields which sprang up. Now, however, it seems that we've shifted to yet another era, where we are inversely using VR to emancipate people from their assumptions about reality.

HM: Artificial Intelligence may be one good example, but at the moment of its conception, what was most pronounced were its missing links, and they formed the basis for the research that is going on today. VR research is still dominated by discussions about the visual field, but additionally, research into aural, haptic and other forms of "presence" have also emerged. I find it interesting that the moment when our senses are able to be synthesized using engineering methods is the same moment that we return to seeking new propositions CABINiComputer Augmented Booth for Image NavigationjExterior View about exactly what our senses are.

MC: While it seems like we are dealing with issues of the "real world,"in fact, we are actually dealing with those of the human interior. Or to put it another way, gaining further insights into the incredible extent to which we had no idea about what we are.

HM: We come to many practical applications for research into sound. There are many technologies presently available to us, such as digital processing technologies which can simulate the resonances of being in a cathedral while a choir sings, or technologies for having a sound seem to emanate from a virtual object placed directly in front of us. With haptic technologies, much of the research is still in a more interim phase. It seems that the sense of touch is really just beginning to be brought into the digital realm.

MC: The visual faculties play an inordinately important role in helping the human brain recognize reality. Another is the sense of touch. I've heard it said that the greatest concentration of nerve endings on the surface of our skin is in the tips of our fingers.

HM: Surface sensations are said to be almost equal in the visual and haptic domains.