Virtual Reality: Fictional all the Way Down (and that’s OK)


Jesper Juul: "Virtual Reality: Fictional all the Way Down (and that’s OK)". Disputatio journal, 2019. https://www.jesperjuul.net/text/fictionalalltheway/ https://doi.org/10.2478/disp-2019-0010

Written in response to David Chalmer's 2017 paper "The Virtual and the Real".


Abstract

Are virtual objects real? I will claim that the question sets us up for the wrong type of conclusion: Chalmers (2017) argues that a virtual calculator (like other entities) is a real calculator when it is “organizationally invariant” with its non-virtual counterpart—when it performs calculation. However, virtual reality and games are defined by the fact that they always selectively implement their source material. Even the most detailed virtual car will still have an infinite range of details which are missing (gas, engines, pistons, fuel, chemical reactions, molecules, atoms). This means that even the most detailed virtual object will still have fictional aspects. Rather than argue that virtual objects are, or aren’t, real, it is preferable to think of overlaps and continuities between the fictional and the real, where even the most painstakingly detailed virtual reality implementation of a non-virtual object is still art: a human process of selection and interpretation. Virtual reality should therefore not be philosophically understood just as a technological implementation on a trajectory to perfection, but as a cultural artifact which derives its value in part from its simplification and difference from its source material.

 

Keywords

Virtual reality, games, fiction, art.

Introduction

I would like to thank David Chalmers for his convincing paper, ‘The Virtual and the Real’ (Chalmers 2017), which I will proceed to criticize. I understand that Chalmer’s interests are somewhat orthogonal to mine. Virtual reality (and games) are a primary focus for me, whereas virtual reality serves as a case in a philosophical argument for Chalmers.

Nevertheless, as a description of virtual reality, I think Chalmers’ argument suffers from the flaw of focusing on the hype of virtual reality, rather than on the actual experience or implementation of virtual reality. I will argue that virtual reality may well be real and valuable, but in different ways and for reasons mostly opposite from what Chalmers suggests.

Virtual reality is not, and is not becoming, indistinguishable from regular reality, and that is the point. There are many promises that technological progress is making VR indistinguishable from “the real thing”. Yet in actuality, VR, like all human art, is selective in the way it represents, and this makes a virtual reality object more readable, more predictable, more easily perceived as part of causal chains than regular objects. Like a theater prop, a VR calculator stands for calculators1, and its implementation is specific to the demands of the specific VR experience. It follows that a VR object does not have functions or complexity matching its non-virtual counterpart. And this means that any value of VR—like with art—will have to come in part from the difference between VR and non-VR.

Let me explain this by pointing to what we seem to agree on, namely that there are different types of objects and events in games and VR (Juul 2005). We need not agree on nomenclature, but I have used the term fiction from the theory of fictional worlds (Pavel 1989) to describe the fact that games and VR contain worlds seen as distinct from the regular world, and the term rules to describe what is implemented in the programming (rules are arguably equal to the lower-level constituents of Chalmers’ digital objects).



Figure 1: Three classes of events and objects: Events not explained by fiction, events explained by fiction, and events suggested by fiction but not implemented (adapted from Juul 2014)


Figure 1 shows three classes of objects/events in games and VR (Juul 2005, chap. 5):

  1. Rules not explained by fiction: Objects and events that are not easily explained by reference to the fiction, but which are implemented in the programming and perceived as coming from game structure, or from a level above the VR experience. These includes lives, extra lives, double-jumping in the air with no physical support, and receiving money past go in Monopoly.
  2. Fiction implemented in rules: The objects and the aspects of objects that are implemented in the rules and are available to the user, such as a virtual car we can drive, a light switch we can turn on, an object we can push. We informally describe this as “interactive” or “virtual”. I call this half-real for reasons I will discuss below.
  3. Fiction not implemented in game rules: Objects that we cannot interact with the way we would expect from a non-virtual version of the object. In video games, the term “invisible walls” describes when the “representation gives no clue that the fictional world space ends, but for no apparent reason, the game space ends” (Juul 2005: 165), delineating between what we can and cannot interact with. The fiction that we cannot interact with also includes background stories (as discussed by Chalmers).

It is then important to note that even objects and events that we can interact with will have aspects not implemented: we can rarely desert from a war, change the oil on a car, or disassemble a desktop calculator. Even in the middle case, when the fiction is implemented in rules, no object is ever “fully” implemented—rather game and VR development consist of creative choices about what aspects of an object to implement and what aspects of an object to omit. The level of abstraction (Juul 2007)is the line below which game or VR objects omit the properties or affordances that we would expect for the “full” object. (I will return to what would constitute “full” implementation.) VR has fictional aspects all the way down.

Continuity or discontinuity?

Is VR therefore fiction? There is a game to play, where we all choose our favorite definition of fiction, and thereby prove that VR is, or isn’t “fiction”. Let’s leave that aside.
The more important question is whether there is continuity or discontinuity between the three classes of objects/events shown in Figure 1. In effect, Chalmers argues for discontinuity between the middle and the rightmost classes: the fact that we can interact, in specific prototypical ways, with VR objects, moves them from fiction into the realm of the real. But is that the whole story? Could they not be both?

Consider Chalmers’ example of a calculator in virtual reality. Chalmers is of course right that a virtual calculator may perform the kinds of calculation that a non-virtual calculator may perform. However, this may be a too-narrow view of desktop calculators.
Consider what it would take for a virtual calculator to be a fullblown virtual counterpart: there are many virtual desktop calculators that implement calculation, but no virtual desktop calculator has been made that allows use in all the other ways that we can use desktop calculators: as paperweights (common), bottle openers, as utensils for spreading marmalade on toast, as something we can smash, burn, sell, paint, disassemble, change battery on, spill orange juice on, short-circuit.

If this sounds silly, it’s just because we take it for granted that a virtual object has a limited implementation—including only prototypical affordances but ignoring most other properties: a VR calculator calculates, but not much else; a VR axe chops wood, but cannot be used as a hammer; a VR car can drive, but usually can’t be repaired or run out of gas. We understand that a virtual object has been designed for particular limited set of interactions, and the aspects of an object that are irrelevant to these interactions will usually be left out; they will be purely fictional.

The impossibility of full-blown virtual counterparts


Figure 1: Unity3d VR shooting gallery tutorial (Unity n.d.)


public event Action OnClick; // Called when click input is detected whilst
the gaze is over this object.


public bool IsOver

get { return m_IsOver; } // Is the gaze currently over this object?

}

Table 1: The implementation of shooting in the Unity3d tutorial. Shooting and hitting are discrete events, unlike how actual lasers work. (Unity n.d.)


But what would a full-blown virtual counterpart with the complexity and functions of an X be like? Such a virtual calculator would not only functionally be useable for everything a non-virtual calculator can be used for; in complexity it would not just simulate calculation, but also the electric circuits of a non-virtual calculator to a subatomic level.

As a measure of the complexity of actually existing VR, consider the activity of shooting objects with lasers in the Unity3d VR tutorial (Figure 1). In the tutorial, shooting is implemented as two discrete and binary states—the shooting button is either clicked or not clicked, and the reticule controlled by the headset’s gaze either hits a target or it does not (Table 1), which is determined by raycasting, mathematically extending the line from the camera viewpoint through the crosshair into the world and calculating which objects are hit by this ray.

A full-blown virtual X in this case depends on what we think of as the non-virtual case, but let’s for simplicity’s sake assume that this is a laser shooting range. This full-blown virtual X would simulate the laser as a device that emitted coherent photons, and simulate the path of each photon at the speed of light towards potentially hitting the target. The virtual laser could thus be in any number of states of only partially touching a target, and the photons of the laser beam would be subject to reflection, scattering, gravity, and dust particles. This is exponentially more complex than the code shown above and requires computational powers far beyond current technology2.

There are three lessons from this:

  1. No Virtual X’s are full-blown virtual counterparts with the complexity and functions of an X. As we see in the Unity example, VR is usually implemented as simple discrete states, and objects are never simulated on a particle or quantum physical level—Virtual X’s never have the all complexity and functions of their non-virtual counterparts.
  2. This need not be a problem, because we understand that VR experiences are designed with specific interactions in mind. A virtual X may well have the functions and complexity of the general prototypical idea of a specific X—for a calculator or a laser, but not the functions or complexity of the actual physical counterpart. The VR shooting gallery lets us shoot things with lasers in a general abstracted sense, and the lack of simulation of the photonic laser beam is perfectly acceptable for this purpose, as we as players, or programmers, may mentally conceive of shooting with lasers as two simple binary states corresponding to the code above. However, this simplicity would be completely unacceptable for a VR simulation meant to teach us about relativity, or about lasers on quantum level.
  3. The sophisticated user understands the limitations of a virtual X. Adding to Chalmers’ discussion of “naïve” and “sophisticated” users, the more consistent and demonstrable journey of users is towards understanding the limitations of the virtual objects that they interact with. Naïve users, primed by the hype of virtual reality, may approach their first VR experience in the belief that virtual Xs are full-blown counterparts to non-virtual Xs, but will quickly be deprived of that belief as virtual Xs turn out to be greatly simplified.

We can compare this to the way fictional worlds are by necessity incomplete—Thomas Pavel notes that incompleteness is often seen as a distinctive feature of fictional worlds (Pavel 1989: 107), and though we know that Lady Macbeth has children, we do not know how many (Pavel 1989: 75). The game above is similarly incomplete. It has a laser beam, but it lacks the photons that comprise an actual laser beam, and even if it did include photons, these would lack the interactions that photons have with air particles, and even if they did interact with air particles, they would lack relativistic and quantum effects, and even if they did include those effects, and so on. Every virtual object will have fictional and unimplemented aspects compared to its non-virtual counterpart, but even the implemented aspects will in turn be simplified and contain purely fictional and unimplemented aspects, all the way down. If this is not a problem most of the time, it’s because we understand VR and game experiences to be intentionally designed for specific activities3. If we are playing an action game, we don’t expect to be able to pick flowers. In a more poetic game, we don’t expect to be able to break things.

Half-real

There are thus arguments for seeing the relation between purely fictional objects and objects that we can interact with as one of continuity. This does not negate Chalmers’ argument about VR objects being real, it is just that they are real in a different special sense, where we as users understand that they are specific, stylized versions of their non-virtual counterparts. But where Chalmers argument for the reality of VR objects was based on their (relative) fidelity, I have here shown that they are always marked by simplification.

Virtual reality, and games, may be seen as perfectly real to the extent that our actions can influence the underlying game state (rules/digital objects), but also like much art a different kind of real by virtue of corresponding more cleanly to human concepts and by being more easily understood. David Bordwell uses the term expressive amplification (Bordwell 2000: 232) to describe the way film is structured and edited in order to be more easily readable. And so it is with virtual reality.

As users we maintain a double consciousness where we think of an object in virtual reality as a full fictional entity, as well as something we can interact with in the specific way determined by the rules of the virtual world. This is why I describe game and VR objects as half-real (Juul 2005). Though the exact choice of terms is not paramount, it’s important to note the continuity between a virtual object that exists only visually, and a virtual object also subject to causal influence such as user interaction. Even the object with which we can interact will still be simplified and have non-implemented fictional aspects.

Again, I do not use the term fiction as a negative label; I am referring to the theory of fictional worlds (Pavel 1989) to describe worlds we see as distinct from the regular one.

We can also see the duality in the way such software is developed: A game or virtual space is rarely if ever designed as elementary particles interacting, but rather, as in the shooting gallery example, programmed as objects in human categories (cars, dragons, people, targets, lasers), with events in human categories (driving, flying, talking, shooting).

A compelling argument says that future technology will eventually enable objects that are full-blown virtual counterparts, but this is neither desirable nor feasible. If we consider the complexity of simulating anything down to the subatomic level (where the science is uncertain anyway), then the full-blown virtual counterpart to even the simplest microscopic object is centuries into the future. “But we do not care about that level of detail, only about higher-lever properties and events”, the objection may be. Yes, that is precisely the point. We neither perceive virtual objects as full-blown counterparts, nor are we hoping that they would be.

If we sometimes discuss virtual objects as if they were full-blown virtual counterparts, it may be because new technology is always promoted on the new things it enables: “this is 3d”, “this is stereoscopic”, “there are proper shadows”, “I can move my head”, “my controller recognizes the position of my thumb”, and so on. The history of video games, from Pong to text adventures to the present day, contains a series of such developments, and video games are often promoted on promises of infinite possibilities, but a game is still designed such that only specific fictional objects, and specific aspects of the fictional objects are implemented in the game rules/programming. This makes games playable; it allows players to make inferences about strategies, to repeat actions that are discrete, and to discuss the games with other players. This is also the case for virtual reality.

My points here do not invalidate Chalmers’ excellent paper. I just hope to have pointed out that virtual objects are quite different from what Chalmers supposes. I agree with Chalmers that any reality of virtual objects derives from the fact that users can influence the virtual worlds, effecting change on the underlying digital state of the world. I also agree that there are classes of object and events whose prototypical properties can easily be implemented in VR—calculators are one such case. But in the general case, the value of virtual reality comes not from the kind of similarity to non-virtual reality that Chalmers discussed, but from its selective difference. And that is the whole point of virtual reality, and games.

Towards the end of his paper, Chalmers makes the following case:
"Non-virtual reality and virtual reality are just two different implementations of closely related structures. There may be some differences, but these differences are not enough to make one real and valuable while the other is not." (Chalmers 2017: 9)

I have argued that the conclusion can be broadly similar, but for inverse reasons:
Virtual reality is a heavily selective and simplified implementation of nonvirtual reality as seen through human categories, and for the purpose of the activities of the given experience. This simplification can be framed as both a problem (lack of fidelity) and a value (added readability, art), but is not by itself sufficient to decide whether VR is valuable or not.4

 

References

1. This example was suggested to me by Frank Lantz.

2. According to a simple calculation, a weak 10mw laser emits 3.15 * 10^16 photons per second, while a generous evaluation of 2018 computers puts them at 10^13 floating point calculations per second, a gap of many orders of magnitude as even the most, yes, simplified simulation would require untold numbers of calculations per photon.

3. Theists would argue that the regular worlds is also designed with specific activities in mind, of course.

4. Thanks to Frank Lantz, Paweł Grabarczyk, the seminar participants, and the anonymous reviewer for discussion and suggestions.