“The Game of Video Game Objects: A Minimal Theory of When We See Pixels as Objects Rather than Pictures.” In Extended Abstracts of the 2021 Annual Symposium on Computer-Human Interaction in Play. CHI PLAY ’21. https://www.jesperjuul.net/text/gameofobjects/ 

We’ve discussed immersion (for and against), but I argue that we’ve overlooked a much more fundamental question: Why and when do we think of pixels on the screen as objects, rather than as pictures of objects?

During the pandemic, I built a game for exploring this question and wrote an accompanying essay. This extends my previous paper Virtual Reality: Fictional all the Way Down (and that’s OK).

The game presents a series of game objects, and asks the player to consider their status:

1. When would you describe something as an object, rather than a picture?
2. When do you think of it as the type of object it represents – like a rock or a lamp?
3. And when would you argue that an onscreen object is the type of object it claims to be, such as a calculator?

The conclusion is not just that games and VR are cultural forms (obviously), but that we judge game objects based on what we are trying to use them for, and game worlds are always designed for particular kinds of uses. And we know this. There thus can be no universal metaverse, only different worlds built for different purposes.

From the abstract:

“While looking to the future, we have overlooked what is right before us. With new technology, haptics, rendering, virtual reality, we have spent much energy discussing immersion and presence, thinking sometimes about current technology, but often about a hypothetical perfect experience or future perfect technology.

In this, we have forgotten something rather fundamental: How do we in the first place decide to see a group of pixels on a screen as an object to which we have access, rather than as a picture of an object? This paper explores this question through a playable essay. At first, we may think that we will identify anything interactive as an object, but the playable essay demonstrates that this is much more complex and pragmatic, and that this identification has three steps – identifying pixels as an object rather than a picture, reasoning about the object as a specific type of object (such as a ball), and identifying it as a real instance of a type of object (such as a calculator).”

Thanks to IVD at The Royal Danish Academy, Nick Montfort, Stefano Gualeni, Pawel Grabarczyk, Dooley Murphy, and Jan-Noël Thon for comments; to Andrés Cabrero Rodríguez-Estecha for visual design; Stephane Bersot for the calculator asset. The project was made with Unity3D and Low Poly Game Kit by JayAnAm.

I was reading Raph Koster’s notes on the launch of Ultima Online back in 1997, and it made me realize that the online world that I programmed also launched just over 15 years ago, nearly at the same time as UO. If you didn’t grow up in Denmark, you have certainly not heard about it, but it’s called Højhuset (literally high-rise, from the metaphor that it was a series of stacked rooms). It’s still running at www.n.dk (only go there if you speak Danish).

This is what it looks like: It’s made up of non-scrolling rooms in a diagonal grid. Users can dress up, chat, and so on as expected. Users have their own apartments which they can decorate. Here is a screen shot with a celebrity visiting:

And most importantly: You can have nice things. The world was always a bit of a compromise between a chat system and game-like elements such as inventories and a currency, but it turned out that this was quite a feature. There have historically been long-running feuds between users who think of it as a chat system, and those who think of it as a game with the goal of amassing the most items. I initially thought that this would be a problem, but in practice this created social cohesion in each group – this was a valuable lesson for a game designer, that an external enemy does give users reason to come back.

As someone who is into game definitions, the height of the “is-it-a-game-or-not” feud was when a user had found a “Player” class in my program, and used this as proof that yes, this was a game. (New game definition: A game is a piece of software that declares itself to be a game.)

My role in this was always as a subcontractor, but I have been providing support and updates for 15 years now. One of the things I did learn as a programmer was to document my code and avoid any quick & dirty fixes which could come back to bite me. The main program (in Java) has always run on a single server. At the height of popularity, there were 2000 simultaneous users, but the improving speed of servers always just always made it unnecessary to spread across multiple machines.

Of course, there were also numerous attempts at hacking the system, which always is a point of pride for a programmer. People still try, here is even someone posting some debug output from such an attempt on Pastebin.

There were also microtransactions going back to the late 1990’s (this mostly paid via text messages).

Having read & written so much about video games since, it is hard to remember what thoughts went into my head when I was first starting out on this project, but I had played MUDs at the time, and I am sure I had read an article about the need for artificial scarcity in virtual worlds. And the strength of scarcity was one of the things that made the biggest impression on me. In the very early versions, there was no automatic dropping of items – this had to be done manually by a superuser referred to as the “superintendent” (“vice”). When going online, that user would always be met with cries encouraging the dropping of items (“smid!” in Danish). I leave you with a bit of user art, in which the superintendent gets fed up with being asked to drop items.

(There was actually a brief period of time in which a new chat system was introduced on the site to replace the one I made, but users demanded the old one back. Warms your heart.)

Gary Foshee, a collector and designer of puzzles from Issaquah near Seattle walked to the lectern to present his talk. It consisted of the following three sentences: “I have two children. One is a boy born on a Tuesday. What is the probability I have two boys?”

“The first thing you think is ‘What has Tuesday got to do with it?'” said Foshee, deadpan. “Well, it has everything to do with it.” And then he stepped down from the stage.

This is the answer: 13/27.

Many people will intuitively say that the answer is 1/2 (=the chance of having a boy or a girl), but probability aficionados will give the answer 1/3, since this is the Boy or Girl Paradox: We are not told that the speaker has a child and is waiting for another, but that he already has two children. Two children can come in four configurations: 1) boy/girl, 2) girl/boy, 3) girl/girl, 4) boy/boy. Since he has one boy, we are looking at the options 1, 2, or 4. Only the boy/boy combination includes two boys, so the probability is 1/3. In other words, order matters and completely changes probability.

So what has being born on a Tuesday got to do with it? Why would the answer not still be 1/3? The New Scientist has a good explanation toward the bottom of the article. Simply count the different combinations of genders and weekdays, which gives the result (number of combinations with two boys, at least one of which was born on a Tuesday) / (number of combinations with at least one boy born on a Tuesday). The result really is 13/27.

This is the best illustration I have found: This shows all the boy/girl pairs as well as the possible weekdays on which they could be born. Green represents situations with two boys, at least one of which was born on a Tuesday. Yellow represents at least one boy born on a Tuesday. Red is neither. Hence the answer is green/(green+yellow)= 13/(13+14)  = 13/27.

But again, what has Tuesday got to do with it?

More below.

Discussion

It is an appealing puzzle because even once you (or at least I) accept the 13/27 calculation and visualize it, it still is wildly counter-intuitive that being born on a Tuesday influences the probability of having a brother. (Actually it doesn’t – see below.) What’s also strange is that with a rarer trait (say, being born on February 29), the probability of having two boys approaches 1/2. (For a trait with probability 1/A, the probability of having two boys is 2A-1/4A-1.)

Other people with more skill in probability theory have explained it well (search for “cocktail party” here), but I thought about it moving backwards from the result: Our intuition (which is correct) is that boys born on Tuesdays do not have a larger chance of having a brother than boys born on other days. Therefore the 13/27 result seems absurd because we could repeat it with all days of the week, add up the results, and reach the conclusion that the answer to the boy or girl paradox was 13/27 and not 1/3. And that is the hint: We are not really calculating what we think we are calculating.

It is really a question of formulation. The puzzle is not “My firstborn is a boy born on Tuesday” or “I have exactly one boy born on a Tuesday”, but “One [or more] is a boy born on a Tuesday [but I am not telling you which one].” Which means that we did not calculate whether being born on a Tuesday influences the probability of having a brother.

As I stated above, the absurdity comes from the fact that we might imagine that we had sliced the possibility space (2 genders, 7 weekdays) into 7 equally big slices, but did we? Try repeating the calculation for “boy born on a Tuesday”, “boy born on a Wednesday” etc…

Now consider the case of the oldest child being a boy born on a Tuesday, and the youngest child being a boy born on a Wednesday. This will count toward both the probability of “two boys when at least one child is a boy born on a Tuesday” and “two boys when at least one child is a boy born on a Wednesday”. Many cases are counted for several weekdays. This is why the probability goes up from the expected 1/3, approaching 1/2.

Why this is such a good puzzle:

• It is a puzzle within a puzzle.
• Like the Boy or Girl Paradox, which encourages us to answer from the knowledge that the boy/girl probability is (roughly) 50%, this puzzle encourages us to answer from the knowledge that being born on a given weekday generally doesn’t influence other probabilities. In both cases, that is not what the puzzle is actually asking us. We are tricked into using a stock answer that is not appropriate for the given puzzle.
• It appeals to puzzle and probability theory fans by building on an existing puzzle.
• It gives a result (approaching 1/2) which is exactly the result that connoisseurs know should not be the answer.
• It uses a counting technique specifically designed to prevent from counting the same case twice, and we may be thinking that “at least I am not counting anything twice”, but that is exactly what we are doing.
• The very terse explanation of the puzzle is deliberately misleading. When Foshee says ‘One is a boy born on a Tuesday. … What has Tuesday got to do with it? Well, it has everything to do with it.“, it makes it sound like we are doing a probability calculation depending on one particular boy, when in fact we are doing a calculation depending on “one or more children being a boy born on a Tuesday” – hence we end up counting some combinations for several weekdays.

Here is Alec Meer’s post at Rock, Paper, Shotgun, where the game was presented under the heading of “PC Gaming In 2010”. A long flame war ensued on the site, showing that some people clearly felt cheated by the game. Which they were, of course, but that sort of is the game. One commenter compares 4:32 to rickrolling and is taking the whole thing personally:

… I bet there will be many other people disappointed or outright enraged by this. This has got nothing to do with “playground attitude” as it deserves to be called out for what it is. There are good and there a bad jokes, but this one is completely off the mark.

Part of the strong reactions at Rock, Paper, Shotgun seemed to come from people who felt that the game had crossed some line by forcing them to install and uninstall plugins.

Over at Play This Thing, Greg Costikyan’s post about the game immediately led to a flame war about PC vs. console gaming.

Petri Purho also blogged about it here.

I am pretty happy with the response, but it’s also interesting to see how players interpret the game in their own way. I didn’t think it was about PC vs. console gaming, and I didn’t think that a conceptual piece like this could get people that agitated. But then again, it’s a long-standing observation that some audiences will get angry if they feel a specific “work of art” breaks the conventions they expect – and the audience may even take it as a personal insult. That wasn’t intended, but it seems I did succeed in making something that played with player expectations!

Statistics: 13.000 games played. 513 games completed. (Cheating may have been involved, but I think that’s part of the game.)

The 2010 Global Game Jam took place this weekend at over 100 locations around the world.

I made a small conceptual game called 4:32.

The theme of the Global Game Jam 2010 was deception, and the constraint for the time zone of the NYU Game Jam was “Rain, Spain or Plain”. This game furthermore fulfills the achievement “instant gratification” by being playable in the browser.

Without giving too much away, 4:32 is a response to Petri Purho’s game 4:33, which in itself is a response to John Cage’s silent composition 4:33.

4:32 won the vote for “most innovative game” at the NYU game jam.

Play it here.

Update February 21, 2010: I have changed the game a little to prevent some of the more obvious ways of cheating. (Linking to the final page, for example.)

Pierre was created at my previous workplace, the Singapore-MIT GAMBIT Game Lab.

I functioned as a product owner on this game, which at GAMBIT means that I posed a research question that the game should answer.

Research question: How do different ways of communicating failure influence the player’s experience and performance?

The excellent team was then free make a game (more or less) within the constraints of that research question. You can play the game here.

For more about the creation of the game, have a look at the GAMBIT updates this week. Introduction video here.

As usual when I am posting games for research purposes, I will keep the further details close to the vest, and then post results later.

In the meantime, please play the game! Thanks!

Direct download link: http://www.jesperjuul.net/download/thebeat_1.2_setup.exe

This was a superbly productive weekend – I am happy about my team’s game, The Beat, which is really a two-player puzzle-rhythm game.

(The current download is little rough – I will post a new version with an installer in a few days.)

All the games can be tried from the Global Game Jam site. I am looking forward to seeing what teams came up with in the other locations around the world.

Thanks, and thanks to all the organizers!

Here’s a photo of the final playing of the games:

I do think it would be much more powerful if it started by giving players a more personal relation to the animals that they prepare and then made players kill them. (Meet the sweet little turkey … now kill it.)

I speculate that PETA faced the problem that they wanted to convince players that animals are treated cruelly – but on the other hand they did not want to make a game in which players could do just that. A shame really.