In 2019, after our Facebook Reality Labs project was over at Artefact, I had some deep thoughts on this new Metaverse / Mirrorworld that people were expecting to rule the future. We saw some dark outcomes and we were thinking about how we might point us toward a preferable outcome for AR. While we were on the Magic Leap project in 2016 John Rousseau wrote a great thought piece called The Laws of Mixed Reality — without the rose-colored glasses I wrote a follow-on piece called The Mirrorworld Bill of Rights based on how the metaverse was shaping up at the time. You can read the final version on the Artefact site, but I thought I'd put the full version here. Note that the references to "Meta AR" is not referencing the current company called Meta.
The Mirrorworld Bill of Rights:
1. One Reality for All
2. Public Privacy
3. Digital Property Rights
4. Freedom to Assemble
The Metaverse, the AR Cloud, The Magicverse, Mirrorworld—many names for the same concept. A digital layer we see over the real world. Many have called this the new digital frontier—the next step in computing. The hype might be overblown at the moment, but I do believe that ten years from now this digital layer or AR Cloud will be a key piece of the computing landscape. But with all the value that the AR Cloud could create comes the possibility for abuse and unforeseen negative consequences. It will need to be created responsibly to avoid locking in surveillance capitalism and further polarizing our tribalistic world. This article will explore four “rights” to help the AR Cloud avoid these pitfalls. These are simply my musings and just the start of the conversation. If the AR Cloud ever becomes a reality the community will need to work together to establish the actual rights of Mirrorworld. But that won’t stop me from trying now.
One Reality for All
Many have predicted that AR glasses will be the primary computing platform in the coming decades. In this future, augmented reality will cease be to seen as augmented, and will simply become reality. When this happens human consciousness will become a mediated consciousness––controlled in part by the technology we create and mount to our faces. What could go wrong? Our screen-based digital services have already created filter bubbles that have led to Measle outbreaks, breakdown of public discourse, and erosion of truth. What happens when we live inside these filter bubbles? Our ability to have freedom of thought could be limited.
To make matters worse, AR systems are inherently private. Two people standing next to each other cannot see each other’s content or activity. This uncertainty leads to social anxiety and the glasshole effect. Meta AR wrote a wonderful UX Guideline on the subject entitled, Public by Default. An excerpt:
We’ve evolved to continually scrutinize the posture, movements, and actions of those around us in order to determine their intent, developing our theory of other people’s minds. Within augmented reality, if a user’s content is made invisible to others, perhaps due to a difference in permissions, their actions will appear to be performed on empty space, making them inherently confusing and confounding expectations learned in the physical world.
The mystery of invisible content can also raise other forms of social anxiety. For instance, a user might wonder if their colleague is secretly engaged in something hostile or invasive, or if they’re even paying attention to the task at hand in the first place.
Furthermore, the companies creating these AR products have a track record of creating walled gardens that don’t allow sharing between platforms. An iOS device can call an Android device, but that’s about where cross-compatibility stops between the competing operating systems. Now imagine a world where your friend has Apple Specs, you have Google Glasses, and the lady on the bus has Magic Leap Shades. Unless a bridge between these devices is engineered, it would be impossible to have a shared experience. You’d be stuck in your Google bubble, unable to communicate with those fancy Apple people or those crazy Magic Leapers. The world will be further divided. Today people hide behind cell phones, but if you’re sitting next to someone you can at least show them a picture of your dog on your phone. With AR, that’s not the case. There would be absolutely no way to share that cute dog pic.
How can we make it so you can share that doc picture with a random person you sit next to if you decide to share it with them? We would need a standard method for shared experiences between devices––a single shared framework that any AR headset can leverage. And if sharing is harder than holding up a phone to someone next to you, we will be training people not to share. We need one reality for all. A world where sharing between competing AR glasses is the default experience. You could imagine it as a single massive multiplayer online game in the real world where everyone is playing together. Who will lead the charge in creating this open framework? Magic Leap? The W3C Immersive Web Group? Microsoft and their new open philosophy? No one is there yet, but at least the conversations are happening and companies like 6D.ai are starting to create the technical layers.
One reality for all doesn’t mean that two people next to each other would always see the same things. I love what Neal Stephenson proposed in his description of the MagicVerse. Layers that can be turned on and off depending on the task and mode of the user. Perhaps there will be a public layer, a transportation layer, a learning layer, an Uber layer, a social layer, and a private layer.
Image credit Magic Leap
It will be important that people can exist inside layers together and share the same experiences. What might this look like? Imagine you’re in the park and you see someone walking by. You can see they are in the Pokemon layer. You don’t need to see exactly what they are doing, but their activity status might reduce your social anxiety toward them and become an affordance for joining. You choose to join them and their layer expands to surround you as well. Now you can see what Pokemon they are battling and you can choose to challenge them when they finish.
Created with Torch 3D on iOS using free Sketchfab assets
Public Privacy
So the first step is allowing content and experiences to be shared between all devices, but we should take it further if we want our technology to connect us rather than divide us. We should make everything public by default in AR in a single shared reality. The alternative is human consciousness diverging further into tribal realities. You might be thinking, “I don’t want everything I do to be seen by everyone around me. What about my right to privacy?” Public by default doesn’t mean that your actions in AR are always visible. You should be able to be private. But your privacy should be public. What does that mean? Meta AR has/had three great rules that I think solve this problem nicely:
1. Ensure content can be viewed by anyone else wearing a headset by default
All users should be exposed to the same holographic surroundings, just as we all see the same physical surroundings in the real world. Actions performed on “invisible” content (or within empty contours) are inherently disorienting to those nearby, …
All users should be exposed to the same holographic surroundings, just as we all see the same physical surroundings in the real world. Actions performed on “invisible” content (or within empty contours) are inherently disorienting to those nearby, …
2. When privacy is necessary, use a blocking object, like curtains or dividers
Do not simply make sensitive content invisible from the perspective of unauthorized users, as this creates a disparity between users that can lead to confusion and distrust. Instead, just as in real life, establish privacy with holographic curtains or dividers. …
Do not simply make sensitive content invisible from the perspective of unauthorized users, as this creates a disparity between users that can lead to confusion and distrust. Instead, just as in real life, establish privacy with holographic curtains or dividers. …
3. Privacy is a function of etiquette, not a feature of the interface
...For example, a user who wishes to view financially sensitive information should do so within an interface that features privacy measures, such as a curtain, divider or a display that can be tilted away from unwanted viewers. Users should design their own privacy within the rules of the shared space, not by breaking them.
...For example, a user who wishes to view financially sensitive information should do so within an interface that features privacy measures, such as a curtain, divider or a display that can be tilted away from unwanted viewers. Users should design their own privacy within the rules of the shared space, not by breaking them.
We can have privacy, but the affordance of privacy needs to be public. Perhaps a virtual door and walls could signal that a private work session is in process. This door could create the social affordance for knocking and joining the private session. Using Torch’s iOS AR prototyping tool I mocked up this scenario in about 5 minutes to get a feel for privacy boundaries. If we completely hide private content we increase social anxiety between users and remove the affordances for bridging between the users. These privacy veils and dividers might be able to utilize existing social norms around privacy, enabling people to naturally share and hide their private work and content. I’ll try a couple of tests to get a feel for it...
After trying this experience it felt a bit weird that there was no presence or awareness of the person before I knocked on the door. It would be better if the glass on the door was transparent so I could see the person, but still obscure private content.
What happens if these privacy dividers are overused? What if a group decided to use them to spitefully exclude others? If social groups are now shown visibly around the physical groups, does this lead to more social anxiety? My guess is that the barriers would be used sparingly and social norms would arise to dissuade spiteful exclusion. Here’s a silly example of a few teenagers excluding everyone to look at the latest memes.
This experience felt about the same as if these three had gone to a room and closed their door. If another friend had been excluded it would be obvious and everyone could have had a conversation about it. Me being the parent the social norm here at my house would be for me to knock or just walk in. If these were my kids and their friends I would probably set some sort of “open door” policy in the house. Meaning that perhaps privacy dividers don’t work when on the property for underage users. Or maybe they always open up to the parents. Finding a balance between trust, control, and protection is the challenge.
Digital Property Rights
In order to get the AR Cloud to work we’re going to need to maintain a highly detailed 3D model of the world and everything in it. This will be created by the cameras we will be wearing that capture the world and figure out where we are looking at any given moment. This plus centralized control of the network has all the ingredients for the ultimate surveillance state. How might we protect our privacy? What if we applied the 4th amendment to the AR Cloud to prevent governments and corporations from violating our privacy?
The right of the people to be secure in their persons, houses, papers, and effects against unreasonable searches and seizures shall not be violated, and no warrants shall issue but upon probable cause, supported by oath or affirmation, and particularly describing the place to be searched and the persons or things to be seized.
This would mean that no government, individual, or corporation would have the right to search the interior 3D model of your dwelling, workplace, audio, user location, and property without a warrant. Could we design the storage of the 3D world in a way that would meet this standard, while still allowing for SLAM to work between all users in all environments? What if we want to search and access volumetric video recordings in our own homes, but not give Google, Facebook, Amazon, or Microsoft access?
I’m not a developer, but it smells like the 5G edge network could be part of the answer. What if the 3D world model was decentralized and distributed out to the edge of the network? You could limit access to the portions of the 3D map to those who are close by. This gives only those who already have physical access, digital access. One step in the right direction away from that surveillance state. But the map would still be owned and managed by the 5G service provider. That might be fine for public spaces, but what about our private spaces?
What if each home and business hosted its own edge server that stored the SLAM map of its interiors? Then we could really own our data and trust the cameras on our glasses. What about visitors to your home? How might we make it easy for them to use your SLAM map to help them watch that show with you on their glasses? What if a rule could be set up so that if the location services of the 5G network see someone inside your property line, then they could get temporary access to your local SLAM map? Or in addition, maybe they need to one-time authenticate. This would give families, businesses, and individuals control over who they share their homes with. This personal data vault could also store 3D digital memories recorded in our private spaces. Baby’s first steps, puppy’s every movement, replay that winning ping pong point.
But how would we match up owners of physical property with their digital content? This sounds like a job for Blockchain. A public ledger of who owns or is leasing what property seems like a good fit for Blockchain. This wouldn’t need to be updated very frequently and would act as a lookup table for anyone building on the AR Cloud. Organizations like Bitland are already making headway. Once a record of private and public lands is available it makes writing AR applications or layers much more interesting.
For instance, maybe I want to build an art layer that allows anyone to post 3D sculptures in public spaces. By default the layer could be used in parks, public spaces, and public airspace, but not inside buildings, on roads, or in any private spaces. I as the developer publish this layer to the AR Cloud and now anyone can view and access this layer when they are in one of those spaces. Perhaps a company wants to open its private atrium to this art layer and have a contest for their employees. They could choose to share that space with the layer and anyone in that space could now add art.
Freedom to Assemble
So digital property rights might work for private property. But what about public spaces? Parks, streets, airspace? Who owns and maintains those? Today cities, counties, and countries manage these spaces through zoning laws, etc. Should they maintain the AR Cloud layers in these spaces in the future as well? Probably yes, but not for a long while. Cities have many other things to manage these days. Until cities are ready to manage their AR Cloud layers we will need to rely on good defaults to set the norm. Here are a few thoughts.
Pulling from the first amendment we could give “rights of the people peaceably to assemble”. What if we allowed anyone to virtually assemble in public spaces? This would enable anyone to virtually travel around the world. But unlike Google Earth, you might be able to see people walking around the streets of the cities in real-time. And if you landed you could go have a conversation with them. Inversely if you were on the city street you could look up and see people flying over your city. Here is a very rudimentary video showing the concept using Torch AR and Google Earth VR.
Remote users don’t have to be their normal size as shown in this travel concept where they are dragon sized. Or the remote players could be miniature. Imagine that a new Fortnite AR or Battle Royal game allows for remote players to come into a public space and play at a miniature scale. Remote players would play in a giant world. Players who are physically present at the time would look like giants. Perhaps the giant’s goal is to smash as many remote players as possible while the teams battle it out.
This excites me because now public spaces become more valuable. They bring more people together in new ways. If you can’t physically travel to a place, you can still enjoy it in a social environment. I could go on a walk in the park with my wheelchair-bound father. I could play Fortnight in Central Park with my remote friends and family. I could dogfight in and around the Eiffel Tower.
We might be able to open this up to private spaces as well. Perhaps my kids and I could build out a physical Fortnight level in the backyard using wood, dirt, and bricks. Once it’s constructed and scanned we could share out that particular part of the yard publicly and invite players from around the world to play.
Moving Forward
We’re at the start of this journey and the rules are still being set. Let’s try to create a world that bridges the gaps between people, rather than simply strengthening the bonds we already have. Let’s create a world that promotes the ability to think for ourselves. A world where individuals have control and ownership over the reality they live in. A world where the public commons are safe places where anyone can travel to learn, discuss, and play together.
While I do believe that these “rights” are on the correct track, they are simply guesses and thought experiments. I don’t pretend to have it all figured out and would love to discuss it further on Twitter @paulhoover or in the comments below.