The Cognitive Impact of Mixed Reality

While it may seem like the stuff of science fiction, we’re already in the foundational world of mixed reality. Consider any of the times you’ve used a mapping service—Apple Maps, Waze, Google Maps, toggled on Street View; played a location-driven augmented reality game—Pokemon Go, Harry Potter: Wizards Unite; or used a voice UI to answer a question or complete a task—Apple Siri, Google Home, Amazon Echo, Microsoft Cortana. These instances blend our daily experiences with technology-driven enablers, allowing us to complete tasks, connect with other individuals, or be entertained in ways that expand our perception and abilities within our everyday environments. We are experiencing a future where digital, physical, and the space in-between are beginning to merge. Our understanding of what is real, what is expected, and what can help or harm us is becoming muddled in our everyday experiences. The reach of these mixed reality environments has the potential to impact us all deeply on a cognitive level in order to both foster rich and meaningful experiences and connect us with others.

The most commonly referenced mainstream mixed reality interfaces have been centered around elementary task management, gaming, and entertainment; all exciting, playful, and constructive domains for us to build toward. While these are valuable and engaging experiences, what inspires me most about mixed reality experiences is how they open the door for us to take stock of the way that we learn and educate ourselves and how we adapt our bodies and brains for the future generations to come. There’s a real opportunity to create more immersive, inclusive, and all accessible environments where individuals can better digest, learn, comprehend, complete, recall, and participate in experiences.

Before I break down the ways in which we can engage this platform, let’s get to a shared understanding of the capabilities we have to work within the mixed reality environment:

—Augmented reality (AR) enhances a real-world experience with an additional layer of (typically digital) context, effectively changing the way in which an individual perceives or interacts with this experience. For example, Pokemon Go adds characters, gaming, and artificial overlays to an otherwise familiar map interface.

—Virtual reality (VR) steeps individuals deeper into a realm of lucid immersion, where we as the participant experience an entirely artificial environment, one which we know is not real, yet our brains are fooled into believing it is, like gaming environments using Oculus, PlayStation VR, or Valve Index VR Kit.

—Mixed reality (MR) can be considered as an umbrella term encompassing the capabilities of both AR and VR, but also as an environment leveraging auxiliary methods of interaction such as haptics (the physical and sensory feedback we get through touch or vibration), and auditory experiences, like sound design and the feedback we get through hearing or spoken environments (Voice UI) that don’t require the additional content of a visual display.

Within these environments, we have the ability to design for three major use cases that could have a large impact on education, the workplace, and aging in place.

Education

Consider the pressures and distractions that come along with learning environments these days, especially at a young age, where many students also have access to digital devices and technologies on their person. Mixed reality environments have the ability to connect students via artificial environments that allow them to learn through a multifaceted approach by enabling them with the technologies they’re already used to. AstroReality’s EARTH, for instance, integrates digital technologies and overlays to allow students to explore, learn, and interact with science, space, and life on earth.

Classroom environments can also be chaotic to our brain’s understanding of what it is we’re trying to learn. Balancing instructor presentations with textbook readings, in-class demonstrations with classmate interjections—all of the factors can exhaust the brain and lessen the rotation of learned materials. Mixed reality environments have the ability to engage individuals more deeply through immersion, which could help to counter the cognitive burden students face in the chaos. Compiling the disparate form factors into a singular immersive environment, one that allows each student to choose the context and framing of learning they’re interested in, allows the cognitive load to remain manageable and move students towards higher levels of achievement, and dare I say, enjoyment. For true enjoyment, check out frog’s experiment with mixed reality environments at SFMOMA with the Magritte Interactive Gallery.

Workplace

The natural segue pushes us from development into workplace performance, and understanding how cognitive burden can turn into cognitive overload. At its core, cognitive overload arises when you have a situation where you have too much information or too many tasks to learn simultaneously, which results in the learner being unable to process the information. Mixed reality environments are creating experiences where individuals can excel at the task at hand while simultaneously seeing a real reduction in cognitive overload by providing workers with the context they need in the order and at the moment they need that information. A number of mixed reality options already exist commercially, including Microsoft HoloLens, Google Glass Enterprise Edition, and Magic Leap One. This isn’t to say we’re heading into (or have the shoulder strength to bear) a Minority Report-style future where our workplace environments turn into gestural handwaving of floating components. Rather, these mixed reality environments allow workers to stay hands free by having digital devices render context and instructions via glasses in the worker’s immediate visual purview. This removes the cognitive burden of disengaging from tasks to find relevant information on how to complete the next task. It merges the focus of the digital and physical environments together. What’s important when considering how these devices enable performance is that cognitive burden and cognitive overload don’t arrive because I have to unscrew a bolt and remove a panel. What is burdensome is knowing which bolt, which panel, in what sequence, all while trying to complete a singular task at hand. This is the kind of workflow and burden that can easily be offloaded from individuals using mixed reality.

Additionally, setting up these mixed reality environments can take the cognitive load off of imagining things not yet built, or painting not just a picture but a whole reality where a prototype comes alive. At frog, we use mixed reality in pitches to bring ideas to life. Rather than making costly or time consuming models, we can actually build a prototype in VR that enables a client to understand almost exactly how the product will live and work in reality. We used this approach when designing the Touchtunes digital jukebox. Our VR model allowed investors to quickly understand the updates in a more meaningful way. If a picture is worth a thousand words—a VR simulation could be worth millions.

Aging in Place

Finally, beyond how we learn and excel at our jobs today, mixed reality environments open the door for a new model of thinking about how we age. These environments allow us to consider different methods of rehabilitation, ones that can leverage the breakthroughs in neuroplasticity and sensory function, allowing individuals to recover faster and more effectively. Similarly, these environments also have the capability to improve cognitive function, stimulating memory and response—which could one day significantly counter the effects of Alzheimer’s and Dementia. Bravemind, a clinical approach through USC, is using mixed reality exposure therapy to help assess and treat PTSD in soldiers in a controlled environment. The use of this technology platform is unique not only because it’s interactive, multisensory, and immersive, but it can also be tailored to each individual’s needs and gives the clinicians the flexibility they need to control and document patient responses. Neuro-Insight has found that mixed reality environments drive almost double the visual attention in the brain (as opposed to non-MR tasks) and these experiences store up to 70 percent more into memory.

As designers, technologists, and makers creating the future that we will exist in as a society, we have an obligation towards the intentional execution of mixed reality interfaces. We need to build environments that don’t just integrate new ways for monetization or entertainment but take into consideration the effect these interfaces will have on the brain. How will we improve the experience within learning environments, reduce the collective cognitive burden that we face as we complete tasks throughout our day, and contribute to the longevity of life and memory function?

We humans are resilient, flexible, and capable of being molded. In mixed reality environments, we have the opportunity to improve cognition through immersion and by creating environments that reiterate and reinforce our lived experiences. With the right priority, this platform has the ability to help and not hurt our brains.