Extended Realities Become Reality

When we talk about enabling human superpowers, we mean using Extended Reality technology as an intelligent, robust, and insight-driven aide that gives people the ability to do more, better. XR allows for the creation of layered realities that can both respond to and inform human users, efficiently and effectively increasing their agency within the connected environment.

In our partnerships with healthcare clients, frog has demonstrated the potential of XR systems to expand and improve therapeutic options and outcomes.  One such proof of concept project aimed to develop a new pain management mechanism for individuals undergoing burn care therapy. While opioids are the mainstay of pain management, current pain control protocols are inadequate for many burn patients. Alternative measures such as distraction, however, are known to be effective. With this in mind, the frog team set out to create a low-cost headset and virtual reality game suitable for burn care patients. The flexibility of XR technology allowed for design solutions that accommodate burn patients’ needs and mobility constraints during treatment while also providing an open-source gaming concept that gives other developers the opportunity to create their own experiences for patient use. In addition to expanding the range of marketable healthcare technology, this application of Extended Reality demonstrates the viability of safer, more affordable means of pain management. By immersing patients in a virtual reality experience, we can extend their innate pain management capabilities, allowing them to take advantage of the brain’s ability to divert attention away from sensations like pain and consciously refocus it on other information-processing activities.

Current Landscape

PwC’s “The Effectiveness of Virtual Reality Soft Skills Training in the Enterprise” study focused on inclusive leadership training in one of three methods—classroom, e-learning, and v-learning. The study found that VR learners were 275 percent more confident to act on what they learned in the training, four times faster to train than classroom learners and four times more focused than e-learners. Additionally, these VR learners felt 3.75 times more emotionally connected to the material than classroom learners. 

Future Considerations

Extended Reality experiences must be engaging, usable, and provide continual value. Introducing XR purely for the sake of innovative technology has the potential to create over-orchestrated solutions and new hurdles, hindering capabilities rather than enhancing them. Any XR solution should help deliver on the core promise of the business and enhance the overall user experience.

As technology merges more seamlessly with individual users, data collection and privacy practices will become increasingly important. Collections of human metadata from connected XR devices could be exploited for marketing purposes, while VR hardware enhancements such as eye tracking (for foveated rendering or interpupillary adjustments) might also allow for unwanted data collection. Especially as more data migrates to the cloud, systems that collect and protect user data must be carefully considered.

frog believes that storytelling is key for the success of any new venture or design solution, regardless of whether we’re working with a cultural institution, a new startup or a Fortune 100 conglomerate. If we can’t convince a customer of the value inherent in a new experience, or if the corporate board can’t see the benefits of a new product design, then innovative initiatives may not succeed. Extended Reality enables organizations to tell compelling stories by immersing the audience in narratives set in dynamic virtual environments.

frog worked with TouchTunes to redesign their digital jukebox system, combining its hardware, software and services into a seamless, connected experience that helped establish TouchTunes as a leader in digital entertainment services for venues across North America. In order to communicate the intent and features of the redesign, frog built a VR demonstration of the new product that the client could walk through and interact with. The virtual experience allowed the stakeholder teams to personally engage the new product at scale and told a visceral story about the future environments the product would live in.

The San Francisco Museum of Modern Art, one of the most influential cultural institutions in the US, is a recognized leader in using technology to engage both art lovers and new audiences. For its Spring 2018 special exhibition, Rene Magritte: The Fifth Season, SFMoMA asked frog to help them create an “interpretive gallery” in which museum visitors could experience “the paradoxes at work in reality” that Magritte explored in his late-career work. This immersive space drew the audience into a spatial and participatory narrative by compelling them to solve augmented reality puzzles based on motifs found in Magritte’s paintings, engaging viewers in a completely different way than exhibitions of the past.

Current Landscape

Extended Reality adaptations to traditional storytelling formats like film, video games and journalism have been key to the rise of XR experiences in recent years. For example, the New York Times has embedded augmented reality elements to enhance reader context in stories covering topics such as the Olympics and COVID-19. In the latter, readers can visualize the recommended social distancing range by placing a ring around themselves using an augmented reality overlay. Societal expectations for these types of immersive and convergent storytelling experiences will continue to grow as long as they enhance comprehension and provide value to the end user. 

Future Considerations

Game engines and real-time rendering methods are evolving rapidly. Epic Games’ Unreal Engine–the foundation of the hugely popular game Fortnite–is one of the key technologies being used to build new XR experiences and speed the production of hit shows like The Mandalorian on Disney+. With Unreal Engine 5, Epic Games achieved a major breakthrough in real-time rendering, enabling the visualization of billions of polygons in each frame. Such advances will speed the convergence of film, game design and XR, unleashing immersive, cinema-quality and hyper-realistic XR product storytelling experiences in the future. The physics and lighting built into these game-engine driven XR visualizations will allow consumers to explore products in highly dynamic and customizable ways, enhancing comprehension of fit, feel and scale of everything from clothing to automobiles.

Designing tangible new products for our clients has been core to frog’s legacy since the founding of the company 50 years ago. Close collaboration between cross-disciplinary team members and rapid iteration during the product design process have been key to frog’s success and to our ability to bring new products to market quickly. Extended Reality tools and processes have even further enhanced and improved the way we work. We use XR systems in diverse ways, from testing new design simulations with future customers to iterating industrial design and architecture at scale.

Recently frog designed a state-of-the-art collaborative lab for one of the leading genome research groups in the world. Their lead investigator and founder wanted to “create a signature, intelligent, sentient work environment where people come to do the best work of their life.” frog designed new workflows, collaboration tools, rituals and spaces to support and evoke that ethos, and then built a full-scale VR mock-up of the entire space, complete with functioning prototypes of the workbench, storage systems and lighting. By sharing our design with the client in VR, we used XR technology in the same collaborative spirit that the lab was intended to embody.

In 2019 frog helped an autonomous scooter company design a vehicle that transcends the 3-wheeled vehicle stereotype into a vehicle that people will love to drive. We were asked to evaluate the client’s current autonomous scooter prototype and develop a design that would be more competitive in today’s market. Because building a vehicle appearance model requires a lot of time, money and resources, we instead used VR models to assess our vehicle designs on a daily basis, leveraging the latest technology to review the form, iterate quickly and save time and money for the client. 

Current Landscape

The current landscape of XR product design requires that designers and developers reconcile competing AR and VR hardware platforms, a limited selection of collaboration software, and a low degree of interoperability. Current CAD software is slowly adopting XR systems as another output for display and collaboration, but improvements to XR software and hardware standards are necessary before we are able to collaborate with teams more holistically in a less restrictive workflow. The OpenXR runtime standards are currently contributing to greater hardware and software interoperability, which will help make XR collaboration more easily achievable. This means that present day XR collaboration applications can be easily developed to run across a range of hardware from VR and AR headsets to mobile AR devices. This could have profound implications given our new remote-working norms and varying levels of access to XR hardware for distributed employees. 

Future Considerations

In the future we will be able work on complex design problems untethered from the large, physical 2D screens we use today. Some design processes will take place in virtual 3D spaces from beginning to end, with entire teams joining from their preferred work location using technology no bigger than a pair of eyeglasses. Continued hardware evolution will allow designers to flip between XR modes at will. AR will allow for enhanced in-person collaboration, while VR will allow us to change context and simulate design outcomes without switching between hardware or software.

Extended Reality—whether augmenting our daily lived experiences to allow us to work in a more familiar and connected environment, enabling us to imagine the possibilities of something new within our own home, or refocusing our consciousness to experience the absence of pain–enables us as humans to forge connections and experience the environments around us in a more informed, efficient, and empowered manner. With the additional opportunities in using Extended Reality to aid in creating more efficient product delivery methods or enhance and engage us through new means of storytelling, the opportunities seem almost boundless in what it will enable us to do next. The important thing is to always remember that this technology alone doesn’t solve anything. Only by exploring this emerging tech through the lens of human-centered design can we arrive at solutions that add true value to the people that need them.