Early Challenges in AR UX
“Scene over screen” and other key obstacles designers must solve before AR can reach its full potential
Augmented reality—a technology that superimposes virtual objects into screen views of the real world—is poised to explode into mobile experiences. Just as skeuomorphic design helped transition users from file folders and physical desktops to clicking and tapping digital simulacra, today’s AR designers will need to move people from the now familiar world of smartphones into the new and unfamiliar realm of mobile AR. At Daydream, we've been working with mobile AR for several years, first with Project Tango, and now with ARCore. We've worked on and tested the full spectrum of available mobile AR experiences, and witnessed firsthand the differences design can make in delivering not only a better user experience, but responsibly redefining the boundaries of this new medium.
We see it over and over in research studies: users are blown away by what they experience in smartphone AR apps—everything from immersive animations to lifelike 3D objects, and movement-based interactions. People finally feel like they’re holding the future in their hands. But that enthusiasm has a downside. New AR users are often so impressed with the idea of AR, that they’re unable to be critical of the experience and articulate what isn’t working. The wow-factor creates a perilous blind spot not only for users, but for AR designers too. It’s an obstacle to truly tracking and addressing some of the core challenges of the medium.
Research conducted by the Daydream team reveals that after the wow-factor wears off, the people using AR are sometimes left with high-friction experiences that are difficult to get into, even harder to stay in, and unstable or unusable in certain environments. AR is a medium full of potential, but compared to their non-AR counterparts, these apps have to overcome significant headwinds in order to build an engaged base. These obstacles include everything from the social awkwardness of having to hold your phone up in a “photo-taking” position, to the fear of an app draining your device’s battery. To help get AR past the biggest of these initial hurdles, here are three of the most common challenges designers need to consider when building a mobile AR product.
Challenge 1: Regression to familiar smartphone interactions
When put into a new experience, people often default to familiar interactions and patterns. Mobile AR is an inherently environmental medium that invites movement and motion from the user. This is completely different from almost all traditional smartphone apps and experiences, where the interaction is constrained to small hand gestures on the surface of a device or interface. As a result, we’ve frequently observed people trying to rotate an AR object by swiping, or pinching to zoom rather than physically moving around the virtual object or walking closer. When we asked people to use an AR app to “scan” the room, they often treated their view like a photosphere and just spun in one place rather than walking around and exploring. User behavior will evolve over time, but at our current early state, it’s important to build in extra guidance and reasons to change position. While we recommend AR apps support touch-prompted rotation, our research shows that apps explicitly instructing users to move closer or move around are more engaging and successful than those that don’t. Think about finding ways to build such instructions into the experience itself, like asking users to follow a virtual character in space or by encouraging them to explore their environment.
AR apps tend to be more successful when they guide people to move around a space, as in the image above.
Challenge 2: Scene over screen
In traditional 2D smartphone apps, the entire user experience is contained within the screen. In mobile AR apps, however, that same 4-6 inch piece of glass also serves as a window into the world, forcing people to divide their attention between the interactive “scene” displayed on their device, and the traditional, non-AR parts of a phone’s UI. In our research studies, we’ve frequently observed “cognitive tunneling”—with people struggling to jump between 2D screen elements (like pop-up notifications or on-screen buttons) and the 3D “scenes” presented in AR. This leads people to miss 2D messages and calls-to-action on the screen’s surface. Designers need to consider the importance of keeping users immersed in the scene and avoid flows that take them back-and-forth between scene and screen. For instance, if you have an AR character that’s talking to the user through text, it’s more effective to display a 3D talk bubble next to the character's head, rather than use a traditional 2D subtitle at the bottom of the screen or UI.
It can be difficult for users to correctly perceive the depth and scale of virtual objects. Designers can help users out by adding visual cues like shadows and surface patterns, or deploying the use of characters (like the dinosaur shown here) or objects for scale.
Challenge 3: Three-dimensional objects in a two-dimensional space
People struggle with perceiving depth in mobile AR—making it more difficult to understand where virtual objects exist in relation to the physical environment. This means that people need additional cues to help them recognize what these virtual objects are and how they relate to the surrounding scene. Depth perception relies on binocular vision—seeing with two eyes and overlapping fields of vision—to triangulate objects in 3D space. But because smartphones only have a single camera and a single screen, the virtual space gets flattened, so that 3D virtual objects might still look two-dimensional. Adding effects like lighting and shadows cast onto the ground help users localize objects, while creating the impression of depth. For functional or informational apps, AR objects with a commonly understood scale—like humans or cars—are easier to perceive because they fit into a user’s mental model of those objects in space. So, if it’s critical to accurately convey depth or distance in your AR app (for turn-by-turn directions for instance), consider using a character like a human or animal pointing the way, rather than 3D symbols or arrows that lack an established and understandable size.
Stay tuned
AR can provide a more immersive and rich experience than traditional 2D applications, and as the designers and developers building these experiences, it's our responsibility to help people over the initial friction of a new modality. Many of these behaviors and observations will continue to evolve over time as users internalize the new features and capabilities of any medium. With ARCore, we're helping to build the next evolution of immersive UX—and providing a place to share even more knowledge and best practices. For us, it’s exciting to see the launch of so many AR apps, especially those that feature innovative experiences, and we’re looking forward to even more creative interactions as the field expands.