Tuesday, 16 June, 2020 UTC
Summary
Source
This winter, I designed my first app in augmented reality (AR) to help my company, Lextech, understand Apple’s ARKit’s capabilities. In my experience as a UX designer at Northwestern’s Knight Lab , I was aware of emerging best practices in AR, but had never designed in it myself.
Designing for AR has radically different requirements than typical mobile app design, and in my case, it was up to me to find the right tools and questions to shape the experience effectively. As the developer on my team explored object recognition with ARKit, I explored popular AR apps and tools to design in AR. In one day, I downloaded over 10 new apps to test. I explored how Pokemon Go utilized AR and played with 4 tools to design in AR.
Taking a video to mimic the in app experienceDiscussing what the developer would need to build our app in ARKit guided my direction for this project. Many tools out there help designers build functioning AR experiences on their own, but that was not going to be the best use of my time. Instead of building a full working prototype on my own, I’d be more effective if I focused on explaining the general direction and creating 3D assets he could use. Had my project goals been different, a tool like WiARframe or Torch might have been effective, but those tools are more focused on building out the interactions than providing complex 3D assets.
Prototyping for AR - WWDC 2018 - Videos - Apple Developer
Low Fi Prototyping for AR
With the new goal in mind, I started with my camera and paper. Inspired by the video above, I had a friend film as I moved paper to imitate buttons. This helped me answer basic questions such as “What buttons would I expect to see in this experience?” and “What do they do when I move my phone?” Paper was extremely helpful in understanding how 2D elements would move in a 3D space, so having a second set of hands rotate paper buttons towards the camera as it moves was critical.
When I didn’t have another person to hold paper, I used Keynote to quickly overlay objects over a video. I would make buttons and animate them to move as the video moved to test different interactions. Throughout my process, I routinely came back to both pen and paper and Keynote to test design changes before building the assets in Figma.
A screen shot from a video I made in Keynote while exploring the app’s basic functionality.In addition to using tools differently for low-fidelity prototypes, I found myself asking different questions. A user’s surroundings became even more important because they not only created the context for using the app, but would appear on screen. How a person might hold their phone or tablet, what they might be aiming it at, and how they might move it while using the app were things I paid particular attention to.
Trending AR VR Articles:
1. Designing for a modern 3D world: A UX design guide for VR
2. Scripting Javascript Promise In Spark AR For Beginners
3. Build your first HoloLens 2 Application with Unity and MRTK 2.3.0
4. Virtual Reality: Do We Live In Our Brain’s Simulation Of The World?
Learning How to Edit 3D Models
The interactions in our demo were fairly simple — we needed to expand pieces of a 3D model and rotate them. Instead of focusing on designing stellar interactions, I prioritized finding and editing a detailed model of phone hardware to highlight the potential of AR to see what’s under the hood of a machine. Since none of the 3D prototyping tools had an existing model of phone hardware, I searched sites like CGTrader to purchase a 3D model.
After testing the model in the app, it was clear the model needed editing. Some parts had colors or textures that weren’t visible in ARKit. In addition, I had to group the parts to allow users to select different pieces of the model. There are many options for editing 3D models, including Autodesk 3DS and SketchUp. I used Autodesk Maya, mostly because I had the easiest time opening my files in Maya.
Maya is traditionally used for making and animating 3D models. It provided a lot of flexibility, despite being complicated to learn. Relying on Autodesk YouTube videos, figuring out how to edit an object in Maya was by far the most difficult part of the process for me. Between the unfamiliar software and foreign file types, it took a lot of back and forth with my developer to understand the best ways to utilize Maya. There were times when I thought I had fixed a problem, only to realize I saved it incorrectly or moved a part to make the whole model completely unusable in ARKit.
A screenshot in Maya with annotations for the features I used frequently: selecting an object, grouping objects, rotating objects, and changing their color and texture.As with any design project, communication was the most important part of the project. The top 3 conversations my team discussed to keep the project progressing smoothly from the design side were:
- What happens once I move my phone? Just as with any design project, most of my focus was making sure my team had good communication about what we envisioned in the product. The biggest way these conversations differed from most projects was ensuring we were all on the same page about what happened when things moved in 3D space. Here’s where pen and paper and Keynote went a long way with aligning my team.
- What assets do you need? Where are you at now? These questions were another way of getting on the same page as my developer and allowed me to understand what the biggest gaps were that I needed to fill in. These questions made me realize that focusing on an accurate, usable model would be more helpful than building out every interaction in an AR prototyping tool.
- What file types can you use? Are you able to open this new file? This question was a lifesaver when it came to figuring out what problems I ran into when learning Maya. Making sure the new file worked well was also critical, because it would often alert me of a new issue in the file I wasn’t aware of before. Because the developer and I were working with the models in different software, I had to ensure that whatever he was seeing matched what I saw.
These questions allowed me to continue making progress as he figured out to recognize an object in AR Kit, which also was not an easy task.
In mobile design, you have to understand a person’s surroundings to create a great experience. This is so much more true for AR experiences, where the surrounding becomes a part of the app. AR has the potential to truly push boundaries of how apps can take advantage of their contexts and keep users focused on their surroundings.
However, to get to that point, new skill sets must be added to the field of experience design. Low fidelity prototype will include a whole new set of methods to learn about 3D space. 3D modeling will start showing up on some UX job descriptions. And I’m excited to see where these skill sets take the world of experience design in AR.
Screen shot of our 3D model in appDon’t forget to give us your 👏 !
https://medium.com/media/1e1f2ee7654748bb938735cbca6f0fd3/hrefWhat I Learned Designing My First AR App was originally published in AR/VR Journey: Augmented & Virtual Reality Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.