Getting started

Table of contents

• What is AR, VR & MR
• Augmented Reality
• How AR works
• AR Fundamentals
• Spatial computing techniques
• Types of AR targets
• Types of AR
• AR SDKs
• Applications of AR
• Advantages of AR
• Disadvantages of AR

What is AR, VR & MR

• Virtual Reality (VR) — It teleports the user into a completely immersive world that replaces the physical world. Users can use HMD devices such as Google Cardboard, Oculus Quest, Rift or HTC Vive to experience an imaginary environment. It has applications in training and educational use cases.
• Augmented Reality (AR) — It overlays digital content in the real world. Users can use their smart phones to experience AR. Unlike VR, it does not require any HMD devices. Pokemon Go is the best example for an augmented reality game. It has applications for creating digital advertisements, visualization etc.
• Mixed Reality (MR) — It is a combination of both AR and VR. It enables users to interact with virtual objects in the real-world environment. Microsoft Hololens is a key player in the MR market.

Augmented Reality

How AR works

• When the app starts, the device sets its initial position and orientation.
• AR app captures the environment using a camera on a smartphone / HMD device.
• It scans the captured environment to identify a point as to overlay virtual objects using markers.
• After successful point identification, AR app requests predefined content to overlay on the identified points. It displays additional information regarding the captured environment.
• It overlays virtual objects in the real-world points. It tracks the device movement to adjust the augmented content position.

AR Fundamentals

Spatial computing techniques

Simultaneous Localization and Mapping (SLAM)

SMART

Types of AR

• Marker based — A visual marker or a distinct image that the camera recognizes and processes to reveal information about an object. The marker is usually a QR code or a custom logo or artwork.
• Marker less — Unlike marker based, it scans the real environment and places digital elements based on geometry.
• Projection based — Directly overlays digital projections onto the physical world by making use of varied machine vision technology, often by combining visible light cameras with 3D sensing systems such as depth cameras. It uses a technique called projection mapping, where the projected image is mapped onto physical objects, creating direct digital overlays.
• Location based — Allows developers to add digital content onto geographical points in the real world. It uses GPS and compass sensor data to position the virtual object at a point of interest. Pokemon Go is a popular example that uses location-based AR.
• WebAR — A web based augmented reality experience that is accessed through a web browser. It does not require a mobile application to function, and users can experience AR directly from their smartphone using the native camera and a web browser.

Types of AR targets

• Marker — It is a basic target with a wide border. It can be readily recognized by the software with very little processing overhead. Coded markers can be used to create 2D barcode patterns within the borders. Vuforia includes a powerful marker system called VuMark that makes it very easy to create branded markers.
• Images — Images can be used as targets. Image tracking falls into the category of natural feature tracking (NFT). The images with a well-defined border, irregular asymmetrical patterns, and good contrast is a good AR target. AR engine locates unique points within the image called feature points which is used to recognize the image while scanning.
• Text recognition — Some AR SDKs allows you to configure your app to read text in specified fonts. Vuforia has an option to recognize text and augment content on top of it.
• Shapes — Basic shapes such as a cube or cylinder with specific relative dimensions can be used as an AR target. It's not just the shape but its the measurements that distinguishes one target from the other. When the app is pointed to a shape which have enough textured surface for good edge detection, it will recognize and overlay the virtual object on top of it. Vuforia has implementation of basic shapes.
• Object recognition — 3D model can be used as AR target. It is useful to augment some content on a 3D model instead of just 2D content. Vuforia offers Object Scanner to create object data files that can be used in your app as targets.
• Geolocation — AR apps can also use the GPS locations to identify to annotate what is in view. The geo-coordinates can be tagged with an augment content and triggered once the user reaches a specific point. Pokemon Go is the best example that uses located based this type of targets.
• Multi-targets — Multiple markers can be used in a scene to pop up multiple virtual objects in the scene simultaneously. These markers can be printed, folded or pasted on geometric objects, such as product labels or toys.

AR SDKs

• ARKit — ARKit is one of the powerful AR SDK released by Apple on 2017 to build AR solutions using iOS devices. ARKit combines device motion tracking, advanced scene processing, camera scene captures and display conveniences to simplify the task of building an AR experience. It also provides space recognition using SLAM, objection detection, light estimation.
• ARCore — ARCore is an AR SDK released by Google in 2018 for building augmented reality experiences. It uses different sensor readings of the device to sense its environment, understand the world and interact with information. It also has features like motion tracking, environmental understanding, and light estimation to integrate virtual content with the real world.
• Vuforia — Vuforia is an AR SDK that enables developers to position and orient virtual objects in real world environment. It uses computer vision technology to recognize and track planar images and 3D objects in real time. It supports a variety of 2D and 3D target types including marker less image targets, 3D model targets, and VuMark.
• Wikitude — Wikitude is an AR SDK recently acquired by Qualcomm. Initially it was focused on providing location-based augmented reality experiences. It released a development framework utilizing image recognition and tracking, and geolocation technologies in 2012. The Wikitude SDK includes image recognition & tracking, 3D model rendering, video overlay, location based AR and SLAM technology which enables object recognition and tracking, as well as marker less instant tracking. The cross platform SDK is available in the market for Android, iOS and Windows operating systems.
• MAXST — MAXST AR SDK is an AR engine for developing augmented reality solutions. It comes with SDKs for both 2D and 3D content augmentation. It is capable of recognizing and tracking 2D images and 3D spaces. It includes a cloud server-based image recognition feature and AR Fusion Tracker with the powerful tracking of ARKit / ARCore. MAXST comes with a cross-platform development compatibility with Android, iOS, Mac OS, and Windows. It is available as both free and paid versions.
• Amazon Sumerian — Amazon Sumerian enables users to create and run virtual reality (VR), augmented reality (AR), and 3D applications quickly without requiring any specialized programming expertise. It is a web-based platform which uses WebGL support. It can run on AR/VR hardwares. It is used to create engaging 3D experiences and provides easy access to machine learning, chatbots, code execution etc.

Applications of AR

• Data Visualization — Being able to visualize data is an important tool to make important decisions. The visualization of data in VR and AR transforms plain and boring numbers into captivating stories and helps users to understand the data more efficiently. When AR is integrated with AR glasses or contact lenses, they can provide a hand-free interaction experience. Using accessories, AR allows users to interact with the data securely that prevents the risks of data leakage. Salesforce uses Oculus Rift to create an immersive 3D environment for analyzing data.
• Gaming and Entertainment — Both AR and VR have made mainstream adoption in the last couple of years. Gaming vendors can run both VR arcade centers as well as profit-off from ad-sales generated within the game. Instead of leasing out costly real-estate and props, AR VR escape rooms enables can offer developers an end-to-end digital experience. Wearable devices such as Heads-Up Displays (HUD) and AR glasses are expected to be widely adopted with nation-wide 5G installations.
• Training — Augmented and virtual reality can revolutionize the training sector by making it more engageable. It is useful to simulate situations that are too dangerous or difficult to recreate in real life. Imagine an accident on nuclear plants or a military attack, AR/VR is opening a way to simulate these scenarios. AR and VR training programs are able to expose employees to realistic and diverse situations, preparing them for any of the variables they might encounter.
• Education — The AR VR was adopted in the education sector during the pandemic. Augmented and virtual reality have the potential to bring more students into the classroom and create more engaging and exciting classroom experiences. Using AR and VR, teachers could create an immersive learning environment, allowing students to explore the ocean floor or unfamiliar parts of the globe. Both AR and VR can provide improvements in the areas of educational content creation, recreational purposes and running scientific simulations. Biju’s, Quiver, DAQRI, Arloon and Blippar are the key players in the education industry who use AR to educate students.
• Retail and Digital Commerce — Retailers have an advantage as mobile-friendly AR can be a new standard for the in-store shopping experience. Customers can scan items and discover the latest deals, prices, and upcoming offers. IKEA was one of the early adopters of AR wherein customers get a well-informed view of how the furnitures would appear in their homes. Dior also enabled customers to try sunglasses virtually using AR specs. E-Commerce industry is making the most of the latest technology including Augmented Reality to engage customers. Virtual trial is one of the widely appreciated feature from the retail side.
• Healthcare — Healthcare is one the main industries that has adapted AR and VR to fit its needs with at most success. Clinicians can use AR VR apps during exposure therapy to help people overcome certain fears. Some people with phantom limb pain use VR to help the complete tasks with the phantom limb. Doctors are also using AR to examine a patient before a surgery and project images of surgery to help other surgeons and project images on internal systems onto indivituals skin.
• Travel and Tourism — AR can be used to augment the travel experiences. Take an example of Google Translate app that uses augmented reality to let the users convert symbols printed in foreign languages to native languages. In various scenarios, AR acts as a virtual tourist guide. AR apps that are location based (like Wikitude) enable users to overlay the required information like nearby ATMs, restaurants, etc. onto the actual camera view. Google recently launched Immersive view in Google maps to augment virtual content over map data.

Advantages of AR

• Enhanced Experience — Augmented Reality provides enhanced experience, users can visualize information which is overplayed on real environment by using their phones while scanning the marker. For example, while using maps, directions are overplayed on the destination routes.
• Easy to Use — Augmented reality is easy to use. Point the camera at an object and the app will display the usage of the object. For instance, Scanning the image of a device will display a list of information regarding the object is overplayed. Users can use AR to furniture or clothing to visualize how the furniture would look or how the clothes will fit on them.
• Learning And Instruction — Augmented Reality enhances the classroom environment by making the whole learning experience more immersive and engaging. It enables learners to engage with information thereby making them active participants rather than passive receivers of teaching. Learning content visualization, training simulations, laboratory exercises and experiments are some of the applications of AR in the learning environment.
• Communication — Augmented Reality brings in a new method of computer-mediated human connection. It makes communication more involved than just sending text messages through chat applications. With the help of AR, remote collaborations become more immersive with digital features. Visual pictures are superimposed in front of the participants during the conversation to assist verbal communication. Social medias like Facebook, Instagram, snapchats are using AR emojis and face mesh to create different filters thereby making our content more attractive.
• Risk free workforce training — There are high-risk activities involved in medical sectors which have to do a lot with human anatomy, military training, where one has to learn to combat on the battlefield. AR as a medium, can get an immersive experience of performing training without being physically hurt. It enables superimposing feedbacks in real time thereby overcoming the barrier of having separate devices unlike the traditional way.

Disadvantages of AR

• Addiction — AR is fully immersive thereby making it more addictive to users. People who are addictive to their phones constantly check for updates regarding game updates. This addiction has been deemed as “smartphone fever” by psychologists.
• Expensive — Augmented reality is expensive to create. The cost of an app can be up to $100k depending on how complicated it is. The device required to run AR apps are costly thus making it unaffordable to small companies.
• Dangers reality modification — AR narrows the line between the real world and the digital world. AR enables users to modify the real world leading to possible dangers. For example, Pokemon Go game has created controversy due to associated accidents and even deaths. Overlaying digital elements on the real environment make users less cautious and lead to dangerous situations. For example, factory workers may lean over too far and end up injuring themselves due to vision being blocked by their glasses/contact lenses, while operating machinery at high speeds giving them no time to react.
• Privacy issues — Augmented reality is based on the collection, analysis, and redistribution of different types of data thus raising concerns over privacy rights and security. For example, some AR devices record the environment in real time to map and augment the content. They may raise potential privacy concerns. Some AR systems collects and analyze information regarding the users such as device usage history among other users which raises GDPR issues.