Building ARCore apps using Sceneform — Part 1
Fundamentals of ARCore
In this following series, we’ll learn about the fundamentals of Augmented Reality, ARCore SDK, Sceneform Framework — using simple examples. You will be walking away with an AR app at the end of this series.
Who Should Read?
Any Android Developer, even if you don’t have experience with AR or Graphics, this is definitely for you. Other AR developers, this may sound basic to you, but nothing wrong in brushing up your concepts and activating your neurons :-P. I’m also a beginner in this and I request the experienced ARCore builders to help this series readers with answers and improve the content.
Series Pit Stops
- Part 1 — ARCore Fundamentals (you’re here)
- Part 2 — Sceneform vs OpenGL
- Part 3 — Hello AR using SceneForm I
- Part 4 — Hello AR using SceneForm II
Prerequisites
- Android development experience (SDK, library usage, Fragment etc.)
- Kotlin experience (codebase will be in Kotlin)
- Augmented reality support device or the emulator supporting them
My Personal Experience
Google launched a new 3D framework called Sceneform at Google IO 2018. I’m more of a design-centric developer, fascinated by all visual elements and experiences. Augmented Reality was really inspiring to me and I always wanted to learn and shift my focus permanently — like an Android Developer, specialized in AR. This agenda was in my pipeline for sometime when AR core was initially released.
I was really excited but I did not pursue it immediately, as the support and the content was huge for C# -> Unity, but not for Java. That’s because of this robust framework called OpenGL, which literally scared everybody. It’s totally a different domain and I had no clue about it. So I postponed learning OpenGL(though I have to at one point) and my agenda for ARCore kept hopping.
Finally, Google released Sceneform — a 3D framework for Java/Android developers to easily create AR apps using ARCore (they also released support for Augment Images and cloud anchors, which we’ll talk another day). I was really excited and got my hands dirty with it, personally, I’m very impressed. It’s almost like I’m writing an Android App.
Immersive Computing Spectrum
Immersive computing is a new term used to describe applications that provide an immersive experience for the user. This may come in the form of an augmented or virtual reality experience. In order to better understand the spectrum of immersive computing, let’s take a look at this diagram
The preceding diagram illustrates how the level of immersion affects the user experience, with the left-hand side of the diagram representing more traditional applications with little or no immersion, and the right representing fully immersive virtual reality applications. For us, we will stay in the middle sweet spot and work on developing augmented reality applications.
AR, VR & MR
Augmented reality (AR) is a direct or indirect live view of a physical, real-world environment whose elements are “augmented” by computer-generated perceptual information, ideally across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. — Wikipedia
If that definition is complex, let me put it up in simple words. Augmented Reality is nothing but a mix of real-world objects and computer-generated objects. It gives an immersive experience to the user. The difference between AR and VR is that VR does not blend with the real world. It’s a mimic of the real world. Whereas AR blends and works with the real world — projecting virtual objects within the real-world.
In short, Matrix movie is virtual reality. In Ironman — Jarvis projects data to him when he is in the suit, which is augmented reality (ignore holographs). In Ironman 2, when he creates a new element, he expands the atoms and molecules by interacting with the virtual object — which is merged or mixed reality.
ARCore
ARCore is a google platform for building immersive augmented reality experiences. It enables our phones to see the world, understand it and also to interact with the objects and pieces of information.
How does it work?
ARCore does couple things..
- Understands the position of the phone as it moves, with respect to the world that is subjected to.
- Form that position, it learns and understands what world it is — building it’s own understanding
ARCore’s understanding of the real world lets you place objects, annotations, or other information in a way that integrates seamlessly with the real world. You can place a napping kitten on the corner of your coffee table, or annotate a painting with biographical information about the artist.
ARCore : Fundamental Concepts
Before diving into ARCore, it’s helpful to understand a few fundamental concepts. Together, these concepts illustrate how ARCore enables experiences that can make virtual content appear to rest on real surfaces or be attached to real-world locations.
ARCore has its origins in Tango, which is/was a more advanced AR toolkit that used special sensors built into the device. In order to make AR more accessible and mainstream, Google developed ARCore as an AR toolkit designed for Android devices not equipped with any special sensors. Where Tango depended on special sensors, ARCore uses software to try and accomplish the same core enhancements. For ARCore, Google has identified three core areas to address with this toolkit.
Motion Tracking
Tracking a user’s motion and ultimately their position in 2D and 3D space is fundamental to any AR application. ARCore allows us to track position changes by identifying and tracking visual feature points from the device’s camera image.
In the figure, we can see how the user’s position is tracked in relation to the feature points identified on the real couch. Previously, in order to successfully track motion (position), we needed to pre-register or pre-train our feature points. Now, ARCore does all this automatically for us, in real time, without any training
Environmental Understanding
The better an AR application understands the user’s reality or the environment around them, the more successful the immersion. ARCore looks for clusters of feature points that appear to lie on common horizontal or vertical surfaces, like tables or walls, and makes these surfaces available to your app as planes. ARCore can also determine each plane’s boundary and make that information available to our app. We can use this information to place virtual objects resting on flat surfaces. ARCore does this using a technique called meshing.
If you look at the illustration image, the triangle distribution on the planes (identified by ARCore) are the detected surfaces, where you can place the virtual objects.
Light Estimation
Fortunately, ARCore provides us with a way to read or estimate the light in a scene. We can then use this lighting information in order to light and shadow virtual AR objects. It can detect information about the lighting of its environment and provide us with the average intensity and color correction of a given camera image.
Summary
- Augmented reality is way to project virtual objects in the real-world and possibly make them interact with real-world entities such as position, light etc.
- ARCore is a platform created by Google to help developers, deliver this AR/immersive experience to the users
- Google has identified three core areas to address ARCore — Motion tracking, Environment understanding and Light estimation
- Motion Tracking — Through camera, ARCore learns and identifies visual feature points. These are the points of the real-world objects, that stays with respect to the phone’s position in the real-world
- Environmental Understanding — ARCore looks for clusters of feature points that appear to lie on common horizontal or vertical surfaces, like tables or walls, and makes these surfaces available to your app as planes. This technique is called Meshing.
- Light Estimation — ARCore provides us with a way to read or estimate the light in a scene. We can then use this lighting information in order to light and shadow virtual AR objects.
What’s Next?
Next pit stop, we’ll be seeing more about sceneform SDK and we’ll compare it with OpenGL.
