Interior Design with Augmented Reality

Ananda Poudel and Omar Al-Azzam

Department of Computer Science and Information Technology
Saint Cloud State University
Saint Cloud, MN, 56301
{apoudel, oalazzam}@stcloudstate.edu

Abstract

This paper presents an application of Augmented Reality (AR) for interior design. Due to huge
advancements in computer vision algorithms and cheap hardware, Augmented Reality is becoming
mainstream. Furniture retail is a significant aspect of USA economy with over $100 billion in
sales. However, most of the sales come from physical stores. Buying furniture from brick and
mortar shops is cumbersome and time consuming. AR is changing the furniture industry. Big
players like IKEA, Wayfair, and Target are using this technology to boost sales by creating
engaging Omni channel experience. In an AR environment virtual furniture could be placed and
manipulated in the physical world in real time which allows the user to have an interactive
experience. Users would be able to visualize exactly how a table would look like in their kitchen.
When people can place an actual couch on the living room or visualize how a bookshelf would
look in different color. The risk of product return and logistics is drastically reduced. As online
stores replace brick and mortar shops, AR will play a vital role in the furniture sales. This paper
provides new ways an individual/enterprise could utilize AR to design interiors.

KEYWORDS
AR, ArCore, Augmented Reality, Interior Design, Marker-Less AR, Marker-Based AR

1. INTRODUCTION
Augmented Reality (AR) is a technology that superimposes a virtual world on a user’s real
world. It is also known as Mixed Reality because it mixes both the real and the virtual world.
AR is being adopted in many industries such as Health, Education, Retail, Manufacturing, and
Interior Design. The different types augmented reality are as follows

Projection Based AR
As its name suggests, it is based upon projection onto object surface. For instance you can
project a virtual keyword onto a wall and use computer vision to track your finger movements
when pressing something.
Figure 1. Projection based Augmented Reality

Recognition Based AR
Recognition based augmented reality relies on recognizing real world items such barcodes,
images, texts, etc. to provide information. The most common type of Recognition Based AR App
is a language translation app which uses camera to read the text, translates it and overlays on top
of an image.

Location Based AR
Location based augmented reality relies on location of the device, orientation, sensor data to find
out where the user is looking and provides meaningful information based on that. Famous
example of location based augmented reality is Pokémon Go.

Superimposition Based AR
Superimposition based AR allows users to merge real and virtual worlds. It allows the user to get
an alternate of the object by replacing either a portion or an entire view with an augmented view.

Figure 2. Superimposition Based Augmented Reality

1.1 COMMON TYPES OF AUGMENTED REALITY TECHNIQUES

Marker-based AR
Marker-based AR application usually involve image recognition, the images that are to be
recognized are predefined in the application. During runtime the app analyzes the camera stream
and try to find marker (target). Once it detects the marker, the AR markers are used as the
location for rendering virtual objects. AR markers can include wide range of images but can be
as simple as barcode.

Marker-less AR
Marker-less AR applications recognizes objects that were not predefined. Application recognizes
different features, patterns, colors etc. As there is no predefined image targets on the app. During
runtime the app has to analyze the different variables in camera frame to trigger AR actions.

2. RESEARCH MOTIVATIONS
Augmented Reality (AR) technology is well developed and widely used as an alternative to
virtual reality on mobile devices. Design industries are extensively adopting AR to create
meaningful user experience because AR allows companies to create apps that enables user to
interact and visualize actual products. With the advancement in computer vision algorithms and
cheap hardware, AR has finally become mainstream. Furniture buyers always had concerns
regarding product returns because whenever they would buy furniture online, they weren’t sure if
the piece of furniture will fit into a particular space. With an AR app, users can place an actual
table in their dining room to find out if the table fits in the space and looks good. This idea
drastically reduces the risks of product return and logistics cost. In this paper, we are exploring
how AR could be used for interior design and retail by employing different AR techniques.
Furniture retail is significant aspect of the U.S. economy with over $100 billion in sales.
However, most of the sales come from physical stores. Buying furniture from brick and mortar
shops is cumbersome and time consuming. According to the Wall Street Journal report, furniture
business is among the fastest growing product categories online with 15% of revenue solely
coming from online sales. AR has been seen as the technology that will boost furniture sales
because AR enables users to visualize furniture. It also enables users to try out different interior
designs. For instance users can visualize wall painted in different colors with no effort. As online
shopping popularity increases and replaces brick and mortar stores, AR could be seen as a major
contributing factor to the online sales. Therefore, it would worth the effort to build solutions
targeting interior design and retail.

3. WHAT ARE WE TRYING TO BUILD?

We are building a mobile app that allows user to visualize and interact with furniture using their
smartphones in augmented reality environment. AR allows an application to use the phone’s
camera and sensors to directly place a virtual object in the space seen by the camera. This is done
by identifying a marker in the AR space and using that as a reference for the rest of the room.
Our application utilizes marker-less AR technology to place objects in a 3d space without the
need of a marker. We are using this technology to develop an interior design application for
previewing furniture virtually in your own home.

3.1 Goals of the Project

The main goals of this research is to build an app that is able to render different furniture models
and be able to interact with them using different gestures. We determined that the system will
perform the following tasks:
1. Select a furniture from a menu
2. Place a furniture by tapping on the screen
3. Remove rendered furniture
4. Change material of rendered furniture
5. Move the location of furniture
6. Take a picture of the current environment and store in the local storage
7. Load 3d models from local storage

While there have been number of attempts in creating AR app in interior design space using
different platforms, tools and techniques. We are trying to find an elegant and simple solution to
this problem.

3.2 Outcomes of the project

The main outcomes of this project are as follows:
1. A GUI that allows user to use select, place, remove and modify 3d models in an
interactive way by using different touch gestures, at any time, more than 90% of the
screen is filled by camera stream and rendered models. We have adopted “more is less”
approach to design this GUI.
2. A solid backend that handles app’s core functionality such as detecting the gestures,
modifying rendered 3d models, loading 3d models from local storage.

4. DESIGN and CONSTRUCTION

We implemented our Interior design app using Marker-less AR, we initially started with Marker-
based approach because of the lack of support for Marker-less augmented reality. Mixed Reality
headsets like HoloLens could easily solve the problem because it uses spatial mapping using
different sensors, however HoloLens is still in its infancy, not a lot people have access to it and it
is very expensive. Recently, Google announced its ArCore platform that would allow developers
to create Marker-less AR experiences so we built our app around android ecosystem primarily
because of its availability, discoverability, reliability, and agility. The development environment
included Unity Game Engine, C# Programming Language, Visual Studio IDE and we
currently only support android devices.

The app’s core requirements could be summarized using the following diagram
 Figure 3. App’s core requirements

4.1 ARCHITECTURE
The entire app is developed using Model View Controller architecture because whenever user
invokes a functionality, UI Controller receives requests for the application and then works with
the Model to prepare any data needed by the View. The View uses generated data to create the
final presentable response and updates the UI. For instance, when the user selects a 3D Furniture
from the UI, It is mapped to the Controller which grabs the 3D Furniture from the Model and
updates the View component which is UI.

Figure 4. Model View Architecture Diagram

The design and construction can be categorized into two main parts:
4.1.1 Front End Development
We have created minimal User Interface for our app. Our primary rule was that at any time the
user should be able to view the camera stream and the rendered model. We designed transparent
menus and sliders that would only come up when the user needs it. Instead of having buttons we
used touch gestures to manipulate and select rendered 3d models.

Figure 5. Menu Invoking Gear and Colored plane

We created simple, easy-to-use gear to pull up the menu. When the user selects the gear menu it
renders a slider with available furniture like shown in the following picture.
 Figure 6. Menu Slider with list of available furniture

With the menu slider, the user can select different furniture. Once the user selects the furniture
one can just tap on the colored plane. A 2-d vector is then shown from the tap action which we
convert into vector3 that gives the actual location where the furniture is going to be placed. Once
the furniture has been rendered, the user can view the furniture from different perspectives. In
addition, the user can manipulate the furniture by using gestures. The furniture can be dragged to
move it around the environment. The user can utilize pinch gestures to rotate the furniture. The
app provides variations of tap actions to remove or select the rendered furniture. Once the user
selects one of the rendered furniture, an additional button is displayed through which user can
change the material of the selected furniture.
Figure 7. App rendering different furniture

4.1.2 Backend Development

As we are using MVC architecture, there is separation of concern. Frontend and Backend are
loosely coupled which allows us to make improvements to either stack without affecting the
other. As of now, backend logic primarily consists following components:

Interface to Local Storage

This component primarily deals with 3d models and it’s storage. It receives data request from the
controller regarding 3d models. It helps exporting the 3d model from local storage to augmented
reality environment. In addition, it is utilized to save the screenshots of the augmented reality
environment.

AR Package
It is the most significant aspect of the App. Everything we have built so far is on top of Google’s
ArCore package which does motion tracking, floor detection and light estimation. We use
ArCore APIs to render 3d models. Once the app starts, we scan the floor through ArCore API.
Once the floor is detected, we provide feedback to the user by rendering a colored plane. After
this the user can place 3d objects in the environment.

Input Controller
It can be seen as the mediator between the UI and the backend logic. Its primary function is to
handle the UI actions and trigger events in the backend. This component is responsible for
detecting gestures for rotating, moving the 3d models in the environment. It also contains logic
for address conversion.
5. FUTURE WORK
We are trying to integrate photogrammetry to our existing platform which will allow us to
reconstruct a 3d model of furniture from pictures. As of now the user can only visualize the 3d
models that are in the local storage, we would like to expand this functionality. We intend to
connect the app to a cloud repository from which a user could browse furniture and import it
during runtime. In addition, we are planning to revolutionize the way we share things by using
photogrammetry principles.

6. CONCLUSIONS
With advancements in computer vision technology and cheaper hardware, AR can only flourish.
AR has its unique advantages and is very good at tackling especially visualization problems. In
an AR environment, buying furniture could be convenient and easy while saving costs by
completely lowering the risk of product returns. In this research paper, we examined how a
marker-less AR could be used for online furniture sales. We proposed a mobile application that
enables users to visualize furniture in virtual reality.

7. REFERENCES
Milgram, P., Takemura, H., Utsumi, A., and Kishimo, F. 1994. Augmented reality: A class of
displays on the reality virtual continuum. In Proceedings Telemanipulator and
Telepresence Technologies: 2351-34
Phan, V. T., and Choo, S. Y. (2010). Interior Design in Augmented Reality Environment.
International Journal of Computer Applications, 5(5), 16-21

Oluwaranti, A. I., Obasa A. A., Olaoye A. O. and Ayeni S., (2015, July). Architectural Model
For
An Augmented Reality Based Mobile Learning Application. Multidisciplinary
Engineering Science and Technology (JMEST), 3159:0040
A deep dive into the U.S. furniture market from
https://www.lifung.com/wp- content/uploads/2016/02/U.S.-Furniture-Market-Report-by-FBIC-
Global-Retail-Tech-Feb-9-2016.pdf