World Academy of Science, Engineering and Technology

International Journal of Computer and Information Engineering

Vol:15, No:8, 2021

A Simulation Tool for Projection Mapping Based on

Mapbox and Unity
Noriko Hanakawa, Masaki Obana

 surrounding environment. We should judge the availability of

Abstract—A simulation tool is proposed for big-scale projection projection mapping to the target at the first fieldtrip. The second
mapping events. The tool has four main functions based on Mapbox fieldtrip is a test of projection to the target building with real
and Unity utilities. The first function is building three-dimensional equipment such as projectors and PCs. We should move a lot of
models of real cities using Mapbox. The second function is movie
equipment to the projection area in front of the target building.
projections to some buildings in real cities using Unity. The third is a
movie sending function from a PC to a virtual projector. The fourth The third fieldtrip is a rehearsal with actual equipment such as
function is mapping movies with fitting buildings. The simulation tool projectors, speakers, PCs, audio amplifiers, and actual movies
Open Science Index, Computer and Information Engineering Vol:15, No:8, 2021 publications.waset.org/10012238/pdf

was adapted to a real projection mapping event held in 2019. The event and music. Then, the fourth fieldtrip is an actual performance
completed, but it faced a severe problem in the movie projection to the on the event day. The four fieldtrips require much time and
target building. Extra tents were set in front of the target building, and effort from teachers and students. In this way, projection
the tents became obstacles to the movie projection. The simulation tool
mapping events are efficient education techniques. However,
developed herein could reconstruct the problems of the event.
Therefore, if the simulation tool was developed before the 2019 severe problems exist in projection mapping events.
projection mapping event, the problem of the tents being obstacles Usually, we project movies onto the surfaces of real
could have been avoided using the tool. Moreover, we confirmed that buildings. At the planning phase of the event, the developers of
the simulation tool is useful for planning future projection mapping projection mapping must set projectors at mapping positions
events to avoid various extra equipment obstacles, such as utility near the buildings. Further, they must test the projection
poles, planting trees, and monument towers.
conditions at the real projection locations. Therefore, they have
excessive workload for setting up the equipment, which
Keywords—Avoiding obstacles, projection mapping, projector
position, real 3D map.
requires a considerable time period. Moreover, they need to
examine the movie projection condition onto the building and
I. INTRODUCTION obstacles. For example, obstacles are event stall tents, traffic
lights, and utility poles. Sometimes, a movie may be interrupted
P ROJECTION mapping events are popular worldwide,
especially in higher education institutes, such as
universities. Abilities of movie creation, music creation, and
by such obstacles. Furthermore, obstacles may be moveable,
e.g., event stall tents. Even if no moveable obstacles are present
in the previous local field survey, they may be present on the
event management are essential for higher education. At first, event day. However, we cannot perform the local field survey
students plan a projection mapping event with customer frequently because the workload and cost of previous local field
meetings and local field surveys. Subsequently, they develop a surveys is very high. Therefore, we developed a simulation tool
theme and scenarios for the movie. Further, students create for projection mapping tests. Using the simulation tool,
movies and music using various multimedia tools. After developers can test projection conditions onto virtual buildings
creating movies and music, the projection mapping movies are and obstacles several times without a heavy workload. The
evaluated by customers. On the event day, students prepare the requirements of equipment, transportation cost, labor cost, and
equipment for projection mapping and operate the event. the cost of renting a place for projection mapping in a real local
Moreover, because general event audiences directly evaluate area are eliminated. The simulation tool has four main
the projection mapping events, the students may have more functions: building modeling function, movie projection
satisfaction and accomplishment. It is a high-performance function, sending movie function of capturing a window, and
interdisciplinary education in higher education [1]. However, mapping function of fitting building surfaces. Especially, the
efforts required for this education are considerable for teachers building modeling function is based on Mapbox SDK [2].
and students alike. This effort significantly influences other Virtual and movie projections are realized on UNITY [3].
education. For example, we need fieldtrip for investigating the The remainder of the paper is organized as follows. Section
target buildings such as train stations, historic castles, and II presents related work. In Section III, we explain the
modern tower buildings. The fieldtrip should be held several simulation tool, including the four functions. The adaptation
times at one projection mapping event. results of the tool are presented in Section IV. Finally, Section
The first fieldtrip is a preview of the target buildings and V presents the summary and future studies.
Noriko Hanakawa is with Hannan University Matsubara, Oasaka 580-8502
Japan (e-mail: [email protected]). II. RELATED WORK
Masaki Obana is with Osaka Institute of Technology, Hirakata, Osaka Many studies have been conducted on high-quality
573-0196 Japan (e-mail: [email protected]).
projection mapping. Several algorithms for analyzing

International Scholarly and Scientific Research & Innovation 15(8) 2021 518 ISNI:0000000091950263
 World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:8, 2021

three-dimensional (3D) model techniques have been proposed III. A SIMULATION TOOL FOR PROJECTION MAPPING
[4], [5]. The proposed algorithms can decrease developers’
A. An Overview of the Simulation Tool
workload by autofitting movies on complicated buildings.
Although the algorithms are significant, the proposed Fig. 1 shows an overview of the simulation tool. The
simulation tool can be implemented with significant simulation tool has four main functions, and it was constructed
algorithms. The simulation tool is highly expandable with the based on Mapbox and Unity. The following sections explain
algorithms. Moreover, Battulga and Fujimoto proposed a new the details of the functions.
method for avoiding blank areas generated by obstacles’ and B. A Building Modeling Function
viewers’ positions [6]. The method can generate natural views The building modeling function is based on Mapbox SDK
even if a viewer moves anywhere because obstacles’ for Unity. Mapbox can generate 3D models of real cities using
backgrounds are projected onto natural movies and obstacles’ map data. Fig. 2 shows a 3D model of a San Francisco city
surface. Simizu et al. proposed a calculation method for generated automatically using the Mapbox SDK for Unity. In
optimizing two projectors’ positions for natural views [7]. the function, longitude and latitude values on the map data
These studies are useful for achieving high-quality projection indicate the views of reproducing the cityscape.
mapping and movies. Our simulation tool is related to these
Open Science Index, Computer and Information Engineering Vol:15, No:8, 2021 publications.waset.org/10012238/pdf

studies. However, our simulation tool is based on the real 3D C. An Overview of the Simulation Tool
map data of real cities. Therefore, our simulation tool can Unity supports a standard projector component. The movie
support large-Sscale projection mapping events, e.g., real projection function is developed using the projector component
building projections and real tower buildings. Further, real in Unity. However, the projector component of Unity cannot
obstacles in the real 3D map can be reconstructed in the tool. project movies when there is an obstacle in front of the
For example, developers try projecting movies on a 3D virtual building. Fig. 3 shows the failure projection case in the
building numerous times until a good projector position can be projector component in Unity.
clarified in the tool.

(3) A sending movie function to projector

( A building
(1)
m
modelling function
Virtual Projector

Movie

(2) A movie projection function

(4) A mapping function of
fitting building
Fig. 1 Overview of the simulation tool

Surface of building

Projector

Fig. 2 3D model of a San Francisco town

International Scholarly and Scientific Research & Innovation 15(8) 2021 519 ISNI:0000000091950263
 World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:8, 2021

The building surface

Projector

An obstacle

The nose of the dog is projected on the building surface

although an obstacle cube is set at the front of the building
Open Science Index, Computer and Information Engineering Vol:15, No:8, 2021 publications.waset.org/10012238/pdf

Fig. 3 Problem of the projector component of Unity

The reason for the problem is a difference in the projection training in real projection mapping events.
method in Unity. Naturally, the projector in Unity should Fig. 5 shows an image of the mapping operation in the
project using the perspective projection method. The mapping function. Users operate various mapping trials to fit
perspective projection method involves a natural view using a the surface of the target building in the tool. The movie of the
perspective. It implies that a different perspective can be a mapping operation is projected obstacles. Therefore, users can
different building view. Therefore, the behind building surface train the mapping operation in a virtual 3D building without
by a cube obstacle also projected the dog nose image. visiting the real building.
Therefore, the projection method was changed from the
perspective projection method to the orthographic projection IV. ADAPTATION OF THE SIMULATION TOOL AND DISCUSSION
method that can generate a 2D view like camera images.
A. Adaptation
Additionally, the orthographic projection method can generate
the shadows of the obstacle on the building surface. The The simulation tool was adapted to a real projection mapping
shadows are calculated through the depth information, i.e., the event. The event was held in November 2019 in Osaka, Japan.
distance between the projector and surfaces. The event finished, but it faced a significant problem. Several
event stall tents were set up in front of the target surface of the
D. A Sending Movie Function to Projector building. It implies that the event stall tents were obstacles to
The projector in Unity project movies captured the images of the movie projection to the target surface of the building. Fig. 6
a PC window. Usually, movies are directly set to the projector shows a real projection mapping picture and a screenshot image
component in Unity. However, the projector projects images of of the simulation tool. In the real event, the tent roofs of the
a PC window because various image and movie types are stores were obstacles to the movie projection. The tent roof
supported in our simulation tool. We developed a window obstacles were unexpected. The tent roof obstacles were found
image capturing tool with User Interface (see Fig. 4). The on the event day. These obstacles were clearly simulated (see
image capturing tool runs on a desktop computer. The images the (2) image of Fig. 6); and had we run the simulation before
of the selected window of the computer are captured. the projection mapping event, we could have prepared a
Subsequently, the images are sent to the projector component in projector position and projection ways to avoid these obstacles.
Unity. In this way, the projector in Unity can project various Here, a procedure of adaptation of the simulation tool is
movies or images like various application software images and explained. First, we prepared the 3D city map data in Mapbox
movies. (see Fig. 7). The target city is Matsubara-shi, Osaka. The city is
near Hannan University. Inset (1) of Fig. 7 is a real picture of
E. A Mapping Function of Fitting Building
the mapping building. Image (2) of Fig. 7 is a screenshot of the
Usually, projection mapping must perform mapping mapping building in the 3D city, and it shows the position of the
operations. The mapping operation implies that a movie is fit to real projector at the event. The projector position can move
building surfaces. For example, a building surface is a trapezoid everywhere. However, to reconstruct the projection condition
although the shape of a movie is rectangular. Therefore, the at the event, the projector position in the simulation tool is at the
shape of the movie is changed to a trapezoid. Changing the real position of the event. Therefore, the problem of the tent
movie shape represents a “mapping operation”. roof obstacles could be reconstructed in the simulation tool, as
The changing movie shape function was developed using shown in Fig. 6. Further, the projector position moves in the
“OPENCVSharp getPer-spectiveTransform()” method. Our simulation tool. As shown in Fig. 8, the projector moves from
simulation tool users perform the mapping operations. The (a) position to (b) position. The projector moves by the x-axis,
mapping operation is essential for users because it is essential y-axis, and z-axis. Thereafter, the movie projection starts. We

International Scholarly and Scientific Research & Innovation 15(8) 2021 520 ISNI:0000000091950263
 World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:8, 2021

can confirm the projection condition of the movies in the projector repeatedly moves until the projection condition is
simulation tool. Here, we confirmed that the tent roof obstacles good. Finally, the position of the projector was decided through
did not influence the projection. However, the position of the the simulation run in the tool.

UI of the function
Design of the sending image to the projector function
(3) Input a window name
(4) Start capturing
12 sending image to
࢟ࣕࣉࢳࣕࡋࡓ⏬ീ
projector
ࢆࣉࣟࢪ࢙ࢡࢱ࡬

11capture OK? 5 input name

Capture Dummy
7input name

(1) A list of window names
ධຊᩥᏐิ

10 capture OK? 4 window name

8࢘࢕ࣥࢻ࢘ྡ
window  9capture
࢟ࣕࣉࢳࣕ
EP S 
 3 window name
name bmp
6,start capture

(2) Selecting a window Content

2. window name
Object name
Script name
C#scriptname 1. node generation
Open Science Index, Computer and Information Engineering Vol:15, No:8, 2021 publications.waset.org/10012238/pdf

Fig. 4 Design of the sending image function

Mapping operation

Obstacles

Fig. 5 Example of the mapping operation

(2) An image of the simulation tool

Obstacles

(1) A real projection mapping picture

Fig. 6 Comparing the real projection mapping and an image of the simulation tool

International Scholarly and Scientific Research & Innovation 15(8) 2021 521 ISNI:0000000091950263
 World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:8, 2021

(3) Projector position

(1) A real city picture

(2) 3D city model

Fig. 7 3D city map and the projector position in the tool

Open Science Index, Computer and Information Engineering Vol:15, No:8, 2021 publications.waset.org/10012238/pdf

(b)
(a)(1) A real city picture

Fig. 8 Moving the projector in the tool for avoiding the obstacles

The simulation details of the projection moving are C. Discussion with 3D City Map Model
presented on YouTube [8], [9]. Refer to the links for more The simulation tool depends on the 3D map model. The real
details. In this manner, we can simulate the condition of the buildings and real city equipment are important to the
movie projection on the projection mapping event. Thus, the simulation. In reality, some projection mapping events cannot
projector moving can be simulated without the workload and be simulated in the tool. The reason is that the 3D city model
cost of developers. The simulation tool is useful to reconfirm was not adapted to the simulation model. For example, if the
the projection conditions. building surface is curved, the 3D city model from the Mapbox
B. Discussion with Performance of a Projector does not fit the simulation tool and the curved surfaces of the
buildings are omitted from the 3D city model. It implies that the
Usually, we use a high-level projector in the real event
projection mapping areas with curved surfaces cannot be
because the high luminance and resolution of a projector are
adapted to the simulation tool. The 3D city map model from
required. The high-level projector has various high-
Google is supported to the curved surfaces of buildings.
performance functions, such as lens-shift, automapping, and
However, the cost of the 3D city model of Google is high.
multiprojection. We often use such high-performance functions
Although famous tourist spots are available in the 3D city
of a high-level projector. However, a projector component of
model in Google, the local areas that are not famous tourist
Unity does not have such high-performance functions. The
spots are not available to the simulation tool. Therefore, we
difference between a projector component of Unity and a real
should construct a 3D city model with real buildings. There are
high-level projector is a significant problem in useful
several software that can construct a 3D model from the
projection simulations. Especially, the lens-shift function is
pictures and movies a user takes. The function of making a 3D
essential for determining projector position. The lens-shift
model from original pictures and movies will be embedded in
function implies that the projection angle can be moved up,
the simulation tool to improve the generality of the tool.
down, left, and right by approximately 40% with only lens
angles. This function is very useful when the projector position
V. SUMMARY
is limited by various obstacles, such as fences, trees, and utility
poles. Therefore, we must implement the lens-shift function in We propose a simulation tool for projection mapping using
the simulation tool. Further, the angles of the projector body Mapbox and Unity. The simulation tool has four main
and lens differ. The maximum value of the lens-shift angle functions: the building modeling function, movie projection
differs for each projector. Therefore, the simulation tool will function, sending movie function from PC to the projector, and
implement a lens-shift function so that the maximum angle of mapping function of fitting building surfaces. The
the lens-shift can be set to various values. characteristics are using a real city 3D model from Mapbox and
the virtual projector moving in the simulation tool.

International Scholarly and Scientific Research & Innovation 15(8) 2021 522 ISNI:0000000091950263
 World Academy of Science, Engineering and Technology
International Journal of Computer and Information Engineering
Vol:15, No:8, 2021

The tool was adapted to a real projection mapping event held He is an associate professor in a university of Japan. Journal articles are
[1] “Process evaluation based on meeting quality of requirement analysis
in November 2019. The tent roof obstacles could be phase in software development projects,” International  Journal of
reconstructed in the simulation tool. The virtual projector Software Engineering and Applications, vol. 7, no. 10, Sep. 2014, pp.
component moved to the other position that avoided the tent 828–843.
[2] “A process complexity- product quality (PCPQ) model based on process
roof obstacles in the simulation tool. We can try setting the fragment with workflow management tables,” International Conference
projector in the tool to investigate good positions without on Product Software Development and Process Improvement Profes2011,
visiting the real target building. Moreover, the simulation tool pp. 171–185, Jun. 2011.
is useful for reconfirming movie projection conditions even [3] “Process Fragment Based Process Complexity with Workflow
Management Tables,” International Workshop on Empirical Software
with various obstacles, such as planting trees, utility poles, and Engineering in Practice IWESEP2010, pp. 7–12, Dec. 2010㸬
event stall tents. He is interested in software process and computer system development
In future, the projector component will be added to the new techniques.
Assoc. Prof. Dr. Obana, Masaki, member of IPSJ JSSST.
functions of a real high-level projector, such as lens-shift,
automapping, and multiprojections. Moreover, the original 3D
map model-generating function will be implemented using the
original photos and movies of target buildings.
Open Science Index, Computer and Information Engineering Vol:15, No:8, 2021 publications.waset.org/10012238/pdf

ACKNOWLEDGMENT
A part of the work is supported by the research foundation of
Hannan University. Mr. Mitsuda implemented a part of the
simulation tool.

REFERENCES
[1] N. Hanakawa, “An education model for project-based learning using
ability map in various target products,” the 27th Asia Pac. Software Eng.
Conf. (APSEC2020), Dec. 2020, pp. 296–305.
[2] Available at: https://docs.mapbox.com/jp/android/maps/overview/,
17/07/2021.
[3] Available at: https://unity.com/ja, 17/07/2021.
[4] G. Qinglian and N. Matsui, “Three indispensable shapes supporting a
successful projection mapping,” Kit Prog., vol. 27, Mar. 21–30, 2019.
[5] G. Qinglian and H. Nagasse, “Component technologies and education
activities required for projection mapping,” Kit Prog., vol. 26, 41–51
Mar, 2018.
[6] K. Battulga and T. Fujimoto, “View-dependent projection mapping
enhanced by real background,” J. Soc. Art Sci., vol. 20, no. 1, pp. 55–71.
[7] S. Shimazu, D. Iwai, and K. Sato, “Optimal placement of multiple
projectors for projection based mixed reality by considering spatial
resolution,” Transaction of Virtual Reality Society Japan, vol. 17, no. 3.
261–268, 2012.
[8] Available at: https://youtu.be/Zp6ITYW93ok, 17/07/2021.
[9] Available at: https://youtu.be/WLORt17r4gA, 17/07/2021.

Noriko Hanakawa, the first author, was born in Japan on Aug. 21, 1961. The
highest educational qualification is a doctor’s degree from the Nara Institute of
Science and Technology in Japan. Ph.D. degree of engineering, Nara, Japan,
2000. Major Field is software engineering.
She is a professor in a university of Japan. Journal articles are:
[1] “An education model for project-based learning using ability map in
various target products,” the 27th Asia-Pacific Software Engineering
Conference (APSEC2020), Dec. 2020, pp. 296–305.
[2] “Towards Integrating Software Development and Multimedia Content
Creation,” Ninth International Symposium on Business Modeling and
Software Design, June 2019, pp. 188–200.
[3] “A Computer System Quality metric for Infrastructure with Configuration
Files’ Changes,” 2019 International Conference on Software Engineering
and Information Management (ICSIM 2019), B1-0019, pp. 39–43, Jan.
2019㸬
She is interested in simulation and software processes and multimedia
techniques.
Prof. Dr. Hanakawa, Noriko, member of IEEE, IPSJ JSSST, IEICE.

Masaki Obana, the second author, was born in Japan on Sep. 3, 1984. The
highest educational qualification is a doctor’s degree from the Nara Institute of
Science and Technology in Japan. Ph.D. degree of engineering, Nara, Japan,
2013. Major Field is software engineering.

International Scholarly and Scientific Research & Innovation 15(8) 2021 523 ISNI:0000000091950263