Realtime Adaptive Virtual Reality for Pain

Reduction
Alexiei Dingli Luca Bondin
Department of Artificial Intelligence Department of Artificial Intelligence
University of Malta University of Malta
[email protected] [email protected]

Abstract— Recent years have seen digital game mediums and provide a personalized experience that goes beyond
taking conventional amusement, entertainment and leisure anything available to the public up until today.
industries by storm. They have revolutionized the system to the The upcoming sections present research critical to the
extent that the industry cannot now even dream to do without study and the methodology to reach the desired final
this overwhelming reality. The same game mediums that have objectives.
capitalized on intrinsic leisure aspects have simultaneously
focused with equal vigor on other equally, if not more, II. LITERATURE REVIEW
important collateral objectives. This paper builds on this
A. Related Work
concept and discusses a work in progress currently being
carried out at the University of Malta. It proposes the use of It is widely acknowledged that by stimulating the visual,
games as a means of distraction therapy for individuals auditory, and proprioception senses, VR acts as a distraction to
undergoing painful clinical treatment procedures. The creation of limit the user’s processing of nociceptive stimuli [3]. The idea
an adaptive Virtual Reality (VR) game within an Artificial of using VR as a distraction tool during patient
Intelligence framework will without doubt be of a significantly rehabilitation and treatment procedures is not entirely new.
greater benefit to the community than mere entertainment VR coupled with medication, was found to be effective
applications. in reducing the pain symptoms of patients during bandage
changes for severe burns [4,5]. In other cases, VR has been
Keywords—Artificial Intelligence, Affective Computing, used as a tool to distract young children during vaccination.
Adaptive Games, Serious Games, Virtual Reality, Deep Ogilvy Brasil designed a game whereby a child about to be
Learning vaccinated is transported into a virtual world. The patient
assumes the role of a hero who is about to receive a powerful
I. INTRODUCTION shield to beat the villain in the virtual environment. The
There is an ever-growing realization that the power games experience that the child goes through is matched step by step
have to engage and motivate individuals to accomplish by the nurse that vaccinates the child. In this case, the VR
specific tasks is by far more attractive and challenging than game was found to significantly reduce the elements of fear
that of any other medium previously explored. This ability to and anxiety that the child usually goes through during such
engage individuals provides a platform to gain experiences procedures.
that go far beyond the expectation of what one had come to The work done by Kind VR [6] whose work is
expect from games for decades. In light of this, research has documented by Birnie et al. and Agrawal et al. is one of
started looking at concepts such as ‘Serious Games’ and particular interest. In the work by Agrawal et. al. for example,
‘Gamification’. This realization is further corroborated by the the patient is immersed in an underwater scenario and can
increasing number of games with a purpose that are being navigate around and interact with the environment during the
released in a wide range of application areas. Perhaps one of therapy procedure which is similar to what is being proposed
the most popular examples of the use of games for a specific by this study. More importantly, however was that both
purpose is Microsoft’s Flight Simulator which gave players a documented studies showed positive results in terms of
unique and realistic experience into how to fly aircraft, and acceptability and effectiveness.
inspired many of the pilots of tomorrow. Similarly, gamified
experiences have been used in healthcare [6,19], helping B. Affective Computing
patients in both physical and mental health, and in the field of When going through present literature one trend becomes
classroom education. glaringly apparent. While studies show positive results in
The motivation behind using VR as a tool to help patients terms of effectiveness in what they want to achieve, they all
is a very strong one mainly because VR offers a safe way with seem to adopt a one size fits all approach. The direct result of
which to help individuals without the side effects that other this could be that while some patients might be positively
medication might have on the patient. Distraction therapy affected by the VR experience, other individuals may be
through the use of VR games has recently been given an completely unaffected, or worse, incur a negative experience
increased degree of importance. Companies and researchers that would cause more distress and make treatment more
have started looking at this field more thoroughly [1, 2]. In our dreadful. For this reason, we are proposing the notion of an
study, however, we want to take what has been done already adaptive game driven by Affective Computing (AC).
to the next level by means of Artificial Intelligence (AI). AC is the study of trying to assign computers the human-
Through the addition of AI concepts such as Affective like capabilities of observation, interpretation and generation
Computing, we envisage the creation of a game that is able to of affect features [7], and is currently one of the most active
adapt itself according to the affective state of the individual, research topics. It is an important topic for the harmonious

978-1-7281-1884-0/19/$31.00 ©2019 IEEE

human-computer interaction that aims to increase the quality elements are the participants, the audience that will be making
of human-computer communication and improve the overall use of the VR experience; the creators themselves, that need to
intelligence of the computer. Software is nowadays able cater for the audience they are targeting; and the virtual
to detect human emotions with remarkable accuracy environment where participants will find themselves in once
through sensing manifestations of physiology, speech they are making use of the VR. The most important in the
and body motion amongst others. Through analysis of context of this research are perhaps immersion and
this data, software is now capable of inferring the emotional interactivity. Immersion is considered the degree to which an
state one finds himself in. Yannakakis [8] goes as far as individual feels engrossed or enveloped within the virtual
describing AC as a multidisciplinary field that studies various environment. In addition to immersion, researchers also take
ways by which computational processes are able to elicit, into consideration the element of presence which is defined “as
sense and detect manifestations of human emotion. the experience of one’s physical environment; It refers not to
one’s surroundings as they exist in the physical world, but to
When discussing AC, the main concern is the the perception of those surroundings as mediated by both
implementation of what is known as the affective loop [9]. The automatic and controlled mental processes” [12]. While
affective loop is a four-stage loop that defines the key phases presence is the subjective value of the illusion one experiences
with which to achieve affect enabled software and is made up when using the system, an increase in immersion often leads
of the below phases; to an increase in the presence felt by the user [13, 14].
1. Affect Elicitation phase Interactivity is the element that sets virtual reality apart from
other communication mediums. For VR to seem authentic, it
2. Affect Sensing phase should be able to respond to an individual’s actions like a real-
world environment would typically react, and herein lies
3. Affect Modeling phase
perhaps the biggest challenge in the creation of an effective
4. Affect Driven Adaptation phase and truly immersive virtual experience.
The Affect Elicitation phase, as the name suggests, is the In the same manner designing for Gamification requires
phase that involves evoking and eliciting emotions in a human particular considerations if it is to be ‘meaningful’.
through various mediums such as images, sound and video. ‘Meaningful gamification’ signifies that it can provide
The Affect Sensing phase follows whereby the software, genuine motivation and is effective on the individual. Among
having successfully elicited an emotion in an individual, must the elements that require consideration when designing for
be equally capable of knowing how to capture that emotion for gamification are engagement and choice. These two elements
can be considered to be the most in line with what was
further processing through measurement of physiological discussed previously in relation to interaction within the
responses. The Affect Modeling phase is the phase which context of virtual reality. The element of engagement defines
makes sense of the physiological data. Calvo & Mello [10] both the game’s ability to provide an engaging gameplay
further dissect this phase into a signal processing phase, experience and the game’s ability to create opportunities for
a feature extraction phase and a machine learning phase. players to engage with others in meaningful ways. People
Finally, completing the loop is the Affect Driven Adaptation have a more positive mental well-being when they feel
phase which is the phase where the software starts altering its connected to the world around them [18] and therefore
behavior according to the affective state of the individual. engagement on this level is crucial to the effectiveness of a
Adaptation can take place through a multitude of approaches gamified system. When discussing engagement, Nicholson
mostly depending on what the software in question is, for reintroduces the concept of flow. Many gamification systems,
Nicholson argues, do not get more challenging which results
example if the affective computing has been put in place in an element of boredom. Engagement, in this context,
inside a game, adaptation might occur through alteration in the is reached when challenges match the skill level of the player.
digital content presented to the individual. Finally, Choice is another crucial element. Choice puts the
C. Virtual Reality & Gamification player in control of how one engages with the gamified system
and stems from the reasoning that a person will have a more
Two key areas of study in the context of this research are positive sense of self-being if one has autonomy. This
VR and Gamification. The reason why these two areas are autonomy often translates in a player being able to make
discussed together in this section is that VR has provided a meaningful choices within the system.
platform for Gamification to thrive on. VR offers capabilities
that are ideal for the design and implementation of effective III. METHODOLOGY
Gamification. Virtual reality is one of the areas attracting the Having highlighted the main areas of research for this
interest of AI researchers for its enormous potential and the study, this paper will in this section discuss the methodology
countless scenarios where it can be deployed. The ability to behind the implementation of the system being proposed.
transport an individual to a completely different world which
can be controlled and with which the user can interact
has provided researchers a very useful tool that can be utilized
at fairly limited costs. On the other hand, Gamification is
the concept of using game elements to motivate an individual
into performing a particular task to achieve a predetermined
goal. Gamification uses game-based elements and strategies
to increase engagement, motivation, learning, and even
solve problems.
When discussing the implementation of VR systems
Sherman & Craig [11] mention five elements that are crucial
to the creation of a VR experience. The first three of these five
 cater for prior to risking the patient getting alienated from the
A. Design Specification game and starting to focus on the pain symptoms instead.
In the scheme of things being proposed, there exist 3 key
components that make up the system. First is the patient, who The second sub-component is the game controller itself. Its
function is to react to the adaptation cues that are the result of
will be making use of the VR experience during the procedure, the Affect Detection sub-component. The component alters
second is a wearable device that will read bio-feedback from dynamics of the game that are identified as influential and
the individual, and finally the VR game itself that will act as relevant to the changes in the patient’s affective state.
the distraction therapy tool for the patient. The general idea is
quite straightforward; the patient will wear the wearable B. Biofeedback sensing
device during the therapy procedure, and the wearable device Biofeedback sensing is one of the main topics of
will in turn read bio-signal data off the patient, to promptly discussion within this study. It forms the basis of all that is to
feed the game that adapts its behavior to keep the patient be achieved by the proposed system and drives its affective
engaged. This loop remains active until the patient effectively computing capabilities. While this study opts for a wearable
decides to stop using the VR experience. device approach to gather the data, other researchers have
investigated other approaches to bio feedback data gathering.
Dingli & Giordimaina [15], for example opt for a non-contact
approach, making use of webcams to monitor the heart rate of
the individual by extracting the RGB channels from the face
region. In the context of this study however, this was not
Realtime Affective State
Estimation Controller
Game Mechanics
Controller
suitable, since the individual’s face will be covered by the VR
Emotion
headset most of the time. As a result, a contact-based
Biofeedback
from wearable Estimation
Updated Game
dynamics approach was adopted, where the data is retrieved by smart-
device watch like devices.
For the first iteration of the research, only heart-rate data
is considered. Several researchers have highlighted other bio-
Figure 1: Description of the components that make up the game feedback data relevant to the goals to be achieved. In line with
module (Source: Dingli & Bondin, Realtime Adaptive Virtual this consideration was taken of what Picard & Healey [16]
Reality for Pain Reduction) describe as “affective wearables”, that is, wearable devices
capable of reading various physiological data from the
Naturally, most of the processing that will take place individual. In addition to heart-rate data, other researchers
during execution of the system will be handled by the game have investigated the use of galvanic skin response, an
component. In order to make the implementation of the indicator of skin conductance that increases linearly with
component more understandable and manageable, the tasks a person’s level of overall arousal, and Electromyography,
that the game component must perform are assigned to two which measures muscle activity and has been shown to
separate sub-components residing in the game. correlate with negatively valanced emotions [17]. However,
First, an Affect Detection sub-component is tasked with the acquisition of such data usually requires specialized
receiving the information from the wearable device, and equipment which is not always easily accessible by the public
performing actions akin to the phases that make up the and therefore limits the audience that the system might
otherwise reach. Hence, a decision was made to make use of
affective loop discussed earlier. In short, these tasks will heart-rate data since heart rate sensors are commonly available
include cleaning the bio-signal data coming in, training a in many of the commercially available smart watches on the
model that is able to detect the affective state of the individual market.
and drive the adaptation in the VR experience. In addition to
these tasks in line with what was proposed by Sundstrom [9] C. Affect Modeling
an extra phase is to be added that will act as a prediction phase. There exist many theories as to how affect can be modeled.
This prediction phase will predict the individual’s affective While some researchers attempt to classify emotions into a
state as he or she goes through the VR experience. The idea is number of categories, such as “joy” and “anger”, recent work
that in time, the game will be driven by this prediction rather tends to focus more on sensing the state one is in, for instance,
than by the bio-feedback data coming in from the wearable when all is going well or when going badly. It is on this line of
device. When the patient is making use of the system, thought that the research is built. It is important for the
the predictor will unobtrusively be trained to predict the system to notice the affective states the individual finds
himself in when faced with specific events and then model the
patient’s affective state journey. For this task, we propose the experience accordingly.
use of Generative Adversarial Networks (GANs). During the
first runs, the predictor will in all probability fare relatively In order to achieve this, a classifier is first trained to
badly, but performance is expected to improve drastically recognize valence and arousal on data belonging to a dataset
after a number of runs which refine the predictor’s results. obtained online. The idea is that at the end of this training
Once the predictor’s accuracy reaches sufficiently accurate phase, the system will be able to make sense of the stream of
levels, the main driving behind the adaptation of the game will heart-rate data being fed. With this classifier in place the game
shift from the data being read from the wearable to the data can start recording the user’s affective journey and associate
generated by the predictor. This does not mean that the affective states with particular events occurring in the VR
wearable will be eliminated completely from the equation, but experience, i.e modeling. The game will know that, for
that it will now act as a validator of the predictor rather than example, putting the participant in an environment high up in
being the main actor. Through this, any changes in the the mountains has a much more calming effect on the
affective state of the individual can be anticipated before individual as opposed to putting the individual in the middle
actually occurring, and
 [1] Agrawal, A.K., Robertson, S., Litwin, L., Tringale, E., Treadwell, M.,
of a bustling metropolis. This data will then be used to achieve Hoppe, C. and Marsh, A., 2019. Virtual reality as complementary pain
affect driven adaptation. Affect modeling, in the context of this therapy in hospitalized patients with sickle cell disease. Pediatric blood
study is seen as an autonomous process that is & cancer, 66(2), p.e27525.
continuously ongoing without requiring explicit human [2] Birnie, K. A. et al. (2018) ‘Usability Testing of an Interactive Virtual
intervention. Reality Distraction Intervention to Reduce Procedural Pain in Children
and Adolescents With Cancer ’, Journal of Pediatric Oncology
D. Game Adaptation Nursing, 35(6), pp. 406–416. doi: 10.1177/1043454218782138.
[3] Li A, Montaño Z, Chen VJ, Gold JI. Virtual reality and pain
For all the good work done by the other components of the management: current trends and future directions. Pain Manag 2011
system, the game adaptation component is tasked with Mar;1(2):147-157 [FREE Full text] [doi: 10.2217/pmt.10.15]
perhaps the most important work of all. Having successfully [Medline: 21779307]
modeled affective states of an individual and “learned” what [4] Carrougher GJ, Hoffman HG, Nakamura D, Lezotte D, Soltani M,
causes arousal or valence in the individual himself, it is the Leahy L, et al. The effect of virtual reality on pain and range of motion
game itself that has to make use of this information to in adults with burn injuries. J Burn Care Res 2009;30(5):785-791
eventually reach the desired goals of the study. The idea in this [FREE Full text] [Medline: 19692911]M. Young, The Technical
case is to have an infinite runner game where scenarios and Writer’s Handbook. Mill Valley, CA: University Science, 1989.
characters change dynamically according to the affective state [5] Hoffman HG, Doctor JN, Patterson DR, Carrougher GJ, Furness TA.
of the patient. The user will be asked to perform tasks and Virtual reality as an adjunctive pain control during burn wound care in
carry out challenges along the way until the game eventually adolescent patients. Pain 2000 Mar;85(1-2):305-309. [Medline:
comes to an end concurrently with that of the therapy session. 10692634]
[6] KindVR. (2019). KindVR. [online] Available at:
The design and implementation of the game component https://www.kindvr.com/ [Accessed 24 Apr. 2019].
brings together numerous established concepts in the field of [7] Tao, J. and Tan, T., 2005, October. Affective computing: A review. In
game design most notably the theory of “flow”. Adaptation International Conference on Affective computing and intelligent
capabilities must ensure that when a game is becoming too interaction (pp. 981-995). Springer, Berlin, Heidelberg.
difficult, or too easy, game dynamics adjust accordingly so [8] Yannakakis, G.N., 2018. Enhancing health care via affective
that the user remains engaged in that tiny segment we call computing. Malta Journal of Health Sciences, (5), p.38.
“flow”. During the affect modeling phases, the system will [9] Sundström, P., 2005. Exploring the affective loop (Doctoral
dissertation).
also learn what colors, sounds, or activities have an effect on
the patient and therefore all this will be taken into [10] Calvo, R. A. & D’Mello, S. (2010), ‘Affect detection: An
interdisciplinary re-view of models, methods, and their applications’,
consideration when adapting the game in real time to cater for Affective Computing, IEEE Transactions on 1(1), 18 – 37.
the individual’s affective state at that current point in time. [11] Sherman, W.R. and Craig, A.B., 2003. Understanding virtual reality.
San Francisco, CA: Morgan Kauffman.
IV. CONCLUSION [12] Gibson, J.J., 2014. The ecological approach to visual perception:
Virtual reality as a tool for pain distraction therapy has for classic edition. Psychology Press.
long been a topic of discussion and a field of research that has [13] Hoffman, H.G., Seibel, E.J., Richards, T.L., Furness, T.A., Patterson,
been explored by numerous researchers. Studies have shown D.R. and Sharar, S.R., 2006. Virtual reality helmet display quality
influences the magnitude of virtual reality analgesia. The Journal of
VR to be effective in this area with studies reporting Pain, 7(11), pp.843-850.
significant drops in pain scores. However, we firmly believe [14] Slater, M. and Wilbur, S., 1997. A framework for immersive virtual
that more can be done in this respect. The one size fits all environments (FIVE): Speculations on the role of presence in virtual
approach that has for long characterized the implementation of environments. Presence: Teleoperators & Virtual Environments, 6(6),
such systems has provided the ground work for this area of pp.603-616.
study but has also highlighted shortcomings that need to be [15] Dingli, A. and Giordimaina, A., 2017. Webcam-based detection of
addressed to encourage wider acceptance of such treatment emotional states. The Visual Computer, 33(4), pp.459-469.
methods. For this reason, this study is proposing a novel way [16] Picard, R.W. and Healey, J., 1997. Affective wearables. Personal
Technologies, 1(4), pp.231-240.
into the creation and utilization of VR systems for pain
reduction in the belief that the approach chosen will provide [17] Nakasone, A., Prendinger, H. and Ishizuka, M., 2005, September.
Emotion recognition from electromyography and skin conductance. In
the desired results and lay the foundations for a new standard Proc. of the 5th international workshop on biosignal interpretation (pp.
in the creation of such systems. 219-222).
[18] Nicholson, S., 2015. A recipe for meaningful gamification. In
V. FUNDING Gamification in education and business (pp. 1-20). Springer, Cham.
This research is funded by the Vodafone Malta Foundation [19] Breton, Z., Zapirain, S., Zorrilla, A.: Kimentia: Kinect based tool to help
and has received backing from the Sir Anthony Mamo cognitive stimulation for individuals with dementia. In: 2012
Oncology Centre in Malta. IEEE 14th International Conference on e-Health Networking,
Applications and Services, Healthcom 2012, pp. 325–328. IEEE
REFERENCES Computer Society (2012)