Journal of Pharmaceutical Sciences 106 (2017) 3438-3441

Contents lists available at ScienceDirect

Journal of Pharmaceutical Sciences

journal homepage: www.jpharmsci.org

Special Topic Commentary

Mixed Reality Meets Pharmaceutical Development

William P. Forrest 1, *, Megan A. Mackey 1, Vivek M. Shah 1, Kerry M. Hassell 2,
Prashant Shah 3, Jennifer L. Wylie 1, Janakiraman Gopinath 3, Henning Balderhaar 4, Li Li 1,
W. Peter Wuelfing 1, Roy Helmy 1
1
Analytical Sciences, MRL, Merck & Co., Inc., Rahway, New Jersey 07065
2
Chromatography and Mass Spectrometry Division, ThermoFisher Scientific, Somerset, New Jersey 08873
3
IT Architecture, MRL, Merck & Co., Inc., Rahway, New Jersey 07065
4
Analytical Development Commercialization-Pharm, MMD, Merck Sharp & Dohme, Schachen, Switzerland

a r t i c l e i n f o a b s t r a c t

Article history: As science evolves, the need for more efficient and innovative knowledge transfer capabilities becomes
Received 20 July 2017 evident. Advances in drug discovery and delivery sciences have directly impacted the pharmaceutical
Revised 24 August 2017 industry, though the added complexities have not shortened the development process. These added
Accepted 25 August 2017
complexities also make it difficult for scientists to rapidly and effectively transfer knowledge to offset the
Available online 1 September 2017
lengthened drug development timelines. While webcams, camera phones, and iPads have been explored
as potential new methods of real-time information sharing, the non-“hands-free” nature and lack of
Keywords:
Analytical chemistry viewer and observer point-of-view render them unsuitable for the R&D laboratory or manufacturing
dissolution setting. As an alternative solution, the Microsoft HoloLens mixed-reality headset was evaluated as a more
formulation efficient, hands-free method of knowledge transfer and information sharing. After completing a tradi-
process analytical technology (PAT)
tional method transfer between 3 R&D sites (Rahway, NJ; West Point, PA and Schnachen, Switzerland), a
microprocessors
high throughput technologies retrospective analysis of efficiency gain was performed through the comparison of a mock method
image analysis transfer between NJ and PA sites using the HoloLens. The results demonstrated a minimum 10-fold gain
analysis in efficiency, weighing in from a savings in time, cost, and the ability to have real-time data analysis and
discussion. In addition, other use cases were evaluated involving vendor and contract research/
manufacturing organizations.
© 2017 American Pharmacists Association®. Published by Elsevier Inc. All rights reserved.

Introduction advances in drug delivery systems, targeted modalities, and

increased regulatory requirements have greatly contributed to the
The 18th Century Age of Enlightenment, an intellectual move- lengthening of drug development timelines. With these added
ment responsible for the progression of science, flourished with the complexities, scientists are continually confronted in their
rise of the print culture and open forums to share information. everyday routines with the problem of rapidly communicating
Today, we are on the cusp of another “Age of Enlightenment”, with complex experimental data, observations, and theories. This type of
immersive (virtual [VR], augmented [AR], and mixed realities [MR]) scientific information is often best shared with pictures or video.
technologies ushering in a new era of functional communication Abstract theories are traditionally illustrated with diagrams or
technologies. The cornerstone of evolving science and business is models, and background information filled with technical jargon is
fundamentally governed by knowledge management; how infor- usually transmitted through books, journal articles, and standard
mation is generated, stored, and transferred. operating procedures (SOPs). Can immersive technologies help
As in many industries, inefficient knowledge transfer can derail with this problem?
key business and R&D processes. An example of this was recently Every reader has more than likely heard of the international
highlighted by the QuintilesIMS Institute and STAT in a study phenomenon Poke mon GO, which is an AR-based smartphone
demonstrating that drug development is no faster today than it was game where players can superimpose Poke mon characters onto the
back in the 1990s (1). How is this possible? It is undeniable that real world that they are viewing on their phone screens. In addition
to the gaming sector, AR (Google Glass and Epson's Moverio)
Conflicts of interest: The authors declare no competing financial interest.
technology's unique ability to blend virtual items into a real world
* Correspondence to: W. Peter Forrest (Telephone/Fax: þ1-7325945315). view have made it applicable to various fields such as medical and
E-mail address: [email protected] (W.P. Forrest). surgical training,1-4 combat training,5,6 manufacturing,7-10

http://dx.doi.org/10.1016/j.xphs.2017.08.020
0022-3549/© 2017 American Pharmacists Association®. Published by Elsevier Inc. All rights reserved.
 W.P. Forrest et al. / Journal of Pharmaceutical Sciences 106 (2017) 3438-3441 3439

architecture,11 journalism,12 and interior design.13 A recent com- Wireless broadband internet (802.11ac) or mobile Wi-Fi hotspot
mentary by the Editor of C&EN magazine highlighted the re-advent (4G) was utilized for data transfer and video-streaming services. Data
of VR and AR- “smart glasses” such as Google Glass and their ability visualization and sharing was conducted using the Microsoft Excel
to be potential “game changers” for scientists.14 Emerging as a software plug-in. SOP and protocol visualization were conducted
combination of AR and VR technologies, MR technology is poten- using Adobe PDF Viewer and Microsoft Word software plug-ins.
tially better equipped to address the needs of the modern day
scientist by enhancing perception of virtual data in real space. MR
technology seemlessly blends both synthetic (virtual) and natural Results and Discussion
light thus providing a scientist an unobstructed view of their lab-
oratory surroundings (an added safety benefit). An example of such To evaluate the capabilities of the HoloLens for application to the
MR technology is the Microsoft HoloLens (Microsoft Corporation, laboratory setting, a “mock” method transfer between R&D scien-
Redmond, WA) mixed-reality headset. Utilizing the HoloLens, a tists from 2 different MSD sites (Rahway, NJ and West Point, PA)
scientist would be able to connect to his or her target persons to was performed. It should be noted that while the “mock” experi-
perform an experiment and view the associated data all in real ment chosen to be conducted herein was an analytical method
time. MR also allows the target person to manipulate the scientist's transfer, many other CMC-based applications can be envisioned.
field-of-view through annotation or by placing virtual data in the Several of which are currently under investigation within MSD labs.
altered environment all while communicating like a live-stream The most attractive capability of the HoloLens MR headset is the
video call. To date, MR technology has found practical applica- ability to live-stream video to a target person's laptop (or any Skype
tions within the oil industry, architecture, and construction. enable device), rather than another head-mounted display.15 This
One specific application in the pharmaceutical industry where gives a “birds-eye” view to the target person from wherever they
we envision MR technology to be greatly beneficial is the transfer are located, while also allowing the scientist in the laboratory to see
of experimental methods or processes across multiple labora- and converse with the target person. This was the first functionality
tories around the globe. Method transfers can span many core to be tested. Using the Skype beta plug-in for HoloLens, the scientist
areas. Of particular interest is chemistry, manufacturing, and in the laboratory in NJ started a live video call to the colleague in
control (CMC) activities which include synthesis (both chemical their office in PA (as depicted in Fig. 1). The “hands-free” head-
and biological), analytical and formulation development, engi- mounted display nature of the HoloLens allowed the scientist to
neering, manufacturing, packaging, and shipping. All of which are continue to move freely throughout the laboratory, interacting with
critical for pipeline drug discovery and development activities. equipment and another scientist while still communicating with
The current landscape of information sharing leaves room for the colleague in PA. The colleague in PA was able to open an Excel
improper interpretation of written protocols/procedures resulting data file from her computer and then share it instantaneously with
in inefficient method transfer and ultimately generating the road the scientist by placing it in the scientist's 3D MR space. The
blocks during drug product development. opposite scenario was also performed where the scientist opened a
PDF file and placed it into the 3D space for viewing by both parties.
It should be noted that the digital windows (Excel plot and PDF file)
Materials and Methods depicted in Figure 1 can be moved and placed anywhere in the
field-of-view of the wearer. They can be pinned to one location or
All case studies were performed using developer edition allowed to travel with the user. It should also be noted that all
HoloLens MR headsets operating on a Windows 10 platform. digital data images that are viewed through the HoloLens are in an
HoloLens to computer (Lenovo T450, Windows 10) live video essence “see through” allowing for an unobstructed viewing field,
streaming was conducted using the Skype beta software plug-in. which is critical for a laboratory scientist (Fig. 2). To support the use

Figure 1. Theoretical rendering of the capabilities of the HoloLens (a mixed-reality headset) which was evaluated for its utility in assisting method transfer. The figure illustrates a
hypothetical example of a method transfer from one MSD R&D location to another.
3440 W.P. Forrest et al. / Journal of Pharmaceutical Sciences 106 (2017) 3438-3441

first example was the realization of the need for a specialty holder
for the dosage form during the IVR test which was not originally
needed during testing in the R&D laboratory. It was discovered by
the GMP scientist that the dosage form needed to be suspended in
the media in such a manner that there is no interference with the
autosampling and media replacement needle, and it is not floating
to the top or sinking to the bottom of the vessel, where it would be
in contact with a stir bar. The GMP scientist designed several
custom sample holders for this specific dosage form and showed
them to the R&D scientist for her input through traditional
methods such as emailed photos and webcam. However, utilizing
the HoloLens, the GMP scientist would have been able to not only
share the designs but demonstrate how each worked in the auto-
Figure 2. Illustrative example of the user's view through the HoloLens showing the mated testing setup. This would have provided the R&D scientist
“see-through” nature of the digital data in the 3D space. with a first-hand view to advise in real time on necessary design
modifications based on their expertise regarding the chemical/
physical properties of the dosage form and the release mechanism,
of the HoloLens in the laboratory setting, MSD IT developed the thus simplifying the process.
MSD HoloBot, an intelligent personal assistant. The MSD HoloBot Once the precise details of the IVR testing method transfer were
allows the scientist to access files stored on Microsoft OneDrive by confirmed between the R&D scientist and GMP scientist and testing
use of voice command, allowing recall of PDFs, Microsoft Word and began, another instance was identified where the HoloLens could
Excel files, etc… This enables the scientist to remain “hands-free” have helped significantly. During testing, data were constantly
and contribute to an increase in laboratory safetydmore or less being shared by email or in web-based meetings by the GMP sci-
“Siri for the lab” (Fig. 3). entist. Any inconsistencies that were observed between the new
The last feature of the HoloLens that was tested was the ability and the previously generated release were discussed and analyzed
for both parties to annotate. As is shown in Figure 1 (orange circle), after the occurrence. This again is where the utilization of the
the colleague in PA was able to use her cursor to highlight certain HoloLens would have been ideal. The R&D scientist could have been
focus areas within the scientist's 3D space. Once again, the reverse observing the testing in real time from their office in the United
scenario was also performed. Unlike other digital data that are States, whereas the GMP scientist shows the current, up-to-date
placed into the HoloLens user's 3D environment, annotations are data from the lab without going back to their office to email the
fixed to a certain location. This again allowed the scientist to move data. As the GMP scientist would be troubleshooting the cause of
freely throughout the lab but return to the original 3D setting and the data inconsistencies, given the R&D scientist's expertise, they
still view the highlighted annotations that were placed by the would be able to walk the GMP scientist through potential problem
colleague in PA. Now having first-hand experience with the capa- areas using the live-viewing, annotation, and document-viewing
bilities of the HoloLens, a team retrospectively analyzed a tradi- functions. Additional to data being shared, the SOPs for daily veri-
tional method transfer process that was completed between fication of the automated IVR testing system and method protocols
multiple MSD R&D laboratories in the United States and a com- could have been displayed in the GMP scientist 3D MR setting for
mercial GMP (good manufacturing practice) facility in Switzerland constant referral, increasing compliance. The workflow involving
and then identified where the HoloLens technology could have the HoloLens would have resulted in a substantial reduction in
been utilized to make this process more efficient. multiple scientists' time and capital required for travel, which
The traditional method transfer was completed for an in vitro allowed for a faster turn-around time and mitigation of complex-
release (IVR) study which involved multiple meetings, pre- ities or delays to the clinical batch release. It is estimated that the
sentations, and an on-site visit. Besides the obvious reduction in method transfer process using the HoloLens technology was
travel time and cost associated with sending the R&D scientist to minimally 10 times more efficient than the traditional method
the GMP facility in Switzerland for 1 week to assist with the transfer process. While definitive quantification of the efficiency is
method transfer process, several examples where the HoloLens difficult given multiple contributions to a single efficiency gain
could have made the process more efficient were the identified. The value, the rationale behind the estimation is simply rooted in the
time savings. While the mock method transfer was conducted over
the course of 1 h, the total travel time from the United States to
Switzerland for the analyst was greater than 12 h.
Another cornerstone of the pharmaceutical R&D process is
based on interactions with external partners (scientific vendors and
Contract Research/Manufacturing Organizations). Collaborating
with our colleagues at ThermoFisher Scientific, we were able to
expand this idea of using MR technology for method transfers to
the broader scientific community. From the vendor side, many
resources go into providing the customer with the best service to
make them successful with their science. For instance, if MSD
received a liquid chromatography/mass spectrometry (LC/MS)
platform, a scientist may observe troublesome data following the
installation and initial hands-on training. After a period of time
tinkering with the instrument, the scientist has to schedule a
service call. The lead time for the field service engineer to come out
Figure 3. Illustrative example of the MSD HoloBot app used for document/data and take a look can be up to 1 month, depending on demand. All
retrieval by voice-commands. this time, the instrument is down and critical data are not being
 W.P. Forrest et al. / Journal of Pharmaceutical Sciences 106 (2017) 3438-3441 3441

generated and the program is at a stand-still. This is another place transfer/translate methods and troubleshoot complications, which
where we envision the HoloLens could come to the rescue. The would ultimately prevent delays in clinical timelines.
MSD scientist would be able to use the HoloLens to give the Ther-
moFisher application scientist a “birds-eye view” of the LC/MS Acknowledgments
platform, allowing the ThermoFisher expert to walk the MSD sci-
entist through a routine troubleshooting checklist. Through this The authors would like to acknowledge Sharon O'Brien from
process the cause of the troublesome data could be identified in a MSD Creative Studios for her assistance with the photography and
fraction of the time as compared with the time it would have taken figure renderings and A. Minhas from MSD IT Architecture for
for the service engineer to come on site and diagnose the simple assistance with the MSD HoloBot.
issue. In addition, it is envisioned that in the future, all customers
could potentially be provided with the HoloLens, opening instant References
communication with a ThermoFisher expert for troubleshooting
and data interpretation as if they were really there. 1. Peden RG, Mercer R, Tatham AJ. The use of head-mounted display eyeglasses
for teaching surgical skills: a prospective randomised study. Int J Surg. 2016;34:
169-173.
2. Pensieri C, Pennacchini M. Virtual reality in medicine. In: Sivan Y, ed. Handbook
Conclusions on 3D3C Platforms: Applications and Tools for Three Dimensional Systems for
Community, Creation and Commerce. New York, NY: Springer International
Publishing; 2015:353-401.
While the technology has many unambiguous promotable
3. Rankin TM, Slepian MJ, Armstrong DG. Augmented reality in surgery. In:
advantages for implementation into an R&D setting, there are some Latifi R, Rhee P, Gruessner RWG, eds. Technological Advances in Surgery, Trauma
drawbacks. The headset is quite heavy (particularly for smaller and Critical Care. New York, NY: Springer New York; 2015:59-71.
4. Rosen JM, Laub Jr DR, Pieper SD, et al. Proceedings - Virtual Reality Annual In-
individuals) and when worn for extended periods of time (>1 h) it
ternational Symposium. Los Alamitos, CA: IEEE; 1996:5-13.
causes a distinct red irritation mark on the user's forehead. 5. Livingston M, Brown D, Gabbard J, et al. An Augmented Reality System for Mil-
Furthermore, the “glasses” nature of the headset does not render itary Operations in Urban Terrain. Orlando, FL: Defense Technical Information
them as approved personal protective equipment, meaning normal Center; 2002.
6. Rosenblum A. Augmented reality glasses are coming to the battlefield. Popular
chemical and splash resistant lab safety goggles still need to be Science. 2015. Available at: http://www.popsci.com/experimental-ar-glasses-
worn underneath the headset. As the HoloLens is a wearable offer-marines-hands-free-intel. Accessed September 21, 2017.
computer, the sharing of confidential or proprietary information 7. Khan WA, Raouf A, Cheng K. Augmented Reality for Manufacturing. Virtual
Manufacturing. London: Springer London; 2011:1-56.
between the headset and other devices (within the company or 8. Michael H, Mark B, Bruce T, Holger R. Industrial augmented reality applications.
external partner groups) via the internet is also a key issue. In: Emerging Technologies of Augmented Reality: Interfaces and Design. Hershey,
Potential solutions for this problem should be fairly minimal as PA: IGI Global; 2007:283-304.
9. Nee AYC, Ong SK, Chryssolouris G, Mourtzis D. Augmented reality applications
long as the headset operates on a secure network. Another poten- in design and manufacturing. CIRP Ann Manuf Techn. 2012;61(2):657-679.
tial drawback for the headset would be regarding its implentation 10. Novak-Marcincin J, Barna J, Janak M, Novakova-Marcincinova L. Augmented
in cGMP activities. It is the viewpoint of the authors that the Hol- reality aided manufacturing. Procedia Comput Sci. 2013;25:23-31.
11. Dunston PS, Wang X. Mixed reality-based visualization interfaces for archi-
oLens is not a data generator rather a portal through which the data tecture, engineering, and construction industry. J Constr Eng Manage.
can be viewed, shared, and interpreted (much like a projector). 2005;131(12):1301-1309.
Thus, rendering the HoloLens suitable for cGMP laboratory activ- 12. John VP, Frank B. The emergence of augmented reality (AR) as a storytelling
medium in journalism. Journal Commun Monogr. 2013;15(1):4-59.
ities. Overall, we have experienced many cases in pharmaceutical
13. Ridden P. IKEA catalog uses augmented reality to give a virtual preview of
R&D where the HoloLens would significantly impact how we furniture in a room. New Atlas. 2013. Available at: http://newatlas.com/ikea-
communicate and share information in real time. This is most augmented-reality-catalog-app/28703/. Accessed September 21, 2017.
noteworthy when communication between scientists at different 14. Campos B. Smart glasses are back. Chem Eng News. 2017;95(31):2.
15. Henry C, Austin SL, Mark S, John CT. 3D collaboration method over HoloLens
sites and companies is critical. The HoloLens would cut down on and Skype end points. In: Proceedings of the 3rd International Workshop on
the time and resources typically required (specifically for travel) to Immersive Media Experiences. Brisbane, Australia: ACM; 2015.