UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

$''
&%!%,)!,
#&&
(%!%
&
$&#&!%

+)!%
$(*' &%
$()

Edward Jay Wang1, William Li1, Doug Hawkins4, Terry Gernsheimer3,
Colette Norby-Slycord3, Shwetak N. Patel1,2
1
Electrical Engineering, 2Computer Science & 4
Seattle Children’s Hospital
Engineering, 3Division of Hematology
4800 Sand Point Way NE
Seattle, WA 98195
1 1 3 3 Seattle, WA 98105
{ejaywang , wli1 , bldbuddy , coletten , 4
[email protected]
shwetak1,2}@uw.edu



We present HemaApp, a smartphone application that
noninvasively monitors blood hemoglobin concentration
using the smartphone’s camera and various lighting sources.
Hemoglobin measurement is a standard clinical tool
commonly used for screening anemia and assessing a
patient’s response to iron supplement treatments. Given a
light source shining through a patient’s finger, we perform a
chromatic analysis, analyzing the color of their blood to
estimate hemoglobin level. We evaluate HemaApp on 31
patients ranging from 6 – 77 years of age, yielding a 0.82
rank order correlation with the gold standard blood test. In
screening for anemia, HemaApp achieve a sensitivity and
precision of 85.7% and 76.5%. Both the regression and
classification performance compares favorably with our
control, an FDA-approved noninvasive hemoglobin
measurement device. We also evaluate and discuss the effect
of using different kinds of lighting sources. Figure 1: HemaApp is a smartphone application that
noninvasively estimates blood hemoglobin concentration using
+* &(
.-&()
a smartphone camera. Analysis of the color of the blood in a
Hemoglobin; Mobile Health; Photoplethysmography; user’s finger yields an estimate of the user’s hemoglobin level.
Anemia; Camera; Blood Screening We evaluated the system using the smartphone’s LED flash and
incandescent light bulbs as illuminating sources.

#))!!*!&%
.-&()

H.5.m. Information interfaces and presentation (e.g., HCI): candidates for clinical and remote healthcare platforms. A
Miscellaneous. number of applications have even leveraged only the existing

sensors on a smartphone to achieve results similar to those
Smartphone-based medical devices have grown increasingly from medical devices (e.g., the microphone [15] and camera
common for heartrate monitoring [8,9,14], pulmonology [7–9,14]). However, there has been little work on performing
[15], sleep monitoring [17], point-of-care diagnostics [23], non-invasive blood screening on a smartphone. Most related
and a variety of telemedicine systems [10,27]. They work focuses on blood oxygen saturation. In this paper, we
demonstrate how the computing, telemetry, and sensing present the design and evaluation of a noninvasive technique
capability of modern smartphones make them excellent for sensing hemoglobin levels using an unmodified
smartphone camera. By leveraging the absorption properties
Permission to make digital or hard copies of all or part of this work of hemoglobin and blood plasma at multiple wavelengths of
for personal or classroom use is granted without fee provided that light, our system, HemaApp, can measure hemoglobin
copies are not made or distributed for profit or commercial
advantage and that copies bear this notice and the full citation on concentrations with no augmentation of the phone hardware
the first page. Copyrights for components of this work owned by and can be further improved with minimal augmentation.
others than the author(s) must be honored. Abstracting with credit
is permitted. To copy otherwise, or republish, to post on servers or Hemoglobin is the protein molecule in the blood that carries
to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from [email protected]. oxygen throughout the body. Conceptually, the measure of
UbiComp '16, September 12 - 16, 2016, Heidelberg, Germany hemoglobin is a representation of the oxygen carrying
Copyright is held by the owner/author(s). Publication rights
licensed to ACM. capacity of the patient’s blood. This is distinct from oxygen
ACM 978-1-4503-4461-6/16/09...$15.00 saturation, which measures the oxygen carrying efficiency of
DOI: http://dx.doi.org/10.1145/2971648.2971653

593
 UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

the blood. Currently, options to monitor hemoglobin include

a full blood analysis using a complete blood count (CBC) [6],
blood analysis using a finger prick [20], or optical
measurements through the finger using a specialized finger
probe [1,16,18,24]. Noninvasive measurement is desirable
for both sanitation and ease of use when measuring
frequently because it avoids puncturing the skin. A blood- Figure 2: Three embodiments of HemaApp in increasing
screening tool based on unmodified smartphones has the hardware augmentation.
added advantage of being easily deployable and enables Our research team includes experienced hematologists and
previously unconsidered treatment management options pediatricians who help conduct a clinical study with 31
given the lack of such technology. For example, HemaApp patients. The study population includes a wide spread of
can help community health workers in developing countries hemoglobin levels (8 – 16 g/dL), an age spread from 6 to 77
screen for iron-deficient anemia caused by malnutrition. A years old, and included skin tones ranging from pale to dark
major barrier for these health workers is the number of skin. This demographic spread is in-line with initial
medical tools they need to transport with them on foot for feasibility studies done for other noninvasive hemoglobin
each test. A smartphone is now standard equipment used for measurement devices [13,19]. We find that the combination
telemetry of records; as such, a smartphone-based solution of incandescent and LEDs gives the best estimation accuracy
helps reduce the burden on these workers and reduces the of ±1.26 g/dL (R = 0.82). As a comparison, the Masimo
cost of equipment. Beyond improved deployability in remote Pronto, an FDA-approved optical hemoglobin measurement
areas, the reuse of smartphones also aids people being treated device, achieves an accuracy of ±1.28 g/dL (R = 0.81).
for cases of anemia and need to monitor their condition at
home. Often, these patients are treated with iron supplements The main contribution of this paper is a demonstration that a
and return to the hospital for a blood test every few weeks to smartphone camera can be used in estimating hemoglobin
ensure their treatment is effective. A smartphone hemoglobin concentrations noninvasively with minimal augmentation
test is convenient for at-home monitoring and does not with a few commonly available light sources. This
require a patient to purchase a specialized blood testing contribution comes in three parts: (1) an algorithm to
device that costs hundreds to thousands of dollars. This estimate hemoglobin using a combination of broadband and
allows both the patient and the doctor to track the narrowband light source; (2) a comparison of the hemoglobin
effectiveness of these treatments much more easily and estimation with a clinical blood draw and another
frequently. This can help early detection of any ineffective noninvasive optical hemoglobin measurement system; and
treatment leading to complications. Lastly, the ability to (3) a performance analysis of hemoglobin estimation when
measure hemoglobin noninvasively is useful for measuring different combinations of light sources are used.
hemoglobin more frequently even in a clinical setting. For
example, sickle cell patients often suffer from extreme 


anemia and need frequent monitoring. However, due to The ubiquity of smartphones has lowered the access barrier
treatment to suppress their production of sickled cells, their to computing, telemetry, and the sensors needed to enable
veins are often hardened, making blood draw difficult [4]. medical and health care data for non-clinical settings. Prior
work in this new space has inspired the development and
Our system enables hemoglobin measurement through a design of HemaApp. We divide our summary of related
chromatic analysis of the blood at the user’s fingertip by works into three sections: smartphone-based sensing, camera
measuring the absorption properties of the blood at different as a hematology sensor, and hemoglobin measurement.
wavelengths of light. This is achieved by using the RGB
camera with different light sources illuminating the fingertip. $(*' &%
)
%)!%

We evaluate three different hardware embodiments that vary There are currently three ways in which medical devices
the level of hardware augmentation necessary (Figure 2): (1) integrate smartphones into their systems: with a self-
white flash + infrared emitter, (2) white flash + infrared contained attachment that digitally communicates with the
emitter + incandescent lamp, and (3) white flash + custom phone, with an attachment that supports and uses the phone’s
infrared led array. Most smartphone cameras have a white built-in sensors for measurement, and with an app using only
LED, and many smartphones have started to introduce IR the phone’s built-in hardware to process the specimen.
emitters for autofocus, potentially making the first Self-contained attachments are medical devices that have all
embodiment augmentation free. The second embodiment the necessary hardware to perform the medical analysis, but
requires a low overhead modification by introducing an use the phone as a hub to control the device, receive data,
incandescent lamp for an additional IR spectrum. The third store data, and potentially send data to an outside database
embodiment requires the most hardware augmentation by for access by hospitals and doctors [25]. This model is
introducing a custom array of LEDs as a phone case. gaining popularity in almost all commercial products for
personal health monitoring systems like blood pressure

594
SESSION: HEALTH I

monitors [30] and glucose implants [31], often using pulse oximetry, using a custom set of red and infrared LEDs
Bluetooth as the communication method. to illuminate the finger tip of a patient and the camera to
measure the pulse at the finger under different illuminations
Another class of devices uses supplemental hardware in
[8,11,12,21]. Our work builds upon the insights from these
addition to the smartphone. One such example is an
works in the use of smartphone cameras to detect infrared
attachment that mechanically couples a stethoscope head to
light, but focuses on not modifying the smartphone camera;
a 3D printed attachment that directs the sound to the
instead, we explore multiple ways to illuminate the fingertip,
smartphone’s microphone [23]. More complex attachments,
ranging from lighting sources commonly available on
such as an optics system that illuminates a flow cytometer
smartphones to incandescent lighting for a broad spectrum.
test strip, can be attached to a smartphone camera to perform
point of care blood tests [28]. The optics system helps to $&#&!%
)+($%*

illuminate the strip with the needed wavelength, the test strip Unlike pulse oximetry, which compares the ratio of
provides the needed chemical reaction to bind to the proteins oxygenated hemoglobin to deoxygenated hemoglobin,
and compounds of interest in the blood specimen, and the hemoglobin measurement needs to measure all forms of
smartphone camera acts as the sensor. The advantage of such hemoglobin in reference to the blood plasma.
systems is that attachments can be tailored to specific tasks
Clinically adopted methods to measure hemoglobin are
(e.g., chemical reaction, illumination, mechanical
currently restricted to blood tests. In a lab test, a technician
amplification), and the sensor of the smartphone can be
draws about 3mL of blood for a complete blood count
reused for various purposes and recorded onto the phone.
(CBC). In a CBC, information about red blood cells (RBC),
The last class of systems uses only what is on the white blood cells (WBC), and platelets are measured
smartphone. Researchers have explored various ways to electronically. The hemoglobin concentration is measured
monitor respiratory ailments through spirometry [15] and optically after the blood is mixed with a chemical agent that
auscultation [23] using the smartphone’s microphone. Sleep changes the solution’s density proportionally to the
disorders like sleep apnea have been successfully monitored concentration of hemoglobin. For tests that only require a
using a combination of speaker and microphone on a hemoglobin concentration result, such as those for screening
smartphone as a sonar to detect breathing [17]. The camera during blood donations, a point of care (PoC) device is often
has been used to map melanoma and skin lesion [29]. By used instead of the more time consuming CBC. An example
only using typical hardware on a smartphone, such systems of a PoC is the HemoCue1 device. The HemoCue requires a
have an advantage in terms of being immediately deployable finger prick to draw a small amount of blood; using a
through software, making these systems attractive for mass microcuvette that draws blood and mixes it with a chemical
deployment due to their ease of installation and low cost. reagent on a test panel, a result can be determined in a
minute. This method has been shown to have a rank order of
$(
)

$*&#&.
%)&(

0.89 at a mean accuracy of ±0.5 g/dL when compared to
Most relevant to our work is the use of smartphone cameras
results from a CBC test [5,20].
to noninvasively perform blood analysis. In particular, this
type of work relies on the fact that various compounds in the Most recently, noninvasive measurement of the hemoglobin
blood produce coloration differences under different concentration through optical measurement of the blood at
concentrations. BiliCam is a system that measures bilirubin, the fingertip has been developed. Alam et al. has created a
a compound in the blood that is important to monitor during finger probe with 6 LEDs that cover multiple wavelengths in
the first few days of a newborn’s life [7]. This system the red to IR spectrum (630, 660, 680, 770, 880 and 1300nm)
measures the yellowness of the skin in order to compute the to measure the hemoglobin to water ratio in blood plasma
concentration of bilirubin, using a sheet of white paper for [1,2]. Kraitl et al. conducted a 41 people study using a three
color balancing. Similar to this work is a student project LED (670, 810, 1300nm) finger probe, and found similar
called Eyenaemia [22], which uses a color calibration card success. Timm et al. based their technique off of Kraitl et
placed next to the eyelid of a patient, and measures the al.’s system to create a device called OxyTrue Hb®. A
redness of the underside of their eyelid when flipped over. In commercially available system developed by Masimo is
this case, the concentration of hemoglobin, which is similar to Alam et al.’s system, but instead incorporates 7
responsible for the red coloration of the blood, affects the red LEDs and uses a proprietary algorithm [16]. Validation of
absorption. Our system follows a similar intuition as this system has shown a mean accuracy of ± 1.0 g/dL [16,18].
Eyenaemia, but measures the coloration of the blood at the We draw from these existing works for noninvasive
fingertip. More importantly, Eyenaemia also only focuses on hemoglobinometry in our design of the lighting sources we
detecting the risk of anemia, but cannot estimate the actual incorporated in our study. The aim for our system is to
concentration of hemoglobin as in the case of our approach. provide a similar level of accuracy, but reusing the hardware
Various research projects have also explored the use of of a smartphone to make the system more easily deployable.
smartphone cameras with their IR filter removed to perform

1
http://www.hemocue.com/en/solutions/hematology

595
 UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

Figure 3: System overview of HemaApp, which consists of the phone and lighting hardware, data, and algorithm. The HemaApp
application communicates with an LED attachment. After a user places their finger over the LED and camera, multiple light sources
are cycled through. A video is recorded for each light source. The algorithm then extracts the R, G, and B time series waveform for
each video by averaging each RGB channel independently for each frame. The algorithm then extracts machine learning features
including peak and trough measurements for each light source, and also interaction terms between light sources. Finally, a SVM
based regression is applied to estimate the hemoglobin concentration for the user.

HemaApp is the only noninvasive hemoglobin measurement below 1000nm. Water begins to have a response above
system that leverages the camera of the smartphone to 940nm and has an initial relative maximum at 970nm.
perform similar measurements as the Masimo Pronto and Second, we not only measure the water content in the plasma,
other finger probe systems, utilizing non-LED sources to but also the proteins that make up about 10% of the blood
remove the need for custom hardware. plasma by volume[14] as a proxy for capturing the plasma
volume to compare against hemoglobin concentration.


Figure 3 shows the HemaApp system, consisting of hardware This is accomplished by leveraging the blue absorption of the
components that record data which are processed by an plasma. By illuminating the finger with a white LED (which
algorithm to calculate the hemoglobin concentration. The contains a strong blue component) and an IR light at 970nm,
following hardware section describes the choice of light the system is capable of capturing the plasma response. We
wavelengths and the phone application that controls the also included another IR LED (880nm) to help capture the
lights and records the light reflected from the finger to the different absorption between the various forms of
camera. The data obtained by the phone app are a series of hemoglobin. Both the 970 and 880nm LED are in the range
videos for each light source. The algorithm section describes of IR autofocus LEDs equipped in the current generation of
the process of converting the series of video recordings to smartphones. Our IR LEDs are sourced from the Marubeni
RGB time series waveform and the physical intuition behind SMC2 series.
the machine learning features used in the hemoglobin
estimation. Even though some new smartphones are beginning to be
equipped with IR LEDs, most smartphones only have a white
(-(
LED. To address this issue, we look towards a commonly
Due to the limited access to low level hardware control of the available IR source, the incandescent light bulb.
phone, we choose to develop a hardware add-on to provide Incandescent light bulbs typically have strong IR light
the lights for hemachrome analysis. However, unlike other emission in the NIR range. Our system uses a candelabra 6W
noninvasive optical hemoglobin measurement systems, we incandescent light bulb that can be found in hardware stores.
limit the lights we choose to ones that can be commonly
found on smartphones today and in typical hardware stores. The LEDs are placed in a ring around the phone camera,
Furthermore, we make no modifications to the phone’s while the incandescent light bulb was placed about 3 inches
camera itself, unlike prior work’s use of phone cameras for above the finger.
pulse oximetry, which requires the removal of the IR filter of 


the camera. Instead, we rely on the front facing camera, For the HemaApp prototype, we use a Nexus 5 with an
which tends to have weaker IR cut off than the back facing unmodified front-facing camera. For typical optical
cameras, making it more suitable for our application. hemoglobin measurement, each lighting condition is cycled
 

through at a high rate repeatedly to measure the response at
We address a major limitation of using a smartphone camera each wavelength pseudo-simultaneously. In this way, the
for doing hemoglobin measurement; a lack in sensitivity to lighting conditions are all measured for the same pulse.
wavelengths above 1000nm. First, we use only IR absorption However, due to the limited sampling rate of the camera and

2
http://www.tech-led.com/product/smc/

596
SESSION: HEALTH I

the inability to synchronize the lighting circuit with the The Beer-Lambert law states that the absorption of light is
camera’s frame refresh, it is not feasible to measure all proportional to the concentration and thickness of the
wavelengths of light at the same time without considerable medium, given by:
access to low level hardware control. We discuss the
tradeoffs of multiplexing and series testing in the discussion NHERTQHG k ;  =V ? G (1)
and make suggestions on how multiplexing can be achieved
based on our own experiments. For our prototype, each where ; is the incident light intensity, ' is the absorption
lighting condition shines continuously for 15 seconds and coefficient, nois the concentration, and  is the thickness of
cycles to the next light with an assumption that hemoglobin the medium that the light travels through. When the finger is
concentration and average blood flow does not change illuminated with a single wavelength of light, the measured
significantly during the course of the test, which takes a few intensity NHERTQHG represents the absorption due to tissues,
minutes. hemoglobin, and plasma:
The mobile app is built on the Android platform with the
Android Camera 2 API, which allows for full control over NW k ;W  =GlVf^eeg[h SKRRTH
<
VXZh AF
<
Vc`YeaYh CMERNE m (2)
the white balancing and exposure. This is important because
infrared is generally considered an unwanted spectral where ( is the wavelength of the incident light. To obtain the
response that camera applications will detect and rebalance ratio of no and
n#o, it is necessary to eliminate the
settings to avoid. Prior work has found that certain white attenuation of the intensity signal due to finger tissue. This is
balancing settings, such as the “incandescent mode” can be accomplished by measuring the temporal change of the
used to avoid the rebalancing [9]. The Camera 2 API allows measured intensity as the thickness of the arteries oscillate
for full control over the exposure, white balance, and sensor with respect to the heartbeat.
sensitivity. The hardware gains are manually set for each
RGB channel using presets that are empirically found for
each lighting condition such that the three channels reported
a similar level of light. This is necessary because the red
channel typically has a much stronger response due to the red
blood under the white and incandescent lights. If left to a flat
white balance, the auto exposure will be set to the red
channel, leaving the G and B channels highly underexposed.
Finally, exposure is set using the camera API’s auto-
exposure settings. Once the image is auto-exposed, the
exposure is locked and a 15-second video is recorded for
each of the lighting conditions sequentially.
*

Each light source is cycled through one after another and a
15 second RGB video is recorded for each light source. The Figure 4: Light absorbed by living tissue. Adapted Web09[26].
exposure, frame rate, white balance gain, and ISO settings The absorption of light changes due to the change in volume of
blood when the heart pulses.
are recorded for calibration. At an average resting heart rate
of about 75 beats per minute (BPM), around 15-20 pulses The change in arterial thickness & affects only the path
were captured for each light source. length for  and
#. By measuring the ratio of the
maximum and minimum intensity of the light received, the
#&(!* $

The algorithm section first explains hemachrome analysis, effect of the tissue is removed:
which is the blood color analysis we base our algorithmic
PHELW
implementation on. We then detail the two stages of the k  UGlVXZh AF
<
Vc`YeaYh CMERNE m
(3)
implementation. First, the video processing step extracts the SQOTIJW
pulsatile signal recorded in each video. The second stage is a
feature generation step that involves combining the intensity Where the ratio of intensities can then be expressed as:
values extracted from each wavelength’s video. These
features are then used with an SVM to train regression !(
DW k  k '(   i '!#( #  (4)
models based on the ground truth blood test hemoglobin. $" %(


!

Hemachrome analysis is the study of blood coloration to The measured ratio of maximum and minimum values of
analyze the components in the blood. HemaApp aims to intensity thus provide a measure of absorption due to the
measure the concentration of hemoglobin as compared to the different components of blood. In theory, a system can use
concentration of plasma in the blood. empirically measured absorption coefficients for each

597
 UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

compound at a specific wavelength of light to predict the 


hemoglobin concentration using Equation 4. By measuring In order to estimate the absorption coefficients at the
the response at multiple wavelengths of light, multiple broadband wavelengths used by our system, the HemaApp
DW
can be calculated. Ratios of DW across wavelengths over system cannot rely on the basic derivation of multi-
determine the ambiguity of  allowing an estimate of no wavelength hemachrome analysis. The following is a list of
to be made. However, factors such as the distribution of the the features. The first three features are derived directly from
emitter, sensitivity of the sensor, and complex reflection the Beer-Lambert equations in hemachrome analysis. The
properties of tissue make this infeasible. In our case, the fourth and fifth features aim to capture nonlinear interactions
incandescent and white LED are very broadband emitters between wavelengths.
that produce measurements that are poorly modeled by ,5/.3 k ,+* : The baseline intensity due to tissue.
Equation 4. Using machine learned regressions to estimate
the 'AFW and
'CMERNEW
for each lighting source, we ,5/.3 j
,764801 k ,)* : The amplitude of the pulsatile absorption.
overcome our reliance on predetermined absorption ,5/.39
coefficients at specific wavelengths for blood.  k
,- : The adjusted amplitude of absorption that
,7648019
eliminates the effect of tissue.


For each lighting condition, we extract the intensity of the ,-)* l:  : m k
,)*92 
,)*93
: A pairwise ratio of pulsatile
peak and troughs for each pulse. The center section, absorptions between all wavelengths.
measuring half the width and half the height of the image, is
,-92 =,-93
cropped and an average intensity for each channel is ,-)*+* l:  : m k : Absorption difference across
,+*92 =
,+*93
calculated. This is done because the image from the center of wavelengths, adjusted with baseline.
the image is the most consistent and stable.



The signal processing technique that extracts D
A separate SVM regression (SVR) model is trained for each
1)
High pass filter with a cut off at 0.5Hz. This is to remove embodiment of our proposed system: (1) white + 970 nm
the fluctuations due to breathing, which are around 0.2- LED, (2) white + 970 nm LED + incandescent light, (3)
0.3 Hz. white + 970 nm + 880 nm LED + incandescent light. We will
2)
Apply an FFT on the filtered waveform and extract the refer to each embodiment as EMB#. The regressions are
dominant peak. The dominant peak gives an estimate of made based on ground truth values obtained from a blood
the heart rate. test. During development, we find that a linear regression do
3)
Using the heart rate as a threshold between successive not produce as good of a result as an SVR, but provides good
peaks, false peaks due to the dicrotic notch and the insight into feature significance. As such, the linear
diastolic peak are avoided. The threshold is set at ¾ of regression is used to help in the feature selection process, as
the heart rate to avoid missing actual peaks. it is easier to interpret the resultant model, but the SVR is
4)
Map the index of peaks and troughs onto the unfiltered used to produce a model for evaluation by employing the
signal. We need the original magnitudes of the peaks and features chosen through the linear regression. Table 1
Bc[Y_ displays the feature list for the three embodiments:
troughs to calculate D k 
Bfdbg\]
5) Calculate D for each peak Table 1: Features used for training in each embodiment.
EMB1 EMB2 EMB3
>? lm D>? l m D>? l m
>? lm D>? l
m D>? l m
>? l
m D>? l m D>? l m
D>? l m D>?@? l m D>?@? l m

D>?@? l
m D>?@? l m D>?@? l
m
D>?@? l
m D>?@? l m

D>?@? l m

Figure 5: To calculate the absorption change due to blood, a

The SVM model is tested using a leave-one-subject out
peak detection algorithm is applied on the temporal signal. An
FFT is used to estimate the heart rate. Using the estimated heart
validation. The training is done using the MATLAB
rate as the minimum spacing between two peaks, the peaks and implementation of SVM regression with a Gaussian kernel
troughs of the heart beat signal is extracted.

598
SESSION: HEALTH I

with default parameters, to avoid overfitting due to The in-patient clinics are centered on cancer patients, in
parameter tuning. particular leukemia and bone marrow transplant patients,
because these patients tend to be chronically anemic. As





To evaluate and inform the design of HemaApp, we conduct such, these patients often have CBCs done as part of their
a clinical study with three groups of people: healthy students clinical care. By including both children and adult clinics, the
and staff of the university, in-patients at a children’s cancer study cover a wide range of age and hemoglobin variations,
allowing us to validate on different groups of people. Our
and transfusion clinic, and in-patients at an adult cancer and
study population include about 1/3 pale skintone, 1/3 sepia –
bone marrow transplant clinic. Data collections at these sites
light brown skintone, and 1/3 dark brown skintone. Due to a
provide a diverse dataset paired with ground-truth
hemoglobin concentration from CBC tests. We note that low population of people with black skintones in the city we
typical initial feasibility studies for hemoglobin conduct our study, we are not able to include those with very
measurement systems involve 10 – 60 people and typically dark pigmentation in our study. However, it should be noted
cover population ranges of 8 – 16g/dL hemoglobin that pigmentation on the underside of the hand tends to be
concentration [13,19]. In order to recruit patients from much lighter in comparison to the backside of the hand, as
beyond these ranges, a system must be validated enough for such, it is likely that the effects of very dark pigmentation is
hospital use during surgical procedures, as people who are covered well enough by our inclusion of dark brown
well below the 8 g/dL range are likely in need of a skintones. In our follow up study, we will be expanding to
transfusion or are already in surgery. These initial feasibility other hospitals nationally and internationally to have a wider
spread of pigmentation to validate these claims.
studies are then followed by a series of larger follow up
clinical validations in more diverse populations and repeated *
&##*!&%
(&+(

measurements per person during surgeries such as urologic In order to evaluate the effects of camera hardware and
procedures where hemoglobin concentrations change lighting conditions, we built a setup that allows us to
dramatically [16,19,24]. We base our study design to reflect efficiently cycle through all the combinations for every
the feasibility studies of these previous works. Our study subject in our clinical validation. The setup is an acrylic box
include 31 patients in a range of 8.3 g/dL to 15.8 g/dL. that contains a Bluetooth-enabled microcontroller that
controls each of the light sources. The top of the box has a
Table 2: Demographic information of subjects. 6W incandescent light and a white piece of card stock with a
hole cut in the middle. A Nexus 5 smartphone is placed in
Participant Demographics (N = 31) the box with the camera pointing up to the ceiling of the box.
The LED circuit is then placed over the camera. The box is
Age (years) 6 – 77 (µ=31, σ=17.5) clear except for the portion holding the electronics, which
lets ambient light shine through. The box also has a black
Hemoglobin (g/dL) 8.3 – 15.8(µ=12.1, σ=2.2) cover that is used to block out ambient light.

Gender Ratio 15 male:16 female

Reported Ethnicity (n, %)

East Asian: 7

South East Asian/Indian: 6

Latino: 3

White: 14

Mixed: 1
Figure 6: The experimental set up included an acrylic box
housing a 6W incandescent light bulb of 3000k color
temperature and a Nexus 5 smartphone with an LED array
Each patient’s data set include a series of videos measuring
attached to its front facing camera. The array is equipped with
the absorption change under multiple wavelengths of light. white, 880nm, and 970nm LEDs. The participant places their
We collect videos within 24 hours of the ground truth CBC finger over the camera while covering the LEDs.
blood draw to ensure that hemoglobin measures are as
accurate as possible. Within a day, hemoglobin The subject places the fingertip of the ring finger on their
concentration is typically stable within 0.5 g/dL[3]. Patients non-dominant hand on the camera. The subject is asked to sit
who have hemoglobin transfusion or heavy bleeding still and not speak during the test to reduce movement. Each
between the study and the blood draw were excluded.

599
 UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

Figure 7: (Top) A comparison between predicted hemoglobin levels and ground truth hemoglobin level. (Bottom) A Bland-Altman
plot showing residuals of predicted hemoglobin level against the ground truth hemoglobin. The ±1.96SD is shown for each plot.
Embodiments 1, 2, and 3 shown from left to right. Blue represents the HemaApp results and orange represents the Pronto results.

lighting condition is then cycled through in the following Implications: The improvement from EMB1 to EMB2
order: incandescent, white, 970 nm, 880 nm. shows that simply supplementing the smartphone with an
incandescent light both improves the correlation and
The study consists of taking a series of videos of the
decreases the error significantly. With the addition of an
participant’s finger under various lighting conditions. Tests
extra IR source, thus including both a 970 nm and 880 nm IR
are done during the day, with no particular control over the
LED, the performance of HemaApp is comparable to the
ambient lighting conditions. An optical hemoglobin
results of the Masimo Pronto.
measurement is obtained using the FDA cleared Masimo
Pronto 7 right before the recordings with the HemaApp A Wilcoxon signed rank test and an F-test of the residual
system. The CBC blood test is used as ground-truth data and variances fails to show statistically significant differences
the optical Hb is used as a source of comparison to a between the EMB3 predictions, Pronto’s Hb predictions, and
specialized noninvasive device. ground truth CBC values (p>0.05). An N-way ANOVA on
the residual magnitude (|HemaApp EMB3 – CBC|) also did

not reveal statistically significant effects on the residual
magnitude due to age and race (p>0.05).
(!*!%
$&#&!%
,#)

The top half of Figure 7 shows the predictions for each
(%!%
&(
%$!


HemaApp embodiment comparing to the ground truth CBC,
To evaluate how well HemaApp can be used as a tool for
calculated using a leave-one-subject out cross validation.
The bottom half of Figure 7 shows the corresponding screening for anemia, the results of the regression are
modified Bland-Altman plot, where the residuals classified into two groups: anemic and normal. The
(HemaApp – CBC) are plotted against the CBC hemoglobin. classification is based on the average expected hemoglobin
For comparison, the results of the Masimo Pronto are for each age group and gender.
included in each plot. Figure 8 shows the classification results plotted against age.
Results: HemaApp’s hemoglobin estimations correlate with A different reference line is shown for male and female
the CBC’s predictions with a rank order correlation of 0.69, above the age of 16 as female tends to have lower
0.74, and 0.82 with a mean error of 1.56 g/dL, 1.44 g/dL, and hemoglobin than males. Those classified correctly are shown
1.26 g/dL respectively for each embodiment. The results of in gray. Higher sensitivity signifies better ability to detect the
presence of anemia. Higher specificity signifies better ability
the Pronto are also compared to the CBC, which yields a rank
to detect normal levels of hemoglobin. Table 3 breaks down
order correlation of 0.81 with a mean error of 1.28 g/dL.

600
SESSION: HEALTH I

the results by sensitivity and specificity for each embodiment unnecessary patient burden and hospitals getting frequent
and the Pronto. The 31 participants included 14 with anemia. calls from unnecessarily worried patients.
The Pronto did not produce a hemoglobin measurement for
one of the anemic participant due to low signal quality, thus 

the sensitivity results are for 13. Our analysis of HemaApp shows that with some
augmentation to the current smartphone hardware, our
Results: HemaApp’s classification results for each
smartphone-based hemoglobin system compares favorably
embodiment have higher sensitivity (78.6%, 85.7%, and
with the Pronto predictions. HemaApp cannot replace the
85.7%) as compared to the Masimo Pronto (69.3%). In
CBC blood test, but can be used as an effective screener to
comparison, the Pronto performed better in specificity,
determine whether further blood testing is necessary. The
88.2% compared to HemaApp (70.6%, 70.6%, and 76.5%).
availability of the smartphone makes HemaApp a good
candidate for at-home monitoring of chronically anemic
patients, and its low cost can help equip community health
workers in low resource areas screen for iron deficient
anemia. The findings show that by using another ubiquitous
item, the incandescent light bulb, the accuracy of HemaApp
can be significantly improved. Although such light bulbs are
beginning to be phased out in developed countries for more
efficient alternatives, they are still easily obtainable as
decorative lights in typical hardware stores.

!$!**!&%)
%
+*+(
&("

The current results are based on data collected on a Nexus 5
device and using only one brand of incandescent light bulb.
As such, for the system to be more accessible, it needs to
function on various types of devices and supplemental
Figure 8: Anemia classification of HemaApp using typical lighting. Different brands and models of phone use different
hemoglobin reference ranges for each age and gender groups.
cameras, lenses, and filters. The strength of the IR filter will
Colored lines map incorrectly classified points to ground truth
classification and hemoglobin predictions. strongly influence how strong the IR LEDs need to be for
penetration. The necessary adjustments based on these
Table 3: Anemia classification results for each embodiment.
factors need to be investigated before the results of
HemaApp can be generalized to different hardwares.
Sensitivity Specificity The incandescent light’s properties also present another
variable for future exploration. The current light bulb is a
EMB 1 78.6% (11/14) 70.6% (12/17) 3000K bulb, which places its peak wavelength at about
1000 nm, making it ideal for the detection of water. Another
EMB 2 85.7% (12/14) 70.6% (12/17) variable is the effect of the age of the light bulb. As light
bulbs become less efficient, their emitted spectrum changes.
EMB 3 85.7% (12/14) 76.5% (13/17)
It is possible that in order for the incandescent light bulb
Pronto method to remain accurate over time, a calibration stage
69.3% (9/13) 88.2% (15/17)
would be necessary after some time. Another issue with
using an incandescent light bulb is ambient lighting. When
Implications: The results are in favor of HemaApp as a using the data collected when ambient light is allowed into
useful screener for anemia. This can be seen in the sensitivity the experimental setup, the results of the best case EMB3
of each embodiment being near or above 80%. The Pronto dropped to R = 0.60 and a mean error of ±1.72 g/dL.
on the other hand performed better in terms of specificity, at Although this does not limit the use of EMB2 and 3 in a dark
just under 90%. Comparing HemaApp to Pronto, the two room, it does suggest a user would have to make sure to use
systems are calibrated differently. HemaApp is tuned to be a lamp shade to reduce large fluctuations due to changes in
more cautious in order to reduce missing patients who are ambient lighting. Further experiments will have to be done
anemic, but at the cost of lower specificity. We believe this to determine to what degree ambient light exposure affects
is a safer choice for use cases such as community health the results. We believe using a brighter light bulb and placing
workers screening in the field, as missing an anemic the hand closer to the lamp will reduce this problem
diagnosis is a higher risk. In comparison, the Pronto’s higher significantly, limiting the effect of ambient light.
specificity is useful for situations such as in home 




monitoring. In situations where the patient is measuring The current data collection focuses on a range of 8-16 g/dL,
frequently, having too many false alarms will result in which is sufficient for covering certain levels of anemia and

601
 UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

normal populations; however, this means HemaApp cannot An automated signal quality detector will also be necessary
be used for those who are so gravely anemic that they need for detecting whether a signal is stable enough for analysis.
immediate transfusion or those with elevated hemoglobin. The usual components in the signal consists of the pulse
Our current data collection has come across a few patients in signal and the breathing signal, both of which are typically
these extreme ranges, but we will need a more targeted periodic between 0.3 – 2 Hz. An automated system can
recruitment to have enough data to develop the system to analyze for sudden DC shifts caused by the finger shifting,
work in these ranges. To find people with elevated resulting in a non-periodic signal alteration. This signal
hemoglobin, we will need to expand our clinical study to quality detector then determines whether a segment of
include pulmonary clinics with patients that have high collected data is useable for analysis, prompting the user to
hemoglobin due to advanced pulmonary ailments. For perform the data collection again if it failed.
obtaining extremely low hemoglobin, our collaborators have
suggested taking data at midnight, when most of their 

transfusion patients are admitted and their hemoglobin HemaApp is a smartphone application that noninvasively
measures as low as 4-5 g/dL. For this to work, our system monitors the hemoglobin concentration using the
needs to become much quicker, more robust, and self- smartphone’s camera and different lighting sources.
contained such that running our study does not require the Monitoring hemoglobin is a standard clinical tool for both
participation of the patient and fast enough that we do not screening and assessing a patient’s response to treatment. We
interfere with the patient’s clinical care. evaluate HemaApp on 31 patients ranging from 6 – 77 years


!


of age using three different embodiments of our system. The
Only patients with normal red blood cell traits were recruited first uses LEDs commonly found on smartphones. The
in our initial study, but we plan to expand our evaluation to second augments the phone with an incandescent light
include different blood disorders (e.g., sickle cell, source. Finally, the third both augments the phone with an
thalassemia, and hemolyzing patients). Each of these incandescent source and uses a custom LED setup. The
disorders can potentially cause different optical absorption resulting hemoglobin predictions yield a rank order of 0.69,
variations that may or may not be observable by our current 0.74, and 0.82 correlation with the gold standard blood test.
measurement technique. Further studies of the optical In screening for anemia, HemaApp can achieve 85.7%
properties of different hemoglobin types will have to be done sensitivity and 76.5% specificity when augmented with
through laboratory experiments. On the other hand, patients incandescent lights and IR LEDs. Comparing to an
with hemolysis have ruptured red blood cells that results in specialized FDA-approved noninvasive hemoglobin
free floating hemoglobin in the blood plasma. In the measurement device, which performed at a rank of 0.81 and
measurement of hemoglobin concentration, it is important to an sensitivity and specificity of 69.3% and 88.2%, our results
distinguish between hemolyzed and non-hemolyzed for HemaApp are promising.
hemoglobin. In a laboratory test, the blood is first separated To further validate our results, a larger national and
into the plasma and intact red blood cell components before international study that includes a more global demographic
hemoglobin concentration is determined. HemaApp has not will need to be deployed. The current results can only be
been validated to differentiate the two conditions to produce interpreted under the scope of normal hemoglobin
an adjusted measurement. This is not possible for our system. phenotypes. However, many of the clinical use cases that


!
would highly benefit from daily monitoring of hemoglobin
A major hurdle to making the system truly deployable is in and anemia require the same efficacy for hemoglobins that
making a system that can ensure data quality. All the data are deformed due to changes in its protein structure.
collection is conducted either by the development team or This study is a first step in transferring hemachrome analysis
trained data collection assistants who have experience using techniques to a smartphone platform, introducing
prototype medical devices. The interface produces a real- hemoglobin monitoring to a wide audience, from those who
time visualization of the camera feed that they interpret at the need at home monitoring to community health workers.
time of data collection to ensure that the patient has fully
covered the camera and LEDs and the waveform being  


recorded is void of excessive movement. We thank all the volunteers, and all publications support who
wrote and provided helpful comments on previous versions
To enforce proper finger placement and reduce movement, a of this document. In particular, we thank Manoj Gulati, Alex
finger cuff can be designed to center the finger over the Mariakakis, Lauren Oickle, Hayley Young, Jim Taylor, and
camera. A snug fit for such a cuff would be ideal, but it Jim Stout for their help with the coordination and data
cannot be so tight that the finger loses circulation. collection during the clinical study. This work is performed
Furthermore, the finger cuff will also function as a cover to under the approval from the University of Washington
block the ambient light. A small window can then be Human Subject Division and reliance agreement with the
integrated into the top of the cuff to be opened for the Seattle Children’s Hospital: IRB HSD #49667.
incandescent light.

602
SESSION: HEALTH I

 
and Cornie Scheffer. 2013. Detection of the Optimal

1. Omar Abdallah, Mohammed Natsheh, Kawther Abo Region of Interest for Camera Oximetry.
Alam, Qasem Qananwah, Ahrned a Nabulsi, and Proceedings of the Annual International Conference
Armin Bolz. 2010. Concentrations of Hemoglobin of the IEEE Engineering in Medicine and Biology
Fractions Calculation Using Modified Lambert-Beer Society, EMBS, 2263–2266.
Law and Solving of an Ill-Posed System of http://doi.org/10.1109/EMBC.2013.6609988
Equations. Spie 7715, 1, 1–8. 12. Walter Karlen, Joanne Lim, J. Mark Ansermino, Guy
http://doi.org/10.1117/12.854603 Dumont, and Cornie Scheffer. 2012. Design
2. Kawther Abo Alam. 2011. Fuzzy Logic Hemoglobin Challenges for Camera Oximetry on a Mobile
Sensors. Ph.D Dissertation. Karlsruhe, Germany. Phone. Proceedings of the Annual International
Conference of the IEEE Engineering in Medicine
3. A. Brown and A. L. Goodall. 1946. Normal and Biology Society, EMBS, 2448–2451.
variations in blood haemoglobin concentration. The http://doi.org/10.1109/EMBC.2012.6346459
Journal of Physiology 104, 4, 404–407.
http://doi.org/10.1113/jphysiol.1946.sp004133 13. Jens Kraitl, Ulrich Timm, Elfed Lewis, and Hartmut
Ewald. 2010. Optical Sensor Technology for a
4. H. Franklin Bunn. 1997. Pathogenesis and Noninvasive Continuous Monitoring of Blood
Treatment of Sickle Cell Disease. The New England Components. Library 7572, 757209–757209–10.
Journal of Medicine 337, 11, 762–769. http://doi.org/10.1117/12.843153
http://doi.org/10.1146/annurev.nutr.23.011702.0730
27 14. Sungjun Kwon, Hyunseok Kim, and Kwang Suk
Park. 2012. Validation of Heart Rate Extraction
5. H. Gehring, C. Hornberger, L. Dibbelt, et al. 2002. Using Video Imaging on a Built-in Camera System
Accuracy of Point-of-Care-Testing (POCT) for of a Smartphone. Proceedings of the Annual
Determining Hemoglobin Concentrations. Acta International Conference of the IEEE Engineering in
Anaesthesiologica Scandinavica 46, 8, 980–986. Medicine and Biology Society, EMBS, 2174–2177.
http://doi.org/aas460809 [pii] http://doi.org/10.1109/EMBC.2012.6346392
6. Beverly George-Gay and Katherine Parker. 2003. 15. Eric C Larson, Mayank Goel, Gaetano Boriello,
Understanding the Complete Blood Count with Sonya Heltshe, Margaret Rosenfeld, and Shwetak N
Differential. Journal of Perianesthesia Nursing 18, Patel. SpiroSmart : Using a Microphone to Measure
2, 96–117. http://doi.org/10.1053/jpan.2003.50013 Lung Function on a Mobile Phone. Figure 1.
7. Lilian de Greef, Mayank Goel, Min Joon Seo, et al. 16. Mark R. MacKnet, Martin Allard, Richard L.
2014. Bilicam: Using Mobile Phones to Monitor Applegate, and James Rook. 2010. The Accuracy of
Newborn Jaundice. Proceedings of the 2014 ACM Noninvasive and Continuous Total Hemoglobin
International Joint Conference on Pervasive and Measurement by Pulse CO-oximetry in Human
Ubiquitous Computing - UbiComp ’14 Adjunct, Subjects Undergoing Hemodilution. Anesthesia and
331–342. http://doi.org/10.1145/2632048.2632076 Analgesia 111, 6, 1424–1426.
8. Mathew J. Gregoski, Martina Mueller, Alexey http://doi.org/10.1213/ANE.0b013e3181fc74b9
Vertegel, et al. 2012. Development and Validation of 17. R; Rajalakshmi, D; Nandakumar, and M; Nathaniel.
a Smartphone Heart Rate Acquisition Application 2015. Contactless Sleep Apnea Detection on
for Health Promotion and Wellness Telehealth Smartphones. The 13th Annual International
Applications. International Journal of Telemedicine Conference on Mobile Systems, Applications and
and Applications 2012. Services, 45–57.
http://doi.org/10.1155/2012/696324 http://doi.org/10.1145/2742647.2742674
9. E. Jonathan and Martin Leahy. 2010. Investigating a 18. Mark J. Rice, Nikolaus Gravenstein, and Timothy E.
Smartphone Imaging Unit for Morey. 2013. Noninvasive Hemoglobin Monitoring:
Photoplethysmography. Physiological measurement How Accurate is Enough? Anesthesia and Analgesia
31, 11, N79–83. http://doi.org/10.1088/0967- 117, 4, 902–907.
3334/31/11/N01 http://doi.org/10.1213/ANE.0b013e31829483fb
10. Warren a Kaplan. 2006. Can the Ubiquitous Power 19. Js Ruckman. 2011. A Comparative Study of Total
of Mobile Phones be Used to Improve Health Hemoglobin Measurement Technology:
Outcomes in Developing Countries? Globalization Noninvasive Pulse Co-oximetry and Conventional
and Health 2, 1, 9. http://doi.org/10.1186/1744- Methods.
8603-2-9
20. H. Von Schenck, M. Falkensson, and B. Lundberg.
11. Walter Karlen, J. Mark Ansermino, Guy A. Dumont, 1986. Evaluation of “HemoCue,” a new device for

603
 UBICOMP '16, SEPTEMBER 12–16, 2016, HEIDELBERG, GERMANY

determining hemoglobin. Clinical Chemistry 32, 3,

526–529.
26. John Webster. 2009. Medical Instrumentation:
21. Christopher G. Scully, Jinseok Lee, Joseph Meyer, Application and Design. Wiley and Sons.
et al. 2012. Physiological Parameter Monitoring 27. B. Woodward, R. S H Istepanian, and C. I. Richards.
from Optical Recordings with a Mobile Phone. IEEE
2001. Design of a Telemedicine System using a
Transactions on Biomedical Engineering 59, 2, 303–
Mobile Telephone. IEEE Transactions on
306. http://doi.org/10.1109/TBME.2011.2163157
Information Technology in Biomedicine 5, 1, 13–15.
22. Jarrel Seah and Jennifer Tang. Eyenaemia. Retrieved http://doi.org/10.1109/4233.908361
May 12, 2015 from www.eyenaemia.com
28. Hongying Zhu and Aydogan Ozcan. 2015. Opto-
23. Jeff Thompson. 2015. Smartphone Attachment for Fluidics Based Microscopy and Flow Cytometry on
Stethoscope. Methods in molecular biology 1256, a Cell Phone for Blood Analysis. Methods in
213–29. http://doi.org/10.1007/978-1-4939-2172-0 molecular biology 1256, 213–29.
http://doi.org/10.1007/978-1-4939-2172-0
24. U. Timm, J. Kraitl, H. Gewiss, et al. 2015. Novel
Multi Wavelength Sensor Concept to Detect Total 29. George Zouridakis, Tarun Wadhawan, Ning Situ, et
Hemoglobin Concentration, Methemoglobin and al. 2015. Melanoma and Other Skin Lesion
Oxygen Saturation. Progress in Biomedical Optics Detection using Smart Handheld Devices. In
and Imaging - Proceedings of SPIE 9332, Figure 1, Methods in molecular biology. Clifton, NJ, 213–29.
1–9. http://doi.org/10.1117/12.2080144 http://doi.org/10.1007/978-1-4939-2172-0
25. K Wac. 2012. Smartphone as a Personal, Pervasive 30. Wireless Blood Pressure Wrist Monitor. Retrieved
Health Informatics Services Platform: Literature January 4, 2016 from https://ihealthlabs.com/blood-
Review. Yearbook of medical informatics 7, 83–93. pressure-monitors/wireless-blood-pressure-wrist-
http://doi.org/arXiv:1310.7965v1 [cs.CY] monitor/
31. Wireless Smart Gluco-monitoring System. 53.
http://doi.org/10.1017/CBO9781107415324.004

604