Non-contact sensing of neonatal pulse rate using camera-based imaging: a clinical feasibility study

Clinical studies involving several infants and assessing PR based on PPGI can be found in Vagedes et al (2004), Scalise et al (2012), Aarts et al (2013), Klaessens et al (2014), Mestha et al (2014), Villarroel et al (2014), Fernando et al (2015), Blanik et al (2016), Paul et al (2017), Antognoli et al (2018), Cobos-Torres et al (2018).

Measurements on one or a few infants are included in, for example, Wu et al (2012), Zhao et al (2013), Wang et al (2018). The applicability of PPGI in various clinically relevant scenarios has been researched in Aarts et al (2013). As PPGI requires light, skin pigmentation is one aspect that affects the signal-to-noise ratio of the PPG signal. For instance, a lower signal-to-noise ratio has been reported for a subject with darker skin (Aarts et al 2013). In some publications, the skin tone is explicitly mentioned (Fernando et al 2015, Wang et al 2018, Chaichulee et al 2018). A subject set similar to the one explored here containing Indian neonates was presented in Mestha et al (2014).

Studies differ greatly in terms of recording length. Antognoli and co-workers (Antognoli et al 2018), for example, recorded until a 10 s window of movement-free video was available, Aarts and colleagues (Aarts et al 2013) recorded between 1 min and 5 min, while Villarroel and co-workers (Villarroel et al 2014) recorded for several hours over consecutive days.

In terms of equipment, measurements were conducted with simple webcams (Scalise et al 2012, Antognoli et al 2018), digital cameras (Klaessens et al 2014, Cobos-Torres et al 2018, Gibson et al 2019), industrial and scientific cameras (Zhao et al 2013, Aarts et al 2013) or even augmented reality goggles (Fernando et al 2015). Image resolutions were typically 640 × 480 px and below, while high-definition (HD) resolutions have been rare (e.g. Villarroel et al (2014), or Paul et al (2017) and Antognoli et al (2018)). In the past, charge-coupled device sensors were used, but nowadays, complementary metal-oxide-semiconductor (CMOS) sensors are prevalent.

The PPGI was used with ambient light (natural, artificial, or a mixture of both) (Mestha et al 2014, Villarroel et al 2014, Cobos-Torres et al 2018), using green light (Vagedes et al 2004, Scalise et al 2012) or NIR light (Klaessens et al 2014, Blanik et al 2016).

Raw camera-based PPG time series are normally generated by spatially averaging pixels (e.g. Hülsbusch (2008)). Furthermore, if more than one color channel is available, these time series can be combined, for example, by independent component analysis (Scalise et al 2012, Villarroel et al 2014). Regions of interest (ROIs) have to be identified and tracked to form a continuous time series. The identification is still often performed manually in feasibility studies, and regions are tracked subsequently by typical algorithms.

Wang and colleagues (Wang et al 2018) have recently made one very interesting finding that can be used on single channels: when more non-skin pixels are part of an ROI, the spatial variance yields a better time series compared to spatial averaging. By way of contrast, when skin pixels dominate, spatial averaging would be preferred.

To estimate PR by also exploiting the spatial resolution, our group has used spatio-temporal mappings by applying grid-based approaches (Blanik et al 2016, Paul et al 2017).

Frequency approaches, such as wavelet analysis (Vagedes et al 2004), or Fourier-based analysis, such as power spectral densities/spectrograms (Vagedes et al 2004, Scalise et al 2012, Aarts et al 2013, Mestha et al 2014, Cobos-Torres et al 2018) have often been applied to estimate PR.

An approach that allows compensating for interference if a non-patient ROI is also available was used by Villarroel and co-workers (Villarroel et al 2014): for this, autoregressive models and pole cancellation techniques were used.

The ROIs are often identified manually or by segmentation. Recent research has been directed at segmentation, among others, by a convolutional neural network (Chaichulee et al 2017, Chaichulee et al 2018) or by fuzzy inference systems (Kaur et al 2017) to improve time series extraction.

In this work, the focus is on the feasibility of the PR assessment of neonates using videos exploiting PPG and ballistographic signals without making any further distinction between the two signals. For this, we present an algorithmic processing chain similar to the one used in Mestha et al (2014). Here, we also benchmark signals against a more simplistic approach by tracking the subjects as a whole. Furthermore, the algorithms are challenged by reduced amplitudes due to skin pigmentation in VIS. In addition, we would also like to highlight observations important for the deployment of camera-based sensing in neonatal applications, particularly noise sources for PPGI.

The hospital's standard bed, the Infant Radiant Warmer NWS 102 (Phoenix Medical System Private Limited, Chennai 600032, India) was used as a patient bed. Furthermore, two measurements were conducted in a Transport Incubator TINC 101 (Phoenix Medical System Private Limited, Chennai 600032, India) with two neonates who were born prematurely.

The subject and measurement characteristics of the 24 measurements are given in tables 1 and 2, respectively. If a subject has been measured multiple times, the data has also been included. Weight and size were unavailable for one subject; nevertheless, the measurement is also included. There are regional differences in skin tone in India: geographically, from north to south, skin tones become darker on average. This information was added as darker skin is assumed to post more challenges compared to lighter skin due to the increased amount of melanin.

Multiple cameras were used to record the babies resulting in the images given in figure 1 (radiant warmer) and figure 4 (incubator). Accordingly, specifications of all cameras used in the experiments are given in table 3 and below.

Zoom In Zoom Out Reset image size

Three sensitive CMOS cameras were used for PPGI: two Grasshopper 3 GS3-U3-23S6M-C and one color camera (GS3-U3-23S6C-C; FLIR, USA). The two monochrome cameras were each equipped with an optical filter: 850 nm (BN850) or 940 nm (BN940) (Midwest Optical Systems, Inc. USA). In the following, these cameras are identified as $\mathrm{CAM_{850}}$, $\mathrm{CAM_{940}}$ and $\mathrm{CAM_{RGB}}$, respectively. Similarly, color channels are identified as $\mathrm{CAM_{RGB~X}}$, where 'X' is 'R', 'G' or 'B'. The color channels from short to long wavelengths are characterized as follows: $B\mathrel{\widehat{ = }} 470\; {\rm nm}$, $G\mathrel{\widehat{ = }} 525\; {\rm nm}$ and $R\mathrel{\widehat{ = }} 640\; {\rm nm}$. The lenses used were the CF12.5HA-1 (Fujifilm Holdings K.K., Japan), a 12.50 mm fixed focal lens.

Zoom In Zoom Out Reset image size

Table 3. Cameras used in the experiment.

Fps Resolution ADC Lens Modality Camera Type (Hz) (px × px) Mode (bit) Trigger (mm) Optical filters Model

PPGI	$\mathrm{CAM_{850}}$	mono	25	1920 × 1200	7	12	o	12.5	NIR narrow bandpass; 850 nm	Grasshopper 3 GS3-U3-23S6M-C
	$\mathrm{CAM_{940}}$	mono	25	1920 × 1200	7	12	o	12.5	NIR narrow bandpass; 940 nm	Grasshopper 3 GS3-U3-23S6M-C
	$\mathrm{CAM_{RGB}}$	color	25	1920 × 1200	7	12	o	12.5	(internal) NIR cut-off ca. 700 nm	Grasshopper 3 GS3-U3-23S6C-C
IRT	$\mathrm{CAM_{IRT}}$	LWIR	25	640 × 480	NA	16	o	10	-	Gobi-640-GigE
	$\mathrm{CAM_{IRT,HD}}$	LWIR	30	1024 × 768	NA	16	-	30	-	VarioCAM HD head 820S
none	$\mathrm{CAM_{ref}}$	mono	10	1920 × 1200	7	12	-	12.5	-	Grasshopper 3 GS3-U3-23S6M-C

We used up to two separate cameras for IRT: a Gobi-640-GigE (Xenics nv, Belgium) was used for all measurements and, in the case of the radiant warmer measurements, a VarioCAM HD head 820S (InfraTec GmbH, Germany) LWIR camera was added. These cameras are identified as $\mathrm{CAM_{IRT}}$ and $\mathrm{CAM_{IRT,HD}}$, respectively.

Moreover, another camera, $\mathrm{CAM_{ref}}$, was used to record the reference monitor.

We used matched light sources for the two NIR cameras ($\mathrm{CAM_{850}}$, $\mathrm{CAM_{940}}$) consisting of an S75-850-W ('$\mathrm{LED_{850}}$'), and S75-940-W ('$\mathrm{LED_{940}}$') lamp (Smart Vision Lights, USA) with the intention of testing PPGI for night measurements and reducing possible stray light. These machine lighting lamps each consist of 2 × 3 light-emitting diodes (LEDs). Moreover, the light coming from these lamps was diffused by using Lee 416 filters (LEE Filters, UK) and stray light was reduced by covering all but the front of the lamps. The lamps were positioned above the respective camera to be as close to the matching cameras and provide frontal illumination.

No dedicated light source was used for $\mathrm{CAM_{RGB}}$. Thus, only the ambient light was used which was composed of the following:

• ambient light originating from ceiling-mounted lamps;
• sunlight entering from behind the camera setup; and
• the 'glow' of the heating elements of the radiant warmer.

All cameras, except $\mathrm{CAM_{IRT,HD}}$ and $\mathrm{CAM_{ref}}$, were triggered, and thus synchronized, by a self-built timing controller based on a MSP430F5329 microcontroller (Texas Instruments, USA). The timing controller can synchronize devices and power up to six of them. The timing controller was connected to an external 24 V medical power supply. In addition to the cameras, the LED lamps were powered via the controller. We had decided to forego triggering the LEDs to avoid problems with temporal light modulations.

Recordings were acquired and stored on a notebook, a Thinkpad P50 (Lenovo, China) which was equipped with two 1 TB SSDs (Samsung 960 Evo), a quad-core processor (Intel i7-6820HQ) and 32 GB RAM.

Two PPGI cameras and one IRT camera were connected directly to the notebook. Because the number of ports on the notebook was limited, the other cameras were connected via a separate USB 3.1 docking station (Delock 87298). For each measurement, about 240 GB were recorded totaling about 5.80 TB of raw video data.

A Radical-7 (Masimo, USA) pulse oximeter (the standard monitoring device in the neonatal unit of the hospital) was used for reference. The device displays the following measurements: the PR, perfusion index, peripheral oxygen saturation and the PPG signal. We recorded the monitor with $\mathrm{CAM_{ref}}$ at 10 Hz and extracted the numerical values via optical character recognition and visual inspection to obtain a synchronous reading. Thus, we used these monitor-extracted PR values ($\mathrm{PR}_\mathrm{ref}$) as the basis for the evaluation of the camera-extracted PR estimates $\mathrm{PR}_\mathrm{est}$.

Videos $V_{\lambda}$ were stored as three-dimensional cubes $V_{\lambda}(x,y,t)$ composed of raw pixel data ($2\ {\rm B\ px}^{-1} = 16\ {\rm bit\ px}^{-1}$). This preprocessing stage consisted of a step where each frame was rotated if a camera was mounted upside down, a stage where data was converted to 8 bit (for the tracking but not signal extraction) and a stage which separated different color changes if necessary (e.g. in the case of $\mathrm{CAM_{RGB}}$).

The separation into color channels for $\mathrm{CAM_{RGB}}$ only consisted of extracting the pixels corresponding to the Bayer pattern. Indeed, this is in contrast to other publications that use undefined demosaicing algorithms (e.g. videos were not recorded bayered). Consequently, we used raw pixel values for the videos $V_{\mathrm{R}}$, $V_{\mathrm{G}}$, and $V_{\mathrm{B}}$, where the resolution is only a quarter of $V_{\mathrm{RGB}}$ because conventional RGB sensors use twice the number of green pixels compared to red and blue. Thus, each pixel in $V_{\mathrm{G}}$ was calculated as the mean of the green pixels of these two virtual green channels.

Zoom In Zoom Out Reset image size

We selected several ROIs manually which are presented in table 4. To illustrate those ROIs, a sketch is given in figure 6. The relevance of the ROIs and the process of annotation are described in the following.

Zoom In Zoom Out Reset image size

2.4.2.1. ROI relevance

2.4.2.2. ROI annotation

Afterwards, we applied the kernelized correlation Filter (KCF) tracker on all regions selected (Henriques et al 2015). However, the KCF tracker was implemented without a failure recovery mechanism. Consequently, out-of-view targets became a problem.

The size of the template has a great impact on the processing time. This was considered by scaling the template and image in case a certain area (number of pixels) was surpassed.

Next, the raw signal, a time series, was generated by averaging all pixels of a rectangular region (ROI) spatially. In other words, each time series consisted of the concatenation of spatial averages extracted from a tracked ROI.

Subsequently, the signal was divided into overlapping signal segments of length $t_{\mathrm{seg}} = 10\ {\rm s}$, which, in our case, overlapped for all values except the oldest one (hop-size v = 1). The sampling frequency f_s of the times series was identical to the recording frequency of the videos: f_s = 25 Hz. We designed a finite impulse response bandpass filter of order 3 × f_s and cutoff frequencies $f_{c1} = 1.30\; {\rm Hz}\;\hat{ = }\;78\;{\rm bpm}$ and $f_{c2} = 5\; {\rm Hz}\;\hat{ = }\;300\; {\rm bpm}$ for temporal filtering. This filter was applied after subtracting the mean from a segment. The filter was used for zero-phase filtering using Matlab's function filtfilt, effectively doubling the filter order.

In the next stage, PR was retrieved from a time-frequency representation of several signal segments using the short-time Fourier transform (STFT).

The STFT consists of a series of Fourier transforms (FFTs) of temporal segments that can overlap. Before applying the FFT, a Hamming window was applied to reduce spectral leakage. We specifically chose the segment size to be $t_{\mathrm{seg}} = 10\ {\rm s}$ ($n_{\mathrm{seg}} = 250\ {\rm frames}$), the overlap to be $n_{\mathrm{ov}} = 249\ {\rm frames}$ and the length of the FFT to be $n_{\mathrm{FFT}} = 1024$. Furthermore, because $n_{\mathrm{FFT}}>n_{\mathrm{seg}} = 250\ {\rm frames}$, the remaining values were zero-padded. In addition, the segment size was chosen to cover about 20 heartbeats ($PR = 120\; {\rm bpm}\;\hat{ = }\;2\; {\rm Hz}$) and about six breaths ($BR = 40\; {\rm bpm}\;\hat{ = }\;0.67\; {\rm Hz}$).

2.4.7.1. PR estimation

We estimated the peak frequency $f_\mathrm{peak}$ in each FFT segment by extracting the frequency with the highest signal energy within the heart frequency band ($90\ {\rm to}\ 220\ {\rm bpm}\;\hat{ = }\;1.50\ {\rm to}\ 3.67\ {\rm Hz}$). Thus, a signal of $\mathrm{F}_\mathrm{peak}$ values at sampling frequency was generated. The PR estimates at 1 Hz ($\mathrm{PR}_\mathrm{est}$) were calculated by downsampling the mean estimate of $p = \mathrm{round}(f_s)$ $f_\mathrm{peak}$ values to compare with the pulse oximeter reference ($\mathrm{PR}_\mathrm{ref}$).

In order to estimate the peak frequency, it is sufficient to find the maximum of the absolute values of the positive FFT coefficients; the other computations are used for scaling and visualization using the spectrograms.

2.4.7.2. Artifact detection

Signal artifacts were clearly present in the raw time series. We used a signal quality index (SQI) to assess the ratio of energy $E_{f_{\mathrm{peak}}}$ at $f_\mathrm{peak}$ to the energy $E_{{\mathrm{tot}}}$ of the whole frequency spectrum, similar to that presented in Blanik et al (2016), for artifact detection. Here, we use

$$\begin{equation} \mathrm{SQI} = \frac{E_{f_{\mathrm{peak}}}}{E_{{\mathrm{tot}}}}, f_{\mathrm{peak}}\in [90\ {\rm to}\ 220\ {\rm bpm}]. \end{equation} \tag{ 1 }$$

In other words, low SQI values mark signal segments whose energy outside of the heart band is relatively high and, thus, may not be used to extract an estimate $f_\mathrm{peak}$. We used an empirical value, a lower threshold $\tau_{\mathrm{SQI}} = 0.075$, which was chosen based on one video sequence where there was only a little movement present to identify those signal segments.

The segments identified were then automatically flagged as artifacts: if one such segment was contained in the calculation of $\mathrm{PR}_\mathrm{est}$, the final estimate was flagged both as an artifact and unreliable.

All median values for bbox are below 60%, while most are below 10%, close to 0% coverage (figure 9). Considering the medians, the blue channel of the RGB camera is the best performing. By contrast, $\mathrm{CAM}_\mathrm{940}$ performed worst. The 75th percentiles for all channels of the RGB camera are above 80%. Additionally, the NIR cameras' 75th percentiles are below 10%, while outliers reach close to 70% ($\mathrm{CAM}_\mathrm{850}$) and close to 40% ($\mathrm{CAM}_\mathrm{940}$).

Selecting the best performing ROI improves the coverage compared to bbox (figure 10): medians go as high as 88.68% for $\mathrm{CAM_G}$, and similar values are reached for the other color channels. Moreover, the other medians stay above the best result of bbox. The 75th and 25th percentiles shift to higher values compared to bbox. Regarding the color camera, the 25th percentiles are above 50% coverage. In addition, the 75th percentile of $\mathrm{CAM}_\mathrm{850}$ improves to above 70%. Similarly, the 75th percentile of $\mathrm{CAM}_\mathrm{940}$ is higher than 10%, while occasional outliers reach more than 70%.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

The number of occurrences of the best performing ROIs in the given scenario considering all measurements are shown in figure 11. These ROIs are very often the right foot, nose, torso or face. The right arm and hand satisfied the condition equally often as bbox and the left hand. The head and the left arm were never the best performing.

Zoom In Zoom Out Reset image size

All median values for bbox are below 10% and close to 0% coverage (figure 12). The 75th percentiles for all cameras are below 30% but outliers can reach above 40%, even 70%. In all aspects, the NIR cameras perform better than the different channels of the RGB camera.

Zoom In Zoom Out Reset image size

Similar to $\mathrm{COV}_{\mathrm{3\ bpm/(cam,ref)}}$, selecting the best performing ROI (figure 13) improved the coverage when compared with bbox (figure 12): medians went as high as 24% for $\mathrm{CAM_{RGB,G}}$ and stay above the best result of bbox (~ 7% vs ~ 6% for $\mathrm{CAM_{940}}$). The 75th and 25th percentiles shift to higher values. To be more precise, the 75th percentiles range between 20 and 40% coverage. Occasionally, values above 50% and even above 70% were reached.

Zoom In Zoom Out Reset image size

The ROIs performing best regarding $\mathrm{COV}_\mathrm{cam,ref}$ are given in figure 14. Long coverages could often be achieved with torso, bbox, face or nose. Other ROIs occurred less often. Left leg and foot occurred seldom, while the left arm not even once.

Zoom In Zoom Out Reset image size

Light interacts differently with tissue depending, amongst others, on its wavelength. Shorter wavelengths (e.g. blue) are more strongly absorbed in the upper skin layers compared to longer wavelengths (e.g. NIR). The usage of NIR allows information about the vascular network of the neonate to be retrieved, as can be seen in figure 21. Here, we did not exploit this property, but the visible vascular network could be used as landmarks in tracking algorithms. Another potential application is vascular diagnostics.

Zoom In Zoom Out Reset image size

The recording frame rate was limited to 25 fps due to bandwidth constraints and light which could be used. We could observe by visual inspection that typical movement, for example, of the extremities, resulted in blurry images. We consider this blurring problematic when trying to extract the small signals associated with the PPG signal because the content of the ROI changes a lot relatively and locating the correct ROI position is more difficult. The blurring is caused, for example, by exposure times being too long (19.50 ms) and not matched with the movement speed. The frame rate used in combination with the long exposure time is certainly sufficient to capture the PR at rest. When faster movement is involved, the tuning of these parameters is necessary. On the positive side, a higher frame rate would allow one to shrink the search region for tracking algorithms and lower exposure times would reduce the blurring. On the negative side, both actions reduce the amount of light available for imaging.

We observed strong movement of the neonates themselves and that induced by caregiving, which resulted in light intensity changes. Furthermore, a ceiling fan positioned directly beneath a ceiling light also introduced light changes. Fortunately, the frequency range affected was above the PR band and below the Nyquist frequency (here, $f_\mathrm{nyquist} = 12.50\; {\rm Hz}$) due to the fan's rotating speed. In addition, the room was equipped with air conditioning. Using the IRT cameras, we could see intensity/temperature changes which we, at the moment, attribute to air movement. The effect of these temperature changes on the patient's comfort and signal changes in the PPGI signals need more attention as patient discomfort could cause more movement.

The medical devices used in the study were also identified as noise sources: radiant warmer, incubator and PPG reference. The radiant warmer is built to direct thermal radiation at the baby's body. However, the reflective material of the warmer does not discern between the sources of radiation and just reflects everything, such as light and the thermal radiation of the clinical staff. Specifically, we suspect that light from the glowing heating filaments is modulated at a low frequency and thus disturbed the PPGI cameras.

Similar problems could be observed regarding the incubator: in order to be able to film the inside of the incubator not only with the PPGI cameras but also with an IRT camera, a thermal window was realized by applying a thin but visibly transparent plastic foil. While it helped to sustain the microclimate, it caused unwanted reflections in combination with light from behind the measurement setup and from the measurement lighting. We found that optical measurements from outside the incubator will require additional engineering prior to commercialization, for example, modifications to the incubator. In the meantime, cameras should be positioned as close to the incubator casing as possible.

The reference PPG probe uses pulsed light instead of constant light. Thus, camera PR estimates of this measurement site were influenced by this light source and are, thus, not reliable. Covering the body parts affected should not be the first option. Instead, if the probe used constant light (or was synchronized with the setup), it would have helped the cameras.

All in all, the influence of medical equipment on remote signal retrieval cannot be neglected. Thus, we suggest moving cables and contact-based sensors to locations out of the FOV of cameras when evaluating contact-free methods. By contrast, when deploying the techniques in the field, we recommend to use sensors which can be used as optical markers for the cameras. The same applies to clothing and blankets: a complex pattern should be preferred to uniform colors. If possible, clothing should be chosen in such a way that it moves even under slight body movement (e.g. breathing movement).