Harnessing the Web for Population-Scale Physiological

Sensing: A Case Study of Sleep and Performance

Tim Althoff⇤ Eric Horvitz Ryen W. White Jamie Zeitzer
Stanford University Microsoft Research Stanford Center for Sleep Sciences
[email protected] {horvitz,ryenw}@microsoft.com and Medicine
[email protected]
ABSTRACT 1 INTRODUCTION
Human cognitive performance is critical to productivity, learning, Maintaining optimal cognitive performance has been found to be
and accident avoidance. Cognitive performance varies throughout important in learning [26], productivity [16], and avoiding industrial
arXiv:1701.07083v1 [cs.HC] 21 Jan 2017

each day and is in part driven by intrinsic, near 24-hour circadian and motor vehicle accidents [16, 20]. Studies have demonstrated that
rhythms. Prior research on the impact of sleep and circadian rhythms cognitive performance varies throughout the day [42], likely influ-
on cognitive performance has typically been restricted to small-scale encing the quality of our efforts and engagements–including how we
laboratory-based studies that do not capture the variability of real- use and interact with vehicles, devices, resources, and applications.
world conditions, such as environmental factors, motivation, and Furthermore, cognitive performance is decreased significantly after
sleep patterns in real-world settings. Given these limitations, leading loss of sleep [20]. Understanding the real-world impact of sleep defi-
sleep researchers have called for larger in situ monitoring of sleep ciency is critical. It has been estimated that the cost of fatigue to U.S.
and performance [39]. We present the largest study to date on the businesses exceeds $150 billion a year in absenteeism, presenteeism,
impact of objectively measured real-world sleep on performance workplace accidents, poor and delayed decision-making and other
enabled through a reframing of everyday interactions with a web lost productivity on top of the increased health care costs and risk of
search engine as a series of performance tasks. Our analysis includes disease [24]. Despite the important influences, temporal variations
3 million nights of sleep and 75 million interaction tasks. We mea- of real-world performance are not well understood and have never
sure cognitive performance through the speed of keystroke and click been characterized on a large scale [39].
interactions on a web search engine and correlate them to wearable Models of daily patterns in human cognitive performance rely
device-defined sleep measures over time. We demonstrate that real- typically on representations of three biological processes: circadian
world performance varies throughout the day and is influenced by rhythms (time-dependent, behavior-independent, near 24-hour oscil-
both circadian rhythms, chronotype (morning/evening preference), lations) [42], homeostatic sleep pressure (the longer awake, the more
and prior sleep duration and timing. We develop a statistical model tired you become) [13], and sleep inertia (performance impairment
that operationalizes a large body of work on sleep and performance experienced immediately after waking up) [4, 19].
and demonstrates that our estimates of circadian rhythms, home- While models of these biological processes capture well the pat-
ostatic sleep drive, and sleep inertia align with expectations from terns of cognitive performance in the laboratory [4, 13], they are
laboratory-based sleep studies. Further, we quantify the impact of based on experimental studies in which participants are deprived of
insufficient sleep on real-world performance and show that two con- sleep and undertake regular, artificial tasks to measure performance
secutive nights with less than six hours of sleep are associated with instead of non-intrusively capturing performance through everyday
decreases in performance which last for a period of six days. This tasks in real-world environments. In addition, these studies typically
work demonstrates the feasibility of using online interactions for include participants that fit a specific physical and psychological
large-scale physiological sensing. profile (e.g., those with depressed mood are often excluded). Further,
participants in an artificial setting can be influenced by their under-
ACM Reference format:
standing of the study and subconsciously change their behavior to fit
Tim Althoff, Eric Horvitz, Ryen W. White, and Jamie Zeitzer. 2017. Harness-
ing the Web for Population-Scale Physiological Sensing: A Case Study of
the interpretation of its motivation and goals [35]. While laboratory
Sleep and Performance. In Proceedings of WWW conference, Perth, Australia, studies have been critical in developing understandings of the basic
April 2017 (WWW’17), 10 pages. biological processes that underlie cognitive performance, they fail to
DOI: 10.475/123 4 account for myriad influences in the real-world, including motivation,
mood, illness, environmental conditions, behavioral compensation
⇤
Research done during an internship at Microsoft Research. including caffeine intake, and sleep patterns in the wild that are far
more complicated than those enforced in research studies. How these
and other factors alter real-world cognitive performance is not well
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
understood. Therefore, sleep scientists have called for large-scale
for profit or commercial advantage and that copies bear this notice and the full citation real-world measurements of performance and sleep as a necessary
on the first page. Copyrights for components of this work owned by others than ACM step to “to transform our understanding of sleep” and “to establish
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a how to manage sleep to improve productivity, health and quality of
fee. Request permissions from [email protected]. life” [39].
WWW’17, Perth, Australia
© 2017 ACM. 123-4567-24-567/08/06. . . $15.00
DOI: 10.475/123 4
WWW’17, April 2017, Perth, Australia Tim Althoff, Eric Horvitz, Ryen W. White, and Jamie Zeitzer

This Work. We respond to the appeal from the sleep research com- that may be affected by variations in human performance, and to the
munity with a large-scale study of sleep and performance enabled growing community of researchers who have been exploring uses
through reframing everyday interactions with a web search engine of data from online activities to address questions and challenges in
as a series of performance tasks. In particular, we use individual the realm of public health.
keystrokes when typing a search query and the clicks on search
results as a source of precisely timed interactions. We demonstrate
that the timing of these interactions varies based on biological pro- 2 RELATED WORK
cesses and can be used to study the influence of different quantities
of sleep on performance. Search engine interactions offer insight Circadian Processes in Sleep and Performance. Empirical stud-
about real-world cognitive performance as they are an integral part ies have found daily rhythms in human performance including alert-
of many people’s lives and work every day. More than 90% of US ness, attention, reaction time, memory, and higher executive func-
online adults use web search engines, which now handle billions of tions such as planning [11]. The daily variations in performance
searches each day [38]. have been found to be modulated primarily by two processes [18]:
Our dataset comprises over 3 million nights of sleep tracked by a circadian rhythm (time-dependent, behavior-independent, near
wearable sensors from 31 thousand users over a period of 18 months 24-hour oscillations) [42] and a homeostatic sleep drive (the longer
and 75 million subsequent real-world performance measurements awake, the more tired we become and the more we sleep, the less
based on keystrokes and clicks within a web search engine (Sec- tired we become) [13]. The circadian rhythm acts in opposition
tion 3). This constitutes the largest prospective study of real-world to the homeostatic drive for sleep that accumulates across the day,
human performance and sleep to date (more than 400 times larger enabling a single, consolidated period of wakefulness throughout
than the second largest comparable study which had only 76 partici- the day. A third process has been proposed called sleep inertia [42],
pants [29]). which corresponds to the performance impairment experienced im-
We first demonstrate that real-world human cognitive perfor- mediately after waking up [4, 19]. In addition to the influence of
mance captured through search engine interactions varies throughout daily rhythms on the structure of sleep and performance, there are
the day in a daily rhythm (Section 4). We find that performance is also shorter, 90-minute oscillations, ultradian rhythms, that organize
lowest during habitual sleep times when it is reduced by up to 31%. the occurrence of NREM and REM stages during sleep. Ultradian
Both the shape and magnitude of this temporal variation are consis- rhythms, circadian rhythms, and homeostatic sleep pressure can all
tent with controlled laboratory-based studies, providing validation impact the structure, and likely function, of sleep [17].
of our large-scale performance measures. We also show that per- Human preferences and natural tendency in the relative timing of
formance varies based on chronotype (morning/evening preference) sleep and wake are called chronotypes and are at least partly based
with early risers performing slowest at 04:00 h (4am) and late risers on genetics [40]. Cognitive performance depends on chronotype
performing slowest at 07:00 h. and time of day [31]; that is, early/morning types (“lark”) tend to be
We then develop a statistical model based on chronobiological higher performing earlier in the day while late/evening types (“owl”)
research and demonstrate that it successfully disentangles circa- are higher performing later. Sleep deprivation has been linked to
dian rhythms, homeostatic sleep drive, sleep inertia, and prior sleep significant decreases in cognitive performance that lead to increased
duration—key factors considered in the sleep literature (Section 5). risk for accidents and injury [20].
We quantify that performance varies by 23% based on time of day, A recent study correlated performance on cognitive exercises with
by 19% based on time since wake up, and by 5% based on sleep a sleep measure based on retrospective self-reports of “typical sleep”
duration (Section 5.3). We validate our methodology by demon- in 160 thousand users [41]. However, this measure suffers from
strating close agreement between our model estimates based on a potential biases [28] and does not enable the study of performance
large amount of performance measurements in the wild and smaller variation over time based on time of day and sleep timing. Another
controlled sleep studies in artificial laboratory settings. study showed that insomnia with short sleep is associated with cogni-
After validating our approach, we extend prior laboratory-based tive deficits in 678 subjects [22] but only measured a single night of
sleep research through estimates of how sleep impacts performance sleep to characterize typical sleep patterns after taking performance
in real-world settings. In particular, we quantify the impact of one measurements, leading to similar limitations. According to a recent
or multiple nights of insufficient sleep on real-world performance meta-analysis [29], the largest study that measured both sleep and
(Section 6). We demonstrate that very short and very long sleep performance concurrently had 76 participants.
durations, and irregular timing of sleep are associated with 3%, 4% Technology Use and Interaction Patterns. Interaction patterns of
and 7% lower performance, respectively. We also show that two different devices and applications have been studied on small scale
consecutive nights with fewer than six hours of sleep are associated to better understand mobile device usage [12], to detect stress [43],
with significantly decreased performance for a period of six days. used as biometric signals for authentication [32], and linked to
Our study is also the first to demonstrate that ambient streams of biological processes [33, 34] including alertness [1]. For example,
data, such as patterns of interactions with devices, can be harnessed less sleep was linked to shorter duration of focus of attention in a
as large-scale physiological sensors to study and continuously and study with 40 participants [30]. Large-scale interaction data have
non-intrusively monitor human performance at population scale. been used to gain insights into human behavior in the areas of mood
The insights and methodology developed in this work are relevant to rhythms [23], diet [45], conversation strategies [5], social networks
sleep scientists in pursuit of larger-scale real-world measurements of and mobile games encouraging health behaviors [7, 8], and health
performance, to computer scientists who build tools and applications and disease-related search behaviors [36, 46].
Harnessing the Web for Population-Scale Physiological Sensing WWW’17, April 2017, Perth, Australia

Dataset Statistics gender, body mass index) are self-reported through the Microsoft
Observation period 18 months Health app. While the user age and overweight/obesity status closely
# users 31,793 track official estimates in the United States, we note that our sample
# nights of sleep tracked 3,102,209 is predominantly male.
# queries 24,590,345
# filtered queries with clicks 6,906,791 Performance. We measure performance through the timing of two
# keystrokes extracted 68,779,113 types of interactions with a search engine (Microsoft Bing): (1)
# total interactions 75,685,904 individual keystrokes within the search box that are tracked by the
Average keystroke time 225ms search engine so it can automatically suggest query completions,
Average click time 9.28s and (2) clicks on the result page after a search query. Section 4.1
Median age 38
provides more details on each of these measures and we discuss
% female 6.1%
% underweight (BMI ¡ 18.5) 1.4% how to account for potential confounds such as the type of query
% normal weight (18.5  BMI ¡ 25) 32.4% in Section 5.1. We exclude search engine interactions originating
% overweight (25  BMI ¡ 30) 39.2% from mobile devices since such interaction patterns and timing are
% obese (30  BMI) 27.0% fundamentally different from those on desktop devices. While users
Median time in bed 7.26h could potentially access the search engine from multiple machines,
Table 1: Dataset statistics. BMI refers to body mass index. we note that for most users this is unlikely to be the case and that
using different keyboards and mice throughout the day is unlikely to
0
explain the timing differences observed in this work.
) Sleep. Sleep data from wearable devices provides objective measure-
ments which have been preferred to subjective self-reports that may
be significantly biased [28]. To estimate sleep, we consider signals
from wrist-worn activity trackers (Microsoft Band) that include a
3-axis accelerometer, gyrometer, and optical heart rate sensor. The
Microsoft Band employs internally validated proprietary algorithms
7

for estimation of sleep and we focus on duration of time in bed

(herein referred to as “sleep duration”). Time in bed is delineated
either by manual input of the user (i.e., explicit taps on the device
Figure 1: Average sleep duration across age and gender. Our
before going to sleep and immediately after waking up) or auto-
measurements are consistent with previous estimates [10, 14,
matically based on movement if the user does not provide manual
44] (Section 3). Error bars in all figures correspond to 95%
input. The use of an event marker to denote bed timing is widely
confidence intervals of the corresponding mean estimates.
used in sleep research in lieu of or in concert with sleep diaries [9].
Following standard practice [44], we exclude any sleep duration
This Work. Existing research on sleep and performance is either measurements below 4 and above 12 hours of time in bed.
small-scale and laboratory-based [29] or relies on subjective mea- As evidence that our sleep measurements have face validity, we
sures such as surveys capturing “typical” sleep [41] which do not show that they match published sleep estimates. Figure 1 illustrates
allow for temporal coordination of sleep and performance mea- average time in bed across age and gender. Time in bed decreases
surements. As a complement and extension of research to date on with age and is higher in females than males consistent with pub-
performance in artificial laboratory settings, we study real-world lished estimates [10, 14, 44]. Walch et al. [44] report very similar
cognitive performance which we measure through interactions with times and a difference of 17 minutes between females and males.
a web search engine. We use objective measurements of sleep (time With the exception of 60 to 70 year old subjects, we find differences
in bed) from wearable devices which are preferred to subjective between 12 and 17 minutes. There is no difference for older subjects,
self-reports that can be significantly biased [28] and that enable us which matches survey-based estimates by Basner et al. [10]. We
to study performance variation over time in reference to sleep tim- take these alignments with published research as evidence for the
ing. This work represents the largest study of objectively measured validity of using wearable device-based sleep data for large-scale
sleep and real-world performance to date, employing a subject pool population studies of sleep and performance.
that is orders of magnitude larger than the largest comparable prior
study [29]. Our study demonstrates on a large scale that interactions
with devices are influenced by biological processes and sleep. 4 PERFORMANCE MEASURES BASED ON
INTERACTIONS DURING SEARCH
3 DATASET Next, we describe two human performance measures derived from
Our dataset contains over 75 million search engine interactions and search engine interactions that we use to study daily variation in
sleep measurements for 31,793 US users of Microsoft products who performance. We show how these measures exhibit variations in per-
agreed to link their Bing searches and Microsoft Band data for use in formance over time and based on chronotype (morning/evening pref-
generating additional insights or recommendations about their sleep erence) consistent with findings from laboratory-based sleep studies.
or activity. Basic dataset statistics and demographic information on This demonstrates that performance signals generated from everyday
the users are summarized in Table 1. Demographic variables (age, search engine interactions vary based on biological processes. We
model these processes and influences explicitly in Section 5.
WWW’17, April 2017, Perth, Australia Tim Althoff, Eric Horvitz, Ryen W. White, and Jamie Zeitzer
(a) (b)
Worse performance
H

H
H
H
H

Better performance

H K H H H

Figure 2: Time of day-dependent variation in keystroke (a) and click timing (b). Higher values indicate worse performance. Both the
shape of temporal variation with fastest performance a few hours after wake and slowest performance during habitual sleep times as
well as the magnitude of variation are consistent with controlled laboratory-based studies [3, 18, 20, 47] (Section 4.2).

time between the search query and the first click on any result on
H H HH the first page. Click times over two minutes are excluded since they
might stem from interrupted sessions. We account for click position
and query type as described in Section 5.1.
We believe that investigating measures that capture performance
H

on two different tasks provides robustness and breadth to our analy-

H

ses. The two tasks rely on different mixes of sensing, reflection, plan-
ning, and formulating, executing, and monitoring of motor plans [37].
Studies of the potential subprocesses for each task and how they
H

might be differentially influenced by sleep is beyond the scope of this

paper. However, our search engine interactions capture performance
in everyday tasks that are highly relevant to many occupations, as
H K H captured by typing and searching for information [38], and allow
us to non-intrusively measure changes in real-world performance
Figure 3: Variation in keystroke time throughout the day varies throughout the day.
with chronotype (morning/evening preference) which is defined Note that all timing measurements are taken on the server side
based on the average point of mid sleep (Section 4.3). Users and not the client side. Therefore, it is important to consider the
that typically sleep early (light color) perform slowest at about potential influence of network latency factors. We found that the
04:00 h, while medium or late sleepers (darker colors) perform network latency changes only very little between two consecutive
slowest at 05:00 h and 06:00-07:00 h, respectively. This closely requests (less than 1 millisecond) and thus any latency effects cancel
matches their habitual sleep time and is consistent with con- out when we take the time difference between two requests (details
trolled laboratory-based studies [31]. in online appendix [6]). This demonstrates that variation in network
latency does not affect our analyses. Furthermore, variations in site
4.1 Performance Measures rendering time (i.e., measuring time from first script till page load
completed including dynamic contents) are much smaller (order of
We study two real-world performance measures in this work since
milliseconds) compared to variation in click times.
it is possible that different measures would respond differently to
The temporal variation sensed in performance could potentially
sleep deprivation as sleep studies have shown differential effects of
be an artifact of different users contributing timings at different
sleep deprivation on different measures of cognition.
time points instead of actual within user variation throughout the
Keystroke Time. The first measure is based on keystroke timing. day. However, we verified that the temporal variation we observe
The search engine’s search box registers every single keystroke and is due to within user variation throughout the day by confirming
sends a request for query completions to the search engine’s servers. that the patterns of temporal variation are effectively identical for
We use the timing between two such requests as the time of a single raw measurements and within-user normalized variants (Z-scores;
keystroke if the two queries are different by exactly one character online appendix [6]). We also verified that performance variation
(not every request is received on the server side) and within two during the weekend is similar to variation during the week (online
seconds (larger times indicate longer thought processes or separate appendix [6]) and we therefore do not further differentiate between
sessions). This threshold is sensible as an average keystroke by an performance during weekdays and weekends in this paper. Finally,
average typist takes about 240 milliseconds (50 words per minute at we considered alternative performance measures based on backspace
5 characters per word [15]). usage in keystrokes and spelling errors in search queries. Since
Click Time. The second measure is based on the time to click on a we found results to be similar to keystroke and click timing but
search result after a search result page is displayed. We measure the more noisy due to less frequent measurements, we report results on
keystroke and click timing in this paper.
Harnessing the Web for Population-Scale Physiological Sensing WWW’17, April 2017, Perth, Australia

4.2 Temporal Variation of Keystroke and closely matches each group’s habitual sleep time and demonstrates
Click Times the validity and power of this large dataset; for each chronotype
group, we have millions of measurements even during typical sleep
Next, we validate our methodology by considering the findings ob-
times that allow us to estimate these performance curves. We find
tained from small-scale controlled sleep studies. It is well established
similar results for click times.
that human performance varies over time and follows a circadian
rhythm [3, 47]. Keystroke and click timing also vary throughout the
day in a daily rhythm as illustrated in Figure 2. Keystroke times (Fig- 5 MODELING PERFORMANCE
ure 2a) are on the order of 240 milliseconds which closely matches Having demonstrated that performance of search engine interactions
the expected typing speed of an average typist (240 milliseconds; 50 vary over time and based on biological processes (Section 4), we now
words per minute at 5 characters per word, see [15]). Click times operationalize and extend a conceptual model of sleep and perfor-
(Figure 2b) are on the order of 10 seconds. Note that both measures mance from chronobiology [4, 13] to explain the variation observed
follow a similar pattern throughout the day. Users are fastest to in performance measurements. Classic sleep models are based on
type and click a few hours after typical wake times and the timing circadian rhythms and homeostatic sleep drive [13]. In addition, we
increases again in the evening hours (in particular for click times). consider sleep inertia and sleep duration [4, 42]. Background on
Performance is slowest during habitual sleep times (e.g., 04:00 h) relevant biological processes is covered in Section 2.
closely matching accident risk rates [20] and the anticipated cir-
cadian nadir (i.e., the time of greatest circadian sleep drive) [18].
Furthermore, controlled laboratory experiments have shown that 5.1 Conceptual Model
performance typically varies by 15 to 30 percent over the course of We model the keystroke and click timing based on (1) time of day in
a day across a variety of simple motor and cognitive tasks [3, 47]. local time, (2) time in hours after wake up, and (3) sleep duration the
For keystrokes we measure a variation of 31% and for click times a previous night. We know (1) from the time of the keystroke or click
variation of 12%. time measurement, and (2) and (3) from wearable device-defined
The consistent agreement in shape and magnitude of variation sleep measurements (Section 3).
with controlled lab experiments on human performance and for two Since many people wake up during the same morning hours every
different tasks suggest that these large-scale measures based on day, time of day and time since wake up are naturally correlated and
search engine interactions can be used to study sleep and perfor- challenging to disentangle. In laboratory-based sleep studies, the
mance. The proposed measures can be collected non-intrusively at goal of exploring the distinct influences of the factors is achieved by
unprecedented scale and shine light on how real-world performance “forced desynchrony” protocols [42], where subjects are deprived of
varies throughout the day and with changes in sleep. sleep for extended periods of time. Instead of similar interventions,
we employ mathematical modeling with a large-scale dataset of real-
4.3 Performance Variation by Chronotype world sleep and performance measurements and use the variation
observed across millions of observations to disentangle the relative
A person’s chronotype encompasses the propensity for the individual
contributions of circadian and homeostatic factors. The large-scale
to sleep at a particular time during a 24-hour period and is at least
dataset contains numerous performance measurements during usual
partly based on genetics [40]. Studies have shown that performance
(day) and unusual (late night) times (e.g., Figure 3) that we can use to
depends on the alignment of chronotype and time of day [31]; early
understand the relative contributions of these factors to performance
types tend to be higher performing earlier in the day while late types
in the open world (see formulation of additive model in Section 5.2).
are higher performing later. The individual chronotype of each user
can be defined based on the mid-sleep point on free days (M SF ) Potential Confounding Factors. We control for several factors in
which is the halfway point between going to sleep and waking up [25, our model to avoid confounding. For keystrokes, we control for the
40]. Many people compensate for slept debt accumulated during exact character typed or removed since different characters might
work days by sleeping longer on free days; that is, the sleep midpoint take a varying amount of time (e.g., typing an “a”, or a capital “A”,
we observe is later than the internal biological clock would dictate or hitting backspace). For click times, it is expected that clicking
on the free days. Therefore, sleep scientists use a midsleep point on results further down the list of results will take more time, which
that is corrected for oversleep (indicated by SC) [25]: M SFSC = holds true in our data (online appendix [6]). We therefore control
M SF 0.5(SDF (5 ⇤ SDW + 2 ⇤ SDF )/7), where SDF and for the click position in our model.
SDW are sleep duration and free days and work days, respectively, Clicking on a result link is preceded by a cognitive process–
and SDF (5 ⇤ SDW + 2 ⇤ SDF )/7 corresponds to the difference interpreting the words displayed on links and deciding which link to
in sleep duration on free days and the average day. We compute click–which can be quick in the case of navigational queries (e.g.,
this corrected midpoint for every user in the dataset using weekdays “facebook”) or much slower in the case of informational queries
as work days and weekend days as free days (Median M SFSC = (e.g., “What is the homeostatic sleep drive?”). Formally, this dis-
4.70). tinction can be captured through the concept of click entropy, which
We show that keystroke times throughout the day vary with chro- measures how “surprising” the distribution over clicked URLs for a
notype (Figure 3), matching results from previous sleep studies [31] given query is [21]. We find that informational queries take about
and thus providing further validation of our methods. We find that two seconds longer than navigational queries on average (online
early sleepers are slowest at about 04:00 h, while medium or late appendix [6]). Therefore, we control for the click entropy of the
sleepers are slowest at 05:00 h and 06:00-07:00 h, respectively. This query preceding the click in our model.
WWW’17, April 2017, Perth, Australia Tim Althoff, Eric Horvitz, Ryen W. White, and Jamie Zeitzer
(a) (b)
2.5

Contribution to click timH (sHconds)

2.0
H
H

1.5

1.0
H

0.5

0.0

−0.5
0 6 12 18 24
H H Hours sincH midnight local timH
(c) (d)
1.5

Contribution to click timH (sHconds)

H

1.0
H

0.5
H

0.0

−0.5

−1.0
0 4 8 12 16
H H Hours aftHr wakHup

(e) (f)
0.3
Contribution to click time (seconds)

0.2

0.1

0.0

−0.1

−0.2

−0.3

−0.4
4 5 6 7 8 9 10 11
7 7ime in bed (hours)

Figure 4: Contributions to keystroke (a,c,e; blue) and click time (b,d,f; red) performance of different factors included in our model.
Results are similar for both performance measures and match estimates from controlled sleep studies in the laboratory (Section 5).
For example, variation over the time of day ct (a,b) shows that performance is slowest during habitual sleep times near the presump-
tive circadian nadir (04:00 h; see main text). Variation across time after wake up cw (c,d) shows effects of sleep inertia during the
first two hours after wake. There is relative stability for around eight hours in keystroke time but a steady decline in click time after
that point. Sleep durations cd (e,f) of 7.0-7.5 hours are associated with optimal performance according to our measures. However,
note that the impact on overall variation is smaller compared to time of day (a,b) and time since wake up (c,d).

An extreme way of controlling for varying queries is to compare evidence for improving performance over time for frequently occur-
click times for exactly identical queries (e.g., popular queries such as ring queries. This is likely because most users were fairly proficient
“facebook”). We verified that this yields very similar results, albeit at typing before the start of our observation period.
with larger confidence intervals since the sample size is reduced
dramatically compared to including all queries and controlling for
click entropy, demonstrating that the observed patterns are not due 5.2 Mathematical Formulation
to a particular mix of query types. We now describe the formulation of the model for keystroke timing.
In addition, we tested for learning effects as issuing the same The model for click times is parallel, where we control for the click
query multiple times might lead to improved performance. However, position and click entropy instead of the keystroke type. We are
most queries, 73.1%, are unique in the dataset and only 4.1% of interested in estimating how (1) time of day, (2) time after wake up,
queries occur more than three times. Further, we did not find any and (3) sleep duration influence performance. We assume that all
these effects are additive as supported by evidence presented in [2].
Harnessing the Web for Population-Scale Physiological Sensing WWW’17, April 2017, Perth, Australia

Mathematically, we formulate a fixed-effects model Within the first two hours, performance rapidly improves (i.e., lower
yi = ↵ + f k (xki ) + f t (xti ) + f w (xw d d
i ) + f (xi ) + ✏i ,
timings). This demonstrates a well-known effect in sleep studies
where yi is the keystroke time for observation i, ↵ is a constant in- called sleep inertia (Section 2). After this point, performance is best
tercept, and f k , f t , f w , f d are the unknown functions of interest for and slowly worsens until a point of poorest performance is reached
keystroke type, time of day, time since wake up, and sleep duration, at around 16 hours of wake time, consistent with the homeostatic
respectively, with corresponding input features xki , xti , xw sleep drive [13]. This corresponds exactly to the point when most
i , xi , and
d

✏i is the i-th residual. people would go to sleep again (i.e., a typical sleep duration of 8
Instead of estimating arbitrary functions, we use fine-grained hours). We excluded data beyond the typical wake period of 16
piecewise constant approximations. We discretize each input space hours because the data becomes more sparse and to avoid potential
(e.g., between midnight and 01:00 h, or between 01:00 h and 02:00 h, selection effects with regard to the people who choose to stay awake
or between 0 and 15 minutes after waking up, etc.). We denote the for exceptionally long periods of time. However we found similar
functions mapping input features xti , xw patterns between both keystrokes and click times even beyond this
i , xi to their respective bins
d
t w d k
as b , b , b (note that keystroke type xi is already discrete). Further, point. We note that keystroke time is relatively stable for about six
we use the functions ck , ct , cw , cd to map the discretized features to hours while click times continuously increase, likely due to the dif-
a constant value. The simplified model then becomes ferences in cognitive and motor competencies for the tasks, and due
to differences in the sensitivities of those competencies to status of
yi = ↵ + ck (xki ) + ct (bt (xti )) + cw (bw (xw d d d
i )) + c (b (xi )) + ✏i . sleep and circadian rhythm. In summary, the estimates derived from
The outcome of interest in this modeling task are the functions our model closely capture the initial sleep inertia and the increasing
ct , cw , cd which express the independent impact of (1) time of day, homeostatic sleep drive first discovered through laboratory-based
(2) time since wake up, and (3) sleep duration on performance tim- studies [4, 42, 48].
ings the next day. We estimate all parameters (↵, ck , ct , cw , cd )
including 95% confidence intervals through least squares optimiza- Time in Bed. Keystrokes and click time vary with the amount of
tion. We also experimented with mixed effects models controlling time in bed during the previous night (Figure 4e,f). However, we
for variation across users and across queries through random effects. note that this variation, 12 milliseconds for keystrokes (5%) and
While standard mixed model libraries do not scale well to the size of 0.25 seconds for click times (3%), is much smaller than the previous
our dataset, we found that these models lead to very similar estimates two factors. For both measures, we find a clear U-shaped curve with
compared to the fixed effects model described above when using its center, indicating optimal performance, at 7.0-7.5 hours of sleep.
subsets of the data. Both sleeping too little (under 7 hours) or too much (more than
8-9 hours) are associated with decreased performance. U-shaped
relationships with respect to sleep duration have been reported for
5.3 Results several outcomes (e.g., mortality [27]). We further investigate the
The functions ct , cw , cd modeling the influence on cognitive perfor- impact of insufficient sleep on performance in Section 6.
mance of time of day, time since wake up, and sleep duration are
illustrated in Figure 4. Impact on keystroke timings are shown in 6 INFLUENCE OF INSUFFICIENT SLEEP ON
blue (Figure 4a,c,e) and impact on click times are shown in red (Fig- PERFORMANCE
ure 4b,d,f). Note that the shapes of these functions for keystrokes
and click times are very similar and smooth, even though there are Following our studies to validate the methodology (Section 4 and
no constraints that would force this to occur. Furthermore, we note Section 5), we now extend prior laboratory-based sleep research
that the temporal variation in cognitive performance is not explained with estimates of how sleep influences performance in real-world
by variation in different users that contribute timings at different settings. In particular, we study the impact of one or multiple nights
points throughout the day (i.e., population differences) but are due of insufficient sleep on performance over the following days.
to within user variation (online appendix [6]).
6.1 Single Nights of Insufficient Sleep
Time of Day. Cognitive performance on both keystroke and click
We first consider single nights of sleep and analyze how very short
tasks varies with time of day (Figure 4a,b) and is slowest during
or very long sleep durations, as well as differences in sleep timing
habitual sleep time around 04:00-06:00 h. Performance quickly
from the usual patterns within a user, impact performance. We only
improves after typical wake times and becomes slightly slower in
show results for keystroke timing here; the results are similar for
the evening for both keystroke and click times (19:00 h). The two
click times (e.g., Figure 2 and Figure 4). Figure 5a shows that users
curves consistently match estimates of circadian rhythm processes
performed significantly slower when in bed fewer than 6 or more
in sleep obtained through controlled laboratory experiments [18, 48].
than 9 hours, consistent with the results described in Section 5.3. In
Note that the magnitude of variation is substantial at around 40
those conditions, the average keystroke times were about four and
milliseconds for keystrokes and over 2.1 seconds for click times,
seven milliseconds longer compared to sleeping between 7 and 9
which are changes of 18% and 23%, respectively, relative to average
hours (increases of 2.7% and 4.0%, respectively; both p ⌧ 10 10 ;
timing for each (Table 1).
Mann–Whitney U-test, which is used for all hypothesis tests in this
Time after Awakening. Cognitive performance also varies substan- section).
tially with the time after wake up (Figure 4c,d). The magnitude of Timing of sleep is also a significant factor for performance the
the variation is relatively large at about 42 milliseconds or 19% for next day (Figure 5b). While sleeping earlier than usual makes only
keystrokes about slightly over 1.6 seconds or 17% for click times. a difference of about 1 milliseconds or 0.5% (p ⌧ 10 10 ), going to
bed an hour or more later than usual is associated with significantly
WWW’17, April 2017, Perth, Australia Tim Althoff, Eric Horvitz, Ryen W. White, and Jamie Zeitzer
(a) (b)
6 HHS S HH H
H H H K

H
H

H
7 H H K
H

H
H H H H S
Figure 5: The impact of sleep duration (a) and timing (b) on performance the next day. Sleep timing is measured through difference
from the typical sleep midpoint and we control for sleep duration. We find that sleeping less than 7 or more than 9 hours is associated
with slower performance (a). Sleeping earlier than usual does not make a large difference but going to bed an hour or more later than
usual is associated with significantly worse performance the next day (b).
6.2 Multiple Nights of Insufficient Sleep
6 S SD Above, we reported on the effect of a single night of sleep with
66
particular duration and timing on the next day. Here, we examine
6
whether multiple insufficient nights of sleep measurably affect per-
formance and how long this effect appears to persist. For purposes
of this analysis, we define an “insufficient” night of sleep (“I”) to
have a time in bed of under six hours (as in [22]), and a “sufficient”
night of sleep (“S”) to have a time in bed of at least six hours. We
consider three different scenarios: two nights of sleep with more
than six hours each (SS), one night over and the next night under six
hours (SI), and two nights under six hours of sleep (II). We measure
the performance after those two nights of sleep for a period of seven
D DI S SD
days, reducing the performance on each of these seven days to a
Figure 6: Comparing the impact on performance of zero (SS), single value—the average performance during the first 16 hours after
one (SI), or two (II) consecutive insufficient nights of sleep (less wake up (i.e., typical wake period). We do not consider longer sleep
than six hours of time in bed). One night of insufficient sleep patterns here due to the large number of possible combinations and
is associated with significantly slower keystroke times and two data reduction associated with individual sleep patterns (e.g., a per-
insufficient nights in a row exhibit a significantly larger effect. son might not track their sleep every single night). Intentionally not
Judging by when average keystroke time drops below the hori- controlling for sleep both preceding and following the two nights of
zontal dashed line representing the slowest performance for the interest, we are addressing how insufficient sleep impacts real-world
group with two nights of sufficient sleep (SS), we observe that it performance given real-world choices. We are not, however, exam-
takes six nights of sleep to return to baseline performance levels ining the underlying biological processes of recovery from sleep
after two nights of insufficient sleep (day 7) and three nights to loss. We note that the start of the sleep patterns was distributed all
return to baseline performance levels after one night of insuffi- throughout the week; for example, two nights of sufficient sleep (SS)
cient sleep (day 4) given real-world sleep schedules. did occur both during the week as well as over the weekend. We
define recovery time as the number of days it takes to reach perfor-
worse average performance of about 14 milliseconds or 7.3% longer mance levels comparable to those after a sufficient sleep schedule
keystrokes (p ⌧ 10 10 ). Note that we limited the sleep duration (SS).
to be between 7 and 8 hours long for this analysis so that these
results demonstrate the impact of timing independent of differences Results. Multiple insufficient nights of sleep have a significant im-
in duration (i.e., those going to sleep later had a normal length of pact on average keystroke timing (Figure 6). Performance is best
time in bed despite going to sleep late). We further verified that these after two sufficient nights of sleep, slightly but measurably worse
results are not due to people sleeping later and longer on weekends after one insufficient night of sleep, and significantly worse after two
when they might be typing slower due to less work pressure as we insufficient nights in a row. Over the first 24 hours, having one in-
find similar patterns and effect sizes using just weekday data. Thus, sufficient night of sleep is associated with 1.2% slower performance
these results could point to an interaction between the circadian (p ⌧ 10 10 ) and two insufficient nights of sleep are 4.8% slower
clock and the ultradian rhythm of sleep (i.e., the cycling of sleep (p ⌧ 10 10 ) compared to two nights with longer than six hours of
stages): sleeping at different phases can result in different sleep sleep each (2.7% and 7.3% increases for click times, respectively;
organization [17]. Our findings suggest that sleeping later in one’s both p ⌧ 10 10 ). Note that these effect estimates take into account
circadian cycle does not satisfy the neural recovery needed for proper any real-world behavioral compensation such as increased caffeine
daytime performance, while sleeping earlier does not have the same intake that will help improve performance after sleep loss. The hori-
negative effects. zontal dashed line in Figure 6 corresponds to the slowest keystroke
Harnessing the Web for Population-Scale Physiological Sensing WWW’17, April 2017, Perth, Australia

time after two nights of sufficient sleep (SS), which we use as a Implications. We have demonstrated that human performance can
conservative point of reference to judge when performance after be measured in a real-world setting without any additional hardware
insufficient sleep (SI and II) has returned to a performance below or explicit testing by exploiting existing search engine interactions
this point. We find that, on average, it takes three nights to make up that occur billions of times per day. We have validated our methodol-
one insufficient night of sleep (SI crosses dashed line on day 4) and ogy and shown that human performance, as measured through these
six nights two make up two insufficient nights of sleep in a row (II signals, varies throughout the day and based on chronotype and
crosses dashed line on day 7). We find very similar results for the sleep, in close agreement with controlled laboratory-based studies.
impact on the variance (i.e., instead of mean) of keystroke timing as Beyond the relevance of the results to extending insights about sleep
well as for click times. A version of Figure 6 that visualizes average and performance, our findings more generally highlight the poten-
performance throughout each of the seven days is included in the tial power of harnessing online activities to study human cognition,
online appendix [6]. motor skills, and public health. Large-scale physiological sensing
Note that these results are not simply due to having fundamentally from online data enables
different users contribute to each of the the curves (SS, SI, II). While • studies of sleep and performance outside of small laboratory
some users are more likely to get fewer than six hours of sleep than settings, and without actively inducing sleep deprivation,
others, we do find similar effects by restricting each of the three • non-intrusive measurement of cognitive performance with-
curves to be estimated from the exact same set of users. We note out forcing individuals to interrupt their work to perform
that, since we enforce no constraints on time in bed during the seven separate artificial tasks [39],
days following the sleep pattern, additional nights of insufficient • the identification of realistic measures of real-world cogni-
sleep could occur during the follow-up period, contributing to the tive performance based on frequent tasks and interactions,
duration of the recovery period. Thus, we need to explore whether • and continuous monitoring of such measures.
there is a higher likelihood of sleep deficiencies on days following Suitable examples for such data include continuous usage patterns
the initial observed two-day period of insufficient sleep. We find from computing applications such as email, programming environ-
that, on average, SS is followed by 0.4 nights of insufficient sleep ments, bug report systems, office suites, and others. Any insights
during the following seven days, whereas SI and II are followed by on performance and productivity gained through monitoring these
1.2 and 2.5 such nights. Thus, additional days of insufficient sleep applications could be used to improve the user’s awareness of such
for the SI and II cases may have an influence on the overall time to patterns and to adapt the user experience appropriately (e.g., sched-
returning to baseline performance. Nevertheless, our findings show uling tasks intelligently in order to prevent or minimize human error;
real-world timing of return to baseline performance. We leave to scheduling meetings based on participants performance and chrono-
future work the study of more complex real-world patterns of sleep type profiles). There are great opportunities ahead to investigate
and sleep deficit and the influences of sleep deficits on performance. how such insights could be used to personalize applications based
on relevant biological processes and chronotypes.
7 CONCLUSION Acknowledgments. We thank Jure Leskovec, Emma Pierson, Marinka
Understanding human performance and its relation to sleep is critical Zitnik, David Hallac, David Jurgens and the anonymous reviewers
to productivity [16], learning [26], and avoiding accidents [16, 20]. for their valuable feedback on the manuscript.
Human performance is not constant but exhibits daily variations [42].
Existing research on sleep and performance has typically been re-
stricted to small-scale laboratory-based studies involving artificial REFERENCES
performance tasks in an artificial environment. Therefore, novel [1] Saeed Abdullah, Elizabeth L Murnane, Mark Matthews, Matthew Kay,
methods of large-scale real-world monitoring, like we have pre- Julie A Kientz, Geri Gay, and Tanzeem Choudhury. 2016. Cognitive
sented, are necessary to advance our understanding of sleep and rhythms: Unobtrusive and continuous sensing of alertness using a
performance [39]. mobile phone. In UbiComp.
[2] Peter Achermann and Alexander A Borbély. 1994. Simulation of day-
Summary of Results. We presented the largest study to date on time vigilance by the additive interaction of a homeostatic and a circa-
sleep and performance in the wild. Using a new approach to non- dian process. Biol Cybern 71, 2 (1994), 115–121.
intrusive measurement for both cognitive performance and sleep we [3] Jennifer Ackerman. 2008. Sex Sleep Eat Drink Dream: A day in the life
were able to study more than 400 times the number of users com- of your body. Houghton Mifflin Harcourt.
pared to the second largest study. We correlated human performance [4] Torbjörn Åkerstedt and Simon Folkard. 1997. The three-process model
based on interactions with a web search engine to sleep measures of alertness and its extension to performance, sleep latency, and sleep
detected by a wearable device. We demonstrated that real-world length. Chronobiol Int 14, 2 (1997), 115–123.
performance varies throughout the day and based on chronotype and [5] Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale Analy-
prior sleep, in close agreement with small-scale laboratory-based sis of Counseling Conversations: An Application of Natural Language
Processing to Mental Health. TACL (2016).
studies. We developed a statistical model that operationalizes recent
[6] Tim Althoff, Eric Horvitz, Ryen W White, and Jamie Zeitzer. 2017.
chronobiological research and showed that our estimates of circadian Online appendix of this paper to be released with publication. (2017).
rhythms, homeostatic sleep drive, and sleep inertia closely match http://stanford.io/2ejFPhD.
published results of controlled sleep studies. Further, we contribute [7] Tim Althoff, Pranav Jindal, and Jure Leskovec. 2017. Online Actions
to existing sleep research through quantifying extended periods of with Offline Impact: How Online Social Networks Influence Online
lower real-world performance that are associated with single and and Offline User Behavior. In WSDM.
multiple nights of insufficient sleep.
 WWW’17, April 2017, Perth, Australia Tim Althoff, Eric Horvitz, Ryen W. White, and Jamie Zeitzer

[8] Tim Althoff, Ryen W White, and Eric Horvitz. 2016. Influence of [29] Julian Lim and David F Dinges. 2010. A meta-analysis of the impact
Pokémon Go on Physical Activity: Study and Implications. J Med of short-term sleep deprivation on cognitive variables. Psychol Bull
Internet Res 18, 12 (2016), e315. 136, 3 (2010), 375.
[9] S Ancoli-Israel, R Cole, C Alessi, M Chambers, W Moorcroft, and C [30] Gloria Mark, Shamsi T Iqbal, Mary Czerwinski, Paul Johns, and Akane
Pollak. 2003. The role of actigraphy in the study of sleep and circadian Sano. 2016. Neurotics Can’t Focus: An in situ Study of Online Multi-
rhythms. Sleep 26, 3 (2003), 342–392. tasking in the Workplace. In CHI.
[10] Mathias Basner, Kenneth M Fomberstein, Farid M Razavi, Siobhan [31] Robert L Matchock and J Toby Mordkoff. 2009. Chronotype and time-
Banks, Jeffrey H William, Roger R Rosa, and David F Dinges. 2007. of-day influences on the alerting, orienting, and executive components
American time use survey: sleep time and its relationship to waking of attention. Exp Brain Res 192, 2 (2009), 189–198.
activities. Sleep 30, 9 (2007), 1085–1095. [32] Fabian Monrose and Aviel Rubin. 1997. Authentication via keystroke
[11] Katharina Blatter and Christian Cajochen. 2007. Circadian rhythms in dynamics. In CCS. 48–56.
cognitive performance: Methodological constraints, protocols, theoreti- [33] Elizabeth L Murnane, Saeed Abdullah, Mark Matthews, Tanzeem
cal underpinnings. Physiology & Behavior 90, 2 (2007), 196–208. Choudhury, and Geri Gay. 2015. Social (media) jet lag: How usage
[12] Matthias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, of social technology can modulate and reflect circadian rhythms. In
and Gernot Bauer. 2011. Falling asleep with Angry Birds, Facebook and UbiComp.
Kindle: A large scale study on mobile application usage. In MobileHCI. [34] Elizabeth L Murnane, Saeed Abdullah, Mark Matthews, Matthew Kay,
[13] Alexander A Borbély. 1982. A two process model of sleep regulation. Julie A Kientz, Tanzeem Choudhury, Geri Gay, and Dan Cosley. 2016.
Human neurobiology (1982). Mobile manifestations of alertness: Connecting biological rhythms with
[14] Bureau of Labor Statistics, American Time Use Survey. 2015. Av- patterns of smartphone app use. In MobileHCI.
erage sleep times per day, by age and sex. Archived at: http://www. [35] Martin T Orne. 1962. On the social psychology of the psychological
webcitation.org/6lPcEntyS. (2015). experiment: With particular reference to demand characteristics and
[15] Stuart K Card, Thomas P Moran, and Allen Newell. 1980. The their implications. Am Psychol 17, 11 (1962), 776.
keystroke-level model for user performance time with interactive sys- [36] John Paparrizos, Ryen W White, and Eric Horvitz. 2016. Screening
tems. CACM 23, 7 (1980), 396–410. for Pancreatic Adenocarcinoma Using Signals From Web Search Logs:
[16] Harvey R Colten and Bruce M Altevogt. 2006. Sleep disorders and Feasibility Study and Results. J Oncol Pract (2016), 737–44.
sleep deprivation: An unmet public health problem. [37] June J Pilcher and Allen J Huffcutt. 1996. Effects of sleep deprivation
[17] Derk-Jan Dijk and Charles A Czeisler. 1995. Contribution of the cir- on performance: A meta-analysis. Sleep (1996).
cadian pacemaker and the sleep homeostat to sleep propensity, sleep [38] Kristen Purcell. 2011. Search and email still top the list of most
structure, electroencephalographic slow waves, and sleep spindle activ- popular online activities. Pew Research Center. Archived at: http:
ity in humans. J. Neurosci. 15, 5 (1995), 3526–3538. //www.webcitation.org/5I2STSU61. (2011).
[18] Derk-Jan Dijk, Jeanne F Duffy, and Charles A Czeisler. 1992. Circadian [39] Till Roenneberg. 2013. Chronobiology: The human sleep project.
and sleep/wake dependent aspects of subjective alertness and cognitive Nature 498, 7455 (2013), 427–428.
performance. J Sleep Res 1, 2 (1992), 112–117. [40] Till Roenneberg, Anna Wirz-Justice, and Martha Merrow. 2003. Life
[19] David F. Dinges. 1990. Are you awake? Cognitive performance and between clocks: Daily temporal patterns of human chronotypes. J Biol
reverie during the hypnopompic state. In Sleep and Cognition. 159–75. Rhythms 18, 1 (2003), 80–90.
[20] David F Dinges. 1995. An overview of sleepiness and accidents. J [41] Daniel A Sternberg, Kacey Ballard, Joseph L Hardy, Benjamin Katz,
Sleep Res 4, s2 (1995), 4–14. P Murali Doraiswamy, and Michael Scanlon. 2013. The largest hu-
[21] Zhicheng Dou, Ruihua Song, and Ji-Rong Wen. 2007. A large-scale man cognitive performance dataset reveals insights into the effects of
evaluation and analysis of personalized search strategies. In WWW. lifestyle factors and aging. Front Hum Neurosci 7 (2013), 292.
[22] Julio Fernandez-Mendoza, Susan Calhoun, Edward O Bixler, Slo- [42] Hans PA Van Dongen and David F Dinges. 2000. Circadian rhythms
bodanka Pejovic, Maria Karataraki, Duanping Liao, Antonio Vela- in fatigue, alertness, and performance. Principles and practice of sleep
Bueno, Maria J Ramos-Platon, Katherine A Sauder, and Alexandros N medicine 20 (2000), 391–9.
Vgontzas. 2010. Insomnia with objective short sleep duration is as- [43] Lisa M Vizer, Lina Zhou, and Andrew Sears. 2009. Automated stress
sociated with deficits in neuropsychological performance: A general detection using keystroke and linguistic features: An exploratory study.
population study. Sleep 33, 4 (2010), 459–465. Int J Hum Comput Stud 67, 10 (2009), 870–886.
[23] Scott A Golder and Michael W Macy. 2011. Diurnal and seasonal [44] Olivia J. Walch, Amy Cochran, and Daniel B. Forger. 2016. A global
mood vary with work, sleep, and daylength across diverse cultures. quantification of “normal” sleep schedules using smartphone data.
Science 333, 6051 (2011), 1878–1881. Sci Adv 2, 5 (2016). DOI:http://dx.doi.org/10.1126/sciadv.1501705
[24] Paul Hemp. 2004. Presenteeism: At work-but out of it. Harvard arXiv:http://advances.sciencemag.org/content/2/5/e1501705.full.pdf
Business Review 82, 10 (2004), 49–58. [45] Robert West, Ryen W White, and Eric Horvitz. 2013. From cookies to
[25] Myriam Juda, Céline Vetter, and Till Roenneberg. 2013. Chrono- cooks: Insights on dietary patterns via analysis of web usage logs. In
type modulates sleep duration, sleep quality, and social jet lag in shift- WWW.
workers. J Biol Rhythms 28, 2 (2013), 141–151. [46] Ryen W White, Sheng Wang, Apurv Pant, Rave Harpaz, Pushpraj
[26] Paul Kelley, Steven W Lockley, Russell G Foster, and Jonathan Kelley. Shukla, Walter Sun, William DuMouchel, and Eric Horvitz. 2016. Early
2015. Synchronizing education to adolescent biology:‘let teens sleep, identification of adverse drug reactions from search log data. J Biomed
start school later’. Learn Media Technol 40, 2 (2015), 210–226. Inform 59 (2016), 42–48.
[27] Daniel F Kripke, Ruth N Simons, Lawrence Garfinkel, and E Cuyler [47] John A Wise, V David Hopkin, and Daniel J Garland. 2009. Handbook
Hammond. 1979. Short and long sleep and sleeping pills: Is increased of aviation human factors. CRC Press.
mortality associated? Arch Gen Psychiatry 36, 1 (1979), 103–116. [48] Kenneth P Wright Jr, Christopher A Lowry, and Monique K LeBour-
[28] Diane S Lauderdale, Kristen L Knutson, Lijing L Yan, Kiang Liu, and geois. 2012. Circadian and wakefulness-sleep modulation of cognition
Paul J Rathouza. 2008. Self-Reported and Measured Sleep Duration. in humans. Front Hum Neurosci 5 (2012), 50.
Epidemiology 19, 6 (2008), 838–845.

View publication stats