www.nature.

com/npjdigitalmed

ARTICLE OPEN

Non-invasive diagnosis of deep vein thrombosis from

ultrasound imaging with machine learning
Bernhard Kainz 1,2,3,4 ✉, Mattias P. Heinrich5, Antonios Makropoulos1, Jonas Oppenheimer6, Ramin Mandegaran7, Shrinivasan Sankar1,
Christopher Deane 8, Sven Mischkewitz6, Fouad Al-Noor1, Andrew C. Rawdin9, Andreas Ruttloff10, Matthew D. Stevenson9,
Peter Klein-Weigel10 and Nicola Curry8

Deep vein thrombosis (DVT) is a blood clot most commonly found in the leg, which can lead to fatal pulmonary embolism (PE).
Compression ultrasound of the legs is the diagnostic gold standard, leading to a definitive diagnosis. However, many patients with
possible symptoms are not found to have a DVT, resulting in long referral waiting times for patients and a large clinical burden for
specialists. Thus, diagnosis at the point of care by non-specialists is desired. We collect images in a pre-clinical study and investigate
a deep learning approach for the automatic interpretation of compression ultrasound images. Our method provides guidance for
free-hand ultrasound and aids non-specialists in detecting DVT. We train a deep learning algorithm on ultrasound videos from 255
volunteers and evaluate on a sample size of 53 prospectively enrolled patients from an NHS DVT diagnostic clinic and 30
prospectively enrolled patients from a German DVT clinic. Algorithmic DVT diagnosis performance results in a sensitivity within a
95% CI range of (0.82, 0.94), specificity of (0.70, 0.82), a positive predictive value of (0.65, 0.89), and a negative predictive value of
1234567890():,;

(0.99, 1.00) when compared to the clinical gold standard. To assess the potential benefits of this technology in healthcare we
evaluate the entire clinical DVT decision algorithm and provide cost analysis when integrating our approach into diagnostic
pathways for DVT. Our approach is estimated to generate a positive net monetary benefit at costs up to £72 to £175 per software-
supported examination, assuming a willingness to pay of £20,000/QALY.
npj Digital Medicine (2021)4:137 ; https://doi.org/10.1038/s41746-021-00503-7

INTRODUCTION presenting to their GP in high-income countries with a suspected

Venous thromboembolism (VTE) is associated with a major global DVT will be investigated only to find no evidence of a thrombus5.
burden of disease. Worldwide, the incidence of VTE is 1–3 per Many patients will receive unnecessary anticoagulants with
1000 individuals, rising to 2–7 per 1000 in individuals aged over numerous potential side-effects through an often-painful sub-
70 years, and 3–12 per 1000 in those over 80 years1. VTE, deep cutaneous injection whilst waiting more than the recommended
vein thrombosis (DVT) and pulmonary embolus (PE) are the four hours for their scan. Safely negating this wait would improve
leading cause of hospital-related disability-adjusted life years lost2. patient satisfaction, reduce the burden of high-risk treatment
Using these estimates, and using the most conservative (anticoagulants confer haemorrhagic complication risks) and
incidence figure, globally at least 7.7 million people will require discount healthcare costs. Rapid diagnosis is known to improve
investigation for VTE every year. An ageing population across compliance to regulatory guidelines that state DVT should be
many countries will lead to a greater health burden, particularly in diagnosed within 24 h6–8. Clinical evidence that DVT examinations
middle- and low-income countries where early death from using ultrasound can be performed by nurses has been shown9–11.
infection is decreasing. Mortality from VTE is common, a European However, confidence in acquiring ultrasound images is generally
study estimated 534,000 deaths per year3 and a similar study in low because of the required image interpretation skills and liability
the US reported 300,000 deaths per year4. DVT has a high level of concerns, which inhibits wide-scale adoption of such approaches.
morbidity. 30–50% of the surviving patients develop long-term In this study we evaluate if Machine Learning (ML) technology can
symptoms in their affected leg (post-thrombotic syndrome)5. provide anatomical image acquisition guidance and point of care
In high-income countries, the routine practice to diagnose diagnostic support. Such ML technology is currently often
patients after a positive D-dimer blood test and an indicative summarised as Artificial Intelligence (AI) support systems.
evaluation using the Wells score6 is to confirm or rule out a ML technology has previously been explored in the context of
suspected DVT with a two- or three-point ultrasound scan. VTE, with several studies having shown the potential for ML
Ultrasound scans are most commonly performed in a radiology clinical decision support systems (CDSS) to add incremental value
or cardiovascular department of a hospital by a highly trained in improving VTE risk stratification of patients. Most of these
radiographer/radiologist. proposed CDSS are predominantly based upon the Wells criteria12,
Currently, no reliable test is available that can be used in a whilst others are more complex, taking into consideration a
general healthcare setting (GP practice, community hospital, on a broader range of clinical risk factors for VTE as identified in the
hospital ward) or be used remotely at the point of care (nursing Caprini model (35 discrete clinical risk factors)13,14. However, to
home, patient’s home). Between 85 and 90% of patients the best of our knowledge, no prior study has shown the potential

1
ThinkSono Ltd, London, UK. 2Imperial College London, London, UK. 3FAU Erlangen-Nürnberg, Erlangen, Germany. 4King’s College London, London, UK. 5University of Lübeck,
Institute of Medical Informatics, Lübeck, Germany. 6ThinkSono GmbH, Potsdam, Germany. 7Central Alberta Medical Imaging Services, Red Deer, AB, Canada. 8Oxford Haemophilia
and Thrombosis Centre, Headington, UK. 9The University of Sheffield, School of Health and Related Research, Sheffield, UK. 10Clinic of Angiology - Interdisciplinary Center of
Vascular Medicine, Potsdam, Germany. ✉email: [email protected]

Published in partnership with Seoul National University Bundang Hospital

 B. Kainz et al.
2
1234567890():,;

Fig. 1 Consort diagram for study enrolment. Allocation, and analysis in External Validation Set 1 (EVS1).

benefit of ML to aid in the image-based diagnosis of DVT using Table 1. General population overview for model training and external
ultrasound. Our hypothesis is that ML technology can comple- validation set.
ment the clinical pathway and provide non-specialists with the
necessary confidence and skills to perform ultrasound DVT Age [years] 64.2 ± 17.7
screening autonomously. Early modelling has been undertaken Wells score 1.67 ± 1.12
to assess the potential cost-effectiveness of such an approach.
D-Dimer [micrograms/litre fibrinogen 1870 ± 3070 (one-
equivalent units] sided)
RESULTS Male [%] 46.4
Female [%] 53.5
Study participation
Not stated [%] 0.1
External Validation Set 1 (EVS1). 124 patients who presented to
proximal DVT diagnosis [%] 7.1
the Oxford Haemophilia and Thrombosis Centre, Oxford, UK, with
symptoms suggestive of DVT were approached for inclusion into Demographics
this study. Compression ultrasound has been performed accord- Asian - Any Other Asian Background 0.49%
ing to the standard practice, without software guidance. Patients Asian or Asian British - Indian 0.52%
have first been scanned as part of the standard pathway with Asian or Asian British - Pakistani 1.40%
various scanners, followed by another scan using a provided Black - Any Other Black Background 0.29%
Philips Lumify probe with screen recording software. Black or Black British - African 0.59%
The recorded screen capture videos have been curated to a Black or Black British - Caribbean 0.29%
data set that is similar in nature to one as it would have been Mixed - Any Other Mixed Background 0.20%
acquired with AutoDVT software guidance. Mixed - White and Asian 0.10%
Thirty-six patients have been excluded during the enrolment
Mixed - White and Black African 0.10%
phase for various reasons as summarised in the Consort Diagram in
Fig. 1. Two patients with confirmed DVT have been excluded due to Mixed - White and Black Caribbean 0.13%
imaging conditions that are not covered by the standard compres- Other - Any Other Ethnic Group 0.39%
sion ultrasound DVT protocol (non-echogenic thrombus and super- Other - Chinese 0.10%
ior thrombosis in the iliac vein). Control participants had no DVT Other - Not Known 0.39%
based on comprehensive clinical and laboratory testing performed Other - Not Stated 15.31%
under the supervision of and interpreted by a haematologist. This White - Any Other White Background 3.58%
results in a data set comprising of 88 eligible patients. An overview White - British 74.85%
of patient characteristics in this clinic’s database is given in Table 1. White - Irish 0.65%
It was specified that all examinations that were not performed
Not recorded 0.62%
according to the standard implemented in our study design should

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
3

Fig. 2 Qualitative example images for our model’s segmentation performance. The segmentation is robust throughout compressions. The
vein area is evaluated for complete compressibility to exclude DVT. Device: Clarius L7 (2017).

be omitted, thus, secondary exclusion criteria must be applied. in carotid ultrasound and peripheral arterial cw-Doppler ultra-
Hence, 35 patients (17 DVT positive, 18 DVT negative) have been sound, but without any experience in venous compression
further excluded during the analysis phase, due to, radiologist/ ultrasound performed the scans and compressions. No patients
haematologist confirmed, incorrect/incomplete compression (11), have been excluded in this pilot study since data suitability for
compression on incorrect/missing anatomical location (2), incorrect automatic image processing was ensured by the prospective use
scanner parameters evaluated by 10-point expert image quality of AutoDVT during image acquisition and guidance. Validation
scoring (14). The remaining sequences from positive DVT patients compression ultrasound was performed by an experienced
may not include a video clip that confirms the positive DVT. Thus, angiologist performing at least 1000 duplex and compression
further eight sequences belonging to a positive DVT case have been ultrasound examinations per year.
excluded. Of the remaining 53 patients 34 patients are DVT positive
and 19 DVT negative, confirmed by to the current clinical pathway. Algorithm performance on the internal validation set
This results in 121 individual compression sequences conforming
Figure 2 shows qualitative examples for the segmentation output
the standard implemented in AutoDVT on defined anatomical vessel
of our method. Table 2 shows quantitative results for the
locations in these patients. EVS1 was drawn from the general
anatomical landmark detection task; Table 3 for the vessel
population of 2041 patients with suspected DVT from the
compression task and Table 4 regarding segmentation perfor-
Haemophilia and Thrombosis Centre at University of Oxford. The
mance. Common image evaluation metrics, Sørensen-Dice Coeffi-
characteristics of the entire population during the year 2019 is
cient (Eq. (6)) for segmentation results and F1-score (Eq. (5)) for
summarised in Table 1. The ethical approval in place did not allow
anatomical landmark discrimination and categorical vessel com-
for the collection of these characteristics for the individual patients
pression analysis, are used for quantitative evaluation.
that have been enrolled in this study.

External Validation Set (EVS2). Thirty patients with suspected DVT Algorithm performance on the external validation sets
have been recruited from the Clinic of Angiology, Ernst von EVS1. Quantitative results on EVS1 are summarised in Table 5.
Bergmann Klinikum, Potsdam, Germany. Four of them were Receiver operator curves are shown together with confusion
clinically confirmed DVT positive. In contrast to EVS1, EVS2 used matrices in Fig. 3 on patient level and Fig. 4 on sequence/
AutoDVT during acquisition. Landmarks and compressions have anatomical landmark level. Note that these results are based on
been proposed automatically by the software, however, users can retrospective analysis of prospectively acquired ultrasound videos
always override suggestions by the ML model irrespective of the without using software guidance. In a perfect prospective setting,
validity of the compression. A vascular technician with experience AutoDVT guides the operator to acquire images that are well

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
4
Table 2. Quantitative results for the landmark detection task of the used models.

Background, no LM0-LM1 consolidated LM2-LM3-LM4-LM5 LM6-LM7 LM8-LM9-LM10

landmark consolidated consolidated consolidated

Groin and thigh model (0.95, 0.96) (0.74, 0.82) (0.66, 0.70) (0.67, 0.73) ..
Knee model (0.93, 0.94) .. .. .. (0.72, 0.78)
Evaluation according to Eq. (5) on the internal validation set.
(⋅, ⋅) is the 95% confidence interval range.
Model after 50 training epochs.

successful. Unsuccessful compression attempts were repeated

Table 3.
Quantitative results for the vein compression state task of the
until deemed successful by AutoDVT or a maximum of three
used models.
attempts.
Vein open Vein closed and fully compressed
Cost effectiveness
Groin and thigh model (0.86, 0.90) (0.88, 0.96)
D-Dimer plus ultrasound confirmatory scan for DVT diagnostic
Knee model (0.78, 0.92) (0.86, 0.92) (Fig. 11a) is currently costed at £92–£9716 (Table 14) in the UK
Evaluation according to Eq. (5) on the internal validation set. (⋅, ⋅) is the 95% NHS. Using the sensitivity and the specificity ranges from Table 5
confidence interval range. (EVS1 + EVS2), a maximum positive net monetary benefit (NMB)
between £71–£139 per ML-guided examination can be achieved
when AutoDVT is integrated into clinical pathways according to
suited for algorithmic evaluation. A setup more akin to the latter Fig. 11. We assume a willingness to pay of £20,000 per QALY6,17.
paradigm has been tested in the pilot study in EVS2 (Fig. 5). Figure 6 shows how the NMB changes with different prices for an
ML-guided examination, considering the different diagnostic
EVS2. Results from 30 DVT-suspected cases (four DVT positive) algorithm variants in Fig. 11. Accuracy versus costs is compared
that have been acquired using the AutoDVT software prospec- in Table 6.
tively in the Clinic of Angiology in Potsdam, Germany, are
presented in Table 5 and Fig. 5. We test the same fourfold cross-
validation models as used for EVS1 data from Oxford. DISCUSSION
This study provides a proof of concept that ML-based analysis can
Comparison to a naive black-box classifier distinguish patients with and without DVT while providing image
To evaluate the benefits of our ML architecture and explainable acquisition guidance for non-experts according to the clinical
image segmentation-based approach over a naïve black box standard. Evaluation was performed on a sample size of n = 53
classifier, we have further tested a direct classifier on EVS1 and enroled patients from the same clinic, 34 DVT-positive patients
EVS2. For this experiment, we encoded each image in the and 30 additional patients from another clinic, n = 4 DVT positive.
available sequences into 128-valued image embeddings using a Algorithmic DVT diagnosis results in a sensitivity within a 95% CI
black box encoder from our previous work15 that was trained to range of (0.82, 0.94), specificity of (0.70, 0.82), the positive
predict open or closed vein classification in a healthy volunteer predictive value (PPV) of (0.65, 0.89), and a negative predictive
cohort. These embeddings were stacked per sequence and used value (NPV) of (0.99, 1.00).
as an input for a 3-layer 1D-CNN. This CNN subsequently learned Our method suggests a diagnosis based on robust segmenta-
binary classification into healthy/pathological sequences from the tion in contrast to a direct image discriminator model. Conse-
training set. After 10-fold cross validation this model achieves an quently, our method does not rely on discrimination in the
area under the receiver operating characteristic curve (AUC) of conventional ML sense. Our model learns predominantly from
only (0.65, 0.71) for EVS1 and (0.65, 0.73) for EVS2 on sequence healthy volunteer data how a healthy vessel looks like and uses
level. On patient level this classifier achieves on EVS1 an AUC of this knowledge to identify DVT suspected patients in the test data.
(0.65, 0.77). A Welch’s test comparing predictions between the This is different from traditional decision boundary modelling with
navie black-box model and our specialised and explainable fully supervised learning from a balanced dataset. Thus, our model
architecture results in p = 0.025 for EVS1, hence, a significant is not noticeably affected by class imbalance issues in the training
difference can be assumed. EVS2 has only four positive cases, thus data; if the vein closes, the compression sequence is not DVT-
overfitting prevents such an analysis for the second eternal suspected, otherwise it is. Identifying the correct vessel and
validation set. interpreting the state of the vessel is the challenging part, which is
addressed by our ML model. Data variability is relevant for the
representation of the vessel itself. To improve this learnable
Operator skill level robustness variability, we use data augmentation as commonly used for
Twenty healthy volunteers have been scanned by three senior image data18. All images are resampled to 150 × 150 pixels to
medical students representing non-expert users. The students got facilitate real-time inference capabilities. We use image augmen-
a brief introduction, performed the entire examination using the tation during training: random left/right flipping, ±15 pixel
Clarius L7 device and reported success and software-specific random translation, ±15∘ rotation, random zoom at a maximum
problems. Each participant performed the examination twice in factor of 0.05, intensity re-scaling with a maximum range of ±0.3.
succession on the same five landmarks (guided by AutoDVT). The ML has previously been studied for a variety of diagnostic
average time needed for an examination was 6:16 min (median: approaches19–21. Several studies have applied ML in the context of
3:47 min) during the first run and 5:24 min (median: 4:17 min) VTE, although these ML applications have focused on developing
during the second run. 154 out of 200 (75.7%) recorded guided CDSS that aid clinicians in VTE risk stratification of patients rather
landmark approaches and compression sequences were reported than diagnose VTE12,22. To the best of our knowledge, our work is

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
5
Table 4. Quantitative results for the vessel segmentation task of the used models.

Sørensen–Dice Coefficient Bounding-box intersection over union

Background Artery Vein

Groin and thigh model (0.97, 1.00) (0.82, 0.88) (0.63, 0.79) (0.72, 0.84)
Knee model (0.97, 1.00) (0.63, 0.85) (0.73, 0.83) (0.72, 0.82)
Metric Eq. (6) Eq. (7)
Evaluation according to Eqs. (6) and (7) on the internal validation set.
(⋅, ⋅) is the 95% confidence interval range.

performance is observed in such situations. We mitigate this

Table 5. Values are expressed between [0,1] intervals and (⋅, ⋅) is the
problem through integrating image data from a diverse set of
95% confidence interval range.
devices, covering almost the entire market for mobile ultrasound
Performance metrics for EVS1 EVS2 EVS1 + EVS2 devices. Still, there is no established method for robust domain
Algorithm predictions adaptation23. Hence, a risk of reduced performance remains when
applying the presented algorithms to images from a new device.
Sensitivity (0.89, 0.93) (0.84, 0.92) (0.82, 0.96) This risk must be avoided by deploying these algorithms
Specificity (0.65, 0.83) (0.69, 0.85) (0.70, 0.82) exclusively with thoroughly tested, specific devices.
PPV (0.62, 0.86) (0.53, 1.00) (0.65, 0.89) ML-supported devices such as described here are often
summarised as clinical AI21. A critical element of any AI-based
NPV (0.98, 0.99) (0.98, 1.00) (0.98, 0.99)
support tool is its clinical relevance.
Accuracy (0.78, 0.88) (0.72, 0.78) (0.75, 0.83) DVT has a relatively low prevalence; 7.1% for a selected
AUC (0.79, 0.91) (0.71, 0.89) (0.77, 0.87) population who present to a DVT clinic as in our work and
PPV/NPV has been calculated using the population prevalence from Table <0.003% in the general population. We factor this in when
1, i.e., 7.1%, (and not the prevalence in the case-control data in EVS1 and calculating PPV and NPV, thus providing values that are most
EVS2, i.e., up to 62%). This approach was chosen to provide a meaningful informative for patients. In our case, a NPV of around 99% means
numeric value for PPV and NPV for patients42. that if the software-supported imaging test does not provide
(⋅, ⋅) is the 95% confidence interval after fourfold cross validation. evidence for the presence of a blood clot, that there is an
EVS1 shows results from N = 53 examinations performed with Philips extremely low chance that this prediction is wrong and that the
Lumify in Oxford. EVS2, N = 30, including four positive DVT cases on Clarius patient might still have DVT. Conversely, the PPV of our method is
HD (2020). The combined evaluation is shown in column (EVS1 + EVS2). about 77% with a large 95% confidence interval of 12 percentage
points. This means that if the automated imaging test gives a DVT
positive result, that there is still a 20–30% chance that this
diagnosis is wrong. This is addressed in possible clinical pathway
a pioneering study that shows the potential benefits of ML for the integration strategies in Fig. 11. A positive test with AutoDVT will
diagnosis of DVT through imaging. always lead to a confirmatory scan with an expert, who will also
Our work evaluates all implications for the implementation of a make treatment decisions which may include secondary criteria
ML model in a challenging clinical workflow like DVT diagnosis like for example the age of the thrombus. However, within the
with ultrasound imaging, a pathway that requires direct human- group who tested positive with AutoDVT, the expert’s chance of
machine interaction. This contrasts currently dominating ML seeing an actual DVT-positive patient is more than 80%, which is
methods for retrospective image analysis of tomographic data notably higher than the current 7.1%. Increasing the pre-test
like CT or MRI, which usually presents itself to an algorithm clearly probability for DVT will likely reciprocally increase the diagnostic
without imaging artefacts and in an often canonical orientation. utility and discriminatory power of the expert examination as well.
Free-hand ultrasound poses additional challenges compared to Literature and our own experiments show strong evidence that
these settings. a DVT examination in primary care performed by non-experts is
First, a user needs to be directed and guided to acquire images feasible. We would expect that rapid point of care diagnostics and
that are suitable to make a prediction through a ML model. This wide availability of testing, which is conceivably enabled by our
requires algorithmic provisions to discriminate useful images from approach, would lead to timely treatment, decreased stress, and
images that do not adhere to a clinic standard. We solve this increased patient satisfaction. Furthermore, a cost analysis
problem through training a discriminator ML model, which can simulation model has been evaluated when integrating the
identify predefined anatomical locations along the femoral vein. proposed algorithm into the clinical practice. Assuming a
Second, compression ultrasound requires the analysis of willingness to pay £20 000/QALY17, a maximum NMB between
continuous image sequences which is challenging in a setup that £71 to £139 per examination could be attained when ML guidance
requires real-time feedback. We solve this problem through a is used by non-specialist workers for DVT diagnosis. This assumes
sliding window, multi-channel input approach, which enforces zero costs for the use of the software; thus, it is the maximum
spatio-temporal consistency for a combined vein-segmentation achievable NMB. A DVT examination software tool could cost up
with learned decision boundaries for identifying a vessel as fully to £72–£175 at the sensitivity and specificity levels measured in
closed. Furthermore, mobile ultrasound probes are used and Table 5, before the NMB falls below £0. If the examination costs go
connected to a GPU-accelerated laptop to provide sufficient above £72, then the conclusion that AutoDVT is cost-effective
computational power. becomes more uncertain.
Third, image domain shift is a serious limitation of ML Our study has several limitations. First, in EVS1, we evaluate a
applications in healthcare. Domain shift occurs when a model is prospectively enroled patient cohort retrospectively, on video
trained on images that have been acquired on one device while sequences that have not necessarily been acquired at an optimal
the testing is performed on images from other, previously unseen standard. Therefore, we had to curate the data and automatically
images from different devices. Commonly, a noticeable drop in extract clips from entire exam video recordings that would be

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
6

Fig. 3 Evaluation results for ESV1 on patient level. Receiver operator characteristics on EVS1 resulting from fourfold cross validation (a).
Confusion matrices are shown in (b) for the optimal threshold (* in (a)) in each fold. Frame colours in (b) correspond to ROC fold colours in (a)).
Vessel status is extracted automatically from 53 patients in EVS1 through the combination of fold-specific groin and knee model pairs.

most similar to clips as they would be acquired by the AutoDVT user holding the ultrasound probe. The implications of this are
software guidance method. Furthermore, free-hand ultrasound particularly significant in the context of a false negative outcome
examinations are highly operator-dependent, and every operator given the possibility of DVT progression to PE and even death.
has a unique style of examination. Our proposed approach aims to With this in mind, clinical acceptance is realistically only attainable
standardise these styles to provide optimal input for subsequent if the AI/ML-powered system can achieve exceptionally high NPV,
image analysis parts and to aid clinical audits. as shown in our work.
Second, standardised acquisition has been demonstrated in Hence, this study describes the first step of a larger clinical trial
EVS2 but for both external validation sets, our patient cohorts are programme which we will use to ultimately evaluate the clinical
small, and we compare across-population with findings from efficacy of the AutoDVT software for the diagnosis of proximal DVT.
literature. This limits the types of statistical techniques that can be The study we describe confirms that the AutoDVT software can
employed in this study to evaluate statistical significance. We will diagnose DVT accurately. However, in order for the device to be
soon start a multi-centre prospective trial that will address these accepted within the clinical community a large-scale efficacy study
issues to give further insights into the practical implications of is required to confirm non-inferiority to expert-led compression US
employing AI support for DVT diagnosis. As suggested in the for proximal DVT diagnosis. Once this has been conducted, the
proposed diagnostic DVT decision trees (Fig. 11), the ultimate goal device will be able to be offered as a diagnostic alternative to
of employing AI support for DVT diagnosis would be to develop a hospital clinic-based DVT diagnosis. In conclusion, our study shows
ML-powered system using free-hand ultrasound that enables the potential of a ML-powered system using free-hand ultrasound
healthcare generalists at the point of care to exclude the presence to identify DVT in clinical populations with high-throughput
of DVT in negative cases. If sufficient accuracy is achievable, this requirements and at the primary care level. Since access to
could obviate the need for a diagnostic scan performed by an ultrasound imaging is increasing and amplified through cost-
expert user for DVT negative cases, leading to quicker diagnoses effective mobile ultrasound devices, a ML-supported examination
and further cost-benefits. Achieving this goal will require a by less specialised front line care workers has the potential to be
number of clinical acceptance issues to be overcome. Perhaps adopted for proximal DVT screening before confirmatory tests.
the most important of these is the notion of clinical responsibility.
When an expert user performs a scan, the presence or absence of
DVT is determined by the expert user, who bears the clinical METHODS
responsibility for the outcome of the test. By obviating the need Study design
for an expert user, the clinical responsibility and any associated This study is a primary analysis of compression ultrasound scan recordings
liability must lie with the AI/ML-powered system and hypotheti- performed on prospectively enroled patients at the Oxford Haemophilia
cally associated teleradiology workers, since it is the system that and Thrombosis Centre adult DVT clinic. The University of Oxford, UK,
determines the outcome of the test and not the non-specialist approved the study (Ethics: 18/SC/0220, IRAS 234007). All participants

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
7

Fig. 4 Evaluation results for ESV1 on compression sequence level. Receiver operator characteristics for the correct compression
classification per ultrasound sequence/anatomical landmark on EVS1 (a). Confusion matrices (b) at optimal thresholds (* in (a)) per fold. Frame
colours correspond to ROC fold colours in (a). Vessel status is extracted automatically through the ML models from the 121 available
anatomical landmark sequences in EVS1.

provided written informed consent. Eligible participants were consecu- acquired from the ML model during the scan. Thus, the ML model has
tively recruited between January 2019 and December 2019. Patients were been trained mainly on data from healthy volunteers (n = 246, age range
approached about participation in the study after their routine ultrasound 18–84, BMI < 30) and compression sequences from consented patients
DVT examination. After study information and consent, they were scanned with confirmed DVT (n = 9). An overview over the inclusion criteria is
for a second time by an expert radiographer. During the second scan a given in Fig. 7 and the training data population in Table 7. An overview
mobile ultrasound device was used (Philips Lumify L7). The examinations over the internal validation set is shown in Table 8. All compression
were recorded as mp4 videos. Patient identifying information has not been sequences have been manually annotated (marking pixels that belong to
recorded in the videos but separately in a spreadsheet where it was tagged vein or artery by different colour labels) by a trained workforce (n = 23
with a unique identifier (UID) by co-author Ch.D. Only the UID was used trained labellers) including medical students and employees of
during downstream analysis. ThinkSono Ltd to (a) train the algorithm and (b) evaluate its performance
A second pilot evaluation has been conducted in another clinic, the quantitatively.
Ernst von Bergmann Klinikum Potsdam, Germany, (Ethics: S7(a)/2020). Image quality control has been performed by a medical student
Eligible participants were recruited between November 2020 and April according to a specialist-defined scheme.
2021. Patients were approached about participation and consented in
the study after their routine ultrasound DVT examination. The Quality control scoring system. We use a 10-point expert image quality
examination was conducted with a Clarius L7 HD (2020) by a clinical scoring as it is outlined below to curate video data that has not been
expert. In contrast to the first data collection in Oxford, the AutoDVT acquired under AutoDVT guidance and real-time quality control. The
software has been used by the operator in Potsdam for guidance and quality cut-off, i.e., the minimum required quality has been less or equal to
video acquisition. a total score of 20 in this study.
In this work, we call the data set from the Oxford Haemophilia and
Thrombosis Centre the EVS1 and data from Potsdam EVS2. Since the 1. Vessel boundaries in video frame images
analysed prototype device is based on a ML computer algorithm, training
data and preliminary testing data are required. Thus, preliminary data (a)Fully in Frame: 1
acquisition was performed on healthy volunteers (n = 246) and nine (b)Cut off <50% of vessel size during parts sequence: 2
consenting patient volunteers who were examined for DVT (n = 4 DVT (c)Cut off <50% of vessel size during entire sequence: 3
positive). The acquisition has been performed by two radiologists and (d)Cut off >50% of vessel size during parts of sequence: 4
three trained engineers. We call the data that is used for training of the (e)Cut off >75% of vessel size during part of sequence or >50% over
model training set (Table 7). The volunteers and patients that have been entire sequence: 5
left out from training to monitor the algorithm’s performance during
2. Adherence to regular LM configuration:
development are collected in the internal validation set (Table 8).
The ML model’s task is to annotate vessels, find anatomical landmarks,
and analyse vessel compression state automatically. DVT diagnosis is (a) Strongly adherent to LM configuration: 1
done by automatization of the standard clinical ultrasound compression (b) Different positions of veins and arteries than regular configura-
algorithm in a heuristic computer programme, based on the biometrics tion, mix of LMs: 2

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
8

Fig. 5 Confusion matrices for EVS2 for each fold. Frame colours correspond to cross-validation folds (a–d) from Figs. 3 and 4. Vessel status is
extracted automatically from all 30 patients in EVS2. Note that this experiment comprises of four DVT positive and 26 DVT negative patients,
thus sensitivity/true positive rate is discretised in a ROC curve with only four steps, which makes a plot less meaningful.

Fig. 6 Costs of the guidance tool vs. net monetary benefit (NMB) per examination when implementing ML-guided DVT diagnostics into
clinical diagnostic pathways. The NMB has been simulated with a deterministic model for each of the diagnostic algorithm variants in Fig. 11
at the mean (solid line) and the 95 CI interval (shaded area) from Table 5 to show possible optimistic and pessimistic scenarios. The red lines
on the y-axis mark the maximal attainable NMB range when examination costs are zero.

(c) Loss of LM position for parts of sequence: 3 4. Contrast of arteries to tissue:

(d) Additional/missing (large) vessels for LM: 4
(e) Additional/missing vessels and different positions: 5 (a) Dark arteries, bright tissue: 1
3. Contrast of veins to tissue (b) Somewhat contrasted arteries: 2
(c) Arteries and tissue almost same echogenity: 3
(a) Dark veins, bright tissue: 1 5. Sharpness of vein boundaries
(b) Somewhat contrasted veins: 2
(c) Veins and tissue almost same echogenity: 3 (a) Clear boundaries with strong dorsal echo amplification: 1

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
9
Table 6. Summary of results from the variants in Fig. 11 comparing the calculated total costs per patient pathway and QALYs to the algorithm
accuracy.

Total costs per diagnostic Total QALYs True positive False positive True negative False negative
pathway likelihood likelihood likelihood likelihood

No software £300 11.57 0.10 0.06 0.84 0.00

support
Algorithm 1 £230–£244 11.57 0.09 0.01–0.02 0.88–0.89 0.01
Algorithm 2 £237–£254 11.57 0.09 0.01–0.02 0.88–0.89 0.01
Algorithm 3 £240–£259 11.57 0.09 0.01–0.02 0.88–0.89 0.01
Algorithm 4 £393–£510 11.56 0.10 0.19–0.30 0.60–0.71 0.00
Algorithm 5 £444–£590 11.56 0.10 0.23–0.37 0.53–0.67 0.00
Assumed population statistics according to DVT prevalence in Table 1.

Table 7. Training data overview.

Algorithm training data Groin/thigh area model Knee area model Acquired data
training training

Subjects 245 163 255

Number of compression sequences 1076 616 1500
Annotated scan sequences
Background/no anatomical landmark or compression 169 169 169
LM0—external iliac vein 10 – 10
Start of groin area after the inguinal ligament
LM1—Greater saphenous vein + common femoral vein at saphenofemoral 215 – 215
junction
LM2—common femoral vein and artery 51 – 51
LM3—common femoral vein and superficial and deep femoral arteries 294 – 294
LM4—superficial and deep femoral veins and arteries 141 – 141
End of groin area and beginning of thigh area at entrance to adductor canal
LM5—proximal thigh with superficial vein clearly visible with deep femoral vein 123 – 123
clearly separated in deep tissue
LM6—mid thigh with superficial femoral vein and artery in the adductor canal 288 – 288
LM7—distal thigh, same anatomy as LM6 2 – 2
End of thigh area and beginning of knee area
LM8—proximal popliteal area, with popliteal vein and artery – 130 130
LM9—middle popliteal area, with tibial-fibular trunk and popliteal artery – 141 141
LM10—distal popliteal area, with anterior and posterior tibial and fibular veins and – 186 186
popliteal artery
Total number of manually annotated images 111,546 88,823 167,145
Subjects may contain more than one landmark; thus, subject IDs may be present in the training and internal validation set. Individual sequences are either in
one or the other set. Landmarks used for the groin model in this study are LM0–LM4 and those for the knee area are LM8–LM10.

(b) Well discernible boundaries with some dorsal echo amplification: 2 9. Image artefacts
(c) Poorly visible boundaries without echo amplification: 3
6. Sharpness of artery boundaries (a) Good quality, only minor artefacts: 1
(b) Multiple smaller artefacts, also in/over the vessels: 2
(a) Clear boundaries with strong dorsal echo amplification: 1 (c) Large image problems, i.e., probe not fully on leg: 3
(b) Well discernible boundaries with some dorsal echo amplification: 2 10. Probe Movement in sequence
(c) Poorly visible boundaries without echo amplification: 3
7. Overall gain of image
(a) Medium paced compression and decompression, no lateral or
(a) Medium gain range, good quality: 1 horizontal movements, full vein compression: 1
(b) Image too bright: 2 (b) Very fast or very slow compression and decompression, no lateral
(c) Image too dark: 3 or horizontal movements, incomplete compression on healthy
veins: 2
8. Depth of image
(c) Minimal lateral/horizontal movement: 3
(d) Lots of movement: 4
(a) Image ends about 1 cm below lowest vessel: 1
(b) Image ends within 1–2 cm below lowest vessel: 2
(c) Image ends 2+ cm below lowest vessel: 3 Total: 10–35 Points

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
10
Table 8. The internal validation set represents a random 10% split at the subject level of the overall available training data.

Internal validation data Groin/thigh area validation Knee area validation Acquired internal validation
sequences sequences sequences

Total subjects 25 17 26
Number of compression sequences 88 58 133
Annotated scan sequences
Background/no anatomical landmark or compression 13 13 13
LM0—external iliac vein – – –
Start of groin area after the inguinal ligament
LM1—Greater saphenous vein + common femoral vein at saphenofemoral 17 – 17
junction
LM2—common femoral vein and artery 1 – 1
LM3—common femoral vein and superficial and deep femoral arteries 27 – 27
LM4—superficial and deep femoral veins and arteries 11 – 11
End of groin area and beginning of thigh area at entrance to adductor canal
LM5—proximal thigh with superficial vein clearly visible with deep femoral 10 – 10
vein clearly separated in deep tissue
LM6—mid thigh with superficial femoral vein and artery in the 25 – 25
adductor canal
LM7—distal thigh, same anatomy as LM6 – – –
End of thigh area and beginning of knee area
LM8—proximal popliteal area, with popliteal vein and artery – 20 20
LM9—middle popliteal area, with tibial-fibular trunk and popliteal artery – 11 11
LM10—distal popliteal area, with anterior and posterior tibial and fibular – 16 16
veins and popliteal artery
Total number of manually annotated images 9598 8257 15,523

Landmarks used for the groin model are LM0–LM4 and those for the knee area are LM8–LM10.

Fig. 7 Consort diagram for inclusion of volunteer scans into the training set and internal validation set. Dataset curation for the training
and internal validation data. Our approach can be trained from image data that originates predominantly from healthy volunteers.

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
11

Fig. 8 Examples of the chosen anatomically salient landmarks and overview over the investigated anatomy. Images have been acquired
by different acquisition devices and from different subjects. This figure illustrates the diversity in our dataset. See the overview above the
table and Tables 7 and 8 for a description for the location of these landmarks. These example images have been manually cropped and
contrast normalised for better readability.

Ultrasound protocol recordings from 53 patients, 34 DVT positive, in EVS1, and 18,399 images in
Non-enhanced ultrasound imaging was performed by a research 150 compression video clips from 30 patients, four DVT positive, in EVS2. This
physician or radiologist (at least one year of hands-on ultrasound DVT provides 83 patient observations for algorithm evaluation that have not been
imaging training) using either Clarius L7 (2017) and Clarius L7 HD (2020) part of the algorithm training data. On the patient level, sample sizes like this
or Philips Lumify L7 or GE VScan Extend (scanned with linear probe, only are in line with other studies evaluating algorithmic diagnostic decision
for training data) ultrasound devices. Example images for these scanners support. Some recently reported external validation sample sizes of, e.g.,
are shown in Fig. 8. Two-point compression ultrasound was used for this 5026,27, 9128, and 19829 subjects during retrospective testing and 8030 to 9731
study. Clinically, a compression is deemed adequate when the vein is during prospective testing. Statistical approximations for sample size
compressed fully. A vein that does not compress at the same pressure, at estimation32 suggest that eight patients with the required outcome would
which a healthy vein would collapse, indicates DVT. The femoral vessels be sufficient in an external validation set, given an incident rate of 7.1%
were examined from 2 cm distal to the saphenofemoral junction to 2 cm (Table 1) as observed in our UK thrombosis clinic. Combining EVS1 and EVS2,
proximal from the inguinal band. The superficial femoral vessels were we have included 38 patients with confirmed DVT and 45 patients who were
examined in the adductor canal. The examination of the popliteal vein suspected but did not suffer from DVT. This provides an almost balanced
starts from the distal 2 cm of the popliteal vein and its trifurcation into dataset for testing. For the training of our model, a balanced dataset with
the anterior tibial vein, posterior tibial vein, and the peroneal vein. The respect to diagnostic outcome is not required. The core algorithm focuses on
entire examination has been recorded as screen capture videos, cropped accurate vessel segmentation, which can be learned from compression
to the ultrasound image area without user-interface content and ultrasound imaging from healthy volunteers with manual delineations of the
resampled with bilinear interpolation to 150 × 150 pixels. Participants visible vessels. Diagnosis is suggested indirectly through the compressibility
were positioned in a supine position, with the hip rotated outwards by of the vein. We provide fourfold cross-validation results on the individual sets
about 60–80∘ and the knee flexed at about 60∘. The knee area was (80%:10%:10% training:validation:testing with non-overlapping data splits).
examined either supine with neutral hip and knee flexed at 80–90∘ or This allows to provide ~95% confidence intervals for the core algorithm’s
sitting upright with knee hanging loose over the gurney edge at 90∘. performance for the vessel segmentation and anatomical guidance tasks. The
power of this study is above 0.8 at a significance level of 0.05, with a Cohen’s
d effect size of 0.5, when assuming an effect between 0.9 (without software
Statistical analysis support n9 = 697, n10 = 1107) and 0.95 (with software support, this study n =
Ultrasound has a sensitivity of 94% and a specificity of 97% for DVT 53, n = 30) with a standard deviation of 0.1. For this setting, 51 patients are
detection24,25 when performed by specialised radiologists. Two studies required as a minimum to reach a power of 0.8. We achieve this for EVS1
reported sensitivity of 84.4–90.0% and specificity of 97.0–97.1% when alone (n = 53) and for the combined analysis of EVS1 with EVS2 (n = 83).
intensely trained nurses and GPs were the ultrasound operator9,10. This The R software package (©The R Project for Statistical Computing) has
means that there is strong evidence that non-specialists like nurses and GPs been used for numerical power analysis.
can (a) acquire ultrasound imaging data of high enough quality for diagnosis Algorithms are evaluated at the participant level. To evaluate classifier
and (b) that these operators are able to correctly identify DVT. performance, we calculate sensitivity, specificity, PPV, NPV, and overall
diagnostic accuracy for DVT identification for the internal and external
Sample size. The sample size in this manuscript is 167,145 annotated validation sets.
ultrasound images for model training from 255 volunteers and 15,523 We also generate the ROC of the DVT classification score for the external
annotated images from 26 participants for internal model validation. validation sets and calculated the area under the ROC (AUC). We show
For prospective model evaluation we use 19,134 images in 121 video confusion matrices at the optimal algorithm threshold.

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
12

Fig. 9 Overview over the AutoDVT prototype core algorithm. a whole overview and b overview over the individual branches. A U-Net41
serves as a backbone for automatic delineation of vein and arteries (b). The prediction of the anatomical location of the image is based on
our previous work15. Network branches predict the anatomical location and whether the vessel is open or closed under pressure. Landmark
predictions are performed from the learned numeric representation in the bottleneck layer; vessel compression state is predicted from the
output segmentation mask. The network components are connected and can be trained through back-propagation42 in an end-to-end
manner. The input is a stack of nine images (individual video frame images resampled to 150 × 150 pixels) from an ultrasound video stream
that moves by one in a sliding window fashion. A single segmentation mask is produced for the last-most image within approximately
25 ms. Two separate models with identical architecture are trained, one for the groin area (LM0–LM5) and one for the knee area (LM8–LM10).
Each model holds 31,475,527 parameters. (OC = open/close).

Algorithm design ultrasound machine. The AutoDVT software has two functions: (1) Directing
This study aims to validate the effectiveness of an ML-powered device the user to correctly position the ultrasound to complete a thorough scan,
(AutoDVT) for the diagnosis of proximal DVT. AutoDVT is a CE-marked and (2) analysing the scan results to confirm the presence/absence of a
software product (93/42/EEC 40873) that is coupled to a handheld CE-marked thrombus.

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
13

Fig. 10 Prototype implementation user interface. The AutoDVT software instructs users to locate a given landmark, instructs to perform a
correct compression and evaluates the result automatically.

The software uses a fully automated ML vessel segmentation network with momentum 0.9 to optimise the parameters of the network. Binary
with auxiliary branches that predict the anatomical location of the cross entropy (BCE, Eqs. (1) and (2)) is used for the segmentation task (one-
ultrasound image relative to the deep veins in the leg and the compression hot encoded) and the vein open/closed task. Cross entropy (CE, Eq. (3)) is
status of the vein (open or closed). Veins have been labelled by a used for the anatomical landmark detection task as an error metric.
radiologist to be either open or closed and fully compressed. Two
1X
pixels
networks with identical design/architecture have been trained: one for the LsegmentationðonehotencodedÞ ¼  y log ðpðy i ÞÞ þ ð1  y i Þlog ð1  pðy i ÞÞ
groin/thigh area and one for the knee area. The subject IDs overlap N i¼1 i
between the training set and internal validation set because a sequence (1)
can have multiple landmarks but belong to either a healthy patient or a
patient with confirmed DVT. See Table 7 for an overview over the 1X 11
algorithm training data and Table 8 for the internal validation data. LanatomicallocationðonehotencodedÞ ¼  y log ðpðy i ÞÞ þ ð1  y i Þlog ð1  pðy i ÞÞ
N i¼1 i
Annotations include manual delineations of vein and artery cross sections
in the images as well as discrete image-level labels for eleven anatomical (2)
locations. To facilitate algorithmic evaluation, we have defined anatomi- Where y is the real label and p(y) is the predicted probability for the image
cally salient landmarks (LM0–LM10) on the common femoral vein, belonging to this label.
superficial femoral vein, and popliteal vein. Example images for these
landmarks, acquired with the different ultrasound probes that are used for LðveinopenorclosedÞ ¼ ðylog ðpðyÞÞÞ þ ð1  yÞlog ð1  pðyÞÞ (3)
algorithm training in this study, are shown in Fig. 8.
The total error metric (loss function) for our network results as
To exclude DVT an operator must follow a protocol as instructed by the
software. This protocol resembles the clinical practice of three-point or two- Ltotal ¼ αLsegmenation þ βLanatomicallocation þ γLveinopenrclosed ; (4)
point examinations33–35, which means doing compression ultrasound in two
to three regions where the greatest risk of developing thrombosis occurs. For where α and β are adjustable hyper parameters. We use α = 100 and
three-point compression protocols, these regions include: (1) the common β = γ = 1. The PyTorch deep learning framework36 has been used for our
femoral vein at the level of the inguinal crease (LM0–LM4), (2) the superficial implementation. A series of manually tuned temporal quality control
femoral vein superior in the adductor canal (LM5–LM7), and (3) the popliteal functions ensure robust communication with the user regarding vessel
vein and its trifurcation in the popliteal fossa (LM8–LM10). location in the image, quality of compressions, imaging parameters and
For two-point compression protocols the same regions are examined placement of the probe (Fig. 9).
except (2), i.e., LM0–LM5 in the groin and LM8– LM10 in the knee. To The internal validation set (n = 26 healthy subjects, held out from
maximise the overlap between common procedures in the clinics from training) has been used to test the models’ performance during
where our external validation sets originate, we investigate in this study development by comparing segmentations to manual delineations of
the effectiveness of algorithmically evaluated two-point compression the vessels and manual, categorical image labels with respect to the
DVT examinations. anatomical locations (LM0–LM10) and the vessel compression status (open
Thus, using the training set, the discriminator parts in the ML models are or fully closed). For categorical labels, the F1-score is used,
trained on consolidated groups of landmarks LM0–LM1, LM2–LM3–LM4, F1 ¼
true positive classifications
i.e., two groups, for (1) and one group, LM8–LM9–LM10, for (3). This means true positive classifications þ 12 ðfalse positive classifications þ false negative classifications Þ
three successful vein compressions, two in the groin area and one in the (5)
knee area, are required in total to exclude DVT. All identified anatomical And for segmentation masks the Sørensen-Dice Coefficient is applied
locations must show fully compressible veins, otherwise the participant is per label (background, artery, vein),
categorised as suspected DVT case.
Two deep ML networks with identical architecture as shown in Fig. 9 2 ´ true positive pixels
DICE ¼ (6)
were trained on a GPU server (Nvidia Tesla K80) using the Adam optimizer 2 ´ true positive pixels þ false positive pixels þ false negative pixels

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
14
a Well’s Score b Well’s Score
Unlikely (<=1) Likely (>=1) Unlikely (<=1) Likely (>=1)

D-dimer proximal ultrasound D-dimer AutoDVTsoftware

supported scan

not raised raised negative positive not raised raised negative positive

AutoDVTsoftware confirmatory
no DVT proximal ultrasound D-dimer treat for DVT no DVT supported scan D-dimer
proximal ultrasound
negative positive
negative positive not raised raised negative positive not raised raised
no DVT treat for DVT

treat for DVT no DVT proximal ultrasound (in -8

6 days) confirmatory no DVT AutoDVTsoftware
no DVT no DVT
proximal ultrasound supported scan(in 6-8 days)

negative positive negative positive negative positive

confirmatory
no DVT treat for DVT no DVT treat for DVT no DVT
proximal ultrasound
negative positive
Algorithm 0, current clinical practice Algorithm 1 no DVT treat for DVT

c d
Well’s Score Well’s Score
Unlikely (<=1) Likely (>=1) Unlikely (<=1) Likely (>=1)

AutoDVTsoftware AutoDVTsoftware AutoDVTsoftware AutoDVTsoftware

supported scan supported scan supported scan supported scan

negative positive negative positive negative positive negative positive

confirmatory AutoDVTsoftware confirmatory

D-dimer
confirmatory proximal ultrasound confirmatory supported scan(in 6-8 days) proximal ultrasound
no DVT no DVT
proximal ultrasound positive proximal ultrasound positive
negative negative positive negative
not raised raised
negative positive treat for DVT negative positive treat for DVT
no DVT no DVT
confirmatory
no DVT
no DVT AutoDVTsoftware proximal ultrasound
no DVT treat for DVT no DVT treat for DVT
supported scan(in 6-8 days) negative positive
negative positive
no DVT treat for DVT
confirmatory
no DVT
proximal ultrasound
negative positive
Algorithm 2 no DVT treat for DVT Algorithm 3

e f

Well’s Score Well’s Score

Unlikely (<=1) Likely (>=1) Unlikely (<=1) Likely (>=1)

D-dimer AutoDVTsoftware AutoDVTsoftware

AutoDVTsoftware
supported scan supported scan
supported scan
not raised raised negative positive negative positive
negative positive
AutoDVTsoftware
no DVT AutoDVTsoftware treat for DVT AutoDVTsoftware treat for DVT
supported scan
supported scan(in 6-8 days) supported scan(in 6-8 days)
no DVT treat for DVT
negative positive negative positive negative positive

no DVT treat for DVT no DVT treat for DVT no DVT treat for DVT

Algorithm 4 Algorithm 5

Fig. 11 Possible integration strategies for our approach into DVT diagnostics pathways. a current clinical algorithm to diagnose DVT
without software support according to UK NICE guidelines6 and b–f possible variants to integrate ML software support into the clinical
pathway. Algorithms 1–3 shown in (b–d) generate a positive net monetary benefit (cf. Fig. 6). The examined modifications have been
suggested by health economics and clinical experts. Note that treatment options may further depend on the age of the clot, which might be
manually estimated during confirmatory ultrasound scans43.

In addition, the bounding boxes for the individual segmentation masks are e.g., anatomical landmark locations. Furthermore, it is designed to require
generated and the intersection over union (IoU = Jaccard index) is computed, a reasonably low number of floating-point operations to facilitate real-time
which is a common performance metric for object detection tasks, performance.
F1 Area of overlap with true bounding box
IoU ¼ ¼ (7)
ð2  F1Þ Area of union with true bounding box Cost effectiveness
In an end-user scenario a non-expert operator would have three to five We simulated the potential cost-effectiveness of a ML-enabled approach at
attempts to complete a compression, otherwise referral is recommended. the front line of care, where non-specialists may perform the examination
A screenshot of the AutoDVT software during use is shown in Fig. 10. independently. A decision tree analytic model was designed and
During our experiments, all compressions have been competed in under implemented in Microsoft Excel (©Microsoft Corporation) to estimate the
five attempts. lifetime costs and benefit measured in terms of quality-adjusted life years
(QALYs) for different proximal DVT testing algorithms. The current clinically
Technical uniqueness of the proposed framework. We propose a triple-task used diagnostic DVT algorithm is shown in Fig. 11a and possible
convolutional neural network (CNN) fully integrated into a clinical integration strategies for our method are shown in Fig. 11b–f.
prototype device that jointly classifies the anatomical landmark plane in The cost analysis model adheres to guidelines issued by the National
the current field-of-view, scores vein compressibility and provides Institute of Health and Care Excellence (NICE)6. It uses an UK NHS and
semantic segmentation masks for arteries and veins. The proposed personal social services perspective with costs at 2018/19 prices and with
network architecture can intrinsically learn to interpret video data to discounting for both costs and QALYs being undertaken at 3.5% per
perform localisation, segmentation, local deformation estimation and annum. Note that costs associated with tangible and intangible expenses
classification from weak discrete labels that characterise whole images, that families can incur in the event of disability or even death due to

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
15
misdiagnosis would not be considered by either NHS or personal social We generate four possible outcomes for patients based on their DVT
services expenditure and is commonly excluded from a NICE appraisal 6. status and the results of each diagnostic algorithm: Treated patients with a
The model uses sensitivity (the ability of a test to correctly identify a true DVT (true positive patients), treated patients without a true DVT (false
patient with a true proximal DVT) and of specificity (the ability of a test to positive patients), untreated patients without a true DVT (true negative
correctly identify a patient without a true proximal DVT) as measured on patients) and untreated patients with a true DVT (false negative patients).
the external validation sets in this study. We also include clinical tests Each of the four diagnostic accuracy outcomes have estimated associated
(Wells Score, D-dimer, and proximal ultrasound) that form part of the costs incurred and utility accrued for the patients. These numbers are
diagnostic algorithm. multiplied by the proportion of patients in each outcome and are
Our cost analysis model splits patients into two subgroups at the start of combined with the costs of each test to obtain estimates of the total
each algorithm, a subgroup in which patients have a proximal DVT and a costs and QALYs for the diagnostic algorithm. When costs and QALYs
subgroup in which patients do not have a proximal DVT. Measured are obtained for diagnostic algorithms with and without the ML model, the
sensitivity and specificity values are used alongside an estimate of the estimated incremental cost-effectiveness ratio for AutoDVT can be
prevalence of proximal DVT of 14.7% taken from Kilroy et al.37 to estimate calculated.
the number of patients (from a cohort of user specified size) that receive
each clinical test and their ultimate diagnoses (proximal DVT or not). Parameters for the cost-effectiveness model. Test characteristics have
Patients with a diagnosed proximal DVT will receive treatment. been taken from Goodacre et al.24 and are presented in Table 9
with the statistical distributions used in stochastic analysis presented in
Tables 10–13.
Table 9. Efficiency parameters for each test. Treatment reduces the probability of a patient with a DVT
experiencing a fatal or non-fatal pulmonary embolism (PE) or post-
Description Mean thrombotic syndrome (PTS). However, treatment is associated with risks
of fatal haemorrhage, non-fatal intracranial haemorrhage, and non-fatal
Wells Score: Proportion of patients with a
non-intracranial haemorrhage.
proximal DVT characterised as:
According to Goodacre et al.24 patients who do not experience any of
High risk 0.68 a PE, PTS or a haemorrhage accrue a mean of 11.58 discounted lifetime
Moderate risk 0.25 QALYs. Mean quality of life multipliers for PTS, non-fatal PE and non-
Low risk 0.07 fatal intracranial haemorrhage of 0.977, 0.94, and 0.29 respectively were
Wells Score: Proportion of patients also presented by Goodacre et al.24 with statistical distributions used in
without a proximal DVT characterised as: the stochastic analysis presented in Tables 10–13. These data were used
to estimate total QALYs for the four diagnostic accuracy outcomes.
High risk 0.11
The lifetime, discounted, QALYs accrued by patients in each
Moderate risk 0.41 classification differ based on their true DVT status and their results
Low risk 0.48 from each diagnostic algorithm. Untreated patients with a DVT remain
D-dimer (assumed to be an enzyme- at high risk of PE and PTS but do not have the risks of haemorrhage
linked immunosorbent assay (ELISA) test) associated with treatment. Treated patients with a DVT have reduced
Sensitivity for 0.98 risks of PE and PTS but have the risk of haemorrhage associated with
proximal DVT treatment. Treated patients without a DVT have the same risk of PE and
Specificity for proximal PTS as the general population but are subject to the risks of
DVT when haemorrhage associated with treatment. Untreated patients without a
true DVT will accrue the same discounted lifetime QALYs as the general
Wells Score is high risk 0.34
population. The QALYs accrued in each of the four diagnostic accuracy
Wells Score is 0.45 outcomes are shown in Table 14.
moderate risk
The discounted lifetime costs associated with patient outcomes were
Wells Score is low risk 0.52 taken from24,38. Where appropriate costs were uplifted to 2018/19
Proximal ultrasound values using inflation indices presented in Curtis et al.38.
Test sensitivity 0.95 The lifetime costs associated with PTS and non-fatal intracranial
haemorrhage were both composite costs including the cost of a first
Test specificity 0.94
attendance at a vascular surgery outpatient clinic and the cost of

Table 10. Test Efficiency Well’s test.

Description Mean Distribution Parameter 1 Source

17
Proportion of patients with proximal DVT characterised as high risk 0.68 Dirichlet 105.61
17
Proportion of patients with proximal DVT characterised as moderate risk 0.25 Dirichlet 38.83
17
Proportion of patients with proximal DVT characterised as low risk 0.07 Dirichlet 10.87
17
Proportion of patients without proximal DVT characterised as high risk 0.11 Dirichlet 40.78
17
Proportion of patients without proximal DVT characterised as moderate risk 0.41 Dirichlet 151.99
17
Proportion of patients without proximal DVT characterised as low risk 0.48 Dirichlet 177.94

Table 11. Test Efficiency D-dimer test.

Description Mean Distribution Param. 1 Param. 2 Source

17
Sensitivity for proximal DVT 0.98 Beta 736.91 15.04
17
Specificity for proximal DVT— Wells’s outcome: high risk 0.34 Fixed
17
Specificity for proximal DVT— Wells’s outcome: moderate risk 0.45 Beta 4278.13 5228.83
17
Specificity for proximal DVT— Wells’s outcome: low risk 0.52 Fixed

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
16
Table 12. Proximal DVT prevalence and outcomes associated with treated and untreated proximal DVT.

Description Mean Distrib. Param. 1 Param. 2 Source

Proximal DVT prevalence 0.15 Beta 41 238 (37 and Table 1)

Treated proximal DVT
17
Probability of fatal pulmonary embolus 0 Beta 17 4204
17
Probability of non-fatal pulmonary embolus 0.01 Beta 33.4 4070.6
17
Probability of post thrombotic syndrome 0.05 Beta 28 500
Outcomes associated with warfarin treatment
Fatal haemorrhage 0 Dirichlet a = 37 17

Non-fatal intracranial haemorrhage 0 Dirichlet b = 13 17

Non-fatal non-intracranial haemorrhage 0.02 Dirichlet c = 226 17

No haemorrhage 0.98 Dirichlet d = 10.481 17

Untreated proximal DVT

17
Probability of fatal pulmonary embolus 0.02 Beta 5 263
17
Probability of non-fatal pulmonary embolus 0.09 Beta 25 243
17
Probability of post thrombotic syndrome 0.33 Beta 5.21 10.57

Table 13. QALYs associated with outcomes and QALYs accrued by patient category.

Description Mean Distribution Param. 1 Param. 2 Source

17
Normal age-specific discounted quality-adjusted life 11.6 Fixed
expectancy (QALYs)
Lifetime utility multiplier associated with
17
Non-fatal pulmonary embolus 0.94 Beta 19.43 1.24
17
Non-fatal intracranial haemorrhage 0.29 Beta 8.34 20.41
17
Post thrombotic syndrome 0.98 Beta 232.64 5.48
Lifetime QALY’s accrued by
Patients with a DVT who are treated 11.5 [A]
Patients with a DVT who are untreated 11.2 [A]
Patients without a DVT who are treated 11.5 [A]
[A]: The variance on these parameters is based on the variance of the parameters that make up these values.

Table 14. Estimated QALYs accrued, and costs incurred for each Table 15. Costs associated with outcomes of DVT and complications
diagnostic accuracy outcome. associated with the treatment of DVT.

Description QALYs accrued Cost incurred [UK £] Description Mean [UK £]

True Positive (DVT – treated) 11,464 1321 Treatment of fatal PE 1650

False Negative (DVT – 11,207 2214 Treatment of non-fatal PE 1601
untreated) Lifetime treatment of PTS 4663
False Positive (No DVT – treated) 11,530 975 Fatal intracranial haemorrhage 9334
True Negative (No DVT – 11,580 0 Lifetime treatment for non-fatal intracranial 64,147
untreated) haemorrhage
Non-fatal non-intracranial haemorrhage 805
subsequent vascular surgery outpatient clinics visits for PTS and the cost
of care in the first year and subsequent years in the case of non-fatal
intracranial haemorrhage. The total cost associated with PTS and the The total cost of DVT treatment of £845 is estimated using the same
method used to calculate these were included in Goodacre et al.24 derivation as that used in24 with one change: The current version of the
together with the costs of the components of the total cost. From this, it British National Formulary39 indicates that the initial dose of LMW heparin
was estimated that the expected lifetime of patients with PTS was 11.67 in the treatment of DVT is a large loading dose with subsequent smaller
years. No such information was provided for patients experiencing a non- maintenance doses, thus the initial loading dose will be associated with a
fatal, non-intracranial haemorrhage and thus it was assumed that the greater cost than subsequent maintenance doses. The costs of LWM
same expected lifetime applied when calculating costs. heparin and warfarin are taken from the current version of the British
Treatment for DVT consists of approximately eight days of low National Formulary39. Additional resource use such as GP visits and
molecular weight (LMW) heparin followed by ninety days of Warfarin. anticoagulant clinic visits and their unit costs24,38 and NHS Reference

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital
 B. Kainz et al.
17
12. Fong-Mata, M. B. et al. An artificial neural network approach and a data aug-
Table 16. Costs of the diagnostic tests. mentation algorithm to systematize the diagnosis of deep-vein thrombosis by
using wells’ criteria. Electronics 9, 1810 (2020).
Description Mean Distribution Parameter 1 Parameter 2 Source
13. Caprini, J. A., Arcelus, J. I. & Reyna, J. Effective risk stratification of surgical and
[UK £]
nonsurgical patients for venous thromboembolic disease. Semin. Hematol. 38,
17 12–19 (2001).
Well’s test 9.66 Gamma 25 0.34
14. Molinaro, A. M. Diagnostic tests: how to estimate the positive predictive value.
16
D-dimer 19.5 Gamma 25 0.78 Neuro-Oncol. Pract. 2, 162–166 (2015).
40 15. Tanno, R. et al. Autodvt: joint real-time classification for vein compressibility analysis
Proximal 77.19 Gamma 319.23 0.25
ultrasound in deep vein thrombosis ultrasound diagnostics. In International Conference on
Medical Image Computing and Computer-Assisted Intervention, 905–912 (Springer,
2018).
Costs 2015–201640, where appropriate costs have been inflated to 2018/ 16. Oxfordshire Clinical Commissioning Group. Occg service specification (2018/19),
19 values using inflation indices presented by Curtis et al.38. The costs primary care service for dvt testing (updated september 18). sept., https://www.
associated with outcomes associated with DVT or with treatment for DVT oxfordshireccg.nhs.uk/professional-resources/documents/primary-care/locally-
are shown in Table 15. commissioned-services-2018%20-2019/dvt-testing.pdf (2018).
The costs associated with each diagnostic test have been taken from 17. Goodacre, S. et al. How should we diagnose suspected deep-vein thrombosis? J.
Goodacre24 and the NHS40 and are presented in Table 16. Assoc. Physicians 99, 377–388 (2006).
18. Shorten, C. & Khoshgoftaar, T. M. A survey on image data augmentation for deep
learning. J. Big Data 6, 1–48 (2019).
19. De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in
DATA AVAILABILITY retinal disease. Nat. Med. 24, 1342–1350 (2018).
Algorithm raw performance results of this study are available from the corresponding 20. Bai, W. et al. Automated cardiovascular magnetic resonance image analysis with
author upon reasonable request. Image data access in line with the informed consent fully convolutional networks. J. Cardiovasc. Magn. Reson. 20, 65 (2018).
of the participants, subject to approval by the project ethics boards and under a 21. Akkus, Z. et al. A survey of deep-learning applications in ultrasound: artificial
formal Data Sharing and License Agreement on a case-by-case basis. The data are not intelligence–powered ultrasound for improving clinical workflow. J. Am. Coll.
publicly available due to them containing information that could compromise Radiol. 16, 1318–1328 (2019).
research participant privacy/consent. Qualified researchers can apply for access via 22. Qatawneh, Z., Alshraideh, M., Almasri, N., Tahat, L. & Awidi, A. Clinical decision
email to https://thinksono.com/ ([email protected]). support system for venous thromboembolism risk classification. Appl. Comput.
Inform. 15, 12–18 (2019).
23. Wang, M. & Deng, W. Deep visual domain adaptation: a survey. Neurocomputing
CODE AVAILABILITY 312, 135–153 (2018).
Custom computer software (AutoDVT) paired with ultrasound acquisition devices can 24. Goodacre, S., Sampson, F., Thomas, S., van Beek, E. & Sutton, A. Systematic review
be licensed from ThinkSono Ltd ([email protected]). Data analysis code in R, and meta-analysis of the diagnostic accuracy of ultrasonography for deep vein
Microsoft ExcelⓇ, and Python and model structure in PyTorch format can be thrombosis. BMC Med. Imaging 5, 1–13 (2005).
requested from the corresponding author. 25. Zierler, B. K. Ultrasonography and diagnosis of venous thromboembolism. Cir-
culation 109, I–9 (2004).
26. Patel, B. N. et al. Human–machine partnership with artificial intelligence for chest
Received: 8 January 2021; Accepted: 6 August 2021; radiograph diagnosis. NPJ Digit. Med. 2, 1–10 (2019).
27. Kanber, B. et al. High-dimensional detection of imaging response to treatment in
multiple sclerosis. NPJ Digit. Med. 2, 1–10 (2019).
28. Miura, K. et al. Feasibility of the deep learning method for estimating the ven-
tilatory threshold with electrocardiography data. NPJ Digit. Med. 3, 1–7 (2020).
REFERENCES 29. Huang, S.-C. et al. Penet–a scalable deep-learning model for automated diagnosis
1. ISTH Steering Committee for World Thrombosis Day. Thrombosis: a major of pulmonary embolism using volumetric ct imaging. NPJ Digit. Med. 3, 1–9
contributor to the global disease burden. J. Thromb. Haemost. 12, 1580–1590 (2020).
(2014). 30. Kiani, A. et al. Impact of a deep learning assistant on the histopathologic clas-
2. Jha, A. K. et al. The global burden of unsafe medical care: analytic modelling of sification of liver cancer. NPJ Digit. Med. 3, 1–8 (2020).
observational studies. BMJ Qual. Saf. 22, 809–815 (2013). 31. Choi, D.-J., Park, J. J., Ali, T. & Lee, S. Artificial intelligence for the diagnosis of heart
3. Cohen, A. et al. Vte impact assessment group in europe (vitae). venous throm- failure. NPJ Digit. Med. 3, 1–6 (2020).
boembolism (vte) in europe. the number of vte events and associated morbidity 32. Riley, R. D. et al. Calculating the sample size required for developing a clinical
and mortality. Thromb. Haemost. 98, 756–764 (2007). prediction model. BMJ 368:m441 (2020).
4. Beckman, M. G., Hooper, W. C., Critchley, S. E. & Ortel, T. L. Venous throm- 33. Lee, J. H., Lee, S. H. & Yun, S. J. Comparison of 2-point and 3-point point-of-care
boembolism: a public health concern. Am. J. Prev. Med. 38, S495–S501 ultrasound techniques for deep vein thrombosis at the emergency department: a
(2010). meta-analysis. Medicine 98, e15791 (2019).
5. Plüddemann, A., Thompson, M., Price, C. P., Wolstenholme, J. & Heneghan, C. The 34. Zuker-Herman, R. et al. Comparison between two-point and three-point com-
d-dimer test in combination with a decision rule for ruling out deep vein pression ultrasound for the diagnosis of deep vein thrombosis. J. Thromb.
thrombosis in primary care: diagnostic technology update. Br. J. Gen. Pract. 62, Thrombolysis 45, 99–105 (2018).
e393–e395 (2012). 35. Dehbozorgi, A. et al. Accuracy of three-point compression ultrasound for the
6. The National Institute for Health and Care Excellence. NICE guidance, https:// diagnosis of proximal deep-vein thrombosis in emergency department. J. Res.
www.nice.org.uk/process/pmg9/resources/guide-to-the-methods-of-technology- Med. Sci. 24, 80 (2019).
appraisal-2013-pdf-2007975843781 (2013). 36. Paszke, A. et al. Pytorch: an imperative style, high-performance deep learning
7. Elliott, C. G., Goldhaber, S. Z. & Jensen, R. L. Delays in diagnosis of deep vein library. Adv. Neural Inform. Process. Syst. 32, 8026–8037 (2019).
thrombosis and pulmonary embolism. Chest 128, 3372–3376 (2005). 37. Kilroy, D., Ireland, S., Reid, P., Goodacre, S. & Morris, F. Emergency department
8. AWMF Leitliniensteuerungsgruppe. Venenthrombose und Lungenembolie: investigation of deep vein thrombosis. Emerg. Med. J. 20, 29–32 (2003).
Diagnostik und Therapie, https://www.awmf.org/uploads/tx_szleitlinien/065- 38. Curtis, L. & Burns, A. Unit costs of health and social care 2018, https://www.pssru.
002l_S2k_VTE_2016-01.pdf (2016). ac.uk/project-pages/unit-costs/unit-costs-2018/ (2019).
9. Mumoli, N. et al. Accuracy of nurse-performed compression ultrasonography in 39. Royal Pharmaceutical Society. British National Formulary, https://www.
the diagnosis of proximal symptomatic deep vein thrombosis: a prospective medicinescomplete.com/#/browse/bnf (2020).
cohort study. J. Thromb. Haemost. 12, 430–435 (2014). 40. The UK Department of Health. Unit costs of health and social care 2018, https://
10. Mumoli, N. et al. General practitioner–performed compression ultrasonography www.pssru.ac.uk/project-pages/unit-costs/unit-costs-2018/ (2018).
for diagnosis of deep vein thrombosis of the leg: a multicenter, prospective 41. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for
cohort study. Ann. Fam. Med. 15, 535–539 (2017). biomedical image segmentation. In International Conference on Medical
11. Fox, J. C. & Bertoglio, K. C. Emergency physician performed ultrasound for dvt Image Computing and Computer-assisted Intervention, 234–241 (Springer,
evaluation. Thrombosis 2011, 938709 (2011). 2015).

Published in partnership with Seoul National University Bundang Hospital npj Digital Medicine (2021) 137
 B. Kainz et al.
18
42. LeCun, Y. et al. Backpropagation applied to handwritten zip code recognition. until September 2020. J.O., F.A-.N., and S.M. are employees of ThinkSono Ltd, M.D.S.
Neural Comput. 1, 541–551 (1989). and A.C.R. acted as contractor for ThinkSono Ltd. All authors had full access to all data
43. Needleman, L. et al. Ultrasound for lower extremity deep venous thrombosis: during this study and accept responsibility to submit for publication. B.K., A.M.,
multidisciplinary recommendations from the society of radiologists in ultrasound F.A-.N., and S.M. are joint inventors on a patent held by ThinkSono Ltd. The remaining
consensus conference. Circulation 137, 1505–1515 (2018). authors declare no competing interests. The views expressed are those of the author
(s) and not necessarily those of ThinkSono Ltd, the NHS, the NIHR or the Department
of Health. The remaining authors declare no competing interests.
ACKNOWLEDGEMENTS
This study has been funded by ThinkSono Ltd. ThinkSono Ltd funded the
development of the method. Data were collected at Oxford Haemophilia & ADDITIONAL INFORMATION
Thrombosis Centre independently. ThinkSono Ltd provided ultrasound data Correspondence and requests for materials should be addressed to Bernhard Kainz.
acquisition devices to Oxford Haemophilia & Thrombosis Centre, UK and Clinic of
Angiology, Ernst von Bergmann Klinikum, Potsdam, Germany for this study. Reprints and permission information is available at http://www.nature.com/
reprints

AUTHOR CONTRIBUTIONS Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims
in published maps and institutional affiliations.
B.K., N.C., A.M., S.S., M.H., and M.D.S. conceived and planned the experiments and
devised the main conceptual ideas and proof outline; B.K., R.M., and F.A-.N. conducted
literature research; A.M., B.K. and M.H. designed the model and the computational
framework and analysed the data; S.M. and F.A-.N. carried out the SW tool and UX
Open Access This article is licensed under a Creative Commons
implementation; B.K. wrote the manuscript with input from all authors; A.M., A.C.R., and
Attribution 4.0 International License, which permits use, sharing,
M.D.S. designed and performed cost analysis experiments; J.O. and R.M. acquired and
adaptation, distribution and reproduction in any medium or format, as long as you give
interpreted additional anatomical measurements; A.R. and P.K.W. acquired data at the
appropriate credit to the original author(s) and the source, provide a link to the Creative
Clinic of Angiology - Interdisciplinary Center of Vascular Medicine, Potsdam, Germany.
Commons license, and indicate if changes were made. The images or other third party
A.M., S.S., J.O., S.M., and F.A-.N. verified the data; C.D. managed patient enrolment,
material in this article are included in the article’s Creative Commons license, unless
patient consent, and patient characteristics in Oxford, UK. N.C., M.H., R.M., A.C.R., M.D.S.,
indicated otherwise in a credit line to the material. If material is not included in the
P.K.W., and B.K. revised the final manuscript. All authors discussed the results and
article’s Creative Commons license and your intended use is not permitted by statutory
contributed to the final manuscript.
regulation or exceeds the permitted use, you will need to obtain permission directly
from the copyright holder. To view a copy of this license, visit http://creativecommons.
org/licenses/by/4.0/.
COMPETING INTERESTS
B.K., M.H., R.M., and N.C. are scientific advisors for ThinkSono Ltd. B.K. is also advisor
for Ultromics Ltd and Cydar medical Ltd. A.M. was an employee of ThinkSono Ltd © The Author(s) 2021

npj Digital Medicine (2021) 137 Published in partnership with Seoul National University Bundang Hospital