Cognitive Computation (2021) 13:1–33

https://doi.org/10.1007/s12559-020-09773-x

Deep Learning in Mining Biological Data

Mufti Mahmud1,5 · M. Shamim Kaiser2 · T. Martin McGinnity1,3 · Amir Hussain4

Received: 1 May 2020 / Accepted: 28 September 2020 / Published online: 5 January 2021
© The Author(s) 2020

Abstract
Recent technological advancements in data acquisition tools allowed life scientists to acquire multimodal data from differ-
ent biological application domains. Categorized in three broad types (i.e. images, signals, and sequences), these data are
huge in amount and complex in nature. Mining such enormous amount of data for pattern recognition is a big challenge and
requires sophisticated data-intensive machine learning techniques. Artificial neural network-based learning systems are well
known for their pattern recognition capabilities, and lately their deep architectures—known as deep learning (DL)—have
been successfully applied to solve many complex pattern recognition problems. To investigate how DL—especially its dif-
ferent architectures—has contributed and been utilized in the mining of biological data pertaining to those three types, a
meta-analysis has been performed and the resulting resources have been critically analysed. Focusing on the use of DL to
analyse patterns in data from diverse biological domains, this work investigates different DL architectures’ applications to
these data. This is followed by an exploration of available open access data sources pertaining to the three data types along
with popular open-source DL tools applicable to these data. Also, comparative investigations of these tools from qualita-
tive, quantitative, and benchmarking perspectives are provided. Finally, some open research challenges in using DL to mine
biological data are outlined and a number of possible future perspectives are put forward.

Keywords Brain–Machine Interfaces · Bioimaging · Deep learning performance comparison · Medical imaging · Omics ·
Open access data sources · Open-source tools

Introduction centuries [1]. This accelerated the development of cutting

edge tools and technologies that allow scientists to study
The pursuit of understanding human behaviours, along with holistically the biological systems as well as dig down, in an
the various pathologies, their early diagnosis and finding unprecedented manner, to the molecular details of the living
cures, has driven the life sciences research in the last two organisms [2, 3]. Increasing technological sophistication has
presented scientists with novel tools for DNA sequencing
[4], gene expression [5], bioimaging [6], neuroimaging [7],
M. Mahmud and M.S. Kaiser are joint first authors.
and body/brain–machine interfaces [8].
* Mufti Mahmud These innovative approaches to study living organisms
[email protected]; [email protected] produce huge amount of data [9] and create a situation
* M. Shamim Kaiser often referred as ‘Data Deluge’ [10]. Depending on the tar-
[email protected] get application and experimentation, these biological big
1
data can be characterized by their inherent characteristics
Department of Computer Science, Nottingham Trent
University, Clifton, NG11 8NS Nottingham, UK
of being hierarchical (i.e. data coming from different levels
2
of a biological system—from molecules to cells to tissues
Institute of Information Technology, Jahangirnagar
University, Savar 1342 Dhaka, Bangladesh
to systems), heterogeneous (i.e. data acquired by different
3
acquisition methods—from genetics to physiology to pathol-
Intelligent Systems Research Centre, Ulster University,
Northern Ireland BT48 7JL Derry, UK
ogy to imaging), dynamic (i.e. data changes as a function of
4
time), and complex (i.e. data describing nonlinear biological
School of Computing , Edinburgh, EH11 4BN Edinburgh,
UK
processes) [11]. These intrinsic characteristics of biologi-
5
cal big data posed an enormous challenge to data scientists
Medical Technology Innovation Facility, Nottingham Trent
University, NG11 8NS Clifton, Nottingham, UK
to identify patterns and analyse them to infer meaningful

13
Vol.:(0123456789)
2 Cognitive Computation (2021) 13:1–33

conclusions from these data [12]. The challenges have trig- Since the 1950s, many methods pertaining to both the
gered the development of rational, reliable, reusable, rigor- learning paradigms (i.e. supervised and unsupervised) have
ous, and robust software tools [11] using machine learning been proposed. The popular methods in the supervised
(ML)-based methods to facilitate recognition, classification, domain include: ANN [14] and its variants (e.g. Backpropa-
and prediction of patterns in the biological big data [13]. gation [15], Hopfield Networks [16], Boltzmann Machines
Based on how a method learns from the data, the ML [17], Restricted Boltzmann Machines [18], Spiking Neu-
techniques can be broadly categorized into supervised ral Networks [19], etc.), Bayesian Statistics [20], Support
and unsupervised approaches. In supervised learning, Vector Machines [21] and other linear classifiers [22] (e.g.
objects in a pool are classified using a set of known Fisher’s Linear Discriminant [23], Regressors [24], Naive
annotations or attributes or features, i.e. a supervised Bayes Classifier [25], etc.), k-Nearest Neighbours [26], Hid-
algorithm learns the pattern(s) from a limited number of den Markov Model [27], and Decision Trees [28]. Popular
annotated training data and then classifies the remaining unsupervised methods include: Autoencoders [29], Expecta-
testing data using the acquired knowledge. Instead, in the tion–Maximization [30], Information Bottleneck [31], Self-
unsupervised learning, pattern(s) are first defined from Organizing Maps [32], Association Rules [33], Hierarchical
a subset of the unknown data and then the remaining Clustering [34], k-Means [35], Fuzzy Clustering [36], and
data are classified based on the defined patterns, i.e. an Density-based Clustering [37, 38] (e.g. Ordering Points To
unsupervised algorithm first defines pattern(s) among Identify the Clustering Structure [39]). Many of these meth-
the objects in a pool of data with unknown annotations or ods have been successfully applied to data coming from vari-
attributes or features, and then uses the acquired knowl- ous biological sources.
edge to classify the remaining data. In addition, there For the sake of simplicity, the vast amount of biologi-
is another category called reinforcement learning which cal data coming from the diverse application domains have
is out of the scope of this work, but allows an agent to been categorized to a few broad data types. These data types
improve its experience and knowledge by learning itera- include Sequences (data generated by Omics technologies,
tively through interacting with its environment. e.g. [gen/transcript/epigen/prote/metabol]omics [40]),

Data Deep Learning Applications

Electroencephalogram Human activity recognition

Electromyogram Simple to Abstract Mapping/ Behavioral monitoring
Electrocardiogram features Classification Cognetive state
Single / multiunit activity Anomaly detection
Field potentials
Image reconstruction
Disease diagnosis
DA DBN Organ segmentaion
EM Images Anomaly detection
(f/s)MRI
PET/CT Scan
Radiographs Cell analysis
Fundus Images Cell/tissue classification
Endoscopy Images Cell count
RNN CNN Disease diagnosis

Gene/DNA/RNA seq. Disease prediction

Gene expression Drug design
Molecular components Gene modification
Protein structure Input Output Hidden Visible Kernel Conv/Pool
Alternative splicing
PS interpretation

Control ALU ALU

ALU ALU
Cache
Config.

DRAM DRAM DL4J scikit

CPU GPU FPGA ASIC DEEPLEARNING4J

Hardware Tools / Frameworks / Libraries

Fig. 1  The ecosystem of modern data analytics using advanced [Medical/Bio]-Imaging, and signals from the [Brain/Body]–Machine
machine learning methods with specific focus on application of DL Interfaces) are mined using DL with suitable architectures tailored for
to biological data mining. The biological data coming from various specific applications
sources (e.g. sequence data from the Omics, various images from the

13
Cognitive Computation (2021) 13:1–33 3

Images (data generated by [bio/medical/clinical/health] the components of this ecosystem (see the various compo-
imaging techniques containing [sub-]cellular and diagnos- nents of the ecosystem in Fig. 1). The remaining components
tic images), and Signals (electrical signals generated by the of the ecosystem include open access data sources and open-
brain and the muscles and acquired using appropriate sen- source toolboxes and libraries which are used in developing
sors at the [Brain/Body]–Machine Interfaces or BMI). Each the individual methods. It is therefore of paramount impor-
of these data types originating at diverse biological applica- tance to have a complete understanding of the availability of
tion domains have witnessed major contributions from the datasets and their characteristics, the capabilities and options
specified ML methods and their variants (see for Sequences offered by the libraries, and how they compare with each
[41], images [42–44], and signals [45–47]). other in different execution environments such as central
In recent years, DL methods are potentially reshaping processing unit (CPU) and graphical processing unit (GPU).
the future of ML and AI [48]. It is worthy to mention here The current paper’s novelty lies in being first of its kind to
that, from a broader perspective, ML has been applied to cover comprehensively the complete ecosystem of modern
a range of tasks including anomaly detection [49, 50, 278, data analysis using advanced ML technique, i.e., DL.
283, 290], biological data mining [51, 52], detection of coro- Therefore, with the above aim, this review provides—a
navirus [53, 54], disease detection and patient management brief overview on DL concepts and their applications to vari-
[55–57, 277, 279–282, 284, 286, 287, 289, 291], education ous biological data types; a list of available open access data
[58], natural language processing [59, 285, 288], and price repositories offering data for method development; and a list
prediction [60]. Despite notable popularity and applicability of existing open-source libraries and frameworks which can
to diverse disciplines [61], there exists no comprehensive be utilized to harness the power of these techniques along
review which focuses on pattern recognition in biological with their relative and performance comparison. Towards
data and provides pointers to the various biological data the end, some open issues are identified and some specula-
sources and DL tools, and the performances of those tools tive future perspectives are outlined.
[51]. The remainder of the article is organized as follows:
Also, considering the ecosystem of modern data analy- Section 2 provides the conceptual overview and intro-
sis using advanced ML techniques (such as DL), providing duces the reader to the underlying theory of DL; Section 3
information about methods application only partially covers describes the applications; Section 4 lists the open-source

a b
1%
3% 3%
4%
5%

5%

6%

58%
7%

8%

CNN FCN DA TRL RNN

ANN GAN DNN DBN DBM

Fig. 2  Application of different DL models to biological data. a colours of the individual pies match the colours in the wordcloud.
Wordcloud generated using author keywords extracted from research Legend—CNN: Convolutional Neural Network, FCN: Fully Con-
papers published between January 2011 and March 2020 which men- nected Network, DA[E]: Deep Autoencoder, TRL: Transfer Learn-
tioned analysis of biological data (images, signals and sequences) ing, RNN: Recurrent Neural Network (including Long Short-Term
using DL techniques and indexed in the Scopus database. The key- Memory or LSTM), ANN: Artificial Neural Network, GAN: Genera-
words were pruned to highlight the analysis methods. b Distribution tive Adversarial Network, DNN: Deep Neural Network, DBN: Deep
of published papers mentioning the usage of top 10 techniques. The Belief Network, DBM: Deep Boltzmann Machine

13
4 Cognitive Computation (2021) 13:1–33

data repositories; Section 5 presents the popular open-source BM is that, the learning process is computationally expen-
DL tools; and Sections 6 and 7 compare the most popular sive and quite slow. Due to this, a BM requires a long period
tools from relative and performance perspectives. Section 8 to reach equilibrium statistics [62]. However, this learning
presents the reader with some of the open issues and hints on inefficiency can be solved by forming a bipartite graph (i.e.
the future perspectives, and finally, the article is concluded restricting to have one hidden layer and one visible layer)
in Section 9. [67]. To extend this shallow architecture to a deep one, mul-
tiple RBMs as unitary learning elements are stacked together
and this yields the following two DL architectures.
Overview of Deep Learning
Deep Boltzmann Machine (DBM)
In DL the data representations are learned with increasing
abstraction levels, i.e., at each level more abstract represen- DBM [70] is a stack of undirected RBMs which supports
tations are learned by defining them in terms of less abstract a feedback mechanism among the layers to facilitate infer-
representations at lower levels [62]. Through this hierarchi- ence from higher-level units to propagate to lower-level
cal learning process, a system can learn complex representa- units. This allows an input to be alternatively interpreted
tions directly from the raw data [63]. through concurrent competition at all levels of the model.
Though many DL architectures have been proposed in Despite this powerful inference mechanism, estimating
the literature for various applications, there has been a con- model parameters from data remains a challenge and can-
sistent preference to use particular variants for biological not be solved using traditional gradient-based methods
data. As shown in Fig. 2, the most popular models have been (e.g., persistent contrastive divergence [71]) [70]. Though
identified as—Deep Neural Network (DNN), Deep Boltz- this learning problem is overcome by pretraining each RBM
mann Machine (DBM) and Deep Belief Network (DBN), in a layerwise greedy fashion, with outputs of the hidden
Deep Autoencoder (DA), Generative Adversarial Network variables from lower layers as input to upper layers [67], the
(GAN), Recurrent Neural Network (RNN, including LSTM), time complexity remains high and the approach may not be
and Convolutional Neural Network (CNN). Each of these suitable for large training datasets [72].
models’ architectures and their respective pros and cons are
listed in Table 1. The following subsections introduce each Deep Belief Network (DBN)
of these most frequently used DL architectures in mining
biological data. DBN [73], in contrast to the DBM, is formed by stacking
several RBMs together in a way that one RBM’s latent layer
Deep Neural Network (DNN) is linked to the next RBM’s visible layer. As the top two
layers of DBN are undirected, the connections are down-
A DNN [64] is inspired by the brain’s multilevel visual ward directed to its immediate lower layer [73, 74]. Thus, the
processing mechanism starting with the cortical area ‘V1’ DBN is a hybrid model with the first two layers as a undi-
and then to area ‘V2’, and so on [65]. Mimicking this, the rected graphical model and the rest being directed generative
traditional artificial neural network or NN is extended with model. The different layers are learned in a layerwise greedy
additional hidden layers containing nonlinear computational fashion and fine-tuned based on required output [75]; how-
units in each of these hidden layers to learn a subset of the ever, the training procedure is computationally demanding.
given representations. Despite its successful usage in a range
of different applications, the main drawback has been the Deep Autoencoder (DA)
slow and cumbersome training process [66].
DA is a DL architecture [76] obtained by stacking a num-
[Restricted] Boltzmann Machines ([R]BM) ber of data-driven Autoencoders which are unsupervised
elements. DA is also known as DAE and is designed to
[R]BM represents specific probability distributions through reduce data dimension by automatically projecting incom-
a undirected probabilistic generative model [67]. Considered ing representations to a lower-dimensional space than that
as a nonlinear feature detector, [R]BM is trained based on of the input. In an Autoencoder, equal amounts of units
optimizing its parameters for a set of given observations are used in the input/output layers and less units in the
to obtain the best possible fit of the probability distribu- hidden layers. (Non)linear transformations are embodied
tion through a Markov chain Monte Carlo method known in the hidden layer units to encode the given input into
as Gibbs sampling [68, 69]. With symmetrical connections smaller dimensions [77]. Despite the fact that it requires
among subsequent units in multiple hidden layers, BM has a pretraining stage and suffers from a vanishing error, this
only one visible layer. The main drawback of the standard architecture is popular for its data compression capability

13
Cognitive Computation (2021) 13:1–33 5

Table 1  Keypoints and applications of different deep learning architectures

Architecture Pros. Cons.

DNN - DNN can learn high-level feature - It requires substantial volume of train-
representation and apply transfer ing data.
learning. - Significant computational power is
- It can be used for healthcare and required.
visual recognition. - The learning process is slow.

Input Hidden Output

DBM - Graphical model, undirected links - High time complexity for interference
across a set of visible nodes and a than DBN. ↵ - Learning information
set of hidden nodes. does not reach to the lower layer.
- Used mainly for dimensionality - Tends to overfit.
reduction and classification.

Hidden Visible

DBN - Easy to code and works sufficiently - It can be trained greedily, one layer
well for just a few layers. at a time.
- High performance gain by add- - Hard to deduce posterior distribution
ing layers compared to multilayer for configurations of hidden causes.
perceptron.
- Robust in classification.
Hidden Visible

DA - Learn data encoding, reconstruction - Requires pretraining stage due to the

and generation at same time. chances of vanishing error.
- Training is stable without label data. - Each application requires redesigned
- Variant: sparse, denoising and con- and retrained the model.
tractive DA. - The DA is sensitive to input errors.

Input Hidden Output

GAN - The main benefit is data augmenta- - Difficult to train as optimizing loss
tion. function is hard and requires a lot of
- GAN performs unsupervised learn- trial and error.
ing.
- GAN learns density distributions of
data.
Backfed Input Hidden Input-Output

RNN - It can process inputs of any length. - Computation is slow and training can
- RNN can use internal memory and be difficult.
performs well for stream time series - Processing long sequences is difficult.
data. - Prone to problems such as exploding
and gradient vanishing.

Input Hidden Output

13
6 Cognitive Computation (2021) 13:1–33

Table 1  (continued)
Architecture Pros. Cons.

CNN - CNN can capture hierarchical infor- - Large labelled dataset is required for
Convolution and Pooling Fully connected mation. training.
- CNN can share pretrained weight - The working mechanism of CNN is
which is required for transfer learn- not clear.
ing.
- Requires less connection compared
to DNN.
Input Output
Kernel Conv/Pool Hidden

Legend: DA Deep Autoencoder, DBN Deep Belief Network, RNN Recurrent Neural Network, DNN Deep Neural Network, DBM Deep Boltz-
mann Machine, CNN Convolutional Neural Network.

and has many variants, e.g. Denoising Autoencoder [76], Convolutional Neural Network (CNN)
Sparse Autoencoder [78], Variational Autoencoder [79],
and Contractive Autoencoder [80]. CNN [90] is a multilayer NN model [91] which has gained
popularity in analysing image-based data. Inspired by the
neurobiology of the visual cortex, the CNN consists of con-
Generative Adversarial Network (GAN) volutional layer(s) containing a set of learnable filter banks
and followed by fully connected layer(s). These filter banks
GAN [81] is an effective generative model. Generative convolve with the input data and pass the results to activa-
models perform an unsupervised learning task, where they tion functions (e.g. ReLU, Sigmoid, and Tanh). There also
automatically discover and learn existing patterns in data exist subsampling steps in between these layers. The CNN
and then use that knowledge to generate new examples of outperforms DNNs, which as they do not scale well with
the learnt pattern as if they were drawn from the original multidimensional locally correlated input data. To address
dataset. Using GAN, the problem is seen as a supervised the scaling problem of DNNs, the CNN approach has been
learning problem with two strands: (i) the generator, which quite successful in analysing datasets with a high number
generates new examples as trained, and (ii) the discrimi- of nodes and parameters (e.g. images). As the images are
nator, which classifies generated examples to two classes ‘stationary,’ convolution filters (CF) can easily learn data-
(real or fake). These generator and discriminator models driven kernels. Applying such CF along with a suitable
are trained together in a zero-sum game (i.e. in an adver- pooling function reduces the features that are supplied to
sarial fashion) such that the examples generated by the the fully connected network to classify. However, in case of
generator model maximize the loss of the discriminator large datasets even this can be daunting and can be solved
model [82, 83]. using sparsely connected networks. Some of the popular
CNN configurations include AlexNet [92], VGGNet [93]
Recurrent Neural Network (RNN) GoogLeNet [94], etc. (see Table 2 for a complete list of
CNN’s variations with relevant details).
The RNN architecture [84] is designed to detect spatio-
temporal alignments in streams of data [85]. Unlike feed-
forward NN which performs computations unidirectionally Deep Learning and Biological Data
from input to output, an RNN computes the current state’s
output depending on the outputs of the previous states. Due Many studies have been reported in the literature which
to this ‘memory’-like property, despite learning problems employ diverse DL architectures with related and varied
related to vanishing and exploding gradients, RNN has parameter sets (see section 2) to analyse patterns in bio-
gained popularity in many fields involving streaming data logical data. For most of the DL architectures, as shown in
(e.g. text mining, time series, genomes, financial, etc.). In Fig. 3, the number of publications is increasing steadily over
recent years, two main variants, bidirectional RNN (BRNN) the years. A set of randomly selected representative studies
[86] and Long Short-Term Memory (LSTM) [87], have also from the large amount of reported literature are described
been applied [48, 88, 89]. below and summarized in Table 3. These studies belong to

13
Cognitive Computation (2021) 13:1–33 7

Table 2  Keypoints of different deep CNN architectures

Architecture Network Design Parameters Key points

LeNet (1998) LeNet-5 is the first CNN architecture with 2 convolu- 0.06 million - Feedforward NN.
tional and 3 fully connected layers. - Connection between layers is sparse to reduce
computational complexity.
AlexNet (2012) AlexNet has 8 layers and consists of 5 convolutional 60 million - Deeper than the LeNet and aliasing artifacts in the
and 3 fully connected layers. learned feature maps due to large filter size.
VGG-16 (2014) VGG-16 has 13 convolutional layers (and max pool- 138 million - Roughly twice deeper network can be designed
ing layers) and 2 fully connected layers followed by compared to the AlexNet.
1 output layer with softmax activation. - A deeper variant of VGG-16 is VGG-19.
- Computationally expensive and cannot be used
with low resource systems.
Inception-v1 (2014) Also known as GoogleNet, it has 22 layers with 5 million - It uses sparse connections to overcome redundant
parameters (or 27 when pooling layers are information problem and omits irrelevant feature
included). Towards the end, it employs an average maps.
pooling. - High accuracy with a reduced computational cost.
- It’s heterogeneous topology requires customization.
Inception-v3 (2015) Inception-v3 has 48 layers with a number of incep- 23 million - It increases accuracy and reduces computational
tion modules (each consisting of pooling layers and complexity in comparison to Inception-v1.
convolutional filters with activation functions), - Reduces representational bottleneck.
concatenation layers and fully connected layer(s) - Replaces large size filters with smaller ones.
along with dropout and softmax. - It’s architecture is complex and lacks homogeneity.
ResNet-50 (2015) ResNet-50 has 50 layers with initial convolutional 25.5 mil- - It provides an accelerated training speed.↵
and max-pooling layers, and final average pooling lion -Reduces the effect of Vanishing Gradient Prob-
and fully connected layers. In between, there are 3, lem.
4, 6 and 3 residual blocks separated in 4 stages - Classifies images with high accuracy.
where each block contains 3 convolutional layers.
Xception (2016) The Xception architecture has 36 convolutional 22.8 million - Xception shows small gains in classification per-
layers forming the feature extraction base of the formance on the ImageNet dataset and large gains
network. The 36 convolutional layers are structured on the JFT dataset when compared to Inception-
into 14 modules, all of which have linear residual v3.
connections around them, except for the first and
last modules.
Inception-v4 (2016) Inception-v4 consists of two main sections: a feature 43 million - Deep hierarchies of features, multilevel feature
extractor and a fully connected layer. The feature representation.↵ - Learning speed is slow.
extractor includes various convolutional blocks
such as 1 stem block, 14 inception blocks, 2 reduc-
tion blocks and a pooling layer. The inception
blocks are divided in three categories, namely, A,
B, and C with 4, 7, and 3 blocks, respectively.
Inception-ResNet-v2 Inception-ResNet-v2 consists of 164 layers with 56 million - It improves training speed.↵ - Deep hierarchies of
(2016) several convolutional blocks which include 1 stem features, multilevel feature representation.
block, 20 residual inception blocks, 2 reduction
blocks and a pooling layer. The residual inception
blocks are divided in three categories, namely, A,
B, and C with 5, 10, and 5 blocks, respectively.
ResNeXt-50 (2016) ResNeXt-50 has initial convolutional and max- 25 millions - Has homogeneous topology. ↵ - Performs grouped
pooling layers, and final average pooling and fully convolution.
connected layers. In between, there are 3, 4, 6
and 3 residual blocks separated in 4 stages where
each block contains 3 convolutional layers. In
comparison to ResNet-50, it scales up the number
of parallel towers (cardinality=32) within each
residual block.
DenseNet-121 DenseNet architecture includes 4 dense blocks. Each 8 millions - Introduces depth or cross-layer dimension.↵ -
(2016) layer in a dense block is connected to every other Ensures maximum data flow between the layers in
layer. The dense blocks, consisting of convolution, the network. ↵ - Avoids relearning of redundant
pooling, batch normalization and activation, are feature maps.
separated by transition layers.

13
8 Cognitive Computation (2021) 13:1–33

Images neuroendocrine carcinoma [55, 74, 105]. CNN’s dual pathway

a
100 version was used by Kamnitsas et al. to segment lesions related
to tumours, traumatic injuries, and ischemic strokes [109]. CNN
80
Publications (%)

was also used by Fritscher et al. for volume segmentation [101]

60 and by Cho et al. to find anatomical structures (Lung nodule to
40 classify malignancy) [106] from Computed Tomography (CT)
scans. DBN was applied on MRIs to detect Attention Deficit
20 Hyperactivity Disorder [96] and on cardiac MRIs to segment
0 the heart’s left ventricle [107]. GANs have gained popularity
DNN DBM DBN DA GAN RNN CNN in image synthesis and data augmentation to reduce overfitting.
b Signals GAN’s application in data augmentation and image translation
100 has been reviewed in [143] and data augmentation in the CT
segmentation tasks was done using CycleGAN [144]. GAN-
80
Publications (%)

based framework called MedGAN was proposed for medical

60 image-to-image translation [145]. GAN was used as survival
40 prediction model for chest CT scan images of patients suffer-
ing from idiopathic pulmonary fibrosis [146, 147]. GAN was
20 also used by Halicek for synthesizing hyperspectral images from
0 digitized histology of breast cancer cells [148].
DNN DBM DBN DA GAN RNN CNN

c Sequences Signals
100
80 A stacked DA was employed to detect emotion from Electroen-
Publications (%)

cephalography (EEG) signals after extracting relevant features

60 using Principal Component Analysis (PCA) and reducing non-
40 stationary effect using covariate shift adaptation [119]. DBN was
applied to decode motor imagery through classifying EEG sig-
20
nal [110]. For a similar purpose, CNN was used with augmented
0 common spatial pattern features [111]. EEG signals were also
DNN DBM DBN DA GAN RNN CNN classified using DA after features such as location, time, and
2015 2016 2017 2018 2019 frequency were extracted using CNN [112]. Li et al. used DBN
to extract low-dimensional latent features, and select critical
Fig. 3  Trends in publication involving different DL architectures channels to classify affective state using EEG signals [114].
from 2015 to 2019 in three major types of data—images a, signals b, Also, Jia et al. used an active learning to train DBN and genera-
and sequences c. The numbers of papers have been normalized within tive RBMs for the classification [115]. Tripathi et al. utilized
each data type. However, it is noteworthy that the ratio of number of
publications involving DL techniques applied to different data types
DNN- and CNN-based model for emotion classification [116].
(images, signals, and sequences) are approximately—1:14:101 CNN was employed to predict seizures through synchroniza-
tion patterns classification [118]. DBN [123] and CNN [122]
were used to decode motion action from NinaPro database. The
the three data types we have considered within the context later approach was also used on MIT-BIH, INCART, and SVDB
of this paper, that is, images, signals, and sequences. repositories [122]. Moreover, the Electrocardiogram (ECG)
Arrhythmias were classified using DBN [120, 121] from the
Images data supplied by MIT-BIH arrhythmia database. Zhu et al. used
a GAN model with LSTM and CNN to generate ECG signals
CNN was used by on histology images of the breast to find with high morphological similarity [149]. Another GAN model,
mitosis [108, 142] and to segment neuronal structures in Elec- RPSeqGAN, trained with SeqGAN [150] generated arrhythmic
tron Microscope Images (EMI) [103]. Havaei et al. used CNN ECG data with five periods and showed high stability and data
to segment brain tumour from Magnetic Resonance Imag- quality [151]. GAN is also used by Luo and Lu for EEG data
ing (MRI) [100] and Hosseini et al. used it for the diagnosis augmentation [152]. You et al. [153] and Jiao et al. [154] utilized
of Alzheimer’s disease (AD) from MRI [56, 97]. DBM [98] GAN-based model for detecting seizure using EEG signal and
and RBM [99] were used in detecting AD and mild cognitive Driver sleepiness using EEG and Electrooculography (EOG)
impairment (MCI) from MRI and Positron Emission Tomog- signals, respectively. Singh et al. proposed a new GAN frame-
raphy (PET) scans. Again, CNN was used on MRI to detect work for denoising ECG [155].

13
Cognitive Computation (2021) 13:1–33 9

Table 3  Deep learning applied to biological data

Type Data [base/set] DL architecture Task

Images ABIDE DNN [95] Autism disorder identification

ADHD-200 dataset DBN [96] ADHD detection
ADNI dataset CNN [97], DBM [98], DBN [99] AD/MCI diagnosis
BRATS Dataset CNN [100] Brain pathology segmentation
CT dataset CNN [101] Fast segmentation of 3D medical images
DRIVE, STARE datasets GAN [102] Retinal blood vessel segmentation
EM segmentation challenge dataset CNN [103] Segment neuronal membranes
LSTM [104] Biomedical volumetric image segmentation
IBSR, LPBA40, and OASIS dataset CNN [105] Skull stripping
LIDC-IDRI dataset CNN [106] Lung nodule malignancy classification
MICCAI 2009 LV dataset DBN [107] Heart LV segmentation
MITOS dataset CNN [108] Mitosis detection in breast cancer
PACS dataset CNN [106] Medical image classification
TBI dataset CNN [109] Brain lesion segmentation
Signals BCI competition IV DBN [110], CNN [111–113] Motion action decoding
DEAP dataset DBN [114, 115] Affective state recognition
CNN [116] Emotion classification
DECAF GAN [117]
Freiburg dataset CNN [118] Seizure prediction
MAHNOB-HCI DA [119] Emotion recognition
MIT-BIH arrhythmia database DBN [120, 121] ECG arrhythmia classification
MIT-BIH, INCART, and SVDB CNN [122] Movement decoding
NinaPro database DBN [123], CNN [122] Motion action decoding
Sequences CullPDB, CB513, CASP datasets, CAMEO CNN [124] 2ps prediction
DREAM CNN [125] DNA/RNA sequence prediction
DNN [126] Predict effective drug combination
ENCODE database CNN [127, 128] Gene expression identification
ENCODE DGF dataset CNN [129] Predict noncoding variant of gene
GEO database GAN [130] Gene expression data augmentation
GWH and UCSC datasets DBN [131] Splice junctions prediction
JASPAR database and ENCODE CNN [132] Predicting DNA-binding protein
miRBoost RNN [133] micro-RNA Prediction
miRNA-mRNA pairing data repository LSTM [134] micro-RNA target prediction
Protein Data Bank (PDB) DA [135] Protein structure reconstruction
SRBCT, prostate tumour, and MLL GE DBN [136] Gene/MiRNA feature selection
sbv IMPROVER DBN [137] Human diseases and drug development
TCGA database DA [138] Cancer detection and gene identification
DBM [139]
DNN [140] Drug combination estimation
UCSC, CGHV Data, SPIDEX database CNN [141] Genetic variants identification

Sequences [135]. Lee et al. applied a DBN-based unsupervised method

to perform autoprediction of splicing junction at Deoxyri-
The stacked denoising DA has been used to extract fea- bonucleic Acid (DNA) level [131]. Combining DBN with
tures for cancer diagnosis and classification along with the active learning, Ibrahim et al. devised a method to select fea-
identification of related genes from Gene Expression (GE) ture groups from genes or micro-Ribonucleic Acids (miR-
data [138]. GAN was also used for identifying expression NAs) based on expression profiles [136]. For translational
patterns from GE data [156]. A template-based DA learn- research, bimodal DBNs were used by Chen et al. to predict
ing model was used in reconstructing the protein structures responses of human cells using model organisms [137]. Pan

13
10 Cognitive Computation (2021) 13:1–33

et al. applied a hybrid CNN-DBN model on RNAs for the information from a number of different microscopic image
prediction of RNA-binding protein (RBP) interaction sites acquisition systems.
and motifs [157], and Alipanahi et al. used CNN to predict Another image library, called the Cell Image Library
sequence specificities of [D/R]BPs [125]. Denas and Tay- (CIL) [163], presents more than 10,000 unique datasets and
lor used CNN to preprocess data generated from Chromatin 20 TB of images, videos, and animations data. These data
Immunoprecipitation followed by sequencing (ChIP-seq) belong to a wide diversity of organisms, cell types, and cel-
and created gene transcription factor activity profiles [127]. lular processes.
CNN was used by Kelley et al. to predict DNA sequence The Euro Bioimaging [164] database provides biological
accessibility [128], by Zeng et al. to predict the DBP [132], and biomedical imaging data aiming to provide collabora-
by Zhou et al. [129] and Huang et al.[141] to find non-coding tion among different stakeholders including scientists, indus-
gene variation, and by Wang et al. to predict secondary pro- try, national and European authorities. Its mission is to give
tein structure (2ps) [124]. Park et al. used LSTM to predict access and services to state-of-the-art imaging techniques
miRNA precursor [133] and Lee et al. [134] used it to pre- and bioimaging data for scientists in Europe and beyond.
dict miRNA precursors’ targets. GAN was used by Marouf Euro Bioimaging also includes image analysis tools.
et al. for the realistic generation of single-cell RNA-seq data The HAPS is a histology image database [165] contains
[130], by Jiang et al. to predict disease gene from RNA-seq medium-/high-resolution photograph of microscopic image
data [158], by Zhao et al. as a semi-supervised procedure for of human cells and tissues which are free of any copy-
predicting drug target binding [159], and by Wang et al. for right. Another image database, the Image Data Resource
identifying expression patterns from GE data [156]. (IDR) [166], contains individual datasets of cellular and
tissue images. Various categories of images include time-
lapse imaging, protein localization studies, digital pathology
imaging, yeast study, human high-content screening, etc. It
Open Access Biological Data Sources
is also public API which facilitates viewing, analysis, and
sharing of multi-D image data for cell biology.
Reproducing scientific results, reported as statistically pro-
The SICAS Medical Image Repository (SMIR) is an
cessed quantitative data or carefully selected representative
image repository for medical research purpose. Two of their
qualitative data, has been facilitated greatly by data shar-
featured collections include post-mortem full-body CT [167]
ing initiatives [160]. In the last few decades, many open
scan of 50 anonymized subjects of different age groups and
access data repositories have been made available for this
gender, and CT, micro-CT, segmentation, and shape models
purpose [161]. Indeed, many research funders and journals
of the cochlea [183].
now require data used for studies to be made openly avail-
The Cancer Imaging Archive (TCIA) [168] contains CT,
able for verification. To facilitate method development, here
MRI, and nuclear medicine (e.g. PET) images for clinical
we list the leading and popular open access data repositories
diagnostic, biomarker, and cross-disciplinary investiga-
pertaining to the Sequences, Images, and Signals data which
tion. The Stanford Tissue Microarray Database (TMA)
are summarized in Tables 4, 5, and 6, respectively.
[169] is a source for annotated microscopic tissue images
and associated expression data. The data can be used for
Images studying cell biology. The UCSB bio-segmentation bench-
mark dataset [170] contains 2/3-D cellular, subcellular, and
Table 4 lists the leading open access data sources including tissue images. These datasets can be used for segmentation
databases and individual datasets that provide access to data and classification task.
pertaining to biological image research. For the sake of sim-
plicity, these sources have been grouped to four broad appli- Disease Detection and Diagnosis
cation areas—[bio/medical] image processing and analysis,
disease detection and diagnosis, neuroimage processing and A large amount of imaging data has been acquired from
analysis, and segmentation—and these are briefly described patients with neurological disorders. The Autism Brain
below. Imaging Data Exchange (ABIDE) [171] database includes
autism brain imaging datasets for studying the autism spec-
Bio/Medical Image Processing and Analysis trum disorder. The other dataset pertains to the Attention
Deficit Hyperactivity Disorder (ADHD) [172] and includes
The Cell Centered Database (CCDB) [162] collection 776 resting-state fMRI and anatomical datasets which are
provides high-resolution 3-D light and electron micro- fused over the 8 independent imaging sites. The phenotypic
scopic reconstructions of cells and subcellular structures. information includes age, sex, diagnostic status, measured
It also contains [2/3/4]-D protein distribution and structural ADHD symptom, intelligence quotient, and medication

13
Cognitive Computation (2021) 13:1–33 11

Table 4  Application-wise categorization of open access data repositories and datasets pertaining to [bio/medical/health/clinical] images
Application Name Description Ref.

Bio/medical image processing and analysis CCDB High-resolution 2/3/4-D light and electron microscope images [162]
CIL Cell image datasets and cell library app. [163]
Euro Bioimaging Biological and biomedical imaging data [164]
HAPS Microscopic image of human cells and tissues [165]
IDR Viewing, analysis, and sharing of multi-D image data [166]
SMIR Post-mortem CT scans of the whole body [167]
TCIA CT, MRI, and PET images of cancer patients [168]
TMA Microscopic tissue images of human [169]
UCSB BioSeg 2D/3D cellular, subcellular and tissue images [170]
Disease detection and diagnosis ABIDE Autism brain imaging datasets [171]
ADHD-200 fMRI/anatomical datasets fused over the 8 imaging sites [172]
ADNI MCI, early AD and elderly control subjects’ diagnosis data [173]
BCDR Multimodal mammography and ultrasound scan data [174]
Kaggle CXRayP Chest X-ray scans for pneumonia [175]
MITOS Breast cancer histological images [176]
NAMIC Lupus, brain, prostate MRI scans [177]
nCOV-CXray COVID-19 cases with chest X-ray/CT images [178]
Neurosynth fMRI datasets and synthesis platform [179]
NIH Labelled chest X-ray images with diagnoses [180]
OASIS MRI datasets and XNAT data management platform [181]
Open NI Imaging modalities and brain diseases data [182]
SMIR CT of human temporal bones [183]
Neuroimage processing and analysis IXI It provides neuroimaging data and toolkit software [184]
LPBA40 Maps of brain regions and a set of whole-head MRI [185]
NeuroVault.org API for collecting and sharing statistical maps of brain [186]
NITRC​ MRI, PET, SPECT, CT, MEG/EEG and optical imaging [187]
OpenfMRI Multimodal MRI and EEG datasets [188]
UK data service fMRI dataset [189]
Segmentation DRIVE Digital Retinal Images diabetic patient [190]
IBSR Segmentation results of MRI data [191]
STARE The dataset contains raw/labelled retinal images [192]

Legend: CXRayP Chest X-ray Pneumonia, JHDTI Johns Hopkins Diffusion Tensor Imaging

status. Imaging-based diagnostic classification is the main disease. On the other hand, the Open Neuroimaging (Open
aim of the ADHD 200 dataset. The ADNI (Alzheimer’s NI) [182] dataset contains imaging modalities and brain
Disease Neuroimaging Initiative [173]) is a popular data- diseases data which can be used to study decision support
base and contains neuroimaging datasets from neurodegen- system for disease identification.
erative diseases, in particular, AD, MCI, early and late AD The recent novel coronavirus disease or COVID-19 pan-
and elderly control subjects. The datasets offered by this demic has attracted a number of researchers to focus their
repository are mainly dedicated for development of novel attention on the detection of the novel coronavirus disease.
methods for diseases related to AD. Another dataset focus- The NIH [180]
ing on AD is the Open Access Series of Imaging Studies nCOV chest X-ray database [178] contains COVID-19
(OASIS) [181] dataset. This contains MRI datasets and cases with chest X-ray/CT images. The data can be used
open-source data management platform (XNAT) to study for identifying bacterial vs viral vs COVID-19 pneumo-
and analyse AD. Neurosynth [179] is yet another database nia. Similar chest X-ray datasets [175] are hosted by Kag-
which includes fMRI literature (with some datasets) and gle which include chest X-ray scans data for detecting
synthesis platform to study brain structure, functions, and traditional viral and bacterial pneumonia.

13
12 Cognitive Computation (2021) 13:1–33

Table 5  Application-wise categorization of open access data repositories and datasets pertaining to biological signals
Application Name Description Ref.

Anomaly detection SAD mc-EEG Multichannel EEG data for sustained-attention driving task [193]
TUH EEG Corpus Repository for EEG datasets, tools and documents [194]
MIT-BIH-ARH ECG database containing 48 recordings [195]
PTB D-ECG ECG database containing 549 recordings [196]
TELE ECG 250 ECG recordings with annotated QRS and artifact masks [197]
Human–Machine Interfacing BNCI Various BMI signal datasets [198]
EMG DataRep Various EMG datasets [199]
Facial sEMG Contains EMG data from 15 participants [200]
NinaPro database Kinematic as well as the sEMG data of 27 subjects [201]
Emotion/affective state detection DEAP Simultaneously recorded EMG/EEG data [202]
DECAF MEG, hEOG, ECG, trapezius muscle EMG, face video data [203]
Imagine EEG datasets of 31 subjects while listening voice [204]
MAHNOB-HCI EMG, ECG, and respiration and skin temperature data [205]
SEED EEG dataset for emotion and vigilance [206]
Motor imagery classification EEG-BCI-MI EEG signals from 13 subjects with 60,000 MI examples [207]
EEG-MI-BCI EEG data from BCI for MI tasks [208]
EEG-MMI EEG data from PhysioNet for MI task [209]
Neurological condition evaluation V-P300 BCI 16-electrode dry EEG from 71 subjects (SP mode) [210]
32-electrode wet EEG from 50 subjects (SP mode) [211]
32-electrode wet EEG from 38 subjects (MPC mode) [212]
32-electrode wet EEG from 44 subjects (MPCC mode) [213]
Signal processing and classification BCI competition EEG, ECoG, and MEG data from a range of BCI applications [214]
BCI-NER challenge 56 channel EEG dataset decoded by a P300 speller [215]
DRYAD EEG datasets of 13 subjects recorded under various conditions [216]
PhysioNet Various EEG, ECG, EMG and sEMG datasets [217]
UCI ML Various ECG, EMG, sEMG datasets [218]

Legend: MI Motor Imagery, MMI Motor Movement/Imagery, ERP Event-Related Potentials, SADmc-EEG Sustained-Attention Driving multi-
channel EEG, V-P300 Visual P300, SP Single Player, MP Multiplayer, BCI-SSVEP Steady State Visual Evoked Potentials, EMG DataRep EMG
Dataset Repository, ARH Arrhythmia, D-ECG Diagnostic ECG

Breast cancer is also another important disease which and diffusion weighted images. These images have been
can be addressed through imaging and this has attracted a collected from three different hospitals in London, UK.
number of databased hosting breast cancer images. Another database, called the Loni Probabilistic Brain Atlas
The Breast Cancer Digital Repository (BCDR) [174] (LPBA40) [185], contains maps of brain anatomic regions
database contains multimodal mammography and ultra- of 40 human volunteers. Each map generates a set of whole-
sound scan and patient history data collected from 1734 head MRI, whereas each MRI describes to identify 56 struc-
anonymized patients. The data can be used for disease tures of brain, most of them lies in the cortex. The study of
detection and diagnosis methods. Another dataset, MITOS skull-stripped MRI volumes, and classification of the native-
[176], contains breast cancer histological images (haema- space MRI, probabilistic maps are key uses of LPBA40. The
toxylin and eosin stained slides). The detection of mitosis NeuroVault.org [186] is a web-based repository (API) for
and evaluation of nuclear atypia are key uses. collecting and sharing statistical maps of the human brain to
study human brain regions. The Neuroimaging Informatics
Neuroimage Processing and Analysis Tools and Resources Clearing house (NITRC) [187] pro-
vides range of imaging data from MRI to PET, SPECT, CT,
The Information eXtraction from Images (IXI) dataset [184] MEG/EEG, and optical imaging for analysing functional
provides 600 MRI images from healthy subjects to study and structural neuroimages. The Open fMRI [188] dataset
brain functions. These images saved in NIFTI file format contains MRI images acquired using different modalities
and were acquired using protocol—T1, T2, proton-density including diffusion-weighted, T1-weighted magnetization
weighted images; magnetic resonance angiography images; prepared rapid acquisition with gradient echo (MPRAGE)

13
Cognitive Computation (2021) 13:1–33 13

Table 6  Application-wise categorization of open access data repositories and datasets pertaining to Omics data
Application Name Description Ref.

Bioassay analysis and drug design COVID-19 Gene sequence, pathway, and bioassay datasets of COVID-19 [220]
PubChem Contains compound structures, molecular datasets, and tool [221]
Genetic disorder analysis Cancer GeEx Different cancer genome datasets [222]
IGDD Mutation data on common genetic diseases [223]
TCGA​ Contains cancer genome data [224]
BDTNP 3D Gene expression, DNA-binding data and ChAcD [225]
Nucleic acid research ENCODE Human genome dataset [226]
ESP Contains sequencing data [227]
GEO Contains high-throughput gene expression and functional genomics [228]
datasets
gnomAD Large-scale exomes and genomes sequencing data [229]
GTEx Gene expression datasets [230]
Harmonizome Collection of genes and proteins datasets [231]
INSDC Contains nucleotide sequence data [232]
IGSR Genome data of various ethnicities, age and sex [233]
JASPAR Transcription factor DNA-binding preferences dataset [234]
NIHREM Human genome datasets [235]
NSD Includes omics and health science data [236]
SysGenSim Bioinformatics tools and gene sequence dataset [237]
Protein structure analysis PDB Proteins, nucleic acids, and complex assemblies data [238]
SCOP2 Contains structural classification of proteins [239]
SCOPe [240]
UCI MB 2ps and splice–junction gene sequences [241]
Signal transduction pathway study NCI Nature Molecular interactions and reactions of cells [242]
NetPath Signal transduction pathways in humans [243]
Reactome Database for reactions, pathways and biological processes [244]
Single-cell omics miRBoost The genomes of eukaryotes containing at least 100 miRNAs [245]
SGD Provides biological data for budding yeast and analysis tool [246]

MRI, and multiecho fast low-angle shot (FLASH) MRI. It STructured Analysis of the Retina (STARE) was initi-
also contains biosignal datasets to study brain regions and its ated in 1975. The project contains datasets of 400 raw
functions. These can be used as a benchmark dataset in order retinal images, 10 labelled images of artery/vein, and
to differentiate outcome from various neuroimaging analysis 80 images with ground truth. Each image is annotated
tools. The UK data service [189] contains T1/2, diffusion and features are shown in image by the expert. The data-
tensor imaging, and fMRI datasets from 22 patients suffering set can be used for blood vessel segmentation and optic
from brain tumours which can be useful for studying brain nerve detection.
tumour surgical planning. The Internet Brain Segmentation Repository (IBSR)
gives segmentation results of MRI data. Development of
Segmentation segmentation methods is the main application of this IBSR.

Segmentation is an important step in any image processing Signals

pipeline. Many datasets mentioned above can be used for
segmentation purposes. Table 5 lists leading open access data repositories and
Focusing on eye diseases, the Digital Retinal Images datasets (also referred as data sources) pertaining to bio-
for Vessel Extraction (DRIVE) contains JPEG Com- logical signals. These sources are broadly mapped to six
pressed retinal images of 400 diabetic patients between application areas—anomaly detection, human–machine
25-90 years old. This dataset can be used to understand interfacing which includes brain–machine interfacing as
segmentation of blood vessels in retinal images and well as rehabilitation research, emotion/affective state detec-
identify diabetic retinopathy. Another dataset called tion, motor imagery classification, neurological condition

13
14 Cognitive Computation (2021) 13:1–33

evaluation, and signal processing and classification—which multifinger movements datasets of 2 channels, 10 classes
are described in the following subsections. and 8 channels, 15 classes; single-/multifinger pressure on a
steering wheel; EMG controlled multifunctional upper-limb
Anomaly Detection prostheses and EMG pattern recognition datasets.
For surface EMG (sEMG), the facial sEMG dataset
Anomaly detection is one of the major application areas in contains facial sEMG signals from the muscles corrugator
which scientists have devoted much efforts. In this process, a supercilii, zygomaticus major, orbicularis oris, orbicularis
number of open access data sources, largely containing EEG oculi, and masseter. Archived data are from 15 participants
and ECG data, have been frequently used. (8 females and 7 males) aged between 26 and 57 years (mean
Starting with the EEG signals, the SAD mc-EEG [193] age 40.7 ± 9.6 years). These data can be used for rehabilita-
dataset contains 32 channel EEG signals from 27 subjects tion research. Also, the NinaPro database includes kinematic
recorded while they were test-driving. That is, signals were as well as sEMG data of 27 subjects, while these subjects
acquired when each subject attended two 90-minute virtual were moving finger, hand, and wrist. These data can be
reality session for sustained-attention driving. employed to study biorobotics and activity detection.
The TUH EEG corpus [194] is also an open-source clin-
ical EEG data repository for clinical EEG data, tool and
documentation. The major datasets include seizure detec- Emotion/Affective State Detection
tion, abnormal EEG, EEG with artifacts (introduced by eye
movement, chewing, shivering, electrode pop, electrode Emotion and affective state detection has been a very active
static, and lead artifacts, and muscle artifacts), EEG for epi- research field over the years. A combination of different
lepsy, etc. signals has been utilized in detecting emotion and affective
Regarding the ECG signals, the MIT-BIH arrhythmia states, and a number of data sources providing these signals
[195] arrhythmia database includes 2-channel ambulatory are described below.
ECG recording taken from 47 subjects for studying arrhyth- A Database for Emotion Analysis using Physiological
mia. There are 48 complete ECG recordings and about 24 Signals (DEAP) provides various datasets for analysing the
recordings are freely available. The PTB diagnostic ECG human affective states. It provides EEG and sEMG signals
database [196] comprises 549 ECG recordings taken from of 32 volunteers, while they were watching music videos
290 subjects of age ranged from 17 to 87 years using con- to analyse the affective states. These volunteers also rated
ventional 12 leads and 3 Frank lead ECG recorder. Each the video, and the front face was also recorded for 22 vol-
recording includes 15 signals coming from these leads and unteers. DECAF is a multimodal dataset for decoding user
each subject was represented in 1 to 5 records. Both the data- physiological responses to affective multimedia content. It
sets can be used for anomaly detection. Another ECG data- contains magnetoencephalogram (MEG), horizontal elec-
set, the TELE-ECG dataset [197] includes 250 ECG records trooculogram (hEOG), ECG, trapezius muscle EMG, and
with annotated QRS and artifact masks. It also includes QRS near-infrared face video data to study physiological and
and artifact detection algorithms to study QRS and detect mental states. Another multimodal dataset is the MAH-
artifacts from ECG signals. NOB-HCI [205] dataset which includes ECG, respiration,
and skin temperature data in addition to 32-channel EEG
signals from 30 subjects, while they were watching movie
Human–Machine Interfacing clips and photos. The different sensors were synchronized
to record a synchronized multimodal dataset. The subjects
The application area of Human–Machine Interfacing focuses were asked to label their own emotion state.
on [body and brain]–machine interfacing and rehabilitation. On the other hand, the Imagined Emotion [204] dataset
This is done largely through Electromyography (EMG) and provides EEG signals recorded when subjects were listen-
sometimes with EEG signals. ing to voice recording. The SJTU Emotion EEG Dataset
The BNCI Horizon 2020 database contains more than 25 [206] contains three individual datasets (SEED, SEED-IV
datasets such as stimulated EEG datasets, Electrocorticog- and SEED-VIG) of EEG signals. In the SEED dataset EEG
raphy (ECoG)-based BCI datasets, Event Related Potential signals were recorded, while the subjects were watching
(ERP)-based BCI datasets, mental arithmetic, motor imagery movie clips and annotated their emotional state as positive,
(extracted from EEG, EOG, fNIRS, EMG) datasets, EEG/ negative and neural. In case of SEED-IV, four emotional
EOG datasets of neuroprosthetic control, speller datasets. states such as happy, sad, fear, and neutral were anno-
Modelling and designing of BMI devices are the key appli- tated, whereas the SEED-VIG dataset contains EEG sig-
cation of this database. While the BNCI contains a variety of nals related to vigilance when the subjects were driving.
signals, the EMG Datasets Repository [199] includes single/

13
Cognitive Computation (2021) 13:1–33 15

Motor Imagery Classification BCI-NER dataset [215]. This dataset was mainly intended
for methodological development of an error detection algo-
Motor imagery (MI) is yet another very active area of rithm suitable for the P300-based BCI systems. The BCI
research. As an outcome of a large number of community competition datasets include EEG datasets (e.g., cortical
contributors, many datasets have been developed from negativity or positivity, feedback test trials, self-paced key
which the popular ones are described below. typing, P300 speller paradigm, motor/mental imagery data,
The electroencephalographic brain–computer interface continuous EEG, EEG with eye movement), ECoG datasets
mental imagery (EEG-BCI-MI) [207] dataset contains 60 (e.g., finger movement, motor/mental imagery signals in the
hours of EEG recording from 13 subjects and 75 experi- form of EEG/ECoG), and MEG dataset (e.g., wrist move-
ments. This contains around 60,000 mental imagery exam- ment). These datasets can be used for signal processing and
ples which is approximately 4.8 hours of EEG recordings classification methods for BMI. Similarly, the BCI-NER
(with 4600 MI examples) per participant. The datasets can Challenge [215] dataset provides 56-channel EEG signals
be used for the rehabilitation of patients having move- from 26 subjects using a P300 speller.
ment disorders. Another EEG dataset for MI brain–com- In addition to the datasets released for challenges and
puter interface (EEG-MI-BCI) [208] contains EEG signals competitions, there are repositories which provide rich data-
with 3-D electrode location and EEG for non-task-related sets for this application area. The DRYAD [216] is a versa-
states as well. The dataset was recorded from 52 partici- tile repository which has been recently unveiled. It contains
pants which also contains [physio/psyco]logical data and a range of EEG recorded datasets when 19 subjects listen to
EMG signals in addition to the EEG. The dataset can be natural speech time-reversed speech, cocktail party atten-
employed to find the human factors which influence MI tion, and noisy audiovisual speech. The PhysioNet reposi-
BCI performances. Yet another EEG signal centric data- tory [217] contains a large number of neuroelectric and
set is called EEG motor movement/imagery (EEG-MMI) myoelectric datasets. As the name suggests, it is mainly for
dataset [209] and incorporates 1500 (1–2 minutes) EEG physiological data. These datasets mainly pertain to signals
recordings taken from 109 volunteers. The dataset can be such as EEG, ECoG, EMG, and ECG and are acquired from
used in designing BCI systems for rehabilitation purposes. many diverse experimental settings. The UCI ML repository
[218] contains a large number of diverse datasets with direct
Neurological Condition Evaluation application to machine learning methods. Some relevant
biosignal datasets include ECG, EEG, and (s)EMG signals
A number of visual P300-based datasets are available with from diverse experimental and physiological conditions.
open-access attributes to perform a range of neurological
condition evaluation. These datasets, V-P300 BCI, are com-
posed of data recorded using dry or wet electrode with 16 Sequences
or 32 channels while the subjects were playing the Brain
Invaders game [219]. These datasets were recorded using Table 6 lists the leading popular open access data sources
different playing modalities such as single player (16 dry pertaining to the various omics-related researches which
electrodes [210] from 71 subjects and 32 wet electrodes include genomics, proteomics, and metabolomics. Grouped
[211] from 50 subjects), multiplayer in collaborative mode to six broad application areas, namely, bioassay analysis and
(32 wet electrodes from 38 subjects [212]), and multiplayer drug design, genetic disorder analysis, nucleic acid research,
cooperation and competition mode (32 wet electrodes from protein structure analysis, signal transduction pathway study,
44 subjects [213]). and single-cell omics, the following subsections provide
brief discussions about the leading open access omics data
Signal Processing and Classification sources.

To solve various signal processing and classification prob- Bioassay Analysis and Drug Design
lems, a number of datasets have been made available under
open-access. Most of these problems are released to the Since December 2019, the world has experienced a pan-
community in the form of challenges with relevant datasets demic caused by the SARS-CoV-2 (COVID-19) virus. Trig-
to solve them. The competitions during the BCI meetings gered by the necessity to facilitate the ongoing researches,
have served this purpose for several years and have released the SARS-CoV-2 [220] dataset provides gene sequence, pro-
datasets (the BCI competition datasets [214]) which are still teins, pathway, and bioassay for SARS-CoV-2 along with
available with relevant problem statements and sample codes compounds used in clinical trials. This dataset can be used
for others to use. The challenge dataset provided by the IEEE for studying biological/chemical process and drug design.
Neural Engineering Conference (NER2015) is known as

13
16 Cognitive Computation (2021) 13:1–33

The PubChem database [221] contains millions of com- from different sequencing projects. The dataset can be used
pound structures and descriptive datasets of chemical mol- for disease diagnosis and genetic studies. The Genotype-
ecules and their activities against biological assays. Main- Tissue Expression (GTEx) [230] database contains GE data-
tained by the National Center for Biotechnology Information sets of 54 healthy tissue sites collected from 1000 subjects
of the United States National Institutes of Health, it can be and histology images. It also includes samples from GTEx
freely accessed through a web user interface and down- biobank.
loaded via FTP. It also contains software services (such as The Harmonizome [231] database provides details about
plotting and clustering). It can be used for [gen/prote]-omics genes and proteins from 114 datasets provided by 66 online
study and drug design. resources with 71927784 associations between 295496
attributes and 56720 genes. The International Nucleotide
Sequence Database [232], popularly known as INSDC, cor-
Genetic Disorder Analysis roborates biological data from three major sources: i) DNA
Databank of Japan [247], ii) European Nucleotide Archive
The cancer gene expression (GE) [222] serves as a small [248], and iii) GenBank [249]. These sources provide the
repository containing several cancer GE datasets which can spectrum of data raw reads, though alignments, and assem-
be employed for designing tool/algorithm for cancer detec- blies to functional annotation, enriched with contextual
tion. The cancer genome atlas (TCGA) [224] repository information relating to samples and experimental configu-
contains more than 2.5 petabytes of genomic, epigenomic, rations. Similar to this, the International Genome Sample
transcriptomic, and proteomic data. It contains data about Resource (IGSR) [233] includes genome sequencing data
33 different cancer types and over 20,000 samples. These from 1000 genomes project. The genome data was taken
data are generated by the National Cancer Institute and the from people of various ethnicities, age, and sex with the final
National Human Genome Research Institute. This reposi- dataset contains gene sequencing data from 2,504 individu-
tory is used in facilitating genomic study for improving the als from 26 populations. These data can be used for disease
prevention, diagnosis, and treatment of cancer. To analyse diagnosis and genetic studies. Also, the SysGenSim [237]
region-specific diseases, the Indian Genetic Disease Data- database includes bioinformatics tool, and Pula-Magdeburg
base (IGDD) [223] tracks mutations in the normal genes for single-gene knockout, StatSeq, and DREAM 5 benchmark
genetic diseases reported in India. datasets for studying Gene Sequence.
JASPAR [234] is a database for transcription factor
Nucleic Acid Research DNA-binding profile. The data spans through six different
taxonomic groups covering Vertebrata, Nematoda, Insecta,
The Berkeley Drosophila Transcription Network Project Plantae, Fungi, and Urochordata. The database can be used
(BDTNP) [225] database contains datasets pertaining to 3D for translational genomics research.
Gene expression data, in vivo and in vitro DNA-binding data The NIH Roadmap Epigenomics Mapping repository
as well as Chromatin Accessibility data (ChAcD). Research (NIHREM) [235] includes 2,804 datasets, i.e., 1,821 his-
on GE and anomaly detection is the key application of the tone modification, 360 DNase, 277 DNA methylation,
datasets provided by this database. and 166 RNA-Seq datasets. The repository provides
The Encyclopedia of DNA Elements (ENCODE) [226] 3,174-fold 150.21 billion mapped sequencing the human
is a whole-genome database curated by the ENCODE Con- and tools for analysing these datasets. It can be used
sortium. It contains a large number of datasets pertaining for stem cell mapping and selection of tissues that are
to functional genomics and characterization data including responsible for human disease. Also, the database known
meta-data of human, worm, mouse, and fly. Another data- as Nature scientific data (NSD) [236] includes datasets
base, called the Exome Sequencing Project (ESP) [227], pertaining to omics, taxonomy and species diversity,
includes genome datasets which can be used to find lung and mathematical and modelling resources, cytometry, organ-
blood disorders and their management and treatment. The ism-focused resources, and health science data. This can
Gene Expression Omnibus (GEO) [228] is an open-access be used for studying and modelling different aspects of
functional genomics (microarray and sequence) data reposi- genomics.
tory. This database can be used for functional genomic and
epigenomic studies such as genome methylation, chromatin Protein Structure Analysis
structure, and genome–protein interactions. It is supported
by the National Center for Biotechnology Information at The Protein Data Bank (PDB) [238] contains 3D structural
the National Library of Medicine of the USA [228]. The data proteins and nucleic acids. These data are obtained tools
Genome Aggregation Database (gnomAD) [229] database such as X-ray crystallography, NMR spectroscopy, and cryo-
contains large-scale exome and genome sequencing data electron microscopy. It includes more than 135 thousand

13
Cognitive Computation (2021) 13:1–33 17

data of proteins, nucleic acids, and complex assemblies. and miRNA-related pathology. Saccharomyces Genome
These can be used to understand all aspects of biomedicine Database (SGD) [246] also provides complete biological
and agriculture. information for the budding yeast Saccharomyces cerevi-
Structural classification of proteins (SCOP) is a reposi- siae. They also give an open-source tool for searching and
tory which hosts manually classified protein structure analysing these data and thereby enable the discovery of
datasets. The classification was done based on amino acid functional relationships between sequence and gene products
sequences and their structural similarity. The main objec- in fungi and higher organisms. The study of genome expres-
tive is to find the evolutionary relationship between the sion, transcriptome, and computational biology is the main
proteins. Currently two versions of SCOP are maintained. function of the SGD.
The SCOP Version 2 (SCOP2) [239] is the up-to-date SCOP
database released at the first quarter of 2020. In contrast,
the SCOP-extended (SCOPe) [240] is an extended version Open‑Source Deep Learning Tools
of the original SCOP maintained by UC Berkeley. SCOPe
includes many new classified protein structures via a fusion Due to surging interest and concurrent multidiscipli-
of manual and automation curation. nary efforts towards DL in the recent years, several
Molecular Biology Databases at the UCI (UCI MB) contain open-source libraries, frameworks, and platforms have
three individual databases: i) Secondary Protein Structure [241], been made available to the community. However, for
which is a bench repository that classifies secondary structure of a new user of these tools to mine biological data, it is
certain globular proteins; ii) Splice–Junction Gene Sequences not always straightforward to know their characteris-
[250], which contain primate splice–junction gene sequences tics, advantages, and disadvantages. In this process, one
(DNA) with associated imperfect domain theory; and iii) Pro- of the main hurdles for a new analyst is to select the
moter Gene Sequences [251], which contain E. coli promoter appropriate DL architecture/model and relevant library
gene sequences (DNA) with partial domain theory. Objectives providing suitable implementations of the selected archi-
include i) sequencing and predicting the secondary structure tecture. Towards introducing a beginner to the field of
of certain proteins; ii) studying primate splice–junction gene biological data analysis using these open-source tools,
sequences (DNA) with associated imperfect domain theory; iii) this section describes the tools in a tutorial style indi-
studying E. Coli promoter gene sequences (DNA) with partial cating their characteristics, pros, and cons. The focus
domain theory. of the section has been to review and summarize the
most popular open-source tools, which aim to facilitate
Signal Transduction Pathway Study the technological developments for the community. This
comprehensive collection contains tools (also developed
The NCI–Nature Pathway Interaction Database [242] hosts by individuals) which are well maintained with a rea-
cellular signalling (molecular interactions/reactions) pathways sonable amount of implemented algorithms (i.e., deep
in humans. The database can be employed for cancer research. learning architectures). For the sake of brevity, the indi-
The database was created by the U.S. National Cancer Institute, vidual publication references of the tools are omitted and
NIH, with the collaboration of Nature Publishing Group and interested readers may consult them at their respective
published in the last quarter of 2006. Another database, NetPath websites from the provided URLs.
[243], also contains signal transduction pathways in humans. Table 7 summarizes the main features and differences of
Created jointly by Johns Hopkins University and the Institute the various tools. To measure the impact and acceptability of
of Bioinformatics (IOB) in India; it includes 45 signalling path- a tool in the community, we provide GitHub-based measures
way ranging from protein–protein interactions to enzyme–pro- such as numbers of Stars, Forks, and Contributors. These
tein substrate reactions including 10 major pathway of immune numbers are indicative of the popularity, maturity, and dif-
system and 10 pathway relevant to cancer regulation. The other fusion of a tool in the community.
one, Reactome [244], is an open access database hosting biologi-
cal pathways of metabolic processes to hormonal signalling in Caffe
humans. Created through a collaboration between North Amer-
ica and Europe, it can be used for cancer research and treatment. Caffe (http://caffe.​ berkel​ eyvis​ ion.org/) is scalable, written in
C++ and provides bindings for Python as well as MATLAB.
Single‑cell Omics Dedicated for experiment, training, and deploying gen-
eral purpose DL models, this framework allows switching
The miRBoost dataset [245] contains the genomes of eukar- between development and deployment platforms. Targeting
yotes containing at least 100 miRNAs. This dataset is used computer vision applications, it is considered as the fastest
for studying post-transcriptional gene regulation (PTGeR) implementation of the CNN.

13
18 Cognitive Computation (2021) 13:1–33

Table 7  Summary of Open-Source Deep Learning Tools (* as of July 2020)

Tool Platform Language(s) Stars* Forks* Contrib.* Supported DL Architecture

Caffeb L, M, W, A Py, C++, Ma 30100 18200 266 CNN, RNN, GAN

Chainerc L Py 5300 1400 251 DA, CNN, RNN, GAN
DL4ja L, M, W Ja 11500 4800 32 DA, CNN, RNN, RBM, LSTM, GAN
DyNeta L C++ 3000 687 117 CNN, RNN, LSTM
H2Oa L, M, W Ja, Py, R 4700 1700 132 CNN, RNN
Kerasc L, M, W Py 47500 18000 816 CNN, RNN, DBN, GAN
Lasagnea L, M Py 3700 980 68 CNN, RNN, LSTM, GAN
MCTc W C++ 16720 4400 197 CNN, DBN, RNN, LSTM
MXNeta L, M, W, A, I C++ 18500 6600 780 DA, CNN, RNN, LSTM, GAN
Neona L, M Py 3800 846 78 DA, CNN, RNN, LSTM, GAN
PyTorchb L, M Py 37400 9500 1345 CNN, RNN, LSTM, GAN
Singhaa L, M, W Py, C++, Ja 2000 499 46 CNN, RNN, RBM, DBM
TensorFlowa L, M, W Py, C++ 14300 80600 2450 CNN, RNN, RBM, LSTM, GAN
TF.Learnc L, M Py, C++ 9400 2400 120 CNN, BRNN, RNN, LSTM, GAN
Theanob L, M, W Py 9103 2500 332 CNN, RNN, RBM, LSTM, GAN
Torchb L, M, W, A, I Lu, C, C++ 8495 2400 130 CNN, RNN, RBM, LSTM, GAN
Velesa L, M, W, A Py 891 185 10 DA, CNN, RNN, LSTM, RBM

L Linux/Unix, M MacOSX, W Windows, A Android, I iOS, CP Cross-platform, Py Python, Ja Java, Lu Lua, Ma Matlab
*GitHub parameters (as of 1 April. 2020)
a
Apache2 License
b
BSD License
c
MIT License

Pros. Cons.

– Easy to deploy; – Open Computing Language framework/Open Multi-

– Pretrained models are available; Processing API is not supported.
– Faster training speed;
– Used for feedforward networks. DeepLearning4j

Cons. DeepLearning4j (DL4J, https:​ //deeple​ arnin​ g4j.org/), written

in Java with core libraries in C/C++, is a distributed frame-
– Requires writing code for generating new layers; work for quick prototyping that targets mainly non-research-
– Less support for recurrent networks; ers. Compatible with JVM supported languages (e.g., Scala/
– No support for distributed training. Clojure), it works on distributed processing frameworks
(e.g., Hadoop and Spark). Through Keras (see section 5.6) as
Chainer a Python API, it allows importing existing DL models from
other frameworks. It allows creation of NN architectures by
Chainer (http://chain​er.org/) is a DL framework provided as combining available shallow NN architectures.
Python library. Besides the availability of popular optimiza- Pros.
tion techniques and NN related computations (e.g., convo-
lution, loss, and activation functions), dynamic creation of – Supports integration with Big Data frameworks Apache
graphs makes Chainer powerful. It supports a wide range Spark and Hadoop;
of DL architectures including CNN, GAN, RNN, and DA. – Supports distributed GPU and CPU platforms and capa-
Pros. ble to work with tensor.

– One of the tools for leading dynamic computation graphs/ Cons.

networks;
– Notably faster than other Python-oriented frameworks. – Open Computing Language framework is not supported;

13
Cognitive Computation (2021) 13:1–33 19

– GUI is supported for workflow and visualization. Pros.

DyNet – Rich documentation;

– A high-level API for neural networks;
The DyNet library (https​: //dynet​. readt​h edoc​s .io/), writ- – Ability to run on top of state-of-the-art deep learning
ten in C++ with Python bindings, is the successor of libraries/frameworks such as TensorFlow, CNTK, or
the ‘C++ neural network library’. In DyNet, computa- Theano.
tional graphs are dynamically created for each training
example; thus, it is computationally efficient and flex- Cons.
ible. Targeting NLP applications, its specialty is in CNN,
RNN, and LSTM. – Cannot utilize multi-GPU directly;
Pros. – Requires Theano as backend for OpenMP support and
Theano/TensorFlow/PlaidML as backend for OpenCL.
– Designed to be efficient for running on CPU or GPU.
– Dynamic computation graph like PyTorch and Chainer. Lasagne

Cons. Lasagne (http://lasag​ne.readt​hedoc​s.io) DL library is built on

top of Theano. It allows multiple input, output, and auxiliary
– In terms of TensorFlow, limited functions are available. classifiers. It supports user-defined cost functions and provides
many optimization functions. Lasagne supports CNN, GAN,
H2O RNN, and LSTM.
Pros.
H2 O (http://www.h2o.ai) is an ML software that includes
DL and data analysis. It provides a unified interface to – Lasagne is a lightweight library to build and train DL
other DL frameworks like TensorFlow, MXNet, and algorithms in Theano;
Caffe. It also supports training of DL models (CNN and – Layers, regularizers, and optimizers can be used indepen-
RNN) designed in R, Python, Java, and Scala. dently;
Pros. – Clear documentation is available;
– Supports training the network on a GPU.
– Due to its in-memory distributed parallel processing
capacities, it can be used for real-time data; Cons.
– GUI is supported (called Flow) for workflow and visu-
alization; – Small community than TensorFlow.
– GPU support for Deep Water and NVIDIA;
– Fast training, memory-efficient DataFrame manipulation; Microsoft Cognitive Toolkit
– Easy-to-use algorithms and well documented;
Replacing CNTK, the Microsoft Cognitive Toolkit (MCT, https​
Cons. ://cntk.ai/) is mainly coded in C++. It provides implementations
of various learning rules and supports different DL architectures
– Lacks the data manipulation capabilities of R and Pandas including DNN, CNN, RNN, and LSTM.
DataFrames; Pros.
– Slow in learning and supports limited model running at
a time. – It is a framework for feedforward DNNs, CNN and RNN;
– Can train production systems very fast;
Keras – Can achieve state-of-the-art performance on benchmark
tasks;
The Python-based Keras (https​: //keras​. io/) library is – Allow directed graph visualization.
used on top of Theano or TensorFlow. Its models can be
imported to DL4J (see section 5.3). It was developed as Cons.
a user friendly tool enabling fast experimentation, and
easy and fast prototyping. Keras supports CNN, GAN, – Less community support;
RNN, and DBN [252]. – Difficult to install;

13
20 Cognitive Computation (2021) 13:1–33

– Draw lass interest among the research community. – Easy to create a layer and run on GPU.

MXNet Cons.

MXNet (https:​ //mxnet.​ io/) framework allows defining, train- – Requires writing training code;
ing, and deploying deep NN (DA, CNN, GAN, RNN and – Limited documentation.
LSTM) on a wide range of devices—from cloud infrastruc-
ture to mobile or even embedded devices (e.g. Raspberry Singa
Pi). Written in C++, it is memory efficient and supports
Go, JavaScript, Julia, MATLAB, Perl, Python, R, and Scala. Singa (https:​ //singa.​ incuba​ tor.apache​ .org/), it is a distributed
Pros. DL platform written in C++, Java, and Python.
Its flexible architecture allows synchronous, asynchro-
– A DL framework which has a high-performance impera- nous, and hybrid training frameworks to run. It supports a
tive API; wide range of DL architectures including CNN, RNN, RBM,
– Rich Language support; and DBM.
– MXNet features advanced GPU support; Pros.
– Highly scalable.
– Pretrained models are available;
Cons. – Supports model/data or hybrid partitioning, and synchro-
nous/asynchronous/hybrid training;
– Small community than TensorFlow; – Distributed deep learning system and handle Big data.
– Poor API documentation; – Widely used for healthcare data analytics.
– Less popular with the research community.
Cons.
Neon
– No Open Multi-Processing support.
Neon (www.nerva​nasys​.com/techn​ology​/neon/) is a DL
framework written in Python. It provides implementations TensorFlow
of various learning rules, along with functions for optimiza-
tion and activation. Its support for DL architecture includes TensorFlow (www.tenso​r flow​.org), written in C++ and
CNN, GAN, RNN, LSTM, and DA. Python, was developed by Google and supports very large-
Pros. scale deep NN. Amended recently as ‘TensorFlow Fold’, its
capability to dynamically create graphs made the architec-
– Better visualization properties than other frameworks; ture flexible, allowing deployment to a wide range of devices
– Apply optimization at data loading level, (e.g., multi-CPU/GPU desktop, server, mobile devices, etc.)
without code rewriting [253, 254]. Also it contains a data
Cons. visualization tool named TensorBoard and supports many
DL architectures including CNN, GAN, RNN, LSTM, and
– Small community than TensorFlow; RBMs [255].
– Less popular with the research community. Pros.

PyTorch – Handles large-scale data and operate in heterogeneous

environments;
PyTorch (http://pytor​ch.org/) provides Torch modules in – Faster compile time than Theano;
Python. More than a wrapper, its deep integration allows – Computational graph abstraction;
exploiting the powerful features of Python. Inspired by – Supports parallelism.
Chainer, it allows dynamic network creation for variable – TensorBoard is used for workflow and visualization.
workload and supports CNN, GAN, RNN and LSTM.
Pros. Cons.

– Pretrained models are available; – Large memory footprint;

– OpenCL support via separately maintained package. – Less number of pretrained models are available;
– Easily combine modular pieces; – Computational graph can be slow;

13
Cognitive Computation (2021) 13:1–33 21

– No support for matrix operations; Recently, Facebook’s DL modules (fbcunn) focusing on

– Difficulties in debugging. CNN have been open-sourced as a plug-in to Torch.
Pros.
TF.Learn
– User friendly;
TF.Learn (www.tflea​r n.org) is a TensorFlow (see sec- – Convenient for employ with GPUs;
tion 5.13)-based high-level Python API. It supports fast – Pretrained models are available;
prototyping with modular NN layers and multiple optimiz- – Highly modular;
ers, inputs, and outputs. Supported DL architectures include – Easy to create a layer and run on GPU.
CNN, GAN, BRNN, and LSTM.
Pros. Cons.

– Modular and transparent DL library built on the top of – Special data format and requires conversion;
TensorFlow; – Require to write training code;
– Provides a higher-level API to TensorFlow. – Less documentation available.

Cons. Veles

– Slower compared to its competitors. Veles (https:​ //github​ .com/Samsun​ g/veles)​ is a Python-based
distributed platform for rapid DL application development. It
Theano provides machine learning and data processing services and
supports IPython notebooks. Developed by Samsung, one of
Theano (www.deepl ​ e arni ​ n g.net/softw​ a re/thean ​ o /) is a its advantages is that it supports OpenCL for cross-platform
Python library that builds on core packages like NumPy and parallel programming, and allows execution across heter-
SymPy. It defines, optimizes, and evaluates mathematical ogenous platforms (e.g. servers, PC, mobile, and embedded
expressions with tensors and served as foundation for many devices). The supported DL architectures include DA, CNN,
DL libraries. RNN, LSTM, and RBM.
Pros. Pros.

– High flexibility; – Distributed platform support;

– High computational stability; – Supports Jupyter Notebook;
– Well suited for tensor-based mathematical expressions; – Supports OpenCL for cross-platform parallel program-
– Open-source libraries such as Keras, Lasagne and Blocks ming.
built on the top of Theano;
– Able to visualize convolutional filters, images, and Cons.
graphs;
– High-level wrappers like Keras and Lasagne increases – Less community support;
usability. – Draws lass interest from the research community.

Cons.
Relative Comparison of DL Tools
– Difficult to learn;
– Difficult to deploy; To perform relative comparison among the available open-
– Deployed on single GPU; source DL tools, we selected four metrics which are detailed
– Slower compilation time than TensorFlow. below: trend in their usage, community participation in their
development, interoperability among themselves, and their
Torch scalability (Fig. 4).

Started in 2000, Torch (http://torch​.ch/), a ML library and Trend

scientific computing framework, has evolved as a powerful
DL library. Core functions are implemented in C and the rest To assess the popularity and trend of the various DL
via LuaJIT scripting language made Torch superfast. Soft- tools among the DL consumers, we looked into two dif-
ware giants like Facebook and Google use Torch extensively. ferent sources to assess the utilization of the tools.

13
22 Cognitive Computation (2021) 13:1–33

a b MXNet Theano Others d

Caffe Keras Torch Theano Tensorflow Caffe
100%
80%
Mention (%)

60%
Keras
40% DL4J
Pytorch
DEEPLEARNING4J

20%
0%
TensorFlow
2015-01
2015-04
2015-07
2015-10
2016-01
2016-04
2016-07
2016-10
2017-01
2017-04
2017-07
2017-10
2018-01
2018-04
2018-07
2018-10
2019-01
2019-04
2019-07
2019-10
Time (year-month)

c e
Memory App.
160 Logic I/O

Increased computing capability

Github stars (in thousands)

140 Tensorfrlow ASIC

Config.
DLL PLL I/O
Keras Memory Macro
40 FPGA
Caffe pyTorch

20 MXNet MCT Control ALU ALU

DRAM
Torch Theano 4j H2O TF.Learn
DL4j ALU ALU
Singa Chainer Cache GPU
0 Veles Lasagne DyyNett Neon DRAM

2008 2014 2015 2016 CPU

Year of release Increased energy efficiency

Fig. 4  Relative comparison of DL tools. a Popularity trend of indi- contributors. d As for the interoperability among the DL tools, Keras
vidual DL tools as per mention in google search generated globally allows model importing from Caffe, MCT (CNTK), Theano, and Ten-
(data courtesy: Google Trend). b Mention in articles submitted to sorFlow and lets DL4j to import. e Regarding hardware-based scal-
arXiv preprint server during the first quarter of 2020. c The effect of ability of the DL tools, most of the tools provide CPU and GPU sup-
community’s participation on individual tools is shown by the bub- port, whereas FPGA and ASIC can mainly execute pretrained models
ble size, which is product of normalized number of GitHub forks and

Firstly, we extracted globally generated search data Community

from Google Trends 1 for five years (January 2015 to
December 2019) related to search terms consisting of The community-based development score for each tool dis-
⟨[toolname] + DeepLearning⟩. The data showed a progres- cussed in Section 5 was calculated from repository popu-
sive increase of search about TensorFlow since its release larity parameters of GitHub (https​://githu​b.com/) (i.e., star,
followed by Keras (Fig. 4a). Secondly, mining the content of fork, and contributors). The bubble plot shown in Fig. 4c
around 2,000 papers submitted to arXiv’s cs.[CV | CL | LG depicts community involvement in the development of the
| AI | NE], and stat.ML categories, during the first quarter tools indicating the year of initial stable release. Each bubble
of 2020 (i.e. January to March), for the presence of the tool size in the figure, pertaining to a tool, represents the normal-
names [256]. As seen in Fig. 4b which shows the percentage ized combined effect of fork and contributors of that tool. It
of each individual tool’s mention in the papers, the top six is clearly seen that a very large part of the community effort
tools were identified as: PyTorch, TensorFlow, Keras, Caffe, is concentrated on TensorFlow, followed by Keras and Caffe.
MXNet, and Theano.
Interoperability

In today’s cross-platform development environments, an

important measure to judge a tool’s flexibility is its interop-
erability with other tools. In this respect, Keras is the most
flexible one whose high-level neural networks are capable
1
https​://trend​s.googl​e.com of running on top of either Tensor or Theano. Alternatively,

13
Cognitive Computation (2021) 13:1–33 23

DL4j model imports neural network models originally con- Stacked autoencoders or DA were benchmarked using the
figured and trained using Keras that provides abstraction experimental setup number 1 in Table 8. To estimate the
layers on top of TensorFlow, Theano, Caffe, and CNTK performance of the various tools on implementing DA, three
backends (Fig. 4d). autoencoders (number of hidden layers: 400, 200, and 100,
respectively) were stacked with tied weights and sigmoid
Scalability activation functions. A two-step network training was per-
formed on the MNIST dataset [261]. As reported in Fig. 5
Hardware-based scalability is an important feature of the (a, b), the performances of various DL tools are evaluated
individual tools (Fig. 4e). Today’s hardware for comput- using forward runtime and training time. The forward runt-
ing devices are dominated by graphics processing units ime refers to the required time for evaluating the information
(GPUs) and central processing units (CPUs). But consid- flow through the full network to produce the intended out-
ering increased computing capacity and energy efficiency, put for an input batch, dataset, and network. In contrast, the
the coming years are expected to witness expanded role gradient computation time measures the time that required
for other chipset types including application-specific inte- to train DL tools. The results suggest that, regardless of the
grated circuits (ASICs) and field-programmable gate arrays number of CPU threads used or GPU, Theano and Torch
(FPGAs). So far DL has been predominantly used through outperform TensorFlow in both gradient and forward times
software. The requirement for hardware acceleration, energy (Fig. 5 a, b).
efficiency, and higher performance has driven the develop- Experimental setup number 2 (Table 8) was used in
ment of chipset-based DL systems. benchmarking RNN. The adapted LSTM network [262]
was designed with 10000 input and output units with two
layers and ∼13 millions parameters. As the performance of
Performance of Tools and Benchmark RNN depends on the input length, an input length of 32 was
used for the experiment. As the results indicate (Fig. 5 c-f),
The power of DL methods lies in their capability to rec- MCT outperforms other tools on both CPU and all three
ognize patterns for which they are trained. Despite the GPU platforms. On CPUs, TensorFlow performs little better
availability of several accelerating hardware (e.g., multi- than Torch (Fig. 5 c). On GPUs, Torch is the slowest with
core [C/G]PUs/FPGAs), this training phase is very time- TensorFlow and MXNet performing similarly (Fig. 5 d-f).
consuming, cumbersome, and computationally challeng-
ing. Moreover, as each tool provides implementations of
several DL architectures and often emphasizing separate
components of them on different hardware platforms, Table 8  Hardware configuration of the evaluating setup
selecting an appropriate tool suitable for an application is ESN Processor Memory
getting increasingly difficult. Besides, different DL tools a
1 CPU: E5-1650 @ 3.50 GHz 32 GB
have different targets, e.g., Caffe targets applications,
GPU: Nvidia GeForce GTX Titan ­Xb
whereas Torch and Theano are more for DL research.
2 CPU: E5-2630c @ 2.20 GHz 128 GB
To facilitate scientists in picking the right tool for their
GPU: Nvidia GeForce GTX ­980d
application, scientists benchmarked the performances of
GPU: Nvidia GeForce GTX ­1080e
the popular tools concerning their training times [257,
GPU: Tesla K80 accelerator with GK210 ­GPUsf
258]. Moreover, to the best of our knowledge, there exist
3 CPU: E5-2690c @ 2.60 GHz 256 GB
two main efforts that provide the benchmarking details
GPU: Tesla P100 ­acceleratorg
of the various DL tools and frameworks publicly [259,
GPU: Tesla M40 ­acceleratorh
260]. Summarizing those seminal works, below we pro-
GPU: Tesla K80 accelerator with GK210 ­GPUsf
vide the time required to complete the training process as
a performance measure of four different DL architectures ESN Experimental Setup Numbers
(e.g., FCN, CNN, RNN, and DA) among the popular a
Intel Xeon CPU v2
tools (e.g., Caffe, CNTK, MXNET, Theano, TensorFlow, b
3072 cores, 1000 MHz base clock, 12 GB memory
and Torch) on multicore [C/G]PU platforms. c
Intel Xeon CPU v4
Table 8 lists the experimental setups used in bench- d
2048 cores, 1126 MHz base clock, 4 GB memory
marking the specified tools. Mainly three different set- e
2560 cores, 1607 MHz base clock, 8 GB memory
ups, each with Intel Xeon E5 CPU, were utilized during f
Tesla K80 accelerator has two Tesla GK210 GPUs with 2496 cores,
the process. Though the CPU was similar, the GPU hard- 560 MHz base clock, 12 GB memory
ware was different: GeForce GTX Titan X, GTX 980, g
3584 cores, 1189 MHz base clock, 16 GB memory
GTX 1080, Tesla K80, M40, and P100. h
3072 cores, 948 MHz base clock, 12 GB memory

13
24 Cognitive Computation (2021) 13:1–33

a Stacked Autoencoder (CPU) b Stacked Autoencoder (GPU-GTX Titan X)

256 Gradient Forward 64 256 64
Gradient Forward
128 1 6 12 1 6 12 128 32
32
64 64 16

Time / Batch [ms]

Time / Batch [ms]

Time / Batch [ms]

Time / Batch [ms]

16 8
32 32
4
16 8 16
2
8 8
4 1
4 4 0.5
2
2 2 0.25
1 1 1 0.125

c LSTM (CPU) d LSTM (GPU-GTX980)

16384 1024 Tensorflow
1 2 4 8 16 32 64 128 256 512 1024
4096
256
1024 Theano
Time / Batch [ms]

Time / Batch [ms]

256 64
Torch
64 16
16
4 MCT
4
1 1 MXNet
e LSTM (GPU-GTX1080) f LSTM (GPU-Tesla K80)
4096 16384
64 128 256 512 1024 64 128 256 512 1024
1024 4096
Time / Batch [ms]

Time / Batch [ms]

1024
256
256
64
64
16
16
4 4
1 1

Fig. 5  Benchmarking stacked autoencoder or DA (a, b) and LSTM (c-f) in CPU and GPU platforms. The numbers in (a, c) denote the number of
CPU threads employed in the benchmarking process, and in (d-f) denote the batch size. In case of DA the batch size was 64

Still a large portion of the pattern analysis is done using Theano, and Torch) [264] clearly suggest that network
CNN; therefore, we further focused on CNN and investi- training process is much accelerated in GPUs in com-
gated how the leading tools performed and scaled in training parison to CPUs. Moreover, another important message
different CNN networks in different GPU platforms. Time is that, all GPUs are not the same and all tools don’t
speedup of GPU over CPU is considered as a metric for scale up at the same rate. The time required to train a
this purpose. The individual values are calculated using the neural network strongly depends on which DL frame-
benchmark scripts of DeepMark [259] on experimental setup work is being used. As for the hardware platform, the
number 3 (Table 8) for one training iteration per batch. The Tesla P100 accelerator provides the best speedup with
time needed to execute a training iteration per batch equals Tesla M40 being the second and Tesla K80 being the last
the time taken to complete a forward propagation operation among the three. In CPUs, TensorFlow achieves the least
followed by a backpropagation operation. Figure 6 summa- training time indicating a quicker training of the network.
rizes the training time per iteration per batch for both CPU In GPUs, Caffe usually provides the best speedup over
and GPUs (left y-axis) and the corresponding GPU speedup CPU but TensorFlow and Torch perform faster training
over CPU (right y-axis). than Caffe. Though TensorFlow and Torch have simi-
These findings for four different CNN network models lar performances (indicated by the height of the lines),
(i.e. Alexnet [92], GoogLeNet [94], Overfeat [263], and Torch slightly outperforming TensorFlow in most of the
VGG [93]) available in four tools (i.e. Caffe, TensorFlow, networks. Finally, most of the tools outperform Theano.

13
Cognitive Computation (2021) 13:1–33 25

Open Issues and Future Perspectives identify below shortcomings and bottlenecks of the popu-
lar methods, open research questions, and challenges and
The brain has the capability to recognize and understand pat- outline possible directions which requires attention in the
terns almost instantaneously. Over several decades, scientists near future.
have been trying decode the biological mechanism of natural First of all, DL methods usually require large datasets.
pattern recognition that takes place in the brain and translate Though the computing cost is declining with increas-
those principles into AI systems. The increasing knowledge ing computational power and speed, it is not worthwhile
about the brain’s information processing policies enabled to apply DL methods in cases of small to moderate sized
this analogy to be adopted and implemented in computing datasets. This is particularly so as considering that many of
systems. Recent technological breakthroughs, seamless the DL methods perform continuous geometric transforma-
integration of diverse techniques, better understanding of tions of one data manifold to another with an assumption
the learning systems, declination of computing costs, and that there exist learnable transfer functions which can per-
expansion of computational power empowered computing form the mapping [266]. However, in cases when the rela-
systems to reach human-level computation in certain sce- tionships among the data are causal or very complex to be
narios [265]. Nonetheless, many of these methods require learned by the geometric transformations, the DL methods
improvements. Though admittedly, there are distinctions on fail regardless the size of the dataset [267]. Also, interpret-
how a DL-based method can be used and applied on biologi- ing high-level outcomes of DL methods is difficult due to
cal data, however, the common open issues and challenges inadequate in-depth understanding of the DL theories which
are equally applicable and important for biological data. We causes many of such models to be considered as ‘Black box’

a Alexnet b GoogLeNet
8192 CPU GPU Speedup 100 32768 CPU GPU Speedup 100
Training Time / Iteration / Batch [ms]

Training Time / Iteration / Batch [ms]

Times Speedup Over CPU

Times Speedup Over CPU

4096 16384
2048 75 8192 75
1024 4096
512 50 2048 50
256 Tesla P100 1024 Tesla P100
128 Tesla K80 25 512 Tesla K80 25
Tesla M40 Tesla M40
64 256
32 0 128 0
o r ffe

o r ffe

o r ffe
h

h
To n o

To n o

To n o
ea w

ea w

ea w

o r ffe

o r ffe

o r ffe
h

h
To n o

To n o

To n o
ea w

ea w

ea w
rc

rc

rc

rc

rc

rc
Th flo

Th flo

Th flo

Th flo

Th flo

Th flo
ns Ca

ns a

ns a

ns Ca

ns Ca

ns Ca
C

C
Te

Te

Te

Te

Te

Te

c Overfeat d VGG
16384 CPU GPU Speedup 100 65536 CPU GPU Speedup 100
Training Time / Iteration / Batch [ms]

Training Time / Iteration / Batch [ms]

8192 32768
Times Speedup Over CPU

Times Speedup Over CPU

4096 75 16384 75
8192
2048
4096
1024 50 50
2048
512
Tesla P100 1024 Tesla P100
256 Tesla K80 25 512 Tesla K80 25
Tesla M40 Tesla M40
128 256
64 0 128 0
o r ffe

o r ffe

o r ffe

o r ffe

o r ffe

o r ffe
h

h
To n o

To n o

To n o

To n o

To n o

To n o
ea w

ea w

ea w

ea w

ea w

ea w
rc

rc

rc

rc

rc

rc
Th flo

Th flo

Th flo

Th flo

Th flo

Th flo
ns Ca

ns a

ns a

ns Ca

ns Ca

ns Ca
C

C
Te

Te

Te

Te

Te

Te

Fig. 6  The speedup of CNN training in different DL tools across various GPUs in comparison to CPU. The reported values were calculated for a
batch size of 128, except for VGG for which the batch size was 64

13
26 Cognitive Computation (2021) 13:1–33

[268]. Moreover, like many other ML techniques, DL is also DL tools (e.g., Keras, Chainer, Lasagne) and architectures
susceptible to misclassification [269] and overclassification (e.g., DBN) which need to be benchmarked providing the
[270]. users with a more comprehensive list to choose. Also, the
Additionally, the ability to exploit the full benefits offered currently available benchmarks are mostly performed on
by open access data repositories, in terms of data sharing non-biological data, and their scalability to biological data
and reuse, is often hampered by the lack of unified reporting is poor; thus, specialized benchmarking on biological data
data standards and non-uniformity of reported information are needed.
[271]. Data provenance, curation, and annotation of these In order to derive insights from an image, a sequence or
biological big data are huge challenges too [272]. a signal analysis problem, a selected DL algorithm using a
Furthermore, except for very few large enterprises, the library or a tool (e.g., TensorFlow, Keras, PyTorch, etc.) may
power of distributed and parallel computation through need to integrate with a big data framework (e.g., Hadoop,
cloud computing remains largely unexplored for the DL Spark, etc.). In such cases, troubleshooting in the model and
techniques. Due to the fact that the DL techniques require debugging the code may be very challenging for the system
retraining for different datasets, repeated training becomes designer due to the parallel execution of multiple threads
a bottleneck for cloud computing environments. Also, in which may not always execute in an orderly fashion. The
such distributed environments, data privacy and security lack of documentation and model transparency of these
concerns are still prevailing [273], and real-time process- libraries may make it impossible for the project manager
ing capability of experimental data is underdeveloped to estimate efforts required in successful completion of a
[274]. project.
To mitigate the shortcomings and address the open issues,
the existing theoretical foundations of the DL methods need
to be improved. The DL models are required not only to be Conclusion
able to describe specific data but also generalize them on
the basis of experimental data which is crucial to quantify The diverse biological data coming from different applica-
the performances of individual NN models [275]. These tion domains are multimodal, multidimensional, and com-
improvements should take place in several directions and plex in nature. At present, a huge amount of such data is
address issues like quantitative assessment of individual publicly available. The affordable access to these data came
model’s learning efficiency and associated computational with a huge challenge to analyse and recognize patterns in
complexity in relation to well-defined parameter tuning them which require sophisticated ML tools to do the job. As
strategies, the ability to generalize and topologically self- a result, many ML-based analytical tools have been devel-
organize based on data-driven properties. Also, to facilitate oped and reported over the last decades and this process has
intuitive and less cumbersome interpretation of the analysis been facilitated greatly by the decrease of computational
results, novel tools for data visualization should be incorpo- costs, increase of computing power, and availability of
rated in the DL frameworks. cheap storage. With the help of these learning techniques,
Recent developments in combined methods pertaining machines have been trained to understand and decipher
to deep reinforcement learning (deep RL) have been popu- complex patterns and interactions of variables in biological
larly applied to many application domains (for a review on data. To facilitate a wider dissemination of DL techniques
deep RL, see [276]). However, deep RL methods have not applied to biological data and serve as a reference point,
yet been applied to biological pattern recognition problems. this article provides a comprehensive survey of the literature
For example, analysing and aggregating dynamically chang- on those techniques’ application on biological data and the
ing patterns in biological data coming from multiple levels relevant open-access data repositories. It also lists existing
could help to remove data redundancy and discover novel open-source tools and frameworks implementing various
biomarkers for disease detection and prevention. Also, novel DL methods and compares these tools for their popularity
deep RL methods are needed to reduce the currently required and performance. Finally, it concludes by pointing out some
large set of labelled training data. open issues and proposing some future perspectives.
Renewing efforts are required for standardization, anno-
tation, curation, and provenance of data and their sources Acknowledgements The authors would like to thank the members of
the acslab (http://www.acsla​b.info/) for valuable discussions.
along with ensuring uniformity of information among the
different repositories. Additionally, to keep up with the rap- Author Contributions This work was carried out in close collaboration
idly growing big data, powerful and secure computational among all authors. M.M. and M.S.K. conceived the idea, developed
infrastructures in terms of distributed, cloud, and parallel the method and experiments, analysed the obtained data, and wrote the
computing tailored to such well-understood learning mecha- manuscript. T.M.M and A.H. edited the manuscript. All authors have
contributed to, seen, and approved the paper.
nisms are badly needed. Lastly, there are many other popular

13
Cognitive Computation (2021) 13:1–33 27

Compliance with Ethical Standards 16. Hopfield JJ. Neurons with graded response have collective
computational properties like those of two-state neurons.
PNAS. 1984;81(10):3088–92.
Conflict of Interest The authors declare that the research was con-
17. Ackley DH, Hinton GE, Sejnowski TJ. A learning algorithm
ducted in the absence of any commercial or financial relationships that
for Boltzmann machines. Cogn Sci. 1985;9(1):147–69.
could be construed as a potential conflict of interest.
18. Salakhutdinov R, Mnih A, Hinton G. Restricted Boltzmann
machines for collaborative filtering. In: Proc. ICML; 2007. p.
Ethical Approval This article does not contain any studies with human
791–798.
participants or animals.
19. Maass W. Networks of spiking neurons: The third generation
of neural network models. Neural Netw. 1997;10(9):1659–71.
Informed Consent As this article does not contain any studies with
20. Heckerman D. A Tutorial on learning with Bayesian net-
human participants or animals, the informed consent is not applicable
works. In: Jordan MI, editor. Learning in Graphical Models.
89. Springer Netherlands; 1998. p. 301–354.
Open Access This article is licensed under a Creative Commons Attri- 21. Cortes C, Vapnik V. Support-vector networks. Mach Learn.
bution 4.0 International License, which permits use, sharing, adapta- 1995;20(3):273–97.
tion, distribution and reproduction in any medium or format, as long 22. Yuan GX, Ho CH, Lin CJ. Recent advances of large-scale lin-
as you give appropriate credit to the original author(s) and the source, ear classification. Proc IEEE. 2012;100(9):2584–603.
provide a link to the Creative Commons licence, and indicate if changes 23. Fisher RA. The use of multiple measurements in taxonomic
were made. The images or other third party material in this article are problems. Ann Eugenics. 1936;7(2):179–88.
included in the article’s Creative Commons licence, unless indicated 24. Uysal I, Güvenir HA. An overview of regression techniques for
otherwise in a credit line to the material. If material is not included in knowledge discovery. Knowl Eng Rev. 1999;14(4):319–40.
the article’s Creative Commons licence and your intended use is not 25. Rish I. An empirical study of the naive Bayes classifier. In:
permitted by statutory regulation or exceeds the permitted use, you will Proc. 2001 IJCAI. vol. 3; 2001. p. 41–46.
need to obtain permission directly from the copyright holder. To view a 26. Cover T, Hart P. Nearest neighbor pattern classification. IEEE
copy of this licence, visit http://creat​iveco​mmons​.org/licen​ses/by/4.0/. Trans Inf Theory. 1967;13(1):21–7.
27. Rabiner L, Juang B. An introduction to hidden Markov models.
IEEE ASSP Mag. 1986;3(1):4–16.
28. Kohavi R, Quinlan JR. Data mining tasks and methods: clas-
sification: decision-tree discovery. In: Klösgen W, Zytkow JM,
References editors. Handbook of data mining and knowledge discovery.
New York, NY, USA: Oxford University Press, Inc.; 2002. p.
1. Coleman W. Biology in the nineteenth century : problems of 267–276.
form, function, and transformation. Cambridge; New York: Cam- 29. Hinton GE. Connectionist learning procedures. Artif Intell.
bridge University Press; 1977. 1989;40(1–3):185–234.
2. Magner LN. A history of the life sciences. 3rd ed. New York: M. 30. Dempster AP, Laird NM, Rubin DB. Maximum likelihood from
Dekker; 2002. incomplete data via the EM algorithm. J R Stat Soc Series B
3. Brenner S. History of science. The revolution in the life sciences. Methodol. 1977;39(1):1–38.
Science. 2012;338(6113):1427–8. 31. Tishby N, Pereira FC, Bialek W. The information bottleneck
4. Shendure J, Ji H. Next-generation DNA sequencing. Nat Biotech- method. In: Proc. 37th ACCCC. 1999. p. 368-377.
nol. 2008;26(10):1135–45. 32. Kohonen T. Self-organized formation of topologically correct
5. Metzker ML. Sequencing technologies the next generation. Nat feature maps. Biol Cybernet. 1982;43(1):59–69.
Rev Genet. 2010;11(1):31–46. 33. Agrawal R, Imieliński T, Swami A. Mining association rules
6. Vadivambal R, Jayas DS. Bio-imaging : principles, techniques, between sets of items in large databases. In: Proc. ACM SIG-
and applications. Boca Raton, FL: CRC Press, Taylor & Fran- MOD ’93. 1993. p. 207–216.
cis Group; 2016. 34. Gordon AD. A review of hierarchical classification. J R Stat Soc
7. Poldrack RA, Farah MJ. Progress and challenges in probing Series A General. 1987;150(2):119–37.
the human brain. Nature. 2015;526(7573):371–9. 35. Ball G, Hall D. ISODATA, a novel method of data anlysis and
8. Lebedev MA, Nicolelis MAL. Brain-machine interfaces: from pattern classification. Stanford, CA: Stanford Research Institute;
basic science to neuroprostheses and neurorehabilitation. Phys 1965.
Rev. 2017;97(2):767–837. 36. Dunn JC. A fuzzy relative of the ISODATA process and its
9. Quackenbush J. Extracting biology from high-dimensional use in detecting compact well-separated clusters. J Cybernet.
biological data. J Exp Biol. 2007;210:1507–17. 1973;3(3):32–57.
10. Mattmann CA. Computing: A vision for data science. Nature. 37. John AH. Clustering algorithms. New York, NY, USA: John
2013;493(7433):473–5. Wiley & Sons, Inc.; 1975.
11. Li Y, Chen L. Big biological data: challenges and opportuni- 38. Kriegel HP, Kroger P, Sander J, Zimek A. Density-based cluster-
ties. Genomics Proteomics Bioinformatics. 2014;12(5):187–9. ing. WIRES Data Min Knowl. 2011;1(3):231–40.
12. Marx V. Biology: The big challenges of big data. Nature. 39. Ankerst M, Breunig MM, Kriegel HP, Sander J. OPTICS: order-
2013;498(7453):255–60. ing points to identify the clustering structure. In: Proc. ACM
13. Tarca AL, Carey VJ, Chen Xw, Romero R, Draghici S. Machine SIGMOD’99. 1999. p. 49–60.
learning and its applications to biology. PLoS Comput Biol. 40. Horgan RP, Kenny LC. Omic technologies: genomics, tran-
2007;3(6):e116. scriptomics, proteomics and metabolomics. Obstet Gynecol.
14. Hopfield JJ. Artificial neural networks. IEEE Circuits Devices 2011;13(3):189–95.
Mag. 1988;4(5):3–10. 41. Libbrecht MW, Noble WS. Machine learning applications in
15. Hecht-Nielsen R. Theory of the backpropagation neural net- genetics and genomics. Nat Rev Genet. 2015;16(6):321–32.
work. In: Proc. IJCNN 1989; 1989. p. 593–605.

13
28 Cognitive Computation (2021) 13:1–33

42. Lemm S, Blankertz B, Dickhaus T, Müller KR. Introduc- 63. Goodfellow I, Bengio Y, Courville A. Deep learning. Cambridge,
tion to machine learning for brain imaging. NeuroImage. USA: MIT Press; 2016.
2011;56(2):387–99. 64. Saxe AM, McClelland JL, Ganguli S. Exact solutions to the non-
43. Erickson BJ, Korfiatis P, Akkus Z, Kline TL. Machine learning linear dynamics of learning in deep linear neural nets. In: Proc.
for medical imaging. RadioGraphics. 2017;37(2):505–15. ICLR; 2014. p. 1–22.
44. Kan A. Machine learning applications in cell image analysis. 65. Schmidhuber J. Deep learning in neural networks: an overview.
Immunol Cell Biol. 2017;95(6):525–30. Neural Netw. 2015;61:85–117.
45. Vidaurre C, Sannelli C, Müller KR, Blankertz B. Machine-learn- 66. Zeng D, Zhao F, Shen W, Ge S. Compressing and accelerat-
ing-based coadaptive calibration for brain-computer interfaces. ing neural network for facial point localization. Cogn Comput.
Neural Computat. 2010;23(3):791–816. 2018;10(2):359–67.
46. Mala S, Latha K. Feature selection in classification of eye move- 67. Salakhutdinov R, Hinton GE. Deep Boltzmann machines. In:
ments using electrooculography for activity recognition. Com Proc. AISTATS2009; 2009. p. 448–455.
Math Met Med. 2014 Dec;2014. 68. Geman S, Geman D. Stochastic relaxation, Gibbs distributions,
47. Mahmud M. Vassanelli S. Processing and analysis of multichan- and the Bayesian restoration of images. IEEE Trans Pattern Anal
nel extracellular neuronal signals: state-of-the-art and challenges. Mach Intell. 1984;6(6):721–41.
Front Neurosci. 2016. p. 10. 69. Fischer A, Igel C. An introduction to restricted Boltzmann
48. Lecun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015 machines. In: Proc. CIARP 2012; 2012. p. 14–36.
5;521(7553):436–444. 70. Desjardins G, Courville AC, Bengio Y. On training deep Boltz-
49. Yahaya SW, Lotfi A, Mahmud M. A consensus novelty detection mann machines. CoRR. 2012;abs/1203.4416.
ensemble approach for anomaly detection in activities of daily 71. Tieleman T. Training restricted Boltzmann machines using
living. Applied Soft Computing. 2019;83:105613. approximations to the likelihood gradient. In: Proc. ICML; 2008.
50. Fabietti M, Mahmud M, Lotfi A, Averna A, Guggenmo D, Nudo p. 1064–1071.
R, et al. Neural network-based artifact detection in local field 72. Guo Y, Liu Y, Oerlemans A, Lao S, Wu S, Lew MS. Deep
potentials recorded from chronically implanted neural probes. learning for visual understanding: A review. Neurocomputing.
In: Proc. IJCNN; 2020. p. 1–8. 2016;187:27–48.
51. Mahmud M, Kaiser MS, Hussain A, Vassanelli S. Applications 73. Hinton GE, Osindero S, Teh YW. A fast learning algorithm for
of deep learning and reinforcement learning to biological data. deep belief nets. Neural Comput. 2006;18(7):1527–54.
IEEE Trans Neural Netw Learn Syst. 2018;29(6):2063–79. 74. Bi X, Zhao X, Huang H, Chen D, Ma Y. Functional brain net-
52. Mahmud M, Kaiser MS, Hussain A. Deep learning in mining work classification for Alzheimer’s disease detection with
biological data. arXiv​:20030​0108 [cs, q-bio, stat]. 2020 Feb;p. deep features and extreme learning machine. Cogn Comput.
1–36. ArXiv​: 2003.00108​. Available from: http://arxiv​.org/ 2020;12:513–527.
abs/2003.00108​. 75. Ravi D, Wong C, Deligianni F, Berthelot M, et al. Deep learn-
53. Dey N, Rajinikanth V, Fong SJ, Kaiser MS, Mahmud M. ing for health informatics. IEEE J Biomed Health Inform.
Social-group-optimization assisted Kapur’s entropy and mor- 2017;21(1):4–21.
phological segmentation for automated detection of COVID-19 76. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA.
infection from computed tomography images. Cogn Comput. Stacked denoising autoencoders: learning useful representations
2020;12(5):1011–1023. in a deep network with a local denoising criterion. J Mach Learn
54. Aradhya MVN, Mahmud M, Guru D, S Agrawal B, Kaiser MS. Res. 2010;11:3371–408.
One shot cluster based approach for the detection of COVID-19 77. Baldi P. Autoencoders, unsupervised learning and deep architec-
from chest X-ray images. Cogn Comput. 2020;p. 1–8. tures. In: Proc. ICUTLW; 2012. p. 37–50.
55. Noor MBT, Zenia NZ, Kaiser MS, Mahmud M, Al Mamun S. 78. Ranzato M, Poultney C, Chopra S, Cun YL. Efficient learning
Detecting neurodegenerative disease from MRI: a brief review on of sparse representations with an energy-based model. In: Proc.
a deep learning perspective. In: Liang P, Goel V, Shan C, editors. NIPS; 2006. p. 1137–1144.
Brain Inform. Cham: Springer International Publishing; 2019. p. 79. Kingma DP, Welling M. Auto-encoding variational bayes.
115–125. CoRR. 2014;abs/1312.6114.
56. Ali HM, Kaiser MS, Mahmud M. Application of convolutional 80. Rifai S, Vincent P, Muller X, Glorot X, Bengio Y. Contractive
neural network in segmenting brain regions from MRI data. In: auto-encoders: explicit invariance during feature extraction.
Liang P, Goel V, Shan C, editors. Brain Inform. Cham: Springer In: Proc. ICML; 2011. p. 833–840.
International Publishing; 2019. p. 136–146. 81. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley
57. Miah Y, Prima CNE, Seema SJ, Mahmud M, Kaiser MS. Perfor- D, Ozair S, et al. Generative adversarial nets. In: Advances in
mance comparison of machine learning techniques in identifying neural information processing systems; 2014. p. 2672–2680.
dementia from open access clinical datasets. In: Proc. ICACIn. 82. Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation
Springer, Singapore; 2020. p. 79–89. with conditional adversarial networks. In: Proc.IEEE CVPR;
58. Watkins J, Fabietti M, Mahmud M. SENSE: a student perfor- 2017. p. 1125–1134.
mance quantifier using sentiment analysis. In: Proc. IJCNN; 83. Wang Z, Healy G, Smeaton AF, Ward TE. Use of neural
2020. p. 1–6. signals to evaluate the quality of generative adversarial net-
59. Rabby G, Azad S, Mahmud M, Zamli KZ, Rahman MM. TeKET: work performance in facial image generation. Cogn Comput.
a tree-based unsupervised keyphrase extraction technique. Cogn 2020;12(1):13–24.
Comput. 2020;12(5):811–33. 84. Pascanu R, Gulcehre C, Cho K, Bengio Y. How to construct
60. Orojo O, Tepper J, McGinnity TM, Mahmud M. A multi-recur- deep recurrent neural networks. In: Proc. ICLR; 2014. p. 1–13.
rent network for crude oil price prediction. In: Proc. SSCI; 2019. 85. Elman JL. Finding structure in time. Cognitive Sci.
p. 2940–2945. 1990;14(2):179–21111.
61. Ching T, et al . Opportunities and obstacles for deep learning in 86. Schuster M, Paliwal KK. Bidirectional recurrent neural net-
biology and medicine. bioRxiv. 2017;p. 142760. works. IEEE Tran Signal Proces. 1997;45(11):2673–81.
62. Bengio Y. Learning deep architectures for AI. Found Trends 87. Hochreiter S, Schmidhuber J. Long short-term memory. Neural
Mach Learn. 2009;2(1):1–127. Comput. 1997;9(8):1735–80.

13
Cognitive Computation (2021) 13:1–33 29

88. Lipton ZC, Berkowitz J, Elkan C. A critical review of recur- 109. Kamnitsas K, Ledig C, Newcombe VFJ, Simpson J, et al. Effi-
rent neural networks for sequence learning. CoRR. 2015 cient multi-scale 3D CNN with fully connected CRF for accurate
May;CoRR: 1506.00019. brain lesion segmentation. Med Image Anal. 2017;36:61–78.
89. Ma Y, Peng H, Khan T, Cambria E, Hussain A. Sentic LSTM: 110. Lu N, Li T, Ren X, Miao H. A deep learning scheme for motor
a hybrid network for targeted aspect-based sentiment analysis. imagery classification based on restricted Boltzmann machines.
Cogn Comput. 2018;10(4):639–50. IEEE Trans Neural Syst Rehabil Eng. 2016; 99
90. Wiatowski T, Bölcskei H. A mathematical theory of deep con- 111. Yang H, Sakhavi S, Ang KK, Guan C. On the use of convolu-
volutional neural networks for feature extraction. IEEE Trans tional neural networks and augmented CSP features for multi-
Inf Theory. 2017;64(3):1845–66. class motor imagery of EEG signals classification. In: Proc. 37th
91. LeCun Y, Bengio Y. Convolutional networks for images, IEEE EMBC; 2015. p. 2620–2623.
speech, and time series. In: Arbib MA, editor. The handbook 112. Tabar YR, Halici U. A novel deep learning approach for
of brain theory and neural networks. Cambridge, MA, USA: classification of EEG motor imagery signals. J Neural Eng.
MIT Press; 1998. p. 255–258. 2017;14(1):016003.
92. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification 113. Sakhavi S, Guan C, Yan S. Parallel convolutional-linear neural
with deep convolutional neural networks. In: Proc. NIPS; 2012. network for motor imagery classification. In: Proc. EUSIPCO;
p. 1097–1105. 2015. p. 2786–2790.
93. Simonyan K, Zisserman A. Very deep convolutional networks 114. Li K, Li X, Zhang Y, Zhang A. Affective state recognition
for large-scale image recognition. CoRR. 2014;abs/1409.1556. from EEG with deep belief networks. In: Proc. BIBM; 2013. p.
94. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, 305–310.
et al. Going deeper with convolutions. In: Proc. CVPR2015; 115. Jia X, Li K, Li X, Zhang A. A novel semi-supervised deep learn-
2015. p. 1–9. ing framework for affective state recognition on EEG signals. In:
95. Heinsfeld AS, Franco AR, Craddock RC, Buchweitz A, Proc. IEEE BIBE; 2014. p. 30–37.
Meneguzzi F. Identification of autism spectrum disorder using 116. Tripathi S, Acharya S, Sharma R, Mittal S, et al. Using deep and
deep learning and the ABIDE dataset. NeuroImage: Clin. convolutional neural networks for accurate emotion classification
2018;17:16 – 23. on DEAP dataset. In: Proc. 29th IAAI; 2017. p. 4746–4752.
96. Kuang D, He L. Classification on ADHD with deep learning. 117. Chen G, Zhu Y, Hong Z, Yang Z. EmotionalGAN: generating
In: Proc. CCBD; 2014. p. 27–32. ECG to enhance emotion state classification. In: Proc. AICS
97. HosseiniAsl E, Gimelfarb GL, El-Baz A. Alzheimer’s disease 2019. 2019. p. 309-313.
diagnostics by a deeply supervised adaptable3D convolutional 118. Mirowski P, Madhavan D, LeCun Y, Kuzniecky R. Classification
network. CoRR. 2016;abs/1607.00556. of patterns of EEG synchronization for seizure prediction. Clin
98. Suk HI, Lee SW, Shen D. Hierarchical feature representation Neurophysiol. 2009;120(11):1927–40.
and multimodal fusion with deep learning for AD/MCI diag- 119. Jirayucharoensak S, Pan-Ngum S, Israsena P. EEG-based emo-
nosis. NeuroImage. 2014;101:569–82. tion recognition using deep learning network with principal
99. Li F, Tran L, Thung KH, Ji S, Shen D, Li J. A robust deep component based covariate shift adaptation. Scientific World J.
model for improved classification of AD/MCI patients. IEEE 2014;p. 1–10.
J Biomed Health Inform. 2015;19(5):1610–6. 120. Wu Z, Ding X, Zhang G. A novel method for classification of
100. Havaei M, Guizard N, Larochelle H, Jodoin PM. Deep learn- ECG arrhythmias using deep belief networks [journal article]. J
ing trends for focal brain pathology segmentation in MRI. In: Comp Intel Appl. 2016;15:1650021.
Holzinger A, editor. Machine learning for health informatics: 121. Yan Y, Qin X, Wu Y, Zhang N, Fan J, et al. A restricted Boltz-
state-of-the-art and future challenges. Cham: Springer; 2016. mann machine based two-lead electrocardiography classification.
p. 125–148. In: Proc. BSN; 2015. p. 1–9.
101. Fritscher K, Raudaschl P, Zaffino P, Spadea MF, Sharp GC, 122. Atzori M, Cognolato M, Müller H. Deep learning with convo-
et al. Deep neural networks for fast segmentation of 3D medical lutional neural networks applied to electromyography data: a
images. In: Proc. MICCAI; 2016. p. 158–165. resource for the classification of movements for prosthetic hands.
102. Iqbal T, Ali H. Generative adversarial network for medical Front Neurorobot. 2016;10:9.
images (MI-GAN). J Med Syst. 2018;42(11):231. 123. Huanhuan M, Yue Z. Classification of electrocardiogram signals
103. Ciresan D, Giusti A, Gambardella L, Schmidhuber J. Deep neural with DBN. In: Proc. IEEE CSE; 2014. p. 7–12.
nets segment neuronal membrane in electron microscopy images. 124. Wang S, Peng J, Ma J, Xu J. Protein secondary structure predic-
In: Proc. NIPS; 2012. p. 2843–2851. tion using deep convolutional neural fields. Scientific Reports.
104. Stollenga MF, Byeon W, Liwicki M, Schmidhuber J. Parallel 2016 Nov;6(1).
multi-dimensional LSTM, with application to fast biomedi- 125. Alipanahi B, Delong A, Weirauch MT, Frey BJ. Predicting the
cal volumetric image segmentation. In: Proc. NIPS; 2015. p. sequence specificities of DNA- and RNA-binding proteins by
2980–88. deep learning. Nature Biotechnol. 2015;33(8):831–8.
105. Kleesiek J, Urban G, Hubert A, Schwarz D, Maier-Hein K, Bend- 126. Chen G, Tsoi A, Xu H, Zheng WJ. Predict effective drug com-
szus M, et al. Deep MRI brain extraction: A 3D convolutional bination by deep belief network and ontology fingerprints. J
neural network for skull stripping. NeuroImage. 2016;129:460–9. Biomed Inform. 2018;85:149–54.
106. Cho J, Lee K, Shin E, Choy G, Do S. Medical image deep learn- 127. Denas O, Taylor J. Deep modeling of gene expression regulation
ing with hospital PACS dataset. CoRR. 2015;abs/1511.06348. in an Erythropoiesis model. In: Proc. ICMLRL; 2013. p. 1–5.
107. Ngo T, et al. Combining deep learning and level set for the auto- 128. Kelley DR, Snoek J, Rinn JL. Basset: learning the regulatory
mated segmentation of the left ventricle of the heart from cardiac code of the accessible genome with deep convolutional neural
cine mr. Med Image Anal. 2017;35:159–71. networks. Genome Res. 2016;26(7):990–9.
108. Ciresan D, Giusti A, Gambardella L, Schmidhuber J. Mitosis 129. Zhou J, Troyanskaya OG. Predicting effects of noncoding vari-
detection in breast cancer histology images with deep neural ants with deep learning-based sequence model. Nature Methods.
networks. In: Proc. MICCAI; 2013. p. 411–4188. 2015;12(10):931–4.

13
30 Cognitive Computation (2021) 13:1–33

130. Marouf M, et al. Realistic in silico generation and augmenta- 151. Ye F, Zhu F, Fu Y, Shen B. ECG generation with sequence gener-
tion of single-cell RNA-seq data using generative adversarial ative adversarial nets optimized by policy gradient. IEEE Access.
networks. Nat Commun. 2020;11:166. 2019;7:159369–159378.
131. Lee T, Yoon S. Boosted categorical restricted boltzmann machine 152. Luo Y, Lu BL. EEG data augmentation for emotion recognition
for computational prediction of splice junctions. In: Proc. ICML; using a conditional Wasserstein GAN. In: 2018 IEEE EMBC;
2015. p. 2483–2492. 2018. p. 2535–2538. ISSN: 1558-4615.
132. Zeng H, Edwards MD, Liu G, Gifford DK. Convolutional neural 153. You S, et al. Unsupervised automatic seizure detection for focal-
network architectures for predicting DNA-protein binding. Bio- onset seizures recorded with behind-the-ear EEG using an anom-
informatics. 2016;32(12):121–7. aly-detecting generative adversarial network. Comput Methods
133. Park S, Min S, Choi H, Yoon S. deepMiRGene: Deep neu- Programs Biomed. 2020;p. 105472.
ral network based precursor microRNA prediction. CoRR. 154. Jiao Y, Deng Y, Luo Y, Lu BL. Driver sleepiness detection from
2016;abs/1605.00017. EEG and EOG signals using GAN and LSTM networks. Neuro-
134. Lee B, Baek J, Park S, Yoon S. deepTarget: end-to-end learning computing. 2020;408:100-111.
framework for miRNA target prediction using deep recurrent 155. Singh P, Pradhan G. A new ECG denoising framework using
neural networks. CoRR. 2016;abs/1603.09123. generative adversarial network. IEEE/ACM Trans Comput Biol
135. Li H. A template-based protein structure reconstruction method Bioinform. 2020;p. 3114–3128.
using DA learning. J Proteomics Bioinform. 2016;9(12). 156. Wang X, Ghasedi Dizaji K, Huang H. Conditional generative
136. Ibrahim R, Yousri NA, Ismail MA, El-Makky NM. Multi-level adversarial network for gene expression inference. Bioinformat-
gene/MiRNA feature selection using deep belief nets and active ics. 2018 09;34(17):i603–i611.
learning. In: Proc. IEEE EMBC; 2014. p. 3957–3960. 157. Pan X, Shen HB. RNA-protein binding motifs mining with a new
137. Chen L, Cai C, Chen V, Lu X. Trans-species learning of cellular hybrid deep learning based cross-domain knowledge integration
signaling systems with bimodal deep belief networks. Bioinfor- approach. BMC Bioinform. 2017;18(1).
matics. 2015 sep;31(18):3008–3015. 158. Jiang X, Zhao J, Qian W, Song W, Lin GN. A generative adver-
138. Danaee P, Ghaeini R, Hendrix DA. A deep learning approach for sarial network model for disease gene prediction with RNA-seq
cancer detection and relevant gene identification. In: Proc. Pac. data. IEEE Access. 2020;8:37352–60.
Symp. Biocomput.. vol. 22; 2016. p. 219–229. 159. Zhao L, Wang J, Pang L, Liu Y, Zhang J. GANsDTA: pre-
139. Li Y, Fauteux F, Zou J, Nantel A, Pan Y. Personalized predic- dicting drug-target binding affinity using GANs. Front Genet.
tion of genes with tumor-causing somatic mutations based 2020;10:1243.
on multi-modal deep Boltzmann machine. Neurocomputing. 160. Editorial. Sharing data. Nat Cell Biol. 2009 11;11(11):1273.
2019;324:51–62. 161. Lord PW, et al . Large-scale data sharing in the life sciences: data
140. Zhang T, Zhang L, Payne PRO, Li F. Synergistic drug combina- standards, incentives, barriers and funding models (the ’Joint
tion prediction by integrating multi-omics data in deep learning Data Standards Study’). 2005. Available from: http://www.nesc.
models. arXiv​:18110​7054 [cs, q-bio, stat]. 2018 Nov;ArXiv: ac.uk/techn​ical_paper​s/UKeS-2006-02.pdf.
1811.07054. Available from: http://arxiv​.org/abs/1811.07054​. 162. Martone ME, Ellisman MH, Sosinsky GE, Gupta A, Tran J,
141. Huang Y, Gulko B, Siepel A. Fast, scalable prediction of deleteri- Wong W, et al. Cell Centered Database. UC San Diego Library
ous noncoding variants from functional and population genomic Digital Collections. 2017. https​://doi.org/10.6075/J0S18​0PX.
data. Nature Genet. 2017;49:618–24. 163. Ellisman M, et al. Cell Image Library. 2016. Available from:
142. Le EPV, Wang Y, Huang Y, Hickman S, Gilbert FJ. Artificial http://www.celli​magel​ibrar​y.org/. Accessed on 4 Jan 2020.
intelligence in breast imaging. Clin Radiol. 2019;74(5):357–66. 164. ERIC. EuroBioimaging. 2016. Available from: http://www.
143. Yi X, Walia E, Babyn P. Generative adversarial network in medi- eurob​ioima​ging.eu/. Accessed on 4 Jan 2020.
cal imaging: A review. Med Image Anal. 2019;58:101552. 165. Karkow W. HAPS Histology Image Database. 2008. Available
144. Sandfort V, Yan K, Pickhardt PJ, Summers RM. Data augmen- from: http://hapshi​ stolo​ gy.wikifo​ undry​ .com/. Accessed on 23 Jan
tation using generative adversarial networks (CycleGAN) to 2017.
improve generalizability in CT segmentation tasks. Sci Rep. 166. Dundee U. IDR: Image Data Resource. 2016. Available from:
2019;9(1):1–9. https​://idr.openm​icros​copy.org/.
145. Armanious K, et al. MedGAN: Medical image translation using 167. Kistler M. SMIR Full Body CT. SMIR. 2017. Available from:
GANs. Comput Med Imaging Graph. 2020;79:101684. https​://doi.org/10.1007/BF003​37288​.
146. Uemura T, et al. GAN-based survival prediction model from CT 168. Arkansas for Medical Sciences U. The Cancer Imaging Archive.
images of patients with idiopathic pulmonary fibrosis. In: Chen 2015. Available from: https​://www.cance​r imag​ingar​chive​.
PH, Deserno TM, editors. Medical Imaging 2020: Imaging Infor- net/. Accessed on 4 Jan 2020.
matics for Healthcare, Research, and Applications. vol. 11318. 169. Marinelli RJ, et al. The Stanford Tissue Microarray Database.
SPIE; 2020. p. 354 – 359. 2007. Available from: https​://tma.stanf​ord.edu. Accessed on
147. Thambawita V, Hammer HL, Riegler M, Halvorsen P. GANEx: A 23 Jan 2017.
complete pipeline of training, inference and benchmarking GAN 170. University of California SB. UCSB Bio-Segmentation Bench-
experiments. In: Proc. 2019 CBMI. 2019. p. 1–4. mark dataset. 2008. Available from: https​://bioim​age.ucsb.edu/
148. Halicek M, et al. Conditional generative adversarial network for resea​rch/bio-segme​ntati​on. Accessed on 23 Jan 2017.
synthesizing hyperspectral images of breast cancer cells from 171. ABIDE. Autism Brain Imaging Data Exchange. 2012. Avail-
digitized histology. In: Tomaszewski JE, Ward AD, editors. Med- able from: https​://goo.gl/n694s​N. Accessed on 4 Jan 2020.
ical Imaging 2020: Digital Pathology, vol. 11320. SPIE; 2020. p. 172. Milham MP. ADHD200. 2011. Available from: https​: //doi.
198–205. org/10.1007/BF003​37288​. Accessed on 4 Jan 2020.
149. Zhu F, Ye F, Fu Y, Liu Q, Shen B. Electrocardiogram generation 173. ANDI. Alzheimer’s Disease Neuroimaging Initiative (ADNI)
with a bidirectional LSTM-CNN generative adversarial network. datasets. 2009. Available from: https​: //adni.loni.usc.edu/.
Sci Rep. 2019;9(1):1–11. Accessed on 4 Jan 2020.
150. Yu L, Zhang W, Wang J, Yu Y. Seqgan: Sequence generative 174. Lopez M. Breast Cancer Digital Repository. 2008. Available
adversarial nets with policy gradient. In: Proc. 31st AAAI Conf. from:https​://bcdr.eu/. Accessed on 8 April 2020.
AI; 2017. p. 2852–2858.

13
Cognitive Computation (2021) 13:1–33 31

175. Mooney P. Chest X-Ray Images (Pneumonia) | Kaggle. UNSW QRS detection algorithms. 2016. Available from: http://
2018. Available from: https​://doi.org/10.1007/BF003​37288​. arxiv​.org/abs/1811.07054​.
Accessed on 4 Jan 2020. 198. 2020 BH. BNCIHorizon2020. 2015. Available from: http://arxiv.​
176. MITOS-ATYPIA. MITOS-ATYPIA-14 - Dataset. 2012. Avail- org/abs/1811.07054​. Accessed on 4 Jan 2020.
able from: https​://doi.org/10.1007/BF003​37288​. Accessed on 199. Khushaba RM. Electromyogram (EMG) Repository. 2012. Avail-
4 Jan 2020. able from: http://arxiv​.org/abs/1811.07054​. Accessed on 6 April
177. NAMIC. MIDAS - Community National Alliance for Medical 2020.
Image Computing (NAMIC). 2010. Available from: http://arxiv​. 200. Rantanen V, et al. Mimetic Interfaces: Facial Surface EMG Data-
org/abs/2003.00108​. set 2015. 2015. Available from: http://arxiv.​ org/abs/1811.07054​.
178. Cohen JP, Morrison P, Dao L. COVID-19 image data collec- Accessed on 4 Jan 2020.
tion. 2020. Available from: http://arxiv​.org/abs/2003.00108​. 201. Atzori M. NinaPro database non-invasive adaptive hand pros-
Accessed on 4 Jan 2020. thetics. 2012. Available from: http://arxiv​.org/abs/1811.07054​.
179. Yarkoni T. Neurosynth. 2012. Available from: http://arxiv​.org/ Accessed on 4 Jan 2020.
abs/2003.00108​. Accessed on 4 Jan 2020. 202. Koelstra S, et al. Database for emotion analysis using physi-
180. NIH. NIH chest x-ray datasets. 2017. Available from: http:// ological signals. 2011. Available from: http://arxiv​. org/
arxiv​.org/abs/2003.00108​. Accessed on 4 Jan 2020. abs/1811.07054​. Accessed on 4 Jan 2020.
181. LaMontagne PJ, et al. Open Access Series of Imaging Studies 203. Abadi MK, et al. MEG-based multimodal database for decoding
(OASIS). 2019. Available from: http://arxiv​.org/abs/2003.00108​ affective physiological responses. 2007. Available from: http://
182. Muschelli J. Open Neuroimaging Datasets. 2015. Available from: arxiv​.org/abs/1811.07054​. Accessed on 4 Jan 2020.
http://arxiv​.org/abs/2003.00108​. Accessed on 4 Jan 2020. 204. HeadIT of University of California SD. Imagined emotion. 2009.
183. Reyes M. The HEAR-EU multiscale imaging and modelling data- Available from: http://arxiv​.org/abs/1811.07054​. Accessed on 4
set of the human inner ear. SMIR. 2017. Available from: http:// Jan 2020.
arxiv​.org/abs/2003.00108​6. 205. Soleymani M, Lichtenauer J, Pun T, M P. HCI tagging database.
184. Dataset I. Brain development datasets. 2014. Available from: 2012. Available from: http://arxiv.​ org/abs/1811.07054.​ Accessed
http://arxiv​.org/abs/2003.00108​. Accessed on 4 Jan 2020. on 4 Jan 2020.
185. Shattuck DW, Mirza M, Adisetiyo V, Hojatkashani C, Salamon 206. Lu PBL. SEED Datasets. 2013. Available from: http://arxiv​.org/
G, et al. Construction of a 3D probabilistic atlas of human corti- abs/1811.07054​. Accessed on 4 Jan 2020.
cal structures. NeuroImage. 2008 Feb;39(3):1064–1080. 207. Kaya M, Binli MK, Ozbay E, Yanar H, Mishchenko Y. A large
186. Gorgolewski KJ, et al. NeuroVault. 2015. Available from: http:// electroencephalographic motor imagery dataset for electroen-
arxiv​.org/abs/2003.00108​. Accessed on 4 Jan 2020. cephalographic brain computer interfaces. figshare. 2018;Col-
187. Boekel W. Neuroimaging informatics tools and resources clear- lection. Available from: http://www.nesc.ac.uk/techni​ cal_papers​ /
inghouse dataset. 2015. Available from: https​://doi.org/10.1007/ UKeS-2006-02.pdf.
s1255​9-020-09751​-3. Accessed on 4 Jan 2020. 208. Cho M H amd Ahn, Ahn S, Kwon M, C JS. Supporting data
188. Poldrack, et al. OPEN fMRI: A multi-subject, multi-modal for EEG datasets for motor imagery brain computer interface.
human neuroimaging dataset. 2015. Available from: https​://doi. GigaScience Database. 2017. Available from: http://www.nesc.
org/10.1007/s1255​9-020-09751​-3. Accessed on 4 Jan 2020. ac.uk/techn​ical_paper​s/UKeS-2006-02.pdf.
189. Pernet C, Gorgolewski K, Ian W. Neuroimaging dataset of brain 209. Schalk G, McFarland DJ, Hinterberger T, Birbaumer N, Wolpaw
tumour patients. 2016. Available from: https​://doi.org/10.1007/ JR. EEG Motor Movement/Imagery Dataset. 2009. Available
s1255​9-020-09751​-3. Accessed on 4 Jan 2020. from: http://www.nesc.ac.uk/techn​ical_paper​s/UKeS-2006-02.
190. van Ginneken B, Kerkstra S, Meakin J. DRIVE - Grand Chal- pdf. Accessed on 6 April 2020.
lenge. 2004. Available from: https​://doi.org/10.1007/s1255​9- 210. Korczowski L, Ostaschenko E, Andreev A, Cattan G, Rodrigues
020-09751​-3. PC, Gautheret V, et al. Brain Invaders calibration-less P300-
191. Repository IBS. NITRC: IBSR: Tool/Resource Info. 2007. based BCI using dry EEG electrodes Dataset (bi2014a). 2019.
Available from: https​://doi.org/10.1007/s1255​9-020-09751​-3. Available from: http://www.nesc.ac.uk/techn​ical_paper​s/UKeS-
Accessed on 4 Jan 2020. 2006-02.pdf. Accessed on 4 Jan 2020.
192. Goldbaum M. The STARE Project. 1975. Available from: https:​ // 211. Korczowski L, Cederhout M, Andreev A, Cattan G, Rodrigues
doi.org/10.1007/s1255​9-020-09751​-3. PL, Gautheret V, et al. Brain Invaders calibration-less P300-
193. Cao Z, Chuang M, King JT, Lin CT. Multi-channel EEG based BCI with modulation of flash duration Dataset (bi2015a).
recordings during a sustained-attention driving task. Figshare. 2019. Available from: http://www.nesc.ac.uk/techn​ical_paper​s/
2019;Collection. Available from: https:​ //doi.org/10.1007/s12559​ - UKeS-2006-02.pdf. Accessed on 4 Jan 2020.
020-09751​-3. 212. Korczowski L, Ostaschenko E, Andreev A, Cattan G, Rodrigues
194. Picone J. Temple University EEG Corpus. 2011. Available from: PC, Gautheret V, et al. Brain Invaders Solo versus Collabora-
https​://doi.org/10.1007/s1255​9-020-09751​-3. Accessed on 7 tion: Multi-User P300-based Brain-Computer Interface Dataset
April 2020. (bi2014b). 2019. Available from: http://www.nesc.ac.uk/techn​ical_
195. GB M, RG M. MIT-BIH Arrhythmia Database. 1999. Available paper​s/UKeS-2006-02.pdf. Accessed on 4 Jan 2020.
from: https:​ //doi.org/10.1007/s12559​ -020-09751-​ 3. Accessed on 213. Korczowski L, Cederhout M, Andreev A, Cattan G, Rodrigues
4 Jan 2020. PL, Gautheret V, et al. Brain Invaders Cooperative versus Com-
196. Goldberger A, et al. PhysioBank, PhysioToolkit, and PhysioNet: petitive: Multi-User P300-based Brain-Computer Interface Dataset
components of a new research resource for complex physiologic (bi2015b). 2019. Available from: http://www.nesc.ac.uk/techn​ical_
signals. 2003. Circulation. 101(23):e215-e220. Available from: paper​s/UKeS-2006-02.pdf. Accessed on 4 Jan 2020.
https​://doi.org/10.1007/s1255​9-020-09751​-3. 214. BCI Competitions. BCI Competition datasets. 2008. Available
197. Khamis H, Weiss R, Xie Y, Chang CW, Lovell NH, Redmond from: http://www.nesc.ac.uk/techn​ical_paper​s/UKeS-2006-02.
SJ. TELE ECG Database: 250 telehealth ECG records (collected pdf. Accessed on 4 Jan 2020.
using dry metal electrodes) with annotated QRS and artifact 215. BCI Challenge NER2015. BCI Challenge @ NER 2015. 2015.
masks, and MATLAB code for the UNSW artifact detection and Available from: http://www.nesc.ac.uk/techn​ical_paper​s/UKeS-
2006-02.pdf. Accessed on 4 Jan 2020.

13
32 Cognitive Computation (2021) 13:1–33

216. Broderick MPea. Electrophysiological correlates of semantic dis- 242. Schaefer CFAKKSBJDMHTBKH. NCI-Nature pathway inter-
similarity reflect the comprehension of natural, narrative speech, action database. 2009. Available from: http://hapsh​istol​ogy.
v3, Dryad; 2020. Available from: http://www.nesc.ac.uk/techn​ical_ wikif​oundr​y.com/. Accessed on 10 April 2020.
paper​s/UKeS-2006-02.pdf. Accessed on 4 Jan 2020. 243. Kandasamy K, et al. NetPath. 2010. Available from: http://
217. for Complex Physiologic Signals RR. Physionet. 1999. Available hapsh​istol​ogy.wikif​oundr​y.com/. Accessed on 10 April 2020.
from: http://www.cellim ​ ageli​ brary​ .org/. Accessed on 4 Jan 2020. 244. Stein L, D’Eustachio P, Hermjakob H, Wu G. Reactome. 2010.
218. Aha D. UCI ML repository. 1987. Available from: http://www. Available from: http://hapshi​ stolo​ gy.wikifo​ undry​ .com/. Accessed
celli​magel​ibrar​y.org/. Accessed on 4 Jan 2020. on 10 April 2020.
219. Congedo M, et al. ”Brain Invaders”: a prototype of an open- 245. Tran VD. miRBoost. 2015. Available from: http://hapshi​ stol​ogy.
source P300-based video game working with the OpenViBE wikif​oundr​y.com/. Accessed on 4 Jan 2020.
platform. In: Proc. BCI 2011; 2011. p. 280–283. 246. SGD. Saccharomyces Genome Database. 2012. Available from:
220. PubChem. PubChem Data Sources. 2020. Available from: http:// http://hapsh​istol​ogy.wikif​oundr​y.com/. Accessed on 4 Jan 2020.
www.celli​magel​ibrar​y.org/. Accessed on 4 Jan 2020. 247. DNAD-J. DNA Databank of Japan. 1980. Available from: http://
221. PubChem. PubChem Data Sources. 2005. Available from: http:// hapsh​istol​ogy.wikif​oundr​y.com/. Accessed on 4 Jan 2020
www.celli​magel​ibrar​y.org/. Accessed on 4 Jan 2020. 248. ENA. European Nucleotide Archive. 1990. Available from:
222. Biolab. Bioinformatics Laboratory. 1999. Available from: http:// http://hapsh​istol​ogy.wikif​oundr​y.com/. Accessed on 4 Jan 2020.
www.celli​magel​ibrar​y.org/. Accessed on 4 Jan 2020. 249. GenBank. GenBank; 2013. Available from: https ​ : //idr.
223. Pradhan S, et al. Indian Genetic Disease Database. 2011. Avail- openm​icros​copy.org/. Accessed on 4 Jan 2020.
able from: http://www.celli​magel​ibrar​y.org/. Accessed on 4 Jan 250. Noordewier MO, Towell GG, Shavlik JW. UCI Molecular Biol-
2020. ogy (UCI MB) splice-junction gene sequences data set. 1981.
224. Atlas TCG. The Cancer Genome Atlas Home Page [nciHome]. Available from: https​://idr.openm​icros​copy.org/. Accessed on 4
2005. Available from: http://www.celli ​ m agel ​ i brar ​ y.org/. Jan 2020.
Accessed on 4 Jan 2020. 251. UCI-MB. UCI Molecular Biology (UCI MB) promoter gene
225. Network BDT. Berkeley Drosophila Transcription Network Pro- sequences data set. 1985. Available from: https​://idr.openm​icros​copy.
ject. 2001. Available from: http://www.celli​magel​ibrar​y.org/. org/. Accessed on 4 Jan 2020.
226. ENCODE. Encyclopedia of DNA Elements. 2003. Available 252. Manaswi NK. Understanding and working with Keras. In: Deep
from: http://www.cellim ​ ageli​ brary​ .org/. Accessed on 4 Jan 2020. learning with applications using Python. Springer; 2018. p.
227. NHLBI GO ESP. Exome Variant Server. 2011. Available from: 31–43.
http://www.celli​magel​ibrar​y.org/. Accessed on 6 April 2020. 253. Kunkel R, et al. TensorSCONE: a secure TensorFlow framework
228. GEO. Gene Expression Omnibus. 2000. Available from: http:// using Intel SGX. CoRR. 2019. p. 1–12.
www.eurob​ioima​ging.eu/. Accessed on 4 April 2020. 254. Sun X, Peng X, Ding S. Emotional human-machine conversation
229. Abreu M, et al. gnomAD. 2016. Available from: http://www. generation based on long short-term memory. Cogn Comput.
eurob​ioima​ging.eu/. Accessed on 6 April 2020. 2018;10(3):389–97.
230. of MIT TBI, Harvar. GTEx Portal. 2012. Available from: http:// 255. Hao L, Liang S, Ye J, Xu Z. TensorD: A tensor decomposition
www.eurob​ioima​ging.eu/. Accessed on 4 Jan 2020. library in TensorFlow. Neurocomputing. 2018;318:196–200.
231. Rouillard AD, Gundersen GW, Fernandez NF, Wang Z, Monteiro 256. Karpathy A. A peek at trends in machine learning. 2017. Avail-
CD, McDermott MG, et al. The harmonizome: a collection of able from: https​://idr.openm​icros​copy.org/.
processed datasets gathered to serve and mine knowledge about 257. Bahrampour S, Ramakrishnan N, Schott L, Shah M. Com-
genes and proteins. Database 2016. 2016;07:baw100. parative study of deep learning software frameworks. CoRR.
232. INSDC. The International Nucleotide Sequence Database Col- 2016;abs/1511.06435. ArXiv​: 1511.06435​.
laboration. 2016. Available from: http://www.eurob​ioima​ging. 258. Shi S, et al . Benchmarking state-of-the-art deep learning soft-
eu/. Accessed on 4 Jan 2020. ware tools. CoRR. 2016;abs/1608.07249.
233. Resource TIGS. 1000 Genomes Project. 2008. Available from: 259. Deepmark. The deep learning benchmarks. 2017. Available from:
http://www.eurob​ioima​ging.eu/. Accessed on 6 April 2020. https​://idr.openm​icros​copy.org/. Accessed on 17 Dec 2017.
234. JASPAR. JASPAR 2018: An open-access database of transcrip- 260. Narang S. The source code and experimental data of benchmark-
tion factor binding profiles. 2008. Available from: http://www. ing state-of-the-art deep learning software tools. 2017. Available
eurob​ioima​ging.eu/. Accessed on 4 Jan 2020. from: https:​ //idr.openmi​ crosc​ opy.org/. Accessed on 17 Dec 2017.
235. Consortium NREM. Roadmap Epigenomics Project - Data. 2007. 261. LeCun Y, Cortes C, Burges CJC. The MNIST database of hand-
Available from: http://www.eurob​ioima​ging.eu/. Accessed on 6 written digits. 1998. Available from: https​://idr.openm​icrosc​ opy.
April 2020. org/. Accessed on 4 Jan 2020.
236. NSD. Nature Scientific data. 2014. Available from: http://www. 262. Zaremba W, Sutskever I, Vinyals O. Recurrent neural network
eurob​ioima​ging.eu/. Accessed on 4 Jan 2020. regularization. CoRR. 2014;abs/1409.2329.
237. SysGENSIM. SysGenSIM - Benchmark datasets. 2013. Available 263. Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, LeCun
from: http://www.eurob​ioima​ging.eu/. Accessed on 4 Jan 2020. Y. OverFeat: integrated recognition, localization and detection
238. in Research BMSEB, Education. RCSB Protein Data Bank - using convolutional networks. CoRR. 2013;abs/1312.6229.
RCSB PDB. 2015. Available from: http://www.eurob​ioima​ging. 264. Murphy J. Deep learning benchmarks of NVIDIA Tesla P100
eu/. Accessed on 4 Jan 2020. PCIe, Tesla K80, and Tesla M40 GPUs; 201. Available from:
239. Murzin AG, Brenner SE, Hubbard TJP, Chothia C. Structural https​://idr.openm​icros​copy.org/. Accessed on 4 Jan 2020.
classification of proteins database 2. 2020. Available from: http:// 265. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, et al.
hapsh​istol​ogy.wikif​oundr​y.com/. Accessed on 10 April 2020. Human-level control through deep reinforcement learning.
240. Fox NK, Brenner SE, Chandonia JM. Structural classification of pro- Nature. 2015;518(7540):529–33.
teins database - extended. 2018. Available from: http://hapsh​istol​ogy. 266. Chollet F. The limitations of deep learning. 2017. Available from:
wikif​oundr​y.com/. Accessed on 10 April 2020. https​://idr.openm​icros​copy.org/. Accessed on 12 Dec 2017.
241. Qian N, Sejnowski TJ. UCI Molecular Biology (UCI MB) protein sec- 267. Zenil H, et al . An algorithmic information calculus for causal
ondary structure data set. 1988. Available from: http://hapsh​istol​ogy. discovery and reprogramming systems. bioRxiv. 2017;p. 185637.
wikif​oundr​y.com/. Accessed on 4 Jan 2020.

13
Cognitive Computation (2021) 13:1–33 33

268. Shwartz-Ziv R, Tishby N. Opening the black box of deep neural 282. Nahiduzzaman Md, Tasnim M, Newaz NT, Kaiser MS, Mah-
networks via information. CoRR. 2017 Mar;abs/1703.00810. mud M. Machine learning based early fall detection for elderly
269. Nguyen AM, Yosinski J, Clune J. Deep neural networks are easily people with neurological disorder using multimodal data fusion.
fooled: High confidence predictions for unrecognizable images. In: Mahmud M, Vassanelli S, Kaiser MS, Zhong N, editors.
In: Proc. CVPR; 2015. p. 427–436. Brain Inform. Cham: Springer International Publishing; 2020.
270. Szegedy C, Zaremba W, Sutskever I, Bruna J, Erhan D, Goodfel- p. 204–14.
low IJ, et al. Intriguing properties of neural networks. In: CoRR. 283. Farah L, Hussain A, Kerrouche A, Ieracitano C, Ahmad J, Mah-
vol. abs/1312.6199; 2013. p. 1–10. mud M. A highly-efficient fuzzy-based controller with high
271. Baker NA, Klemm JD, Harper SL, Gaheen S, Heiskanen M, reduction inputs and membership functions for a grid-connected
Rocca-Serra P, et al. Standardizing data. Nat Nanotechnol. photovoltaic system. IEEE Access. 2020;8:163225–37.
2013;8(2):73. 284. Fabietti M, Mahmud M, Lotfi A. Machine learning in analysing
272. Wittig U, Rey M, Weidemann A, Muller W. Data management invasively recorded neuronal signals: available open access data
and data enrichment for systems biology projects. J Biotechnol. sources. In: Mahmud M, Vassanelli S, Kaiser MS, Zhong N,
2017;261:229–237. editors. Brain Inform. Cham: Springer International Publishing;
273. Mahmud M, Rahman MM, Travalin D, Raif P, Hussain A. 2020. p. 151–62.
Service oriented architecture based web application model for 285. Chen L, Yan J, Chen J, Sheng Y, Xu Z, Mahmud M. An event
collaborative biomedical signal analysis. Biomed Tech (Berl). based topic learning pipeline for neuroimaging literature mining.
2012;57:780–3. 2020 Oct 23 [cited 2020 Nov 22]; Available from: https​://www.
274. Mahmud M, Pulizzi R, Vasilaki E, Giugliano M. A web-based resea​rchsq​uare.com/artic​le/rs-95392​/v1
framework for semi-online parallel processing of extracellular 286. Nahian MJA, Ghosh T, Uddin MN, Islam MdM, Mahmud M,
neuronal signals recorded by microelectrode arrays. In: Proc. Kaiser MS. Towards artificial intelligence driven emotion aware
MEAMEETING; 2014. p. 202–203. fall monitoring framework suitable for elderly people with neuro-
275. Angelov P, Sperduti A. Challenges in deep learning. In: Proc. logical disorder. In: Mahmud M, Vassanelli S, Kaiser MS, Zhong
ESANN; 2016. p. 489–495. N, editors. Brain Inform. Cham: Springer International Publish-
276. Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA. Deep ing; 2020. p. 275–86.
reinforcement learning: a brief survey. IEEE Signal Process Mag. 287. Al Banna MdH, Ghosh T, Taher KA, Kaiser MS, Mahmud M. A
2017;34(6):26–38. monitoring system for patients of autism spectrum disorder using
277. Noor MB, Zenia NZ, Kaiser MS, Al Mamun S, Mahmud M. artificial intelligence. In: Mahmud M, Vassanelli S, Kaiser MS,
Application of deep learning in detecting neurological disorders Zhong N, editors. Brain Inform. Cham: Springer International
from magnetic resonance images: a survey on the detection of Publishing; 2020. p. 251–62.
Alzheimer’s disease, Parkinson’s disease and schizophrenia. 288. Adiba FI, Islam T, Kaiser MS, Mahmud M, Rahman MA. Effect
Brain Inform. 2020;7(1):1-21. of corpora on classification of fake news using naive Bayes clas-
278. Al Banna MH, Taher KA, Kaiser MS, Mahmud M, Rahman MS, sifier. International Journal of Automation, Artificial Intelligence
Hosen AS, Cho GH. Application of artificial intelligence in pre- and Machine Learning. 2020;1(1):80–92.
dicting earthquakes: state-of-the-art and future challenges. IEEE 289. Sumi AI, Zohora MostF, Mahjabeen M, Faria TJ, Mahmud M,
Access. 2020;8:192880–192923. Kaiser MS. fASSERT: A fuzzy assistive system for children with
279. Jesmin S, Kaiser MS, Mahmud M. Artificial and internet of autism using internet of things. In: Wang S, Yamamoto V, Su J,
healthcare things based Alzheimer care during COVID 19. In: Yang Y, Jones E, Iasemidis L, et al., editors. Brain Inform. Cham:
Mahmud M, Vassanelli S, Kaiser MS, Zhong N, editors. Brain Springer International Publishing; 2018. p. 403–12.
Inform. Cham: Springer International Publishing; 2020. p. 290. Mahmud M, Kaiser MS, Rahman MM, Rahman MA, Shabut A,
263–274. Al-Mamun S, et al. A brain-inspired trust management model to
280. Ruiz J, Mahmud M, Modasshir M, Shamim Kaiser M, Alz- assure security in a cloud based IoT framework for neuroscience
heimer’s disease neuroimaging initiative for the 3D DenseNet applications. Cogn Comput. 2018 Oct 1;10(5):864–73.
ensemble in 4-way classification of Alzheimer’s disease. In: 291. Kaiser MS, Chowdhury ZI, Mamun SA, Hussain A, Mahmud
Mahmud M, Vassanelli S, Kaiser MS, Zhong N, editors. Brain M. A neuro-fuzzy control system based on feature extraction
Inform. Cham: Springer International Publishing; 2020. p. of surface electromyogram signal for solar-powered wheelchair.
85–96. Cogn Comput. 2016 Oct 1;8(5):946–54.
281. Rahman S, Sharma T, Mahmud M. Improving alcoholism diag-
nosis: comparing instance-based classifiers against neural net-
works for classifying EEG signal. In: Mahmud M, Vassanelli
S, Kaiser MS, Zhong N, editors. Brain Inform. Cham: Springer
International Publishing; 2020. p. 239–50.

13