Vol. 28 no.

23 2012, pages 3081–3088

BIOINFORMATICS ORIGINAL PAPER doi:10.1093/bioinformatics/bts581

Systems biology Advance Access publication October 9, 2012

An unbiased evaluation of gene prioritization tools

Daniela Börnigen1,2,3,y, Léon-Charles Tranchevent1,2,y, Francisco Bonachela-Capdevila4,y,
Koenraad Devriendt5, Bart De Moor1,2, Patrick De Causmaecker4 and Yves Moreau1,2,*
1
Department of Electrical Engineering, ESAT-SCD, 2IBBT-KULeuven Future Health Department, Katholieke Universiteit
Leuven, Leuven, Belgium, 3Biostatistics Department, Harvard School of Public Health, Harvard University, Boston, MA,
USA, 4CODeS Group, ITEC-IBBT-KULEUVEN, Katholieke Universiteit Leuven Campus Kortrijk, Kortrijk, Belgium and
5

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

Center for Human Genetics, Katholieke Universiteit Leuven, Leuven, Belgium
Associate Editor: Martin Bishop

ABSTRACT advances in next-generation sequencing offer promising

Motivation: Gene prioritization aims at identifying the most promising opportunities to explore the genomic alterations of patients
candidate genes among a large pool of candidates—so as to (Schuster, 2008). However, thousands of mutations in hundreds
maximize the yield and biological relevance of further downstream of genes are often detected, among which only a few are in fact
validation experiments and functional studies. During the past few linked to the genetic condition of interest (Lupski et al., 2010).
years, several gene prioritization tools have been defined, and The experimental validation of these candidate genes, for in-
some of them have been implemented and made available through stance, through resequencing, pathway or expression analysis,
freely available web tools. In this study, we aim at comparing is still expensive and time consuming. An efficient way to
the predictive performance of eight publicly available prioritization reduce the validation cost is to narrow down the large list of
tools on novel data. We have performed an analysis in which 42 candidate genes to a small and manageable set of highly promis-
recently reported disease-gene associations from literature are used ing genes, a process called gene prioritization. Prioritization in
to benchmark these tools before the underlying databases are the past was achieved manually by geneticists and biologists and
updated. was mainly based on their own expertise. Nowadays, biologists
Results: Cross-validation on retrospective data provides performance and geneticists can use computational approaches that can
estimate likely to be overoptimistic because some of the data sources handle and analyze the large amount of genomic data currently
are contaminated with knowledge from disease-gene association. Our available.
approach mimics a novel discovery more closely and thus provides In the past few years, many gene prioritization methods have
more realistic performance estimates. There are, however, marked been proposed, some of which have been implemented into pub-
differences, and tools that rely on more advanced data integration licly available tools that users can freely access and use
schemes appear more powerful. (Doncheva et al., 2012; Moreau et al., 2012; Oti, 2011; Piro
Contact: [email protected] et al., 2012; Tiffin, 2011, Tranchevent et al., 2010). Information
Supplementary information: Supplementary data are available at about these tools is summarized in our Gene Prioritization Portal
Bioinformatics online. (http://www.esat.kuleuven.be/gpp) that currently describes 33
prioritization tools. This web site has been designed to help re-
Received on July 10, 2012; revised on September 19, 2012; accepted searchers to carefully select the tools that best correspond to their
on September 20, 2012 needs. For instance, only few tools can prioritize the whole
genome, which can be necessary when no positive regions can
be identified beforehand, or when selecting candidates for a
1 INTRODUCTION medium-throughput screen (instead of low-throughput valid-
A major challenge in human genetics is to discover novel disease- ation). Another example is the study of a poorly characterized
causing genes, both for Mendelian and complex disorders. Iden- disorder for which a prioritization tool not relying on a set of
tifying disease genes is a crucial first step in unraveling molecular known disease genes might be more suited. Recently, several
networks underlying diseases, and thus understanding disease studies have demonstrated that gene prioritization tools can
mechanisms, also toward the development of effective therapies. help geneticists to discover novel disease genes (Calvo et al.,
The discovery of a novel disease gene often starts with a cyto- 2006; Thienpont et al., 2010). For instance, a KIF1A mutation
genetic study, a linkage analysis, a high-throughput omics experi- was discovered in hereditary spastic paraparesis patients after
ment or a genome-wide association study (GWAS). However, KIF1A was predicted to be the best candidate gene from the
these studies do not always pinpoint the disease gene uniquely, locus using multiple prioritization tools (Erlich et al., 2011).
but often result in large lists of candidate genes that are poten- Another study discovered homozygous mutations in the
tially relevant (Hardy and Singleton, 2009). Moreover, recent PTRF-CAVIN gene in patients with congenital generalized
lipodystrophy with muscle rippling after PTRF-CAVIN was pre-
*To whom correspondence should be addressed.
dicted as the most probable candidate gene for high expression in
yThe authors wish it to be known that, in their opinion, the first three muscle and adipose tissue (Rajab et al., 2010). A third
authors should be regarded as joint First Authors. study identified the HHEX gene to be associated with Type

ß The Author 2012. Published by Oxford University Press. All rights reserved. For Permissions, please e-mail: [email protected] 3081
D.Börnigen et al.

2 diabetes (T2D) in a Dutch cohort after investigating the T2D- 2 METHODS

susceptibility loci using candidate gene prioritization
2.1 Gene prioritization tools
(Vliet-Ostaptchouk et al., 2008).
However, beyond these conceptual differences, one essential We aim at comparing the gene prioritization tools that can easily be used,
and therefore, only select the tools for which a free web-based implemen-
parameter to consider when selecting gene prioritization tools is
tation is available. The main objective is to assess the ability of the gene
their respective performance—that is, their ability to identify the prioritization tools to predict potential novel disease genes that can then
true positive genes as promising candidate genes to maximize the be experimentally validated. We have therefore not selected the tools
yield of the follow-up experimental validation. A common standard whose ranking strategies exclusively depend on text as they would most
in bioinformatics is to estimate the performance with a bench- likely work only when the novel disease gene was already considered a
mark analysis. Several publications that introduce a novel pri- good candidate gene before discovery. One exception is Candid, which
oritization approach also describe a comparative benchmark also uses other data sources beside MEDLINE (e.g. protein domains,

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

with several existing methods (Hutz et al., 2008; Köhler et al., interactions and expression data). In total, we have selected eight tools:
Suspects (Adie et al., 2006), ToppGene (Chen et al., 2007), GeneDistiller
2008; Thornblad et al., 2007). However, these benchmarks are
(Seelow et al., 2008), GeneWanderer (Köhler et al., 2008), Posmed
most of the time cross-validations of gold-standard disease data-
(Yoshida et al., 2009), Candid (Hutz et al., 2008), Endeavour (Aerts
sets (e.g. known data). Therefore, the estimation of the et al., 2006) and Pinta (Nitsch et al., 2010). The tools are run with the
performance is likely an overestimate of the real performance settings recommended by the developers. When applicable, multiple con-
(i.e. on novel data). Because different types of data are depend- figurations are defined to explore several possibilities (for instance, several
ent on each other (e.g. Gene Ontology (GO) annotation, Kyoto ranking algorithms within one tool). Originally, Pinta was developed to
Encyclopedia of Genes and Genomes (KEGG) pathway mem- use expression data as input data, but here, we replace the continuous
bership and MEDLINEÕ abstracts), it becomes impossible to data (coming from expression data) with binary data using training genes:
remove all cross-talk effects between data sources (e.g. removing a 1 is inputted for each training gene, and a 0 is associated to the other
genes. For an overview of the tools, please see Supplementary Table S1.
MEDLINE data does not remove all information from the bio-
All tools except Candid are used to prioritize a set of candidate genes
medical literature since much of it is present in GO and KEGG) (from a chromosomal region), and Candid is used to prioritize the whole
to prevent contamination of the prediction of the disease gene by genome. Pinta and Endeavour support both genome-wide and candidate
actual retrospective knowledge of this association. This makes it set based prioritizations, and are used for both in this study
challenging to create benchmarks on retrospective data that are (Endeavour-GW and Pinta-GW for genome-wide prioritization and
indicative of the performance of the method in an actual research Endeavour-CS and Pinta-CS for candidate set prioritization). In addition,
setting. Next to benchmarking, some studies use several GeneWanderer can be run with up to four different ranking strategies
prioritization methods to analyze disease-associated loci, (random walk, diffusion kernel, shortest path and direct interaction). We
present the results for the first two strategies (GeneWanderer-RW for
mostly for Type 2 diabetes and obesity (Elbers et al., 2007;
random walk, GeneWanderer-DK for diffusion kernel) because they
Teber et al., 2009; Tiffin et al., 2006). However, the results
have been showed to outperform the other two, simpler, approaches
have not been experimentally validated, which means that it is (Köhler et al., 2008) and since they can be efficiently used with many
not possible to identify which methods made better predictions. training genes. The performance of Posmed shows a strong dependency
Also, a few studies combine computational and experimental on the set of keywords used as an input, and we ran it twice with different
analysis: in silico-generated hypothesis are then validated inputs. In the first run, we use the complete keyword set (Posmed-KS),
in vivo. We have, for instance, performed a computationally sup- and in the second, we only use the name of the disease (Posmed-DN).
ported genetic screen in Drosophila that led to the identification GeneDistiller is trained with both genes and keywords. These keywords
of 12 novel atonal genetic interactors (Aerts et al., 2009). are then used to find additional genes through the mining of Online
Mendelian Inheritance in Man (OMIM), which, in our case, has less
Although useful, such studies often rely on the use of a single
influence since OMIM is already used to derive the training genes. We
tool and therefore cannot be used to compare different therefore consider that GeneDistiller is trained with genes only. Candid is
approaches. They also give no indication of the performance the only tool that can also be trained with disease-specific tissues, and
of the method in general, but only illustrate it on a single when available, tissues relevant to the disease under study are used.
well-validated case. Notice that suspects went offline during our study after the 27th associ-
In this study, we aim at comparing the performance of several ation and is not supported anymore (E.Adie, personal communication);
freely accessible web-based gene prioritization tools on novel therefore, Suspects results are based on 27 associations over 42.
data, which, to our knowledge, has never been performed
before. To this aim, we selected recently reported disease-gene 2.2 Validation dataset
associations from literature and use several gene prioritization The validation dataset is built by mining the scientific literature to iden-
tools to make predictions immediately after publication (typically tify the recently discovered disease-gene associations. This is achieved
within 2 days). Our approach relies on the fact that, when the manually to avoid false-positive associations. We select six journals
prioritization tools are used, the novel disease-gene association of that frequently publish papers that describe such associations: Nature
interest is not yet included in the databases that underlie these Genetics, American Journal of Medical Genetics (Part A/Part B),
Human Genetics, Human Molecular Genetics, and Human Mutation. We
tools. As a consequence, our approach mimics a novel discovery,
select all novel disease-gene associations regardless of the disease under
and therefore, the estimation of the performance is more
study, of the methodology used, and of whether the findings are con-
accurate. It has to be mentioned that we compare tools and firmed or not. Novelty is assessed by using OMIM (McKusick, 1998), the
not the underlying algorithms (we see a tool as an algorithm Genetic Association Database (Becker et al., 2004), GoPubmed (Doms
plus some data sources), because this is what is most relevant and Schroeder, 2005) and GeneCards (Safran et al., 2010). More pre-
to geneticists. cisely, we assess novelty at the gene level, and therefore, novel mutations

3082
 An unbiased evaluation of gene prioritization tools

Table 1. The validation dataset consisting of 42 recently discovered disease-gene associations

Gene Disease/phenotype Reference(s)

HCCS Congenital diaphragmatic hernia Qidwai et al. (2010)

BRCA2 Bipolar disorder Tesli et al. (2010)
TNFRSF19 Nasopharyngeal carcinoma Bei et al. (2010)
MECOM Nasopharyngeal carcinoma Bei et al. (2010)
ATF7IP Testicular germ cell tumor Turnbull et al. (2010)
DMRT1 Testicular germ cell tumor Turnbull et al. (2010)
FUT2 Crohn’s disease McGovern et al. (2010)
CSF1R Asthma Shin et al. (2010)

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

GLI3 Metopic craniosynostosis McDonald-McGinn et al. (2010)
STOM Nonsyndromic cleft lip/palate Letra et al. (2010)
UTRN Arthrogryposis Tabet et al. (2010)
GABRR1 Bipolar schizoaffective disorder Green et al. (2010)
UBE2L3 Crohn’s disease Fransen et al. (2010)
BCL3 Crohn’s disease Fransen et al. (2010)
EZH2 Myelodysplastic syndromes Nikoloski et al. (2010)
TRAF6 Parkinson’s disease Zucchelli et al. (2010)
IL10 Behcet’s disease Mizuki et al. (2010); Remmers et al. (2010)
DAB2IP Abdominal aortic aneurysm Gretarsdottir et al. (2010)
SPIB Primary biliary cirrhosis Liu et al. (2010)
MMEL1 Primary biliary cirrhosis Hirschfield et al. (2010)
TBX2 Complex heart defect Radio et al. (2010)
RUNX2 Single-suture craniosynostosis Mefford et al. (2010)
CRHR1 Multiple sclerosis Briggs et al. (2010)
IFNG Leprosy Cardoso et al. (2010)
SH2B1 Congenital anomalies of the kidney and urinary tract Sampson et al. (2010)
DISP1 Congenital diaphragmatic hernia Kantarci et al. (2010)
G6PC3 Dursun syndrome Banka et al. (2010)
PQBP1 Periventricular heterotopia Sheen et al. (2010)
CD320 Methylmalonic aciduria Quadros et al. (2010)
CHST14 Ehlers-Danlos syndrome Miyake et al. (2010)
PLCE1 Esophageal squamous cell carcinoma Abnet et al. (2010); Wang et al. (2010)
C20orf54 Esophageal squamous cell carcinoma Wang et al. (2010)
SDCCAG8 Retinal–renal ciliopathy Otto et al. (2010)
TP63 Lung adenocarcinoma Miki et al. (2010)
UBE2E2 Type 2 diabetes Yamauchi et al. (2010)
LPP Tetralogy of fallot Arrington et al. (2010)
RANBP1 Smooth pursuit eye movement abnormality Cheong et al. (2011)
HTR7 Alcohol dependence Zlojutro et al. (2010)
SOX17 Congenital anomalies of the kidney and the urinary tract Gimelli et al. (2010)
ACAD9 Mitochondrial complex I deficiency Haack et al. (2010)
TRAF3IP2 Psoriasis Ellinghaus et al. (2010); Hüffmeier et al. (2010)
WDR62 Autosomal recessive primary microcephaly Nicholas et al. (2010); Yu et al. (2010)

within already known genes are not considered. This process was kept and tool-specific gene identifiers (e.g. EntrezGene or Ensembl identifiers).
active for 6 months (May 15–November 15, 2010) and led to a collection As mentioned earlier, most of the tools in addition require a set of candi-
of 42 associations (see Table 1 and Supplementary Table S2). For each date genes (from the whole genome). Several tools accept chromosomal
association, the tools are run as soon as the association is identified fol- coordinates, whereas some prefer cytogenetics bands. For each associ-
lowing the defined workflow (see later). By doing this, we simulate as ation, we select the cytogenetics bands that cover 10 Mb around the
much as possible the prediction of a novel disease gene, since the under- novel disease gene and derive the chromosomal coordinates. We choose
lying databases are still unaware of the association. Once an association is 10 Mb to obtain on average at least 100 candidate genes. Once again,
identified, the exact inputs for the different tools have to be defined. For BioMart is used to retrieve specific gene identifiers. For an overview of
instance, ToppGene, GeneDistiller, GeneWanderer, Pinta and Endeavour the inputs for the 42 associations, please see Supplementary Table S3.
require training genes (genes already known to be associated to the disease The resulting 42 novel disease-gene associations do not represent a
under study), whereas Suspects, Posmed, GeneDistiller and Candid re- homogeneous set. Therefore, we have divided them into confirmed
quire keywords that describe the disease. Training genes and keywords (for monogenic diseases, the mutation is found in at least two unrelated
are collected from the corresponding OMIM pages, Genetic Association patients; for multifactorial diseases, a GWAS is replicated in a separate
Database (GAD) pages and from recently published reviews when pos- cohort), intermediate (a single study, but additional functional evidence is
sible. BioMart (Haider et al., 2009) is used to map between gene symbols provided) and unconfirmed (a single study) associations.

3083
D.Börnigen et al.

2.3 Performance measures Table 2. Results for the genome-wide and candidate set based prioritiza-
tion tools
For each tool, we then assess its ability to identify the novel disease genes
as promising genes using several statistical measures. We first compute
the median of the rank ratio over all associations. We preferably use rank Median Response TPR in top TPR in top TPR in top
ratio over rank because tools do not necessarily return the same number rate (%) 5% (%) 10% (%) 30% (%)
of candidate genes even when fed with the same inputs. In addition, we
also draw the boxplots of these rank ratios to give a more comprehensive Genome-wide prioritization tools
view of tool performance. Another method to compare the tools is to Candid 18.10 100 21.4 33.3 64.3
Endeavour-GW 15.49 100 28.6 38.1 71.4
build the receiver-operating characteristic (ROC) curves and to compute
Pinta-GW 19.03 100 26.2 31.0 71.4
the area under the curve (AUC) as an estimate of the global performance. Integration 12.45 100 19.1 38.1 78.6
To compare the tools even further, we computed the true-positive rates Candidate set based prioritization tools
when setting the threshold for validation at the top 5% (True Positive Suspects 12.77a 88.9a 33.3a 33.3a 63.0a

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

Rate (TPR) in top 5% of candidates), 10% (TPR in top 10%) and 30% ToppGene 16.80 97.6 35.7 42.9 52.4
GeneWanderer-RW 22.10 95.2 16.7 26.2 61.9
(TPR in top 30%). This is motivated by the fact that, in a real situation,
GeneWanderer-DK 22.97 88.1 11.9 21.4 52.4
the number of candidate genes to assay often needs to be limited because Posmed-DN 45.45 50.0 4.7 11.9 23.8
of financial and time constraints. We have selected three thresholds that Posmed-KS 31.44 47.6 4.7 7.1 23.8
represent reasonable biological hypotheses, as we previously illustrated in GeneDistiller 11.11 97.6 26.2 47.6 78.6
a genetic screen (Aerts et al., 2009). The corresponding TPR measures are Endeavour-CS 11.16 100 26.2 42.9 90.5
used to estimate how efficient the tools are if only the top 5, 10 or 30% Pinta-CS 18.87 100 28.6 31.0 71.4
Integration 6.99 100 40.5 57.1 83.3
candidate genes would be assayed. Notice that these values correspond to
the shape of the lower end of the ROC curve (the sharper the curve, the a
Values computed only on the first 27 associations.
higher the TPR). There are cases for which some tools are unable to
identify the novel disease gene at all; therefore, we include a response
rate. It is defined as the percentage of associations for which each tool
does return a prioritization result for the novel disease gene (in some cases,
a tool will not return any result, for example, because it could not correctly
map the gene identifier or some candidates are otherwise filtered out). For
example, if one of the 42 disease genes could not be ranked (i.e. gene is
missing), the response rate drops down to 98% (41/42).
Lastly, we also derive a heat map to detect any correlation
between tools by computing the pairwise cosine similarity of the rankings
pre-sented in Table 2 (see Supplementary Fig. S1).

2.4 Integration of predictions

To get an estimate of the usefulness of a meta-predictor, the results of the Fig. 1. Boxplots of the 42 novel disease genes from the validation dataset
different tools are combined using the order statistics as within illustrated for the genome-wide (left) and candidate gene set based (right)
Endeavour. Integration happens separately for the genome-wide tools prioritization tools
and candidate set based tools, and tools that return only few rankings
(Suspects and Posmed) were not included. For each experiment, the gene
identifiers of the different tools are mapped using Biomart. To avoid tools (Fig. 1, right). Among the genome-wide tools, Endeavour-
getting artificially favorable rankings, the size of the merged ranking is GW performs slightly better than Pinta-GW and Candid
set to the maximum size of the underlying rankings. (Fig. 1, left).
When considering the response rate, Endeavour (both modes),
Candid and Pinta (both modes) performed the best study with
3 RESULTS
100% closely followed by ToppGene, GeneDistiller and
The overall ranking results of all gene prioritization tools are GeneWanderer-RW with more than 95% (meaning that only
summarized in Table 2, the complete results are presented in one or two associations are missing). At the other hand of the
Supplementary Tables S9 and S10. These results have also spectrum, Posmed-KS and Posmed-DN only work for about half
been added to the Gene Prioritization Portal (http://www.esat. of the experiments in our benchmark (47.6% and 50%,
kuleuven.be/gpp). respectively).
When we compare the tools based on the global AUC (see
3.1 Performance measures Fig. 2), we observe that GeneDistiller appears as the best
When considering the median of the rank ratios, GeneDistiller, performing tool overall with an AUC of 86%. It is followed
Endeavour-CS and Suspects are the tools that perform the best by Endeavour-CS (82%), Endeavour-GW (79%), Pinta-GW
on this benchmark (11.11, 11.16 and 12.77, respectively). They (77%), Suspects (76%), Pinta-CS (75%), Candid (73%),
are followed by Endeavour-GW (15.49), ToppGene (16.8), GeneWanderer-RW (71%), GeneWanderer-DK (67%),
Candid (18.1), Pinta-CS (18.87), Pinta-GW (19.03), GeneWan- ToppGene (66%), Posmed-KS (58%) and Posmed-DN (56%).
derer-RW (22.11), GeneWanderer-DK (22.97), Posmed-KS However, the ROC curves are in general intertwined, meaning
(31.44) and Posmed-DN (45.45). The boxplots presented in that none of the approaches is clearly performing better than the
Figure 1 illustrate that both GeneDistiller and Endeavour-CS other. However, we postulate that, in our case, the most import-
perform better than the other candidate set based prioritization ant section of the ROC curve is the beginning and therefore use

3084
 An unbiased evaluation of gene prioritization tools

associations from literature containing novel genes for both

monogenic conditions and complex disorders. We have selected
novel disease-gene associations regardless of their strength and of
the underlying methodology. To mimic a real discovery, we have
run the tools as soon as the article appeared online so that all
databases used for gene prioritization are still not contaminated
by the knowledge of the novel disease-gene association. This also
means that we had to exclude tools that query MEDLINE online
because their results would be biased.
We want to compare the performance of the tools even if the
inputs are different (genes versus keywords, genome-wide versus

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

Fig. 2. ROC curves of the genome-wide (A) and candidate gene set based candidate set). Among the eight gene prioritization tools that we
(B) prioritization tools have analyzed in this study, only Endeavour, Candid and Pinta
have been used for genome-wide prioritization. The input
three other measures, the true-positive rates at 5, 10 and 30%, data for Endeavour and Pinta are training genes, whereas
respectively. These measures indicate how efficient the tools Candid requires keywords. The gene prioritization tools that
would be if only the top candidate genes would be assayed. we have used to prioritize candidate genes within a region of
Considering the TPR in top 10 and 30%, we can observe a interest are Suspects, ToppGene, GeneWanderer, Posmed,
similar trend. Indeed, at 10%, GeneDistiller is first with a rate of GeneDistiller, and again Endeavour and Pinta. Suspects and
47.6% (20 associations found over 42), followed by both Posmed are trained with keywords, and the other tools require
ToppGene and Endeavour-CS with 42.9% (18 associations). training genes. We have extensively searched through literature
However, at 30%, the best tool is Endeavour-CS (90.5%, 38 and dedicated databases to identify as many reliable training
associations), followed by GeneDistiller (78.6%, 33 associ- genes as possible for the disease of interest as well as a set of
ations). The other tools show smaller TPR at both levels: appropriate keywords to derive fair and meaningful compari-
Pinta-CS (31%, 71.4%), Suspects (33.3%, 63%), sons. However, different, and possibly better, results might be
GeneWanderer-RW (26.2%, 61.9%), GeneWanderer-DK obtained by refining the inputs.
(21.4%, 52.4%), Posmed-KS (7.1%, 23.8%) and Posmed-DN Our validation is too small to claim that the differences among
(11.9%, 23.8%). Among the genome-wide prioritization tools, tools are significant. However, a trend can still be observed,
Endeavour-GW shows the highest TPR in top 10 and 30% GeneDistiller and Endeavour-CS consistently appear as the
(38.1%, 71.4%), followed by Candid (33.3%, 64.3%) and best tools when looking at all performance measures. It is inter-
Pinta-GW (31%, 71.4%). esting to notice that the best results are, in general, obtained with
tools that use many data types in conjunction (up to eight for
3.2 Correlations Endeavour, when compared with the three data sources used by
Posmed), but there is no perfect correlation. This is in agreement
Supplementary Figure S1 shows the heat map of the novel
with the conclusion of the recent review by Tiffin et al. (2009),
disease-gene ranking positions for all tools in this study. For
who indicate that successful computational applications will be
the tools that have two modes (i.e. Posmed, GeneWanderer,
facilitated by improved data integration.
Endeavour and Pinta), the two modes are highly correlated
All tools except Posmed have a high response rate ranging
(40.89). There is also a significant correlation between Candid
from 88 to 100%, meaning that at least 37 of the 42 novel disease
and GeneWanderer-DK (0.82). The other values are within 0.4
genes are prioritized (or 24 of 27 for Suspects). However, the
and 0.7, indicating that all tools are moderately correlated.
response rates for Posmed-KS and Posmed-DN are 47.6% and
50%, respectively, which can be explained by the fact that
3.3 Integration of predictions Posmed also acts as a filter on the candidate genes to obtain a
Our meta-analysis reveals that the best results are obtained when reduced list of genes in the end. There are therefore cases for
predictions are combined over the different tools (see Table 2 and which the novel disease gene has been removed by the filter. This
Supplementary Table S11). For genome-wide tools, all perform- is different from the other tools for which missing genes basically
ance measures are improved by the integrative method (e.g. correspond to genes that are not recognized by the tool (it hap-
median of 12.45 for the meta-predictor versus 15.49 for pens most of the time with poorly characterized genes, such as
Endeavour-GW). Similar results are obtained for the candidate C20orf54). Another special case is Suspects that went offline
set based tools (e.g. median of 6.99 for the meta-predictor versus during the validation and therefore could only be validated
11.11 for GeneDistiller), although the TPR in the top 30% of the with the first 27 associations. We therefore calculated the re-
integrative method is still lower than for Endeavour-CS (83.3% sponse rate only on the first 27 associations.
versus 90.5%). Two types of tools can be distinguished, the ones that are
trained with already-known genes and the ones that are trained
with descriptive keywords. It appears that gene-based tools seem
4 DISCUSSION to work better than keyword-based tools (the average of medians
We aim at assessing the usefulness of eight gene prioritization is 17.2 for gene-based tools and 27 for keyword based tools;
tools that are freely available via web applications. We have built similar results are obtained with the other measures, see
a validation based on 42 recently discovered disease-gene Supplementary Table S8). This could be because we use, in

3085
D.Börnigen et al.

general, more genes than keywords for training (18.8 genes on an us to identify the influence of the size of the gene list to prioritize.
average for six keywords). This also indicates that more key- The median rank ratio is better for Endeavour-CS (11.16) than
words might be needed to model a disease and that a small for Endeavour-GW (15.49) in our benchmark. The difference
text (such as an OMIM entry) might even be necessary (van remains, albeit smaller, when considering the AUC and the
Driel et al., 2006). TPR in top 10 and 30%.
There is in general an agreement between the five performance The same training genes are used, and therefore the observed
measures we use throughout our study. One notable exception difference is only caused by extending the small candidate gene
exists for ToppGene, whose AUC is 66%, and corresponds to set to the whole genome. This confirms previous findings that
rank 10th (out of the 12 prioritization tools). In contrast, its prioritizing the whole genome is more difficult than prioritizing a
associated TPR in top 10% is 42.9%, which corresponds to rather small positive locus. The heat map indicates that the two
rank second. This apparent contradiction can be explained by Endeavour modes are strongly correlated as expected since the

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

observing Figure 2, in which the ROC curve exhibits a core algorithm is the same in both modes (40.9, see
non-convex shape. This is because ToppGene either ranks the Supplementary Fig. S1). In contrary, the results for both Pinta
novel disease gene on top or at the bottom (i.e. the disease genes modes are similar (correlation of 0.99) and seem to indicate that
are rarely ranked in the middle). Therefore, the TPR in top 10% the size of the candidate set does not influence this algorithm.
will be high because it only takes into account the top of the list, In this study, we consider the tools as off-the-shelf solutions
while the AUC will be lower because it basically behaves like an and use them as recommended by the developers without
average over all cases. Another important point is that our ob- fine-tuning of the parameters. However, an important feature
servations are in line with the ‘no free lunch’ theorem. Indeed, that might influence the results is the date of the last data
each tool can perform better than all the others for some cases, update. The latest genomic data (still prior to discoveries con-
or, in other words, none of the tools outperforms another on the sidered in this study) is likely to give the best results because it
complete dataset (if we do not consider the special case of will model more accurately what is currently known, when com-
Posmed that also acts as a filter). pared with data that are 2 years old. In our setup, we have no
Posmed-KS has been trained with the complete keyword set, control over the genomic data used and cannot identify whether
whereas Posmed-DN has been trained only with the disease variation in performance among tools can be explained by this.
name. The median rank ratio is 31.44 when the complete key- In addition, the quality of both the data sources and the inte-
word set is used and drops to 45.45 when only the disease name is gration methodologies are also influencing the outcome of the
inputted. If we only compare the results over the 19 associations prioritization process. However, we aim at estimating the useful-
for which both tools are able to prioritize the novel disease gene, ness of some prioritization tools for geneticists. Therefore an
the difference becomes even larger (29.6 and 50, respectively, for in-depth comparison of the implementation of the tools is
Posmed-KS and Posmed-DN). Altogether, these results indicate beyond the scope of this study.
that Posmed does not rely on the use of the single disease name It is important to notice that the 42 novel disease-gene asso-
and that the extra keywords are indeed important. It can be ciations do not represent a homogeneous set. Indeed, the median
observed that the performance measures for Posmed are worse of the rank ratios over the tools show that some associations
than for the other tools in our benchmark study. However, when seem to be easier to predict than others. This also explains why
looking at the individual ranks, it can be observed that Posmed all tools are moderately correlated on the heat map (40.4). A
returns far fewer genes than the other tools because it also acts as plausible explanation is the disparity in the available data be-
a filter. As a result, the rank ratios are in general larger and the tween the novel disease genes. Since only little data can be gath-
performance measures are therefore worse. As such, it becomes ered for poorly characterized genes, such as C20orf54, they are
difficult to fairly compare Posmed with the other tools because more difficult to prioritize. However, we also hypothesize that
our measures of performance naturally penalize the fact that the nature of the underlying genetic disorder as well as the qual-
Posmed returns prioritizations for a limited set of candidates. ity of the reported association might influence the ability of the
Changing our performance measures to counterbalance this tools to correctly predict that association. We have therefore
effect would then give an unfair advantage to Posmed because divided the associations between confirmed, intermediate and
it returns prioritizations only for the ‘safer bets’. unconfirmed. Among the 42 associations, 23 are confirmed, 8
GeneWanderer has also been run twice with different network are intermediate and 11 are unconfirmed (see Supplementary
algorithms: random walk and diffusion kernel. The respective Table S2). We hypothesize that this might influence our valid-
performances are very similar although the random walk ap- ation since some unconfirmed associations might in fact be spuri-
proach is performing a little bit better than the diffusion kernel ous. We observe that Suspects and ToppGene perform better for
albeit non-significant (22.11–22.97 for median rank ratio – simi- the 23 confirmed associations than for the 19 unconfirmed ones
lar differences are observed with the other measures). The heat (see Supplementary Tables S4 and S5). However, this trend is not
map indicates a strong correlation (40.9, see Supplementary always shared as the situation is opposite for GeneDistiller and
Fig. S1) between the two modes, which was expected since apply- GeneWanderer. Although informative, these comparisons are
ing diffusion to a kernel can be interpreted as equivalent to not significant due to the small number of associations.
applying a random walk on the underlying network. In our validation dataset, there are 17 monogenic diseases and
Altogether, this indicates that these two algorithms are similar. 25 multifactorial disorders (see Supplementary Tables S6 and
Endeavour and Pinta are used to prioritize both the whole S7). It has been shown that it is more difficult to make predic-
genome (Endeavour-GW and Pinta-GW) and the defined tions for multifactorial diseases than for monogenic diseases
chromosomal region (Endeavour-CS and Pinta-CS), allowing (Linghu et al., 2009). Our results however seem to indicate

3086
 An unbiased evaluation of gene prioritization tools

that not all tools are influenced by the intrinsic complexity of projects, G.0318.05 (subfunctionalization), G.0553.06
multifactorial diseases. For instance, Endeavour and ToppGene (VitamineD), G.0302.07 (SVM/Kernel), research communities
seem to perform better for monogenic conditions while (ICCoS, ANMMM, MLDM); G.0733.09 (3UTR); G.082409
GeneWanderer and Suspects perform better for complex dis- (EGFR), IWT: PhD Grants, Silicos; SBO-BioFrame,
orders. However, the size of our validation dataset does not SBO-MoKa, TBM-IOTA3, FOD:Cancer plans, IBBT]; Belgian
allow for a complete statistical analysis. Larger validation data- Federal Science Policy Office [IUAP P6/25 (BioMaGNet,
sets and real predictive studies will be pursued to complement Bioinformatics and Modeling: from Genomes to Networks,
our preliminary study. 2007–2011)]; EU-RTD [ERNSI: European Research Network
We are aware of the limited coverage of available literature in on System Identification; FP7-HEALTH CHeartED].
human genetics in our study that report novel disease-gene asso-
ciations. However, we aimed at estimating the real performance Conflict of Interest: none declared.

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

of gene prioritization tools and therefore have decided to keep
under strict control all the factors that could potentially bias the
REFERENCES
benchmark. We were further interested in finding novel
disease-gene associations for defining a proper benchmark, and Abnet,C.C. et al. (2010) A shared susceptibility locus in PLCE1 at 10q23 for gastric
adenocarcinoma and esophageal squamous cell carcinoma. Nat. Genet., 42,
there is no guarantee that these associations are uniformly dis- 764–767.
tributed over the whole literature. We have used journals about Adie,E.A. et al. (2006) SUSPECTS: enabling fast and effective prioritization of
genetic disorders, in general, and favor journals that report novel positional candidates. Bioinformatics, 22, 773–774.
associations and have avoided specialized journals that focus on Aerts,S. et al. (2006) Gene prioritization through genomic data fusion. Nat.
Biotech., 24, 537–544.
few diseases to avoid introducing bias toward one disease class.
Aerts,S. et al. (2009) Integrating computational biology and forward genetics in
Our choice of the six selected journals may not be perfect, but drosophila. PLoS Genet., 5, e1000351.
they allowed us to cover most disease types and most situations. Arrington,C.B. et al. (2010) Haploinsufficiency of the LIM domain containing
Several studies have shown that combining predictions of sev- preferred translocation partner in lipoma (LPP) gene in patients with tetralogy
eral tools lead to even better predictions (Elbers et al., 2007; of fallot and VACTERL association. Am. J. Med. Genet. A, 152, 2919–2923.
Banka,S. et al. (2010) Mutations in the G6PC3 gene cause Dursun syndrome. Am.
Tiffin et al., 2006). However, no performance criteria were J. Med. Genet. A, 152, 2609–2611.
used to select the tools to be combined. With this comparison Becker,K.G. et al. (2004) The genetic association database. Nat. Genet., 36, 431–432.
of tools, we ease the selection of the most efficient tools, whose Bei,J. et al. (2010) A genome-wide association study of nasopharyngeal carcinoma
combination may lead to more accurate predictions. In addition, identifies three new susceptibility loci. Nat. Genet., 42, 599–603.
Briggs,F.B.S. et al. (2010) Evidence for CRHR1 in multiple sclerosis using super-
we report that the meta-predictors that integrate the predictions vised machine learning and meta-analysis in 12,566 individuals. Hum. Mol.
made by several tools perform better than the best individual Genet., 19, 4286–4295.
tools as already reported (Thornblad et al., 2007). Calvo,S. et al. (2006) Systematic identification of human mitochondrial disease
Our results indicate that cross-validation based benchmarks genes through integrative genomics. Nat. Genet., 38, 576–582.
Cardoso,C.C. et al. (2010) IFNG þ874 TntextgreaterA single nucleotide poly-
tend to overestimate the real predictive performance. Indeed,
morphism is associated with leprosy among Brazilians. Hum. Genet., 128,
all the tools for which such a benchmark exists have lower 481–490.
AUC than anticipated using our dataset (see Supplementary Chen,J. et al. (2007) Improved human disease candidate gene prioritization using
Table S12). We therefore believe that developers should take mouse phenotype. BMC Bioinformatics, 8, 392.
extra care when benchmarking their tools as to avoid these pit- Cheong,H.S. et al. (2011) Association of RANBP1 haplotype with smooth pursuit
eye movement abnormality. Am. J. Med. Genet. B, 156, 67–71.
falls. Also, some hard constraints have made this study small Doms,A. and Schroeder,M. (2005) GoPubMed: exploring PubMed with the gene
enough not to reach significance (e.g. only few tools have a pro- ontology. Nucleic Acids Res., 33, W783–W786.
grammatically queryable interface). Doncheva,N.T. et al. (2012) Recent approaches to the prioritization of candidate
As already discussed in Moreau et al. (2012), this field needs to disease genes. WIREs Syst. Biol. Med., 4, 429–442.
Elbers,C.C. et al. (2007) A strategy to search for common obesity and type 2 dia-
consolidate through improved benchmarking efforts due to the
betes genes. Trends Endocrinol. Metab., 18, 19–26.
lack of a ground truth for evaluating the performance of priori- Ellinghaus,E. et al. (2010) Genome-wide association study identifies a psoriasis
tization methods. Therefore, we see a need for a large-scale com- susceptibility locus at TRAF3IP2. Nat. Genet., 42, 991–995.
munity effort to compare multiple tools across common Erlich,Y. et al. (2011) Exome sequencing and disease-network analysis of a single
prospective benchmarks. We hope our work represents the first family implicate a mutation in KIF1A in hereditary spastic paraparesis. Genome
Res., 21, 558–664.
step toward a collaborative effort to tackle this problem at a Fransen,K. et al. (2010) Analysis of SNPs with an effect on gene expression iden-
larger scale. tifies UBE2L3 and BCL3 as potential new risk genes for Crohn’s disease. Hum.
Mol. Genet., 19, 3482–3488.
Gimelli,S. et al. (2010) Mutations in SOX17 are associated with congenital anoma-
ACKNOWLEDGEMENTS lies of the kidney and the urinary tract. Hum. Mutat., 31, 1352–1359.
Green,E.K. et al. (2010) Variation at the GABAA receptor gene, rho 1 (GABRR1)
The authors thank Peter Konings for his help regarding the associated with susceptibility to bipolar schizoaffective disorder. Am. J. Med.
statistics. Genet. B, 153, 1347–1349.
Gretarsdottir,S. et al. (2010) Genome-wide association study identifies a sequence
Funding: Research Council KUL [CIF/07/02 DE CAUSMAE/ variant within the DAB2IP gene conferring susceptibility to abdominal aortic
DEFIS-SOCK, ProMeta, GOA Ambiorics, GOA MaNet, GOA aneurysm. Nat. Genet., 42, 692–697.
Haack,T.B. et al. (2010) Exome sequencing identifies ACAD9 mutations as a cause
2006/12, CoE EF/05/007 SymBioSys en KUL PFV/10/016 of complex i deficiency. Nat. Genet., 42, 1131–1134.
SymBioSys, START 1, several PhD/post-doc and fellow Haider,S. et al. (2009) BioMart central portal–unified access to biological data.
grants]; Flemish Government [FWO: PhD/post-doc grants, Nucleic Acids Res., 37, W23–W27.

3087
D.Börnigen et al.

Hardy,J. and Singleton,A. (2009) Genomewide association studies and human dis- Remmers,E.F. et al. (2010) Genome-wide association study identifies variants in the
ease. N. Engl. J. Med., 360, 1759–1768. MHC class i, IL10, and IL23R-IL12RB2 regions associated with Behçet’s dis-
Hirschfield,G.M. et al. (2010) Variants at IRF5-TNPO3, 17q12-21 and MMEL1 are ease. Nat. Genet., 42, 698–702.
associated with primary biliary cirrhosis. Nat. Genet., 42, 655–657. Safran,M. et al. (2010) GeneCards version 3: the human gene integrator. Database,
Hüffmeier,U. et al. (2010) Common variants at TRAF3IP2 are associated with 2010: article ID baq020; doi:10.1093/database/baq020.
susceptibility to psoriatic arthritis and psoriasis. Nat. Genet., 42, 996–999. Sampson,M.G. et al. (2010) Evidence for a recurrent microdeletion at chromosome
Hutz,J.E. et al. (2008) CANDID: a flexible method for prioritizing candidate genes 16p11.2 associated with congenital anomalies of the kidney and urinary tract
for complex human traits. Genet. Epidemiol., 32, 779–790. (CAKUT) and hirschsprung disease. Am. J. Med. Genet. A, 152, 2618–2622.
Kantarci,S. et al. (2010) Characterization of the chromosome 1q41q42.12 region, Schuster,S.C. (2008) Next-generation sequencing transforms today’s biology. Nat.
and the candidate gene DISP1, in patients with CDH. Am. J. Med. Genet. A, Methods, 5, 16–18.
152, 2493–2504. Seelow,D. et al. (2008) GeneDistiller—distilling candidate genes from linkage inter-
Köhler,S. et al. (2008) Walking the interactome for prioritization of candidate dis- vals. PLoS One, 3, e3874.
ease genes. Am. J. Hum. Genet., 82, 949–958. Sheen,V.L. et al. (2010) Mutation in PQBP1 is associated with periventricular het-
Letra,A. et al. (2010) Follow-up association studies of chromosome region 9q and erotopia. Am. J. Med. Genet. A, 152, 2888–2890.

Downloaded from https://academic.oup.com/bioinformatics/article/28/23/3081/193845 by guest on 25 November 2020

nonsyndromic cleft lip/palate. Am. J. Med. Genet. A, 152, 1701–1710. Shin,E.K. et al. (2010) Association between colony-stimulating factor 1 receptor
Linghu,B. et al. (2009) Genome-wide prioritization of disease genes and identifica- gene polymorphisms and asthma risk. Hum. Genet., 128, 293–302.
tion of disease-disease associations from an integrated human functional linkage Tabet,A. et al. (2010) Molecular characterization of a de novo 6q24.2q25.3 dupli-
network. Genome Biol., 10, R91. cation interrupting UTRN in a patient with arthrogryposis. Am. J. Med. Genet.
Liu,X. et al. (2010) Genome-wide meta-analyses identify three loci associated with A, 152, 1781–1788.
primary biliary cirrhosis. Nat. Genet., 42, 658–660. Teber,E.T. et al. (2009) Comparison of automated candidate gene prediction sys-
Lupski,J.R. et al. (2010) Whole-genome sequencing in a patient with tems using genes implicated in type 2 diabetes by genome-wide association
Charcot-Marie-Tooth neuropathy. N. Engl. J. Med., 362, 1181–1191. studies. BMC Bioinformatics, 10 (Suppl. 1), S69.
McDonald-McGinn,D.M. et al. (2010) Metopic craniosynostosis due to mutations Tesli,M. et al. (2010) Association analysis of PALB2 and BRCA2 in bipolar dis-
in GLI3: a novel association. Am. J. Med. Genet. A, 152, 1654–1660. order and schizophrenia in a Scandinavian case–control sample. Am. J. Med.
McGovern,D.P.B. et al. (2010) Fucosyltransferase 2 (FUT2) non-secretor status is
Genet. B, 153, 1276–1282.
associated with Crohn’s disease. Hum. Mol. Genet., 19, 3468–3476.
Thienpont,B. et al. (2010) Haploinsufficiency of TAB2 causes congenital heart de-
McKusick,V.A. (1998) Mendelian Inheritance in Man: A Catalog of Human Genes
fects in humans. Am. J. Hum. Genet., 86, 839–849.
and Genetic Disorders. 12th edn. The Johns Hopkins University Press,
Thornblad,T.A. et al. (2007) Prioritization of positional candidate genes using mul-
Baltimore, Maryland, USA.
tiple web-based software tools. Twin Res. Hum. Genet., 10, 861–870.
Mefford,H.C. et al. (2010) Copy number variation analysis in single-suture cranio-
Tiffin,N. (2011) Conceptual thinking for in silico prioritization of candidate disease
synostosis: multiple rare variants including RUNX2 duplication in two cousins
genes. Methods Mol. Biol., 760, 175–187.
with metopic craniosynostosis. Am. J. Med. Genet. A, 152, 2203–2210.
Tiffin,N. et al. (2006) Computational disease gene identification: a concert of meth-
Miki,D. et al. (2010) Variation in TP63 is associated with lung adenocarcinoma
ods prioritizes type 2 diabetes and obesity candidate genes. Nucleic Acids Res.,
susceptibility in Japanese and Korean populations. Nat. Genet., 42, 893–896.
34, 3067–3081.
Miyake,N. et al. (2010) Loss-of-function mutations of CHST14 in a new type of
Tiffin,N. et al. (2009) Linking genes to diseases: it’s all in the data. Genome Med., 1,
Ehlers-Danlos syndrome. Hum. Mutat., 31, 966–974.
77.
Mizuki,N. et al. (2010) Genome-wide association studies identify IL23R-IL12RB2
Tranchevent,L. et al. (2010) A guide to web tools to prioritize candidate genes.
and IL10 as Behçet’s disease susceptibility loci. Nat. Genet., 42, 703–706.
Brief. Bioinformatics, 12, 22–32.
Moreau,Y. et al. (2012) Computational tools for prioritizing candidate genes: boost-
Turnbull,C. et al. (2010) Variants near DMRT1, TERT and ATF7IP are associated
ing disease gene discover. Nat. Rev. Genet., 13, 523–536.
with testicular germ cell cancer. Nat. Genet., 42, 604–607.
Nicholas,A.K. et al. (2010) WDR62 is associated with the spindle pole and is
van Driel,M.A. et al. (2006) A text-mining analysis of the human phenome. Eur. J.
mutated in human microcephaly. Nat. Genet., 42, 1010–1014.
Nikoloski,G. et al. (2010) Somatic mutations of the histone methyltransferase gene Hum. Genet., 14, 535–542.
EZH2 in myelodysplastic syndromes. Nat. Genet., 42, 665–667. Vliet-Ostaptchouk,J.V. et al. (2008) HHEX gene polymorphisms are associated
Nitsch,D. et al. (2010) Candidate gene prioritization by network analysis of differ- with type 2 diabetes in the Dutch Breda cohort. Eur. J. Hum. Genet., 16,
ential expression using machine learning approaches. BMC Bioinformatics, 11, 652–656.
460. Wang,L. et al. (2010) Genome-wide association study of esophageal squamous cell
Oti,M. (2011) Web tools for the prioritization of candidate disease genes. Methods carcinoma in Chinese subjects identifies susceptibility loci at PLCE1 and
Mol. Biol., 760, 189–206. c20orf54. Nat. Genet., 42, 759–763.
Otto,E.A. et al. (2010) Candidate exome capture identifies mutation of SDCCAG8 Yamauchi,T. et al. (2010) A genome-wide association study in the Japanese popu-
as the cause of a retinal–renal ciliopathy. Nat. Genet., 42, 840–850. lation identifies susceptibility loci for type 2 diabetes at UBE2E2 and
Piro,R.M. et al. (2012) Computational approaches to disease-gene prediction: ra- C2CD4AC2CD4B. Nat. Genet., 42, 864–868.
tionale, classification and successes. FEBS J., 279, 678–696. Yoshida,Y. et al. (2009) PosMed (Positional medline): prioritizing genes with an
Qidwai,K. et al. (2010) Deletions of Xp provide evidence for the role of holocyto- artificial neural network comprising medical documents to accelerate positional
chrome c-type synthase (HCCS) in congenital diaphragmatic hernia. Am. J. cloning. Nucleic Acids Res., 37 (Web Server issue), W147–W152.
Med. Genet. A, 152, 1588–1590. Yu,T.W. et al. (2010) Mutations in WDR62, encoding a centrosome-associated
Quadros,E.V. et al. (2010) Positive newborn screen for methylmalonic aciduria protein, cause microcephaly with simplified gyri and abnormal cortical archi-
identifies the first mutation in TCblR/CD320, the gene for cellular uptake of tecture. Nat. Genet., 42, 1015–1020.
transcobalamin-bound vitamin b(12). Hum. Mutat., 31, 924–929. Zlojutro,M. et al. (2010) Genome-wide association study of theta band event-related
Radio,F.C. et al. (2010) TBX2 gene duplication associated with complex heart oscillations identifies serotonin receptor gene HTR7 influencing risk of alcohol
defect and skeletal malformations. Am. J. Med. Genet. A, 152, 2061–2066. dependence. Am. J. Med. Genet. B, 156, 44–58.
Rajab,A. et al. (2010) Fatal cardiac arrhythmia and long-QT syndrome in a new Zucchelli,S. et al. (2010) TRAF6 promotes atypical ubiquitination of mutant DJ-1
form of congenital generalized lipodystrophy with muscle rippling (CGL4) due and alpha-synuclein and is localized to Lewy bodies in sporadic Parkinson’s
to PTRF-CAVIN mutations. PLoS Genet., 6, e1000874. disease brains. Hum. Mol. Genet., 19, 3759–3770.

3088