Combining pattern classifiers Methods and

algorithms Second Edition Kuncheva pdf download

https://ebookname.com/product/combining-pattern-classifiers-
methods-and-algorithms-second-edition-kuncheva/

Get Instant Ebook Downloads – Browse at https://ebookname.com

Instant digital products (PDF, ePub, MOBI) available
Download now and explore formats that suit you...

Hybrid Methods in Pattern Recognition 1st Edition H.

Bunke

https://ebookname.com/product/hybrid-methods-in-pattern-
recognition-1st-edition-h-bunke/

Statistical Pattern Recognition Second Edition Andrew

R. Webb(Auth.)

https://ebookname.com/product/statistical-pattern-recognition-
second-edition-andrew-r-webbauth/

Grokking Algorithms Second Edition Aditya Y Bhargava

https://ebookname.com/product/grokking-algorithms-second-edition-
aditya-y-bhargava/

Medicinal Chemistry Fusion of Traditional and Western

Medicine Third Edition Robert E. Smith

https://ebookname.com/product/medicinal-chemistry-fusion-of-
traditional-and-western-medicine-third-edition-robert-e-smith/
Citizenship and Moral Education Mark Halstead

https://ebookname.com/product/citizenship-and-moral-education-
mark-halstead/

The Colour of Injustice The Mysterious Murder of the

Daughter of a High Court Judge 1st Edition John
Hostettler

https://ebookname.com/product/the-colour-of-injustice-the-
mysterious-murder-of-the-daughter-of-a-high-court-judge-1st-
edition-john-hostettler/

Advances in Experimental Social Psychology 38 1st

Edition Mark P. Zanna (Eds.)

https://ebookname.com/product/advances-in-experimental-social-
psychology-38-1st-edition-mark-p-zanna-eds/

Microsoft Windows 2000 Core Requirements Exam 70 216

Microsoft Windows 2000 Network Infrastructure
Administration 2nd Edition Microsoft Corporation

https://ebookname.com/product/microsoft-windows-2000-core-
requirements-exam-70-216-microsoft-windows-2000-network-
infrastructure-administration-2nd-edition-microsoft-corporation/

Implementing Transfer Associate Degrees Perspectives

From the States New Directions for Community Colleges
Number 160 2nd Edition Carrie B. Kisker

https://ebookname.com/product/implementing-transfer-associate-
degrees-perspectives-from-the-states-new-directions-for-
community-colleges-number-160-2nd-edition-carrie-b-kisker/
Chemostratigraphy Concepts Techniques and Applications
1st Edition Mu Ramkumar

https://ebookname.com/product/chemostratigraphy-concepts-
techniques-and-applications-1st-edition-mu-ramkumar/
Combining
Pattern Classifiers
Methods and Algorithms, Second Edition
Ludmila Kuncheva
COMBINING PATTERN
CLASSIFIERS
COMBINING PATTERN
CLASSIFIERS
Methods and Algorithms

Second Edition

LUDMILA I. KUNCHEVA
Copyright © 2014 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or
by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as
permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to
the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400,
fax (978) 646-8600, or on the web at www.copyright.com. Requests to the Publisher for permission
should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street,
Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in
preparing this book, they make no representations or warranties with respect to the accuracy or
completeness of the contents of this book and specifically disclaim any implied warranties of
merchantability or fitness for a particular purpose. No warranty may be created or extended by sales
representatives or written sales materials. The advice and strategies contained herin may not be suitable
for your situation. You should consult with a professional where appropriate. Neither the publisher nor
author shall be liable for any loss of profit or any other commercial damages, including but not limited to
special, incidental, consequential, or other damages.

For general information on our other products and services please contact our Customer Care
Department with the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax 317-572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print,
however, may not be available in electronic format.

MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does
not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of
MATLAB® software or related products does not constitute endorsement or sponsorship by The
MathWorks of a particular pedagogical approach or particular use of the MATLAB® software.

Library of Congress Cataloging-in-Publication Data

Kuncheva, Ludmila I. (Ludmila Ilieva), 1959–

Combining pattern classifiers : methods and algorithms / Ludmila I. Kuncheva. – Second edition.
pages cm
Includes index.
ISBN 978-1-118-31523-1 (hardback)
1. Pattern recognition systems. 2. Image processing–Digital techniques. I. Title.
TK7882.P3K83 2014
006.4–dc23
2014014214

Printed in the United States of America.

10 9 8 7 6 5 4 3 2 1
To Roumen, Diana and Kamelia
CONTENTS

Preface xv
Acknowledgements xxi

1 Fundamentals of Pattern Recognition 1

1.1 Basic Concepts: Class, Feature, Data Set, 1
1.1.1 Classes and Class Labels, 1
1.1.2 Features, 2
1.1.3 Data Set, 3
1.1.4 Generate Your Own Data, 6
1.2 Classifier, Discriminant Functions, Classification Regions, 9
1.3 Classification Error and Classification Accuracy, 11
1.3.1 Where Does the Error Come From? Bias and Variance, 11
1.3.2 Estimation of the Error, 13
1.3.3 Confusion Matrices and Loss Matrices, 14
1.3.4 Training and Testing Protocols, 15
1.3.5 Overtraining and Peeking, 17
1.4 Experimental Comparison of Classifiers, 19
1.4.1 Two Trained Classifiers and a Fixed Testing Set, 20
1.4.2 Two Classifier Models and a Single Data Set, 22
1.4.3 Two Classifier Models and Multiple Data Sets, 26
1.4.4 Multiple Classifier Models and Multiple Data Sets, 27
1.5 Bayes Decision Theory, 30
1.5.1 Probabilistic Framework, 30

vii
viii CONTENTS

1.5.2 Discriminant Functions and Decision Boundaries, 31

1.5.3 Bayes Error, 33
1.6 Clustering and Feature Selection, 35
1.6.1 Clustering, 35
1.6.2 Feature Selection, 37
1.7 Challenges of Real-Life Data, 40
Appendix, 41
1.A.1 Data Generation, 41
1.A.2 Comparison of Classifiers, 42
1.A.2.1 MATLAB Functions for Comparing Classifiers, 42
1.A.2.2 Critical Values for Wilcoxon and Sign Test, 45
1.A.3 Feature Selection, 47

2 Base Classifiers 49
2.1 Linear and Quadratic Classifiers, 49
2.1.1 Linear Discriminant Classifier, 49
2.1.2 Nearest Mean Classifier, 52
2.1.3 Quadratic Discriminant Classifier, 52
2.1.4 Stability of LDC and QDC, 53
2.2 Decision Tree Classifiers, 55
2.2.1 Basics and Terminology, 55
2.2.2 Training of Decision Tree Classifiers, 57
2.2.3 Selection of the Feature for a Node, 58
2.2.4 Stopping Criterion, 60
2.2.5 Pruning of the Decision Tree, 63
2.2.6 C4.5 and ID3, 64
2.2.7 Instability of Decision Trees, 64
2.2.8 Random Trees, 65
2.3 The Naı̈ve Bayes Classifier, 66
2.4 Neural Networks, 68
2.4.1 Neurons, 68
2.4.2 Rosenblatt’s Perceptron, 70
2.4.3 Multi-Layer Perceptron, 71
2.5 Support Vector Machines, 73
2.5.1 Why Would It Work?, 73
2.5.2 Classification Margins, 74
2.5.3 Optimal Linear Boundary, 76
2.5.4 Parameters and Classification Boundaries of SVM, 78
2.6 The k-Nearest Neighbor Classifier (k-nn), 80
2.7 Final Remarks, 82
2.7.1 Simple or Complex Models?, 82
2.7.2 The Triangle Diagram, 83
2.7.3 Choosing a Base Classifier for Ensembles, 85
Appendix, 85
 CONTENTS ix

2.A.1 MATLAB Code for the Fish Data, 85

2.A.2 MATLAB Code for Individual Classifiers, 86
2.A.2.1 Decision Tree, 86
2.A.2.2 Naı̈ve Bayes, 89
2.A.2.3 Multi-Layer Perceptron, 90
2.A.2.4 1-nn Classifier, 92

3 An Overview of the Field 94

3.1 Philosophy, 94
3.2 Two Examples, 98
3.2.1 The Wisdom of the “Classifier Crowd”, 98
3.2.2 The Power of Divide-and-Conquer, 98
3.3 Structure of the Area, 100
3.3.1 Terminology, 100
3.3.2 A Taxonomy of Classifier Ensemble Methods, 100
3.3.3 Classifier Fusion and Classifier Selection, 104
3.4 Quo Vadis?, 105
3.4.1 Reinventing the Wheel?, 105
3.4.2 The Illusion of Progress?, 106
3.4.3 A Bibliometric Snapshot, 107

4 Combining Label Outputs 111

4.1 Types of Classifier Outputs, 111
4.2 A Probabilistic Framework for Combining Label Outputs, 112
4.3 Majority Vote, 113
4.3.1 “Democracy” in Classifier Combination, 113
4.3.2 Accuracy of the Majority Vote, 114
4.3.3 Limits on the Majority Vote Accuracy:
An Example, 117
4.3.4 Patterns of Success and Failure, 119
4.3.5 Optimality of the Majority Vote Combiner, 124
4.4 Weighted Majority Vote, 125
4.4.1 Two Examples, 126
4.4.2 Optimality of the Weighted Majority Vote
Combiner, 127
4.5 Naı̈ve-Bayes Combiner, 128
4.5.1 Optimality of the Naı̈ve Bayes Combiner, 128
4.5.2 Implementation of the NB Combiner, 130
4.6 Multinomial Methods, 132
4.7 Comparison of Combination Methods for Label Outputs, 135
Appendix, 137
4.A.1 Matan’s Proof for the Limits on the Majority Vote
Accuracy, 137
4.A.2 Selected MATLAB Code, 139
x CONTENTS

5 Combining Continuous-Valued Outputs 143

5.1 Decision Profile, 143
5.2 How Do We Get Probability Outputs?, 144
5.2.1 Probabilities Based on Discriminant Scores, 144
5.2.2 Probabilities Based on Counts: Laplace Estimator, 147
5.3 Nontrainable (Fixed) Combination Rules, 150
5.3.1 A Generic Formulation, 150
5.3.2 Equivalence of Simple Combination Rules, 152
5.3.3 Generalized Mean Combiner, 153
5.3.4 A Theoretical Comparison of Simple Combiners, 156
5.3.5 Where Do They Come From?, 160
5.4 The Weighted Average (Linear Combiner), 166
5.4.1 Consensus Theory, 166
5.4.2 Added Error for the Weighted Mean Combination, 167
5.4.3 Linear Regression, 168
5.5 A Classifier as a Combiner, 172
5.5.1 The Supra Bayesian Approach, 172
5.5.2 Decision Templates, 173
5.5.3 A Linear Classifier, 175
5.6 An Example of Nine Combiners for Continuous-Valued
Outputs, 175
5.7 To Train or Not to Train?, 176
Appendix, 178
5.A.1 Theoretical Classification Error for the Simple Combiners, 178
5.A.1.1 Set-up and Assumptions, 178
5.A.1.2 Individual Error, 180
5.A.1.3 Minimum and Maximum, 180
5.A.1.4 Average (Sum), 181
5.A.1.5 Median and Majority Vote, 182
5.A.1.6 Oracle, 183
5.A.2 Selected MATLAB Code, 183

6 Ensemble Methods 186

6.1 Bagging, 186
6.1.1 The Origins: Bagging Predictors, 186
6.1.2 Why Does Bagging Work?, 187
6.1.3 Out-of-bag Estimates, 189
6.1.4 Variants of Bagging, 190
6.2 Random Forests, 190
6.3 AdaBoost, 192
6.3.1 The AdaBoost Algorithm, 192
6.3.2 The arc-x4 Algorithm, 194
6.3.3 Why Does AdaBoost Work?, 195
 CONTENTS xi

6.3.4 Variants of Boosting, 199

6.3.5 A Famous Application: AdaBoost for Face Detection, 199
6.4 Random Subspace Ensembles, 203
6.5 Rotation Forest, 204
6.6 Random Linear Oracle, 208
6.7 Error Correcting Output Codes (ECOC), 211
6.7.1 Code Designs, 212
6.7.2 Decoding, 214
6.7.3 Ensembles of Nested Dichotomies, 216
Appendix, 218
6.A.1 Bagging, 218
6.A.2 AdaBoost, 220
6.A.3 Random Subspace, 223
6.A.4 Rotation Forest, 225
6.A.5 Random Linear Oracle, 228
6.A.6 ECOC, 229

7 Classifier Selection 230

7.1 Preliminaries, 230
7.2 Why Classifier Selection Works, 231
7.3 Estimating Local Competence Dynamically, 233
7.3.1 Decision-Independent Estimates, 233
7.3.2 Decision-Dependent Estimates, 238
7.4 Pre-Estimation of the Competence Regions, 239
7.4.1 Bespoke Classifiers, 240
7.4.2 Clustering and Selection, 241
7.5 Simultaneous Training of Regions and Classifiers, 242
7.6 Cascade Classifiers, 244
Appendix: Selected MATLAB Code, 244
7.A.1 Banana Data, 244
7.A.2 Evolutionary Algorithm for a Selection Ensemble for the
Banana Data, 245

8 Diversity in Classifier Ensembles 247

8.1 What Is Diversity?, 247
8.1.1 Diversity for a Point-Value Estimate, 248
8.1.2 Diversity in Software Engineering, 248
8.1.3 Statistical Measures of Relationship, 249
8.2 Measuring Diversity in Classifier Ensembles, 250
8.2.1 Pairwise Measures, 250
8.2.2 Nonpairwise Measures, 251
8.3 Relationship Between Diversity and Accuracy, 256
8.3.1 An Example, 256
xii CONTENTS

8.3.2 Relationship Patterns, 258

8.3.3 A Caveat: Independent Outputs ≠ Independent
Errors, 262
8.3.4 Independence Is Not the Best Scenario, 265
8.3.5 Diversity and Ensemble Margins, 267
8.4 Using Diversity, 270
8.4.1 Diversity for Finding Bounds and Theoretical
Relationships, 270
8.4.2 Kappa-error Diagrams and Ensemble Maps, 271
8.4.3 Overproduce and Select, 275
8.5 Conclusions: Diversity of Diversity, 279
Appendix, 280
8.A.1 Derivation of Diversity Measures for Oracle Outputs, 280
8.A.1.1 Correlation 𝜌, 280
8.A.1.2 Interrater Agreement 𝜅, 281
8.A.2 Diversity Measure Equivalence, 282
8.A.3 Independent Outputs ≠ Independent Errors, 284
8.A.4 A Bound on the Kappa-Error Diagram, 286
8.A.5 Calculation of the Pareto Frontier, 287

9 Ensemble Feature Selection 290

9.1 Preliminaries, 290
9.1.1 Right and Wrong Protocols, 290
9.1.2 Ensemble Feature Selection Approaches, 294
9.1.3 Natural Grouping, 294
9.2 Ranking by Decision Tree Ensembles, 295
9.2.1 Simple Count and Split Criterion, 295
9.2.2 Permuted Features or the “Noised-up” Method, 297
9.3 Ensembles of Rankers, 299
9.3.1 The Approach, 299
9.3.2 Ranking Methods (Criteria), 300
9.4 Random Feature Selection for the Ensemble, 305
9.4.1 Random Subspace Revisited, 305
9.4.2 Usability, Coverage, and Feature Diversity, 306
9.4.3 Genetic Algorithms, 312
9.5 Nonrandom Selection, 315
9.5.1 The “Favorite Class” Model, 315
9.5.2 The Iterative Model, 315
9.5.3 The Incremental Model, 316
9.6 A Stability Index, 317
9.6.1 Consistency Between a Pair of Subsets, 317
9.6.2 A Stability Index for K Sequences, 319
9.6.3 An Example of Applying the Stability Index, 320
Appendix, 322
 CONTENTS xiii

9.A.1 MATLAB Code for the Numerical Example of Ensemble

Ranking, 322
9.A.2 MATLAB GA Nuggets, 322
9.A.3 MATLAB Code for the Stability Index, 324

10 A Final Thought 326

References 327
Index 353
PREFACE

Pattern recognition is everywhere. It is the technology behind automatically identi-

fying fraudulent bank transactions, giving verbal instructions to your mobile phone,
predicting oil deposit odds, or segmenting a brain tumour within a magnetic resonance
image.
A decade has passed since the first edition of this book. Combining classifiers,
also known as “classifier ensembles,” has flourished into a prolific discipline. Viewed
from the top, classifier ensembles reside at the intersection of engineering, comput-
ing, and mathematics. Zoomed in, classifier ensembles are fuelled by advances in
pattern recognition, machine learning and data mining, among others. An ensem-
ble aggregates the “opinions” of several pattern classifiers in the hope that the new
opinion will be better than the individual ones. Vox populi, vox Dei.
The interest in classifier ensembles received a welcome boost due to the high-
profile Netflix contest. The world’s research creativeness was challenged using a
difficult task and a substantial reward. The problem was to predict whether a person
will enjoy a movie based on their past movie preferences. A Grand Prize of $1,000,000
was to be awarded to the team who first achieved a 10% improvement on the clas-
sification accuracy of the existing system Cinematch. The contest was launched in
October 2006, and the prize was awarded in September 2009. The winning solution
was nothing else but a rather fancy classifier ensemble.
What is wrong with the good old single classifiers? Jokingly, I often put up a slide
in presentations, with a multiple-choice question. The question is “Why classifier
ensembles?” and the three possible answers are:

(a) because we like to complicate entities beyond necessity (anti-Occam’s

razor);

xv
xvi PREFACE

(b) because we are lazy and stupid and cannot be bothered to design and train one
single sophisticated classifier; and
(c) because democracy is so important to our society, it must be important to
classification.

Funnily enough, the real answer hinges on choice (b). Of course, it is not a matter
of laziness or stupidity, but the realization that a complex problem can be elegantly
solved using simple and manageable tools. Recall the invention of the error back-
propagation algorithm followed by the dramatic resurfacing of neural networks in
the 1980s. Neural networks were proved to be universal approximators with unlim-
ited flexibility. They could approximate any classification boundary in any number
of dimensions. This capability, however, comes at a price. Large structures with
a vast number of parameters have to be trained. The initial excitement cooled
down as it transpired that massive structures cannot be easily trained with suffi-
cient guarantees of good generalization performance. Until recently, a typical neural
network classifier contained one hidden layer with a dozen neurons, sacrificing the so
acclaimed flexibility but gaining credibility. Enter classifier ensembles! Ensembles
of simple neural networks are among the most versatile and successful ensemble
methods.
But the story does not end here. Recent studies have rekindled the excitement
of using massive neural networks drawing upon hardware advances such as parallel
computations using graphics processing units (GPU) [75]. The giant data sets neces-
sary for training such structures are generated by small distortions of the available set.
These conceptually different rival approaches to machine learning can be regarded
as divide-and-conquer and brute force, respectively. It seems that the jury is still out
about their relative merits. In this book we adopt the divide-and-conquer approach.

THE PLAYING FIELD

Writing the first edition of the book felt like the overwhelming task of bringing
structure and organization to a hoarder’s attic. The scenery has changed markedly
since then. The series of workshops on Multiple Classifier Systems (MCS), run
since 2000 by Fabio Roli and Josef Kittler [338], served as a beacon, inspiration,
and guidance for experienced and new researchers alike. Excellent surveys shaped
the field, among which are the works by Polikar [311], Brown [53], and Valentini
and Re [397]. Better still, four recent texts together present accessible, in-depth,
comprehensive, and exquisite coverage of the classifier ensemble area: Rokach [335],
Zhou [439], Schapire and Freund [351], and Seni and Elder [355]. This gives me the
comfort and luxury to be able to skim over topics which are discussed at length and
in-depth elsewhere, and pick ones which I believe deserve more exposure or which I
just find curious.
As in the first edition, I have no ambition to present an accurate snapshot of the
state of the art. Instead, I have chosen to explain and illustrate some methods and
algorithms, giving sufficient detail so that the reader can reproduce them in code.
 PREFACE xvii

Although I venture an opinion based on general consensus and examples in the text,
this should not be regarded as a guide for preferring one method to another.

SOFTWARE

A rich set of classifier ensemble methods is implemented in WEKA1 [167], a collec-

tion of machine learning algorithms for data-mining tasks. PRTools2 is a MATLAB
toolbox for pattern recognition developed by the Pattern Recognition Research Group
of the TU Delft, The Netherlands, led by Professor R. P. W. (Bob) Duin. An industry-
oriented spin-off toolbox, called “perClass”3 was designed later. Classifier ensembles
feature prominently in both packages.
PRTools and perClass are instruments for advanced MATLAB programmers and
can also be used by practitioners after a short training. The recent edition of MATLAB
Statistics toolbox (2013b) includes a classifier ensemble suite as well.
Snippets of MATLAB DIY (do-it-yourself) code for illustrating methodologies
and concepts are given in the chapter appendices. MATLAB was seen as a suitable
language for such illustrations because it often looks like executable pseudo-code.
A programming language is like a living creature—it grows, develops, changes, and
breeds. The code in the book is written by today’s versions, styles, and conventions.
It does not, by any means, measure up to the richness, elegance, and sophistication
of PRTools and perClass. Aimed at simplicity, the code is not fool-proof nor is it
optimized for time or other efficiency criteria. Its sole purpose is to enable the reader
to grasp the ideas and run their own small-scale experiments.

STRUCTURE AND WHAT IS NEW IN THE SECOND EDITION

The book is organized as follows.

Chapter 1, Fundamentals, gives an introduction of the main concepts in pattern
recognition, Bayes decision theory, and experimental comparison of classifiers. A
new treatment of the classifier comparison issue is offered (after Demšar [89]). The
discussion of bias and variance decomposition of the error which was given in a
greater level of detail in Chapter 7 before (bagging and boosting) is now briefly
introduced and illustrated in Chapter 1.
Chapter 2, Base Classifiers, contains methods and algorithms for designing the
individual classifiers. In this edition, a special emphasis is put on the stability of the
classifier models. To aid the discussions and illustrations throughout the book, a toy
two-dimensional data set was created called the fish data. The Naı̈ve Bayes classifier
and the support vector machine classifier (SVM) are brought to the fore as they are
often used in classifier ensembles. In the final section of this chapter, I introduce the
triangle diagram that can enrich the analyses of pattern recognition methods.

1 http://www.cs.waikato.ac.nz/ml/weka/
2 http://prtools.org/
3 http://perclass.com/index.php/html/
xviii PREFACE

Chapter 3, Multiple Classifier Systems, discusses some general questions in com-

bining classifiers. It has undergone a major makeover. The new final section, “Quo
Vadis?,” asks questions such as “Are we reinventing the wheel?” and “Has the progress
thus far been illusory?” It also contains a bibliometric snapshot of the area of classifier
ensembles as of January 4, 2013 using Thomson Reuters’ Web of Knowledge (WoK).
Chapter 4, Combining Label Outputs, introduces a new theoretical framework
which defines the optimality conditions of several fusion rules by progressively
relaxing an assumption. The Behavior Knowledge Space method is trimmed down
and illustrated better in this edition. The combination method based on singular value
decomposition (SVD) has been dropped.
Chapter 5, Combining Continuous-Valued Outputs, summarizes classifier fusion
methods such as simple and weighted average, decision templates and a classifier used
as a combiner. The division of methods into class-conscious and class-independent
in the first edition was regarded as surplus and was therefore abandoned.
Chapter 6, Ensemble Methods, grew out of the former Bagging and Boosting
chapter. It now accommodates on an equal keel the reigning classics in classifier
ensembles: bagging, random forest, AdaBoost and random subspace, as well as a
couple of newcomers: rotation forest and random oracle. The Error Correcting Output
Code (ECOC) ensemble method is included here, having been cast as “Miscellanea”
in the first edition of the book. Based on the interest in this method, as well as its
success, ECOC’s rightful place is together with the classics.
Chapter 7, Classifier Selection, explains why this approach works and how clas-
sifier competence regions are estimated. The chapter contains new examples and
illustrations.
Chapter 8, Diversity, gives a modern view on ensemble diversity, raising at the
same time some old questions, which are still puzzling the researchers in spite of
the remarkable progress made in the area. There is a frighteningly large number of
possible “new” diversity measures, lurking as binary similarity and distance mea-
sures (take for example Choi et al.’s study [74] with 76, s-e-v-e-n-t-y s-i-x, such
measures). And we have not even touched the continuous-valued outputs and the
possible diversity measured from those. The message in this chapter is stronger now:
we hardly need any more diversity measures; we need to pick a few and learn how
to use them. In view of this, I have included a theoretical bound on the kappa-error
diagram [243] which shows how much space is still there for new ensemble methods
with engineered diversity.
Chapter 9, Ensemble Feature Selection, considers feature selection by the ensemble
and for the ensemble. It was born from a section in the former Chapter 8, Miscellanea.
The expansion was deemed necessary because of the surge of interest to ensemble
feature selection from a variety of application areas, notably so from bioinformatics
[346]. I have included a stability index between feature subsets or between feature
rankings [236].
I picked a figure from each chapter to create a small graphical guide to the contents
of the book as illustrated in Figure 1.
The former Theory chapter (Chapter 9) was dissolved; parts of it are now blended
with the rest of the content of the book. Lengthier proofs are relegated to the respective
 PREFACE xix

(a) Discriminant function set 1 class label

A fancy
Priors × p.d.f

0.2
A classifier? feature
combiner extractor?
0.1

0 x
0 2 4 6 8 10 classifier classifier classifier
Posterior probabilities

(b) Discriminant function set 2

1

0.5

feature values
0 x (object description)
0 2 4 6 8 10

1. Fundamentals 2. Base classifiers 3. Ensemble overview

BKS accuracy 0.8948 AdaBoost sampling distribution

0.182
Harmonic mean
Ensemble error

0.181

Geometric mean
(Product)
0.18
Average
rule
0.179
–2 0 2 4
Level of optimism α

4. Combining labels 5. Combining continuous 6. Ensemble methods

Individual Ensemble
4 κ -error bound
0.8
1

2 0.8 0.6
Stability index

0.6
Error

0 0.4
0.4
0.2
−2 0.2

0 0
−4 −1 0 1 0 20 40 60
−4 −2 0 2 4 kappa Number of features
7. Classifier selection 8. Diversity 9. Feature selection

FIGURE 1 The book chapters at a glance.

chapter appendices. Some of the proofs and derivations were dropped altogether, for
example, the theory behind the magic of AdaBoost. Plenty of literature sources can
be consulted for the proofs and derivations left out.
The differences between the two editions reflect the fact that the classifier ensemble
research has made a giant leap; some methods and techniques discussed in the first
edition did not withstand the test of time, others were replaced with modern versions.
The dramatic expansion of some sub-areas forced me, unfortunately, to drop topics
such as cluster ensembles and stay away from topics such as classifier ensembles for:
adaptive (on-line) learning, learning in the presence of concept drift, semi-supervised
learning, active learning, handing imbalanced classes and missing values. Each of
these sub-areas will likely see a bespoke monograph in a not so distant future. I look
forward to that.
xx PREFACE

I am humbled by the enormous volume of literature on the subject, and the

ingenious ideas and solutions within. My sincere apology to those authors, whose
excellent research into classifier ensembles went without citation in this book because
of lack of space or because of unawareness on my part.

WHO IS THIS BOOK FOR?

The book is suitable for postgraduate students and researchers in computing and
engineering, as well as practitioners with some technical background. The assumed
level of mathematics is minimal and includes a basic understanding of probabilities
and simple linear algebra. Beginner’s MATLAB programming knowledge would be
beneficial but is not essential.

Ludmila I. Kuncheva
Bangor, Gwynedd, UK
December 2013
ACKNOWLEDGEMENTS

I am most sincerely indebted to Gavin Brown, Juan Rodrı́guez, and Kami Kountcheva
for scrutinizing the manuscript and returning to me their invaluable comments, sug-
gestions, and corrections. Many heartfelt thanks go to my family and friends for their
constant support and encouragement. Last but not least, thank you, my reader, for
picking up this book.

Ludmila I. Kuncheva
Bangor, Gwynedd, UK
December 2013

xxi
1
FUNDAMENTALS OF PATTERN
RECOGNITION

1.1 BASIC CONCEPTS: CLASS, FEATURE, DATA SET

A wealth of literature in the 1960s and 1970s laid the grounds for modern pattern
recognition [90,106,140,141,282,290,305,340,353,386]. Faced with the formidable
challenges of real-life problems, elegant theories still coexist with ad hoc ideas,
intuition, and guessing.
Pattern recognition is about assigning labels to objects. Objects are described by
features, also called attributes. A classic example is recognition of handwritten digits
for the purpose of automatic mail sorting. Figure 1.1 shows a small data sample. Each
15×15 image is one object. Its class label is the digit it represents, and the features
can be extracted from the binary matrix of pixels.

1.1.1 Classes and Class Labels

Intuitively, a class contains similar objects, whereas objects from different classes
are dissimilar. Some classes have a clear-cut meaning, and in the simplest case are
mutually exclusive. For example, in signature verification, the signature is either
genuine or forged. The true class is one of the two, regardless of what we might
deduce from the observation of a particular signature. In other problems, classes
might be difficult to define, for example, the classes of left-handed and right-handed
people or ordered categories such as “low risk,” “medium risk,” and “high risk.”

Combining Pattern Classifiers: Methods and Algorithms, Second Edition. Ludmila I. Kuncheva.
© 2014 John Wiley & Sons, Inc. Published 2014 by John Wiley & Sons, Inc.

1
 Another Random Document on
Scribd Without Any Related Topics
 CHAPTER XX

POLITICAL REFORMS, 1479-1517

IT has already been pointed out that the union of Castile

and Aragon under the Catholic Kings lacked a real political or Tendency
toward
institutional basis. Both monarchs signed papers applicable to Spanish
the two kingdoms and exercised personal influence, each with unity under
the other, but although Ferdinand assisted his consort in Castile.
Castilian affairs, Isabella was clearly regarded as ruler in
Castile, as Ferdinand was in Aragon. The latter’s will advised Charles I to
maintain the separation of the kingdoms and to conduct their affairs through
native officials. Nevertheless, the long continuance of the same royal family
at the head of both was bound to produce a greater unity eventually. Castile
was drawn into European politics through the medium of the Aragonese
wars in Italy. On the other hand, she tended to become the centre of
authority and influence on account of the greater extent of her territory
(especially with the addition of Granada, Navarre, and the Americas), her
greater wealth, the royal practice of residing in Castile, and the more
advanced social and political condition of Castile as the result of Isabella’s
reforms.
Both sovereigns followed the policy of centralization in
their respective kingdoms. In Castile the major problem was Masterships
of the
the reduction of the oligarchical nobility, for the middle military
classes had already been won over in great part when Isabella orders
ascended the throne. Her success in reducing the lawless incorporated
into the
nobles has already been discussed; it only remains to point crown.
out the significance of the act by which she completed this
task,—her incorporation of the masterships of the military orders into the
crown. The principal element in the three great orders of Santiago,
Calatrava, and Alcántara were the segundones of great noble families and
members of the lesser nobility. Not only by their military power but also by
their numbers and wealth these orders constituted a potential danger to the
crown unless their action could be controlled. An estimate of the year 1493
showed that there were 700,000 members and vassals in the order of
Santiago, and 200,000 and 100,000 respectively in those of Calatrava and
Alcántara. The first-named had annual revenues of some 60,000 ducats
($900,000), and the two last combined, some 95,000 ($1,425,000). With the
masterships in royal hands the probability of civil strife was greatly
lessened.
As regards the towns the Catholic Kings followed
precisely the same practices which had been employed with Increase of
such success in the previous era. It was rare, indeed, that they the royal
authority
suppressed charters, but circumstances like those already and
recorded[52] enabled the corregidores and other royal officers tendency
toward unity
to exercise virtual control. Meanwhile, the process of in municipal
unification was going on through the ordinances of the Cortes life.
and royal decrees, fortified by the unrecorded development of
similarity in customs in Castilian municipal life. This was Decline of
furthered by the representatives of the towns themselves, for the Castilian
Cortes.
royal and municipal interests were usually in accord.
Noteworthy extensions of royal authority appeared in the subjection of local
officials to the residencia (or trial during a number of days after the
completion of a term of office, to determine the liability of an official for
the wrongful acts of his administration) and in the sending of royal
pesquisidores, or enquirers (in cases of crime), and veedores (inspectors),
later more often called visitadores (visitors), to investigate matters of
government, such as the accounts of financial agents and the conduct of
public officers. These institutions were later transferred to the Americas,
becoming an important means of sustaining the authority of the mother
country. In some instances the Catholic Kings resorted to force to reduce
municipalities which were too autonomous in character, notably in the case
of the hermandad of the north coast towns, whose decadence dates from
this reign.
The royalist ideal was manifested strikingly in the relations of the
Catholic Kings with the Castilian Cortes. From 1475 to 1503 the Cortes
was summoned but nine times, and during the years 1482 to 1498, at a time
when Granada was being conquered, America discovered and occupied, the
new Inquisition instituted, and the Jews expelled, it did not meet even once.
Its decline was evidenced still further in the increasingly respectful
language employed whenever it addressed the monarch and its growing
dependence on the Consejo Real, which body subjected the acts of the
Cortes to its own revision and whose president acted in a similar capacity
for the Cortes.
Ferdinand followed the same policy in Aragon. The
various Cortes of Aragon, Catalonia, and Valencia and the Decline of
general Cortes of all three were infrequently called; the king the Aragonese
acted in an arbitrary manner in his methods of raising funds, Cortes and
without observing the spirit of the laws. It was in his dealings of the power
of
with Barcelona that he most clearly manifested the royalist Barcelona.
tendency, for that city was the most powerful element in the
kingdom. Through his intervention the practice of electing the five
concelleres, or councillors, was suspended in favor of royal appointment,
and the Consell, or council of a hundred, was altered so that it was no
longer democratic but represented the will of the monarch. The fact that
these changes were made without provoking resistance and almost without
protest shows how utterly dead were the political ideals of the past.
The concentration in royal hands of so many powers which
were formerly exercised by the lords and towns made The new
necessary the development of a numerous and varied bureaucracy.
officialdom to assist the monarch. As the basis of the new bureaucracy in
Castile the Catholic Kings had at hand the Consejo Real, which with some
changes was admirably adapted to the purpose. The first step was to rid it of
the great nobles. In 1480 the untitled letrados became a majority in this
body. The counts, dukes, and marquises were still allowed to attend, but
were deprived of the right to vote. Shortly afterward they were excluded
altogether, and the Consejo Real now responded without question to the
will of the king. It served as the head of the various branches of the
bureaucratic organization, with the final decision, subject to the wishes of
the king, in all matters of government. Pressure of work led to the formation
of three additional councils, those of the Inquisition (Inquisición), the
military orders, (Órdenes Militares), and the Americas, or Indies (Indias),
while there were still others in the kingdom of Aragon. Particularly
important among the other officials was the monarch’s private secretary,
who came to have a very nearly decisive influence, owing to the favor he
enjoyed with the head of the state. A horde of other officers, old and new,
made up the ranks of the bureaucracy. Among the older group it is to be
noted that the adelantados were supplanted by alcaldes mayores, until only
one of the former was left. Among newer officials the important inquisitors
and veedores, or visitadores, should be noted.
A similar development to that of the executive branch was
experienced in the administration of justice. The fountain- Administrati
head was the chancillería at the capital, Valladolid, to which on of justice.
were subordinate in a measure the several regional audiencias, which were
now established for the first time, besides the hierarchy of the judiciary of
lower grades. In addition to unifying and regulating the judicial system the
Catholic Kings gave attention to the internal purification of the courts, with
a view to eliminating the unfit or undesirable and to checking abuses. The
corrupt practices of those outside the courts were also attacked, especially
powerful persons who attempted to overawe judges or procure a
miscarriage of justice. One of the principal difficulties encountered was that
of conflicts of jurisdiction, notably in the case of the church courts. Good
Catholic though she was, Isabella was determined in her opposition to
ecclesiastical invasions of royal jurisdiction, but despite her energetic
measures the issue was far from being decided in her day. In line with the
royal policy of settling disputes by law rather than by force the use of
firearms was prohibited, gambling was persecuted, and the riepto (or
judicial duel, the last survival of medieval procedure) was abolished. Good
order in the present-day sense was far from existing, and this led to a
revival of the medieval idea of the hermandades for the punishment of
crimes committed in uninhabited places or small villages as well as for the
pursuit and execution generally of those guilty of felony. The Santa
Hermandad, with its capital at Toledo, was created as a kind of judicial
body, sustained by the groups of citizens who formed part of it, employing a
militia of mounted men, and making use of summary methods and extreme
penalties in its procedure. Its life as an effective body was brief, although it
continued to exist for many years. On the other hand the medieval
hermandad of Toledo enjoyed a revival of life and usefulness.[53]
It is hardly necessary to trace the administrative and
judicial reforms of Ferdinand in Aragon. Suffice to say that Reforms in
they followed the Castilian pattern much more closely, Aragon.
indeed, than in the matter of social organization.
The Castilian Inquisition, first created in 1478 for specific
and temporary objects, underwent considerable modification Procedure of
the
when retained as a permanent body to combat heresy in Inquisition.
general. The popes refused to allow it to be in all respects a
royal instrument, and retained the right of appointing or dismissing
inquisitors, permitting the kings to recommend candidates. The expansion
of the institution from Seville to other cities in Spain and the creation of a
supreme council of the Inquisition have already been mentioned. Ximénez,
who became head of the Inquisition of Castile in 1507, extended its
operations to Africa and the Americas. The methods of trial were harsh,
though less so if gauged by the standards of that time. Torture was used as a
means of obtaining confessions. The accused was kept utterly apart from his
family and friends, who did not learn what had become of him until his
liberation or his appearance in an auto de fe. The same secrecy was
employed in dealing with the prisoner, who was informed of the general
charge against him, without the details and without knowing his accuser’s
name. He was allowed to indicate those in whom he lacked confidence, and
if he should chance to hit upon an accuser that person’s evidence was
eliminated. Two witnesses against him were sufficient to outweigh any
testimony he might give. He might have a lawyer, but could not confer with
him in private. He might also object to a judge whose impartiality he had
reason to suspect, and could appeal to the pope. Penalties varied from the
imposition of a light penance to imprisonment or burning to death. Burning
in effigy of those who escaped or burning of the remains of those who had
died was also practised. The auto de fe represented, as the words imply,
merely the decision in the given case, and not the imposition of the penalty
as has often been stated. The general rule was for the executions to take
place on holidays, which in Spain are indeed “holy days,” or days in
celebration of events in church history. A procession was held, in which the
functionaries of the Inquisition took part. A public announcement of the
decisions was made, and those who were condemned to death were turned
over to the civil authorities, who carried out the execution in the customary
place. As has already been said, the imposition of sentences was
accompanied by confiscations or the levy of fines. Since the Inquisition was
supported by these amercements there were numerous scandals in
connection therewith. Certain royal orders implied, and complaints by men
of such standing as Juan de Daza, bishop of Cordova, directly charged, that
the Inquisition displayed a too great eagerness to insure its financial
standing by confiscations. On one occasion it seems that the estate of a
wealthy victim of the Inquisition was divided between Cardinal Carvajal,
the inquisitor Lucero, the royal treasurer Morales, and Ferdinand’s private
secretary. The funds did not belong in law to the Inquisition. That body
collected them and turned them over to the king, who granted them back
again.
The new Castilian and Aragonese states required greatly
increased funds and a royal army, and both of these matters Financial
received the careful consideration of Ferdinand and Isabella. administratio
n.
In financial affairs their activities were twofold: to procure
more revenues; and to bring about greater economy in their collection and
administration. The revocation of earlier land grants was one measure
productive of income, since the taxes from them now went to the crown
rather than to the lords. Two sources of revenue of a religious character
were procured by papal grant. One of these was the cruzada, or sale of
indulgences, based on the crusade (cruzada) against the Moslems. Designed
for a temporary purpose it became an enduring element in the royal income.
The other was the diezmo, or tithe, presumably for the same objects as the
cruzada, although it too was diverted to other uses. Great attention was paid
to the administration of the remunerative alcabala, and to stamp taxes and
customs duties. The treasury department as a modern institution may be
said to date from this era. In addition the Catholic Kings corrected abuses in
the coinage of money. The final result is shown in the increase in the
revenues from about 900,000 reales[54] in 1474 to well over 26,000,000 in
1504. Expenses were so heavy, however, that more than once a resort to
loans was necessary.
The army kept pace with other institutions in the advance
out of medievalism into modernity. The seigniorial levies, Modernizati
unequal in size and subversive of discipline as well as a on of the
army.
potential danger, were virtually done away with after the
Granadine war, although such bodies appeared occasionally The royal
even in the next era. In their place were substituted a larger navy.
royal army at state expense and the principle of universal
military service. One man in every twelve of those between twenty and
forty years of age was held liable, but did not take the field and was not
paid except when specifically called. The glory of the new professional
army attracted many who had formerly served the great lords, including a
number of the nobility and the adventurous element. Under the leadership
of Gonzalo de Ayora and especially of the “great captain,” Gonzalo de
Córdoba, noteworthy reforms in tactics were made. The army was now an
aggregation of equal groups, based on battalions and companies, while the
larger divisions were assigned a proportionate number of infantry, cavalry,
and artillery. From this period date many current military titles: colonel,
captain, and others. Arms and equipment were much improved and military
administration bettered. The importance of firearms was just becoming
recognized; cannon, firing balls of stone, played a prominent part in the war
with Granada. A similar if less pronounced development appeared in the
navy. The admiral of Castile, who had enjoyed a semi-independent
sinecure, now lost much of his authority, for many of his powers were taken
over by the crown.
The reforms which have been chronicled were the result of
a great body of legislation, most of which emanated directly The
Ordinance of
from the crown, although some important laws were enacted Montalvo
in conjunction with the Cortes. Taken with the variety of and other
legislation in preceding years it caused not a little confusion codifications
of the laws.
as to the precise principle governing a specific case. This led
to the compilation by Alfonso Díaz de Montalvo of the Ordenanzas Reales
de Castilla (1484?), or Royal Ordinances of Castile, commonly called the
Ordinance (Ordenamiento) of Doctor Montalvo, in which were set forth
various ordinances of the Cortes since that of Alcalá in 1348 and certain
orders of the kings from the time of Alfonso X, together with some
provisions of earlier date. In all, 1163 laws were included, of which 230
belonged to the era of the Catholic Kings. Although it is not certain, the
Ordenanzas seems to have been promulgated as law, and in any event was
very influential, running through thirteen editions down to the year 1513.
The compilation was far from meeting the full requirement of the times,
however. Besides being incomplete, as was only to be expected, it
contained various inaccuracies of form and substance. Furthermore, with
such varying elements still in effect as the Partidas and the medieval fueros,
besides the unwritten transformation and unification which had been going
on for two centuries (as a result of royalist policies), there was need for a
clear and methodical revision of Castilian legislation. Various other
publications covering special phases of the laws, such as the Ordenanzas de
Alcabalas (1491), or Ordinances of the Alcabala, the already mentioned
Leyes de Toro (1505), and the privileges of the Mesta (1511), date from this
era, while there was a similar tendency toward legislative publication in the
Catalonian and Valencian parts of the kingdom of Aragon.
Although the piety of Ferdinand and Isabella earned them
the sobriquet of the “Catholic Kings,” particularly merited in Relations of
the case of Isabella, they did not let their regard for the church church
state.
and

interfere with their conceptions of the royal authority.

Something has already been said about their resistance to the intrusions of
ecclesiastical courts and their objection to appointments of foreigners to
Spanish benefices. The same conflict with the pope was maintained with
regard to papal appointments of Spaniards. In the case of Granada and the
Americas the crown gained the patronato real, or royal patronage, in such
degree that the monarch became the virtual administrative head of the
church, but the concession for the rest of Spain was not so complete.
Nevertheless, the royal nominees were usually appointed. The Catholic
Kings displayed great consideration for the church when the interests of the
latter did not run counter to the monarchical ideal, and in Castile the
confessors of the queen obtained a certain ascendency which made them
among the most powerful individuals in the state. They proved to be well
deserving of their influence, however, notably cardinals Mendoza, Talavera,
and Ximénez, of whom the last-named was, after the Catholic Kings, by far
the most important figure of the times.
 CHAPTER XXI

MATERIAL AND INTELLECTUAL PROGRESS, 1479-1517

THE Catholic Kings attacked the economic problems of

their era with much the same zeal they had displayed in social Economic
medievalism
and political reforms, but without equal success, for .
medievalism in material affairs was more persistent than in
social, political, and intellectual institutions. The same false Privileges of
economic ideas of the past were still operative. Especially the Mesta.
was this manifest in the belief that legislation and state
intervention in business provided a panacea for all evils, when the real
needs were the development of the wealth at hand and the modification of
geographical conditions in such a way as to permit of additional
productivity. Protection and excessive regulation were the keynote of the
laws. As a result manufactures were stimulated on the one hand, and
various cities of the two kingdoms became notable industrial centres, but on
the other hand, these same industries were hindered by inspections, by laws
regulating the fashion and style of goods and fixing prices, wages, and the
hours of labor, and by a host of other measures which killed initiative and
hindered rapidity of work. In part to promote this artificial industrial life, so
that raw wool might be readily procured, the Catholic Kings recognized and
even extended the privileges of the great corporation of the Mesta. Starting
from La Mancha and Extremadura in April, flocks of sheep annually
ravaged Castile, returning in September to the place whence they had come.
The cañada real, or royal sheepwalk, was set aside for their exclusive use,
and a prohibition was placed on clearing, working, or enclosing any part of
that strip. In fact the sheepmen ventured beyond the legal limits, and
although required by law to pay damages in such cases were so powerful
that they rarely did so. Withal, the stimulus to manufacturing was almost
purely artificial, and the Spanish cities, even Barcelona, found competition
with foreign cloths and other goods too keen. In the main, Spain continued
to be a raw material land, exporting primary articles to foreign countries, in
return for manufactures.
 Attempts were made to encourage agriculture, but the
spirit of legislative interference and the superior importance Lack of
accorded the grazing industry were not conducive to progress. progress in
agriculture.
The menace of the Mesta was responsible for the almost
complete destruction of forestry and agriculture in many regions which
were suitable to development in those respects, while the irrigation ditches
of Andalusia and other former Moslem lands were too often allowed to
decay.
The same royal solicitude appeared, to assist and to retard
commerce. Interior customs lines were to some extent done Vicissitudes
away with, notably on the frontier of Castile and Aragon of commerce.
proper. Shipbuilding was encouraged, but favors were shown
to owners of large ships, wherefore the smaller ship traffic was damaged, at
the same time that the larger boats were too big for the needs of the trade. A
flourishing foreign commerce developed, nevertheless, but it was in the
hands of the Jews and, after their expulsion, of foreigners of Italian,
Germanic, and French extraction. Many laws were passed subjecting
foreigners to annoyances, lest they export precious metal or in other ways
act contrary to the economic interests of the peninsula as they were then
understood. It was in this period that the commerce of the Mediterranean
cities of the kingdom of Aragon sank into a hopeless decline. Other factors
than those of the false economic principles of the day were primarily
responsible, such as the conquests of the Turks, which ended the eastern
Mediterranean trade, and the Portuguese discovery of the sea-route to India,
along with the Castilian voyages to America, which made the Atlantic
Ocean the chief centre of sea-going traffic and closed the era of
Mediterranean supremacy.
Nevertheless, the net result of the period was a marked
advance in material wealth,—in part, perhaps, because the Advance in
false economic ideas of the Catholic Kings were shared by wealth.
them with the other rulers of Europe, wherefore they did not prove so great
a handicap to Spain, and, in part, because some of their measures were well
calculated to prove beneficial. At this time, too, the wealth of the Americas
began to pour in, although the future was to hold far more in store.
Brief as was the span of years embraced by the reign of the
Catholic Kings it was as notable a period in intellectual Extension of
progress as in other respects, bringing Spain into the current intellectual
of modern life. This was due primarily to the rapid extension culture and
the triumph
of printing, which had appeared in the peninsula in the closing of
years of the preceding period, and which now came into such Humanism.
general use that the works of Spanish and classical writers
became available to all. Through private initiative many schools were
founded which later became universities, although this activity was limited
to Castile. Most notable of these institutions was that of Alcalá founded by
Ximénez. This undertaking was due to the great cardinal’s desire to
establish a Humanist centre of learning, where Latin, Greek, Hebrew, and
philology could be studied to the best advantage. The most learned Spanish
Humanists assembled there, together with many foreigners, and works of
note were produced, such as the famous polyglot Bible in Hebrew, Greek,
Chaldean, and Latin, with accompanying grammars and vocabularies. Not a
little of the advancement in intellectual manifestations was due to the
encouragement of the Catholic Kings, especially Isabella. Books coming
into Spain were exempted from duty; ordinances were made regulating
university life, and ridding it of much of its turbulence and abuses; and the
court set an example in showing favor to distinguished scholars, who were
engaged as teachers of the royal children. The great nobles imitated royalty,
and invited foreign savants to Spain, among whom was the Italian, Peter
Martyr of Anghiera, celebrated as the author of the first history of the
Americas, the De orbe novo (Concerning the new world). The most marked
impulse to the spread of Humanist ideals came through Spaniards studying
abroad, and these men returned to give Spain her leading names in
intellectual production for the period. The greatest of them was Antonio de
Nebrija, educated in Italy, a man of such encyclopedic attainments that he
left works on theology, law, archæology, history, natural science, geography,
and geodesy, although particularly noteworthy as a Latin scholar. Cardinal
Ximénez is deserving of a high place in the achievements of the era for his
patronage of letters, for it was through his aid that some of the most
valuable work of the period was accomplished. Education was a matter for
the higher classes only; people had not even begun to think, yet, of popular
education.
Although the extension of intellectual culture and the
triumph of Humanism were outstanding facts of the period, Progress in
there were notable cultivators, too, of the sciences, moral, the sciences.
social, and natural, especially the last-named. Studies in geography,
cosmography, and cartography received a great impulse through the
discovery of America, and many scientific works along these lines were due
to the scholars connected with the Casa de Contratación (House of Trade),
or India House. Medical works were even more prominent, not a few of
them on the subject of venereal disease. A number of these works were
mutilated or condemned altogether by the Inquisition, in part because of
their doctrines, but also because of the anatomical details which they
contained, for they were considered immoral.
In polite literature the leading characteristics were the
complete victory of the Italian influence, the predominance of Polite
Castilian, the popularity of the romances, and the beginning literature.
of the Castilian theatre. The Italian influence manifested itself
both in the translation of Classical and Italian Renaissance La Celestina.
works and in an imitation of their models and forms. Castilian
was employed, not only in Castile and Aragon proper, but History.
even in the literary works of Portuguese, Catalans,
Valencians, and not a few individuals (Spaniards in the main) The theatre.
at the court of Naples, although Catalan and Valencian poetry
still had a vogue. The poetry of the era often exhibited tendencies of a
medieval character,—for example, in its use of allegory. It is curious to note
also the prevalence of two somewhat opposed types of subject-matter,
religious and erotic; in the latter there was a vigorous school which often
went to the extreme of license. The romances of love and chivalry gained
even greater favor than in the preceding period. The Amadís de Gaula
(Amadis of Gaul) of Vasco de Lobeira was translated from the Portuguese
by Garcí Ordóñez de Montalvo, and many other novels on the same model
were written. One of these was Las sergas de Esplandián (The deeds of
Esplandián) by Ordóñez de Montalvo himself, references in which to an
“island California” as a land of fabulous wealth were to result in the naming
of the present-day California, once believed to be just such an island. Much
superior to the amatory or chivalric novels was a remarkable book which
stood alone in its time, the Tragicomedia de Calixto y Melibea (The tragi-
comedy of Calixtus and Melibea), better known as La Celestina (1499),
from the name of one of the characters, believed to have been the work of
Fernando de Rojas. In eloquent Spanish and with intense realism La
Celestina dealt with people in what might be called “the under-world.” This
was the first of the picaresque novels (so-called because they dealt with the
life of pícaros, or rogues), out of which was to develop the true Spanish
novel. History, too, had a notable growth. The outstanding name was that of
Hernando del Pulgar. His Crónica (Chronicle) and his Claros varones de
España (Illustrious men of Spain), besides being well written, noteworthy
for their characterizations of individuals, and influenced by classical Latin
authors, showed a distinct historical sense. The already mentioned De orbe
novo of Peter Martyr and the letters of Columbus were the chief
contributions to the history of the new world. As to the theatre, while the
religious mysteries continued to be played, popular representations in
dialogue, some of them religious and others profane in subject-matter,
began to be written and staged. The most notable writer was Juan del
Enzina (1468-1534), who has been called the “father of Spanish comedy.”
His compositions were not represented publicly in a theatre, but only in
private houses or on the occasions of royal or aristocratic feasts.
The transitional character of the age was nowhere more
clear than in the various forms of art. The principal Plateresque
architectural style was a combination of late Gothic with early architecture.
Renaissance features, which, because of its exuberantly
decorative character, was called plateresque, for many of its Sculpture
and the
forms resembled the work of plateros, or makers of plate. lesser arts.
Structurally there was a mingling of the two above-named
elements, with a superimposition of adornment marked by great profusion
and richness,—such, for example, as in the façade of the convent of San
Pablo of Valladolid. At the same time, edifices were still built which were
more properly to be called Gothic, and there were yet others predominantly
representative of the Renaissance, characterized by the restoration of the
later classical structural and decorative elements, such as the slightly
pointed arch, intersecting vaults, columns, entablatures, pediments, and
lavish ornamentation. Sculpture displayed the same manifestations, and
became in a measure independent of architecture. Noteworthy survivals are
the richly carved sepulchres of the era. Gold and silver work had an
extraordinary development not only in articles of luxury but also in those
for popular use, and as regards luxury the same was true of work in rich
embroideries and textures.
The contest between the Flemish and Italian influences on
Spanish painting resolved itself decidedly in favor of the Advance in
latter, although a certain eclecticism, the germ of a national painting.
school, made itself apparent in the works of Spanish artists.
Characteristics of a medieval type still persisted, such as Music.
faulty drawing, color lacking in energy and richness, a sad
and sober ambient, and a disregard for everything in a painting except the
human figures. Like sculpture, painting began to be dissociated from
architecture, and was encouraged by the purchases of the wealthy. It was
not yet the custom to hang paintings on the walls; they were kept in chests
or otherwise under lock and key except when brought out for temporary
display. Music, employed principally in song as the accompaniment of
verse, enjoyed a favor comparable with that of the plastic arts.
 CHAPTER XXII

CHARLES I OF SPAIN, 1516-1556

FROM the standpoint of European history the period of the

House of Hapsburg, or Austria, covering nearly two centuries, Historical
setting of the
when Spain was one of the great powers of the world, should era of the
be replete with the details of Spanish intervention in European House of
affairs. The purposes of the present work will be served, Austria.
however, by a comparatively brief treatment of this phase of
Spanish history; indeed, the central idea underlying it reduces itself to this:
Spain wasted her energies and expended her wealth in a fruitless attempt,
first to become the dominant power in Europe, and later to maintain
possessions in Italy and the Low Countries which were productive only of
trouble; what she took from the Americas with the one hand, she
squandered in Europe with the other. Internally there were changes which
were to react on the Spanish colonial dominions, wherefore a
correspondingly greater space must be accorded peninsula history than
directly to the wars in Europe. The greatest feature of the period was the
conquest of the Americas, accomplished in part by the spectacular
expeditions of the conquistadores, or conquerors, and in part by the slower
advance of the Spanish settlers, pushing onward the frontier of profits. Not
only was this the most notable achievement when considered from the
American angle, but it was, also, when taken from the standpoint of Spain,
and possibly, too, from that of Europe and the world.
The Italian venture of the Aragonese kings had yielded
probably more of advantage than of harm down to the time of Vast empire
of Charles I
Ferdinand, and it may be that even he did not overstep the of Spain, the
bounds of prudence in his ambitious designs. When his Emperor
policies were continued, however, in the person of Charles I, Charles V.
better known by his imperial title as the Emperor Charles V,
the results were to prove more disastrous to Spain than beneficial. The
circumstances were in fact different for the two monarchs, although their
aims were much the same. Some writers have supposed that Ferdinand
himself recognized the danger of a union of the Austrian, Burgundian, and
Spanish dominions under one king, and they assert that he planned to make
Charles’ younger brother, Ferdinand, ruler of Spain and the Two Sicilies in
case the former should be elected emperor. In his will, however, he
respected the principle of primogeniture, and left all to Charles, eldest son
of Philip the Handsome and Juana la Loca. Through his mother and
Ferdinand, Charles inherited Castile, Aragon, and Navarre, the Castilian
dominions in Africa and America (where the era of great conquests was just
about to begin), the Roussillon and Cerdagne across the Pyrenees, and
Sardinia, Sicily, and Naples in Italy; through his father he had already
become possessed of the territories of the House of Burgundy, comprised of
Flanders and Artois in northern France, Franche-Comté and Charolais in the
east, Luxembourg, and the Low Countries. This was not all, for Charles was
heir of the Emperor Maximilian, and in addition to inheriting the latter’s
Austrian dominions might hope to succeed to the imperial title as ruler of
the Holy Roman Empire. To be sure, the system of electing the emperors by
the electoral princes still obtained, but the Germanic states of the empire
were almost certain to prefer a powerful Hapsburg, with such dominions as
Charles had, to any other candidate, if only to serve as a counterpoise to the
ambitions of France. Nevertheless, the electors did not miss the opportunity
to make a profit out of the situation, and encouraged the candidacy of
Henry VIII of England and especially of Francis I of France as well as that
of Charles, receiving bribes and favors from all. In the end, following the
death of Maximilian in 1519, they decided in favor of Charles. He was now
ruler, at least in name, of one of the most vast empires in the history of the
world.
The mere possession of such extensive domains inevitably
led to an imperialistic policy to insure their retention. Each of Inherent
the three principal elements therein, Spain, Burgundy, and the weakness of
his empire.
Austrian dominions, was ambitious in itself and especially
hostile to France, and all of these aspirations and enmities were now
combined in a single monarch. Charles himself was desirous not only of
conquest but also of becoming the most powerful prince in the world, thus
assuring the Hapsburg supremacy in Europe, and making himself the arbiter
in European political affairs and the protector of Christianity; he may even
have dreamed of a world monarchy, for if he did not aspire to such a state
for himself he believed its attainment possible of realization. In the
achievement of a less vast ideal, however, Charles was certain to experience
many difficulties, and at some point or other was bound to encounter the
hostility not only of France but also of the other states of Europe. If this
were not enough there came along the unforeseen dilemma of the
Reformation. Finally, his own dominions were none too strongly held
together, one with another or within themselves. They were widely
separated, some indeed entirely surrounded by French territory, leading to a
multiplicity of problems of a military and a political nature. The imperial
rank carried little real authority in Germany, and the Burgundian realms
were not a great source of power. It appears, therefore, that the empire was
more a matter of show than of strength, and that Spain, who already had a
surfeit of responsibility, what with her conquests in Italy, Africa, and the
Americas, must bear the burden for all. The reign of Charles would seem to
be the parting of the ways for Spain. If she could have restricted herself to
her purely Spanish inheritance, even with the incubus of her Italian
possessions, she might have prolonged her existence as a great power
indefinitely. A century ahead of England in colonial enterprise, she had such
an opportunity as that which made the island of Britain one of the dominant
factors in the world. Even as matters were, Spain was able to stand forth as
a first rank nation for well over a century. Whatever might have happened if
a different policy had been followed, it hardly admits of doubt that Spain’s
intervention in European affairs involved too great a strain on her resources,
and proved a detriment politically and economically to the peninsula.
Charles had been brought up in Flanders, and, it is said,
was unable to speak Spanish when he first entered the Dissatisfacti
peninsula as king of Spain. His official reign began in 1516, on over
foreign
but it was not until his arrival in the following year that the favorites and
full effect of his measures began to be felt. Even before that increased
taxation.
time there was some inkling of what was to come in the
appointments of foreigners, mostly Flemings, to political or ecclesiastical
office in Castile. At length Charles reached Spain, surrounded by Flemish
courtiers, who proceeded to supplant Spaniards not only in the favor but
also in the patronage of the king. The new officials, more eager for personal
profit than patriotic, began to sell privileges and the posts of lower grade to
the highest bidders. Such practices could not fail to wound the feelings of
Spaniards, besides which they contravened the laws, and many protests by
individuals and towns were made, to which was joined the complaint of the
Cortes of Valladolid in 1518. To make matters worse Chièvres, the favorite
minister of the king, caused taxes to be raised. The amount of the alcabala
was increased, and the tax was made applicable to the hitherto privileged
nobility, much against their will. In like manner the opposition of the clergy
was roused through a bull procured from the pope requiring ecclesiastical
estates to pay a tenth of their income to the king during a period of three
years. Furthermore it was commonly believed, no doubt with justice, that
the Flemish office-holders were sending gold and other precious metals out
of the country, despite the laws forbidding such export. Nevertheless the
Cortes of 1518 granted a generous subsidy to the king, but this was
followed by new increases in royal taxation. Opposition to these practices
now began to crystallize, with the nobles of Toledo taking the lead in
remonstrance against them.
The situation in Castile was complicated by the question of
the imperial election. Between the death of Maximilian in Charles’
January, 1519, and the election of Charles in June of the same manipulation
of the Cortes
year it was necessary to pay huge bribes to the electoral in Galicia.
princes. Once chosen, Charles accepted the imperial honor,
and prepared to go to Germany to be crowned, an event which called for yet
more expenditures of a substantial nature. So, notwithstanding the grant of
1518, it was decided to call the Cortes early in 1520 with a view to a fresh
subsidy. Since all Castile was in a state of tumult it was deemed best that
the meeting should take place at some point whence an escape from the
country would be easy in case of need. Thus Santiago de Compostela in
Galicia was selected, and it was there that the Cortes eventually met,
moving to the neighboring port of Coruña after the first few days’ sessions.
The call for the Cortes provoked a storm of protest not only by Toledo but
also by many other cities with which the first-named was in
correspondence. Messengers were sent to the king to beg of him not to
leave Spain, or, if he must do so, to place Spaniards in control of the affairs
of state, and complaints were made against the practices already recounted
and numerous others, such, for example, as the royal use of the title
“Majesty,” an unwonted term in Spain. From the first, Charles turned a deaf
ear, refusing to receive the messengers of the towns, or reproving them
when he did give them audience, and he even went so far as to order the
arrest of the Toledan leaders. The Cortes at length met, and gave evidence
of the widespread discontent in its demands upon the king. In accordance
with their instructions most of the deputies were disinclined to take up the
matter of a supply for the king until he should accede to their petitions.
Under the royal eye, however, they gradually modified their demands, and
when Charles took it upon himself to absolve them from the pledges they
had given to their constituents they voted the subsidy without obtaining any
tangible redress of grievances. The king did promise not to appoint any
foreigners to Spanish benefices or political holdings during his absence, but
broke his word forthwith when he named Cardinal Adrian, a foreigner, as
his representative and governor during his absence. This done, Charles set
out in the same year, 1520, for Germany.
Meanwhile, a riot in Toledo, promoted by the nobles
whom Charles had ordered arrested, converted itself into a War of the
veritable revolt when the royal corregidor was expelled from Comunidade
s in Castile.
the city. This action was stated to have been taken in the name
of the Comunidad, or community, of Toledo, and served to give a name to
the uprising which now took place in all parts of Castile. Deputies to the
Cortes who had been faithless to their trust, some of whom had accepted
bribes from the king, were roughly handled upon their return home, and city
after city joined Toledo in proclaiming the Comunidad. In July, 1520,
delegates of the rebellious communities met, and formed the Junta of Ávila,
which from that town and later from Tordesillas and Valladolid served as
the executive body of the revolution. For a time the Junta was practically
the ruling body in the state; so complete was the overturn of royal authority
that Cardinal Adrian and his advisers made no attempt to put down the
rebellion. Time worked to the advantage of the king, however. The revolt of
Toledo had begun as a protest of the nobles and clergy against the
imposition of taxes against them. The program of the Junta of Ávila went
much further than that, going into the question of the grievances of the
various social classes. At length many of the comuneros began to indulge in
acts of violence and revenge against those by whom they regarded
themselves as having been oppressed, and the movement changed from one
of all the classes, including the nobles, against the royal infractions of law
and privileges, to one of the popular element against the lords. Thus the
middle classes, who objected to the disorder of the times as harmful to
business, and the nobles, in self-defence, began to take sides with the king.
City after city went over to Charles, and late in 1520 the government was
strong enough to declare war on the communities still faithful to the Junta.
Dissension, treason, and incompetent leadership furthered the decline of the
popular cause, and in 1521 the revolt was crushed at the battle of Villalar.
Charles promised a general pardon, but when he came to Spain in 1522 he
caused a great many to be put to death. Not until 1526 did he show a
disposition to clemency. Moreover he retained his Flemish advisers.
During the period of the revolt of the Comunidades in
Castile even more bitter civil wars were going on in Valencia Social wars
(1520-1522) and Majorca (1521-1523). The contest in in Valencia
and Majorca.
Valencia was a social conflict from the start, of plebeians
against the lords, whereas the Castilian conflict was fundamentally political.
In Majorca the strife began over pressure for financial reforms, but
developed into an attempt to eliminate the nobility altogether. Both
uprisings were independent of the Castilian revolt, although serving to aid
the latter through the necessary diversion of troops. As in Castile, so in
Valencia and Majorca, Charles took sides against the popular element, and
put down the insurrections, displaying great severity toward the leaders.
While the civil wars were at their height Charles was
having more than his share of trouble in other quarters. The Charles’
princes of Germany compelled him to sign a document difficulties in Germany
affirming their privileges, in which appeared many and war with
paragraphs similar to those of the Castilian petitions to the France.
king, together with one requiring Charles to maintain the
empire independently of the Spanish crown. The acceptance of these
principles by the emperor is an evidence of the weakness of his authority in
the subject states of Germany, for not only was he a believer in the divine
origin of the imperial dignity, a doctrine which would have impelled him to
establish his personal and absolute rule in all of his realms if possible, but
he seems also to have intended to make Spain the political centre of his
dominions, because she was, after all, his strongest element of support. At
the same time, a fresh difficulty appeared in Germany with the Lutheran
outbreak of 1521. Charles himself favored reform in the church, but was
opposed to any change in dogma. Before he could confront either the
political or the religious problem in Germany, he found himself attacked on
another quarter. Francis I of France had seized upon Charles’ difficulties as
affording him a rich opportunity to strike to advantage; so in 1521 he twice
sent French armies into Spain through the western Pyrenees on the pretext
of restoring the crown of Navarre to the Labrit family. With all these
questions pressing for solution Charles was in an exceedingly unsatisfactory
position. Thus early in the period lack of funds to prosecute European
policies was chronic. Spain herself, even if there had been no civil wars,
was not united internally like the compact French nation, and the other
Hapsburg dominions could give but little help. Finally, Charles could not
depend on the alliance of any other power, for his own realms were
neighbors of all the others, and his designs were therefore generally
suspected. Nevertheless, Charles brought to his many tasks an indomitable
will, marked energy, a steadfast purpose, and an all-round ability which
were to do much toward overcoming the obstacles that hindered him.
It is profitless, here, to relate the course of the wars with
France and other European states. In the years 1521 to 1529, Wars with
1536 to 1538, and 1542 to 1544, France and Spain were at France, pope, the
the

war, and at other times, down to the death of Francis I in Italian states,
1547, the two countries enjoyed what was virtually no more and German
princes.
than a truce. Meanwhile, Charles was usually in conflict with
the popes, whose temporal dominions in central Italy were
threatened by the growing power of Spain and the empire in The outcome.
the Italian peninsula. Other states in Italy fought now on
Charles’ side, now against him, while the princes of Germany were an
equally variable quantity. England favored each side in turn, but offered
little effective aid to either. As affecting the history of religion these wars
gave Protestantism a chance to develop. Neither Charles nor Francis
disdained the aid of Protestant princes, and the former had little opportunity
to proceed against them on religious grounds. Francis even allied himself
with the Moslem power of Turkey. On the whole, Charles was the victor in
the wars, and could point to the occupation of Milan as a tangible evidence
of his success,—about the only territorial change of consequence as a result
of the many campaigns. Perhaps the most noteworthy fact as affecting the
history of Spain and Spanish America was the financial drain occasioned by
the fighting. Time and again lack of funds was mainly responsible for
defeats or failures to follow up a victory. Spain and the Americas had to
meet the bills, but, liberal as were their contributions, more were always
needed.
The wars with Turkey had a special significance because
of the ever impending peril from Moslem northern Africa. Wars with
The pirates of the Berber, or Barbary, Coast, as the lands in the Turks
northwestern Africa are often called, seemed to be more than and the
Moslems of
ever audacious in the early years of the reign of Charles. Not northern
only did they attack Spanish ships and even Spanish ports, but Africa.
they also made numerous incursions inland in the peninsula.
Aside from the loss in captives and in economic wealth that these
visitations represented, they served to remind the authorities of the Moslem
sympathies of Spanish Moriscos and of the ease with which a Moslem
invasion might be effected. Furthermore the conquests of Isabella and
Ximénez had created Castilian interests in northern Africa, of both a
political and an economic character, which were in need of defence against
the efforts of the tributary princes to free themselves by Turkish aid. The
situation was aggravated by the achievements of a renegade Greek
adventurer and pirate, known best by the sobriquet “Barbarossa.” This
daring corsair became so powerful that he was able to dethrone the king of
Algiers and set up his own brother in his stead. On the death of the latter at
the hands of the Spaniards in 1518, Barbarossa placed the kingdom of
Algiers under the protection of the sultan of Turkey, became himself an
admiral in the Turkish navy, and soon afterward conquered the kingdom of
Tunis, whence during many years he menaced the Spanish dominions in
Italy. Charles in person led an expedition in 1535 which was successful in
dethroning Barbarossa and in restoring the former king to the throne, but an
expedition of 1541, sent against Algiers, was a dismal failure. On yet
another frontier, that of Hungary, Spanish troops were called upon to meet
the Turks, and there they contributed to the checking of that people at a time
when their military power threatened Europe. The problem of northern
Africa, however, had been little affected by the efforts of Charles.
Meanwhile, the religious question in Germany had all
along been considered by Charles as one of his most Charles’
important problems. The first war with France prevented any failure to
stamp out
action on his part until 1529, since he needed the support of Protestantis
the Protestant princes. The movement therefore had time to m.
gather headway, and it was evident that Charles would meet
with determined opposition whenever he should decide to face the issue.
Various factors entered in to complicate the matter, such, for example, as
the fear on the part of many princes of the growing Hapsburg power and the
belief that Charles meant to make the imperial succession hereditary in his
family. A temporary adjustment of the religious situation was made by the
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookname.com