Get the full ebook with Bonus Features for a Better Reading Experience on ebookluna.

com

High Performance Computing: Modern Systems and

Practices 1st edition - eBook PDF

https://ebookluna.com/download/high-performance-computing-
modern-systems-and-practices-ebook-pdf/

OR CLICK HERE

DOWLOAD NOW

Download more ebook instantly today at https://ebookluna.com

 Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Engineering of High-Performance Textiles 1st Edition

Menghe Miao - eBook PDF

https://ebookluna.com/download/engineering-of-high-performance-
textiles-ebook-pdf/

ebookluna.com

(eBook PDF) Regression & Linear Modeling: Best Practices

and Modern Methods

https://ebookluna.com/product/ebook-pdf-regression-linear-modeling-
best-practices-and-modern-methods/

ebookluna.com

(eBook PDF) Modern Blood Banking & Transfusion Practices

7th Edition

https://ebookluna.com/product/ebook-pdf-modern-blood-banking-
transfusion-practices-7th-edition/

ebookluna.com

(eBook PDF) Effective Training Systems Strategies and

Practices 5th Edition

https://ebookluna.com/product/ebook-pdf-effective-training-systems-
strategies-and-practices-5th-edition/

ebookluna.com
(eBook PDF) Performance Management Systems: An
Experiential Approach

https://ebookluna.com/product/ebook-pdf-performance-management-
systems-an-experiential-approach/

ebookluna.com

High performance tubular protonic ceramic fuel cells via

highly-scalable extrusion process - eBook PDF

https://ebookluna.com/download/high-performance-tubular-protonic-
ceramic-fuel-cells-via-highly-scalable-extrusion-process-ebook-pdf/

ebookluna.com

(eBook PDF) Modern Systems Analysis and Design 9th Edition

https://ebookluna.com/product/ebook-pdf-modern-systems-analysis-and-
design-9th-edition/

ebookluna.com

(eBook PDF) Modern Digital and Analog Communication

Systems 4th

https://ebookluna.com/product/ebook-pdf-modern-digital-and-analog-
communication-systems-4th/

ebookluna.com

(eBook PDF) Modern Systems Analysis and Design 8th

Valacich

https://ebookluna.com/product/ebook-pdf-modern-systems-analysis-and-
design-8th-valacich/

ebookluna.com
 High Performance
Computing
Modern Systems and Practices

Thomas Sterling
Matthew Anderson
Maciej Brodowicz
School of Informatics, Computing, and Engineering
Indiana University, Bloomington

Foreword by C. Gordon Bell

Acknowledgments

This textbook would not have been possible in either form or quality without the many contributions,
both direct and indirect, of a large number of friends and colleagues. It is derivative of first-year grad-
uate courses taught at both Louisiana State University (LSU) and Indiana University (IU). A number of
people contributed to these courses, including Chirag Dekate, Daniel Kogler, and Timur Gilmanov.
Amy Apon, a professor at the University of Arkansas, partnered with LSU and taught this course in
real time over the internet and helped to develop pedagogical material, including many of the exercises
used. Now at Clemson University, she continued this important contribution using her technical and
pedagogical expertise. Andrew Lumsdaine, then a professor at IU, cotaught the first version of this
course at IU. Amanda Upshaw was instrumental in the coordination of the process that resulted in
the final draft of the book, and directly developed many of the illustrations, graphics, and tables.
She was also responsible for the glossary of terms and acronyms. Her efforts are responsible in part
for the quality of this textbook.
A number of friends and colleagues provided guidance as the authors crafted early drafts of the book.
These contributions were of tremendous value, and helped improve the quality of content and form to be
useful for readers and students. David Keyes of KAUST reviewed and advised on Chapter 9 on parallel
algorithms. Jack Dongarra provided important feedback on Chapter 4 on benchmarking.
This textbook reflects decades of effort, research, development, and experience by uncounted num-
ber of contributors to the field of high performance computing. While not directly involved with the
creation of this text, many colleagues have contributed to the concepts, components, tools, methods,
and common practices associated with the broad context of high performance computing and its value.
Among these are Bill Gropp, Bill Kramer, Don Becker, Richard and Sarah Murphy, Jack Dongarra and
his many collaborators, Satoshi Matsuoka, Guang Gao, Bill Harrod, Lucy Nowell, Kathy Yelick, John
Shalf, John Salmon, and of course Gordon Bell. Thomas Sterling would like to acknowledge his thesis
advisor (at MIT) Bert Halstead for his mentorship to become the contributor that he has become.
Thomas Sterling also acknowledges Jorge Ucan, Amanda Upshaw, co-authors who made this book
possible, and especially Paul Messina who is his colleague, role model, mentor, and friend without
whom this book would never occurred. Matthew Anderson would like to thank Dayana Marvez, Oliver
Anderson, and Beltran Anderson. Maciej Brodowicz would like to thank his wife Yuko Prince Brodo-
wicz. The authors would like to thank Nate McFadden of Morgan-Kaufmann who provided enormous
effort, guidance, and patience that made this textbook possible.

DEDICATION TO PAUL MESSINA, WRITTEN BY THOMAS STERLING

The authors are pleased to dedicate this book to Dr Paul C. Messina, in acknowledgment of and grat-
itude for his exceptional contributions to and leadership in the field of high performance computing
over a career of more than 4 decades. It is impossible to capture fully the importance of his impact,
but many of the significant national programs have benefited from his guidance. Dr Messina has
been a visionary, a strategist, and a leader of programs, projects, organizations, initiatives, and, perhaps
most importantly, the careers of individual scientists who would come to deliver technical accomplish-
ments and leadership of their own. Dr Messina was the founding director of the Mathematics and
Computer Science Division at Argonne National Laboratory, a leading institution applying

xxvii
xxviii ACKNOWLEDGMENTS

high performance computing to mission-critical problem domains of the US Department of Energy

(DOE). He then founded and directed the Caltech Concurrent Supercomputing Facilities that staged
the Intel Touchstone Delta massively parallel processor, the fastest computer in the world in 1991
and the prototype of a family of massively parallel processors that determined the future direction
of high performance computing for the next 30 years. Caltech Concurrent Supercomputing Facilities
evolved into the Caltech Center for Advanced Computing Research, at which two of the authors spent
some of their most formative years. Paul was particularly instrumental and served as co-principal
investigator for the pioneering NSF Teragrid and the National Virtual Observatory. He directed the
DOE ASCI program for almost 3 years, building up the nation’s high performance computing capa-
bilities toward the leadership-scale computing it currently demonstrates. Most recently Paul Messina
led the Exascale Computing Project, America’s biggest undertaking in achieving exascale computing
performance by the beginning of the 2020s. For some Paul has had a direct and meaningful effect on
their individual careers. To author Thomas Sterling Paul has been a colleague, leader, mentor, and
friend for many years.
Morgan Kaufmann is an imprint of Elsevier
50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States
Copyright © 2018 Elsevier Inc. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying, recording, or any information storage and retrieval system, without
permission in writing from the publisher. Details on how to seek permission, further information about the
Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance
Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the Publisher (other
than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience broaden our
understanding, changes in research methods, professional practices, or medical treatment may become
necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using
any information, methods, compounds, or experiments described herein. In using such information or methods
they should be mindful of their own safety and the safety of others, including parties for whom they have a
professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any
liability for any injury and/or damage to persons or property as a matter of products liability, negligence or
otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the
material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library
ISBN: 978-0-12-420158-3

For information on all Morgan Kaufmann publications visit

our website at https://www.elsevier.com/books-and-journals

Publisher: Katey Birtcher

Acquisition Editor: Steve Merken
Developmental Editor: Nate McFadden
Production Project Manager: Punithavathy Govindaradjane
Designer: Mark Rogers

Typeset by TNQ Books and Journals

 Dedicated to

Dr. Paul C. Messina

Leader, colleague, collaborator, mentor, friend
Foreword

High Performance Computing is a needed follow-on to Becker and Sterling’s 1994 creation of the
Beowulf clusters recipe to build scalable high performance computers (also known as a supercomputers)
from commodity hardware. Beowulf enabled groups everywhere to build their own supercomputers.
Now with hundreds of Beowulf clusters operating worldwide, this comprehensive text addresses the crit-
ical missing link of an academic course for training domain scientists and engineersdand especially
computer scientists. Competence involves knowing exactly how to create and run (e.g., controlling,
debugging, monitoring, visualizing, evolving) parallel programs on the congeries of computational
elements (cores) that constitute today’s supercomputers.
Mastery of these ever-increasing, scalable, parallel computing machines gives entry into a compar-
atively small but growing elite, and is the authors’ goal for readers of the book. Lest the reader believes
the name is unimportant: the first conference in 1988 was the ACM/IEEE Supercomputing Confer-
ence, also known as Supercomputing 88; in 2006 the name evolved to the International Conference
on High Performance Computing, Networking, Storage, and Analysis, abbreviated SCXX. About
11,000 people attended SC16.
It is hard to describe a “supercomputer,” but I know one when I see one. Personally, I never pass up
a visit to a supercomputer having seen the first one in 1961dthe UNIVAC LARC (Livermore
Advanced Research Computer) at Lawrence Livermore National Laboratory, specified by Edward
Teller to run hydrodynamic simulations for nuclear weapons design. LARC consisted of a few dozen
cabinets of densely packed circuit board interconnected with a few thousand miles of wires and a few
computational units operating at a 100 kHz rate. In 2016 the largest Sunway Light supercomputer in
China operated a trillion times faster than LARC. It consists of over 10 million processing cores oper-
ating at a 1.5 GHz rate, and consumes 15 MW. The computer is housed in four rows of 40 cabinets,
containing 256 processing nodes. A node has four interconnected 8 MB processors, controlling 64 pro-
cessing elements or cores. Thus the 10.6 million processing elements deliver 125 peak petaflops, i.e.,
160 cabinets
256 physical nodes
4 computers
(1 control þ 8
8) processing elements or cores
with a 1.31 PB memory (160
256
4
8 GB). Several of the Top 500 supercomputers have
O(10,000) computing nodes that connect and control graphic processing units (GPUs) with O(100)
cores. Today’s challenge for computational program developers is designing the architecture and
implementation of programs to utilize these megaprocessor computers.
From a user perspective, the “ideal high performance computer” has an infinitely fast clock,
executes a single instruction stream program operating on data stored in an infinitely large and fast
single-memory, and comes in any size to fit any budget or problem. In 1957 Backus established the
von Neumann programming model with Fortran. The first or “Cray” era of supercomputing from
the 1960s through the early 1990s saw the evolution of hardware to support this simple, easy-to-use
ideal by increasing processor speed, pipelining an instruction stream, processing vectors with a single
instruction, and finally adding processors for a program held in the single-memory computer. By the
early 1990s evolution of a single computer toward the ideal had stopped: clock speeds reached a few
GHz, and the number of processors accessing a single memory through interconnection was limited to
a few dozen. Still, the limited-scale, multiple-processor shared memory is likely to be the most
straightforward to program and use!

xix
xx FOREWORD

Fortunately, in the mid-1980s the “killer microprocessor” arrived, demonstrating cost effectiveness
and unlimited scaling just by interconnecting increasingly powerful computers. Unfortunately, this
multicomputer era has required abandoning both the single memory and the single sequential program
ideal of Fortran. Thus “supercomputing” has evolved from a hardware engineering design challenge of
the single (mono-memory) computer of the Seymour Cray era (1960e95) to a software engineering
design challenge of creating a program to run effectively using multicomputers. Programs first oper-
ated on 64 processing elements (1983), then 1000 elements (1987), and now 10 million (2016) pro-
cessing elements in thousands of fully distributed (mono-memory) computers in today’s
multicomputer era. So in effect, today’s high performance computing (HPC) nodes are like the super-
computers of a decade ago, as processing elements have grown 36% per year from 1000 computers in
1987 to 10 million processing elements (contained in 100,000 computer nodes).
High Performance Computing is the essential guide and reference for mastering supercomputing,
as the authors enumerate the complexity and subtleties of structuring for parallelism, creating, and
running these large parallel and distributed programs. For example, the largest climate models
simulate ocean, ice, atmosphere, and land concurrently created by a team of a dozen or more domain
scientists, computational mathematicians, and computer scientists.
Program creation includes understanding the structure of the collection of processing resources and
their interaction for different computers, from multiprocessors to multicomputers (Chapters 2 and 3),
and the various overall strategies for parallelization (Chapter 9). Other topics include synchronization
and message-passing communication among the parts of parallel programs (Chapters 7 and 8), addi-
tional libraries that form a program (Chapter 10), file systems (Chapter 18), long-term mass storage
(Chapter 17), and components for the visualization of results (Chapter 12). Standard benchmarks for
a system give an indication of how well your parallel program is likely to run (Chapter 4). Chapters 16
and 17 introduce and describe the techniques for controlling accelerators and special hardware cores,
especially GPUs, attached to nodes to provide an extra two orders of magnitude more processing per
node. These attachments are an alternative to the vector processing units of the Cray era, and typified
by the Compute Unified Device Architecture, or CUDA, model and standard to encapsulate
parallelism across different accelerators.
Unlike the creation, debugging, and execution of programs that run interactively on a personal
computer, smartphone, or within a browser, supercomputer programs are submitted via batch process-
ing control. Running a program requires specifying to the computer the resources and conditions for
controlling your program with batch control languages and commands (Chapter 5), getting the pro-
gram into a reliable and dependable state through debugging (Chapter 14), checkpointing, i.e., saving
intermediate results on a timely basis as insurance for the computational investment (Chapter 20), and
evolving and enhancing a program’s efficacy through performance monitoring (Chapter 13).
Chapter 21 concludes with a forward look at the problems and alternatives for moving supercom-
puters and the ability to use them to petascale and beyond. In fact, the only part of HPC not described
in this book is the incredible teamwork and evolution of team sizes for writing and managing HPC
codes. However, the most critical aspect of teamwork resides with the competence of the individual
members. This book is your guide.

Gordon Bell
October 2017
Index
‘Note: Page numbers followed by “f” indicate figures, “t” indicate tables, and “b” indicate boxes.’

Anatomy of supercomputer, 14e16

A computer performance, 16e21
Accelerator architecture, 451e453
peak performance, 17e18
evolution of graphics processing unit functionality, 466e471,
performance, 16
468fe469f, 470t
performance degradation, 19e20
graphics processing units, 464e466, 465f
performance improvement, 20e21
heterogeneous system architecture, 477e478
scaling, 18e19
historic perspective, 454e463
sustained performance, 18
accelerators in processor I/O space, 461
Antikythera mechanism, 53f
accelerators with industry-standard interfaces, 462e463
Application profiling, significance of, 390e391
coprocessors, 456e461
Application programming, 4, 8e9
Intel 8087, 457e459
Application-level checkpointing, 598e601
Motorola MC68881, 459e461
Architecture overview, essential SLURM, 147, 147f
modern graphics processing unit architecture, 471e477,
Arithmetic logic unit (ALU), 63e64, 70
472t
Arithmetic on variables, Linux, 654
compute architecture, 471e474, 474f
Asynchronous multitasking, 613e615
interconnects, 475e476
actor synchronization, 615
memory implementation, 474e475
global address space, 614e615
programming environment, 476e477
message-driven computation, 613e614
outcomes, 480
multithreaded, 613
Accelerators
runtime system software, 615
with industry-standard interfaces, 462e463
Atomics, 504e505
in processor I/O space, 461
Automated calculators through mechanical technologies,
Actor synchronization, 615
22e23
Advection equation using finite difference, 295e297,
296fe298f
Aggregated storage, 534e544 B
network attached storage, 543e544 Backus, John, 317
redundant array of independent disks, 532f, 534e541 Barrier directive, OpenMP programming model, 243
hybrid redundant array of independent disks variants, Basic linear algebra subprograms (BLAS), 317e322
539e541 Level 1 rotation operations, 320t
RAID 0: striping, 534e535 Level 1 vector operations, 320te321t
RAID 1: mirroring, 535 Levels 2 and 3, 321te322t
RAID 2: bit-level striping with hamming code, 536 routines, precision prefixes used by, 319t
RAID 3: byte-level striping with dedicated parity, Basic methods of use, commodity clusters, 104e113
536e537 compilers and compiling, 112e113
RAID 4: block-level striping with dedicated parity, 537 logging on, 104e105
RAID 5: block-level striping with single distributed package configuration and building, 110e111
parity, 538 running applications, 113
RAID 6: block-level striping with dual distributed parity, user space and directory system, 105e110
539 Basic two-input logic gates, 56f
storage area networks, 541e542 Benchmarking
tertiary storage, 544, 544f, 545t benchmarks used in HPC community, 118t
Allgather, 272e274 Graph500, 132e135
Alltoall, 277e278, 277f highly parallel computing linpack, 120e123
Amdahl’s law, 70e73 high performance conjugate gradients, 126e128
Amdahl’s law plus, 196e199, 197fe199f HPC challenge benchmark suite, 123e124

677
678 INDEX

Benchmarking (Continued ) Clearspeed Advance e710 accelerator board, 463f

key properties of, 117e120 Climate change, understanding of, 12e14
miniapplications as benchmarks, 135e138 CMOS, 55
NAS parallel benchmarks, 130e131 Collective data movement, 265e267, 265fe267f
nonproprietary, 119t Command substitution, Linux, 654
outcomes, 138e139 Command-line utilities, Linux, 659e663
overview, 115e117 data compression and archiving, 662e663
standard HPC community benchmarks, 120 process management, 661e662
top-performing supercomputer, 119t text tools, 659e661
Beowulf class of parallel computer architecture, 77f Commands, SLURM scheduling, 151e166
Beowulf cluster project, 91e93 sacct, 164e165
Bit-level striping with hamming code (RAID 2), 536 salloc, 160e161
Block-level striping sbatch, 161e162
with dedicated parity (RAID 4), 537 scancel, 163e164
with dual distributed parity (RAID 6), 539 sinfo, 165e166
with single distributed parity (RAID 5), 538 squeue, 162e163
Branch prediction, symmetric multiprocessor architecture, srun, 151e160
201e202 Commercial parallel debuggers, 431e432
Breadth first search, parallel algorithms, 306e310, 307fe309f Commercial systems summary, commodity clusters, 95
Brief history of supercomputing, 21e38 Commodity clusters
automated calculators through mechanical technologies, basic methods of use, 104e113
22e23 compilers and compiling, 112e113
communicating sequential processors and very large scale logging on, 104e105
integration, 34e37 package configuration and building, 110e111
instruction-level parallelism, 29e30 running applications, 113
multicore petaflops, 37 user space and directory system, 105e110
neodigital age and beyond Moore’s law, 37e38 Beowulf cluster project, 91e93
single-instruction multiple data array, 33 cluster elements, 85e86
vector processing and integration, 30e33 definition of, 84
von Neumann architecture in vacuum tubes, 24e29 hardware architecture, 93e95
Broadcast, 268e269 commercial systems summary, 95
Buffer flush, 560 node, 93e94
Buffered File I/O, 559e562 secondary storage, 95
buffer flush, 560 system area networks, 94e95
conversion between streams and file descriptors, 561e562 history, 88e90
file open and close, 559 HPC architecture, 77e78
offset update and query, 560 impact on top 500 list, 86e87
sequential data access, 560 motivation and justification for clusters,
Bush, Vannevar, 22e23 84e85
Byte-level striping with dedicated parity (RAID 3), 536e537 outcomes, 113e114
overview, 84e90
programming interfaces, 97e98
C
high performance computing programming languages,
C++ AMP, 486
97
Cannon’s algorithm, 301e303, 301fe307f
parallel programming modalities, 97e98
Catalyzing fraud detection, 10
software environment, 98e104
CDC-7600, 318f
operating systems, 98e99
Cellular automata, 618e619
resource management, 99e100
Checkpointing, 591
debugger, 101
application-level, 598e601
performance profiling, 101
outcomes, 602
visualization, 101e104
system-level, 592e597
Communicating sequential processors and, 34e37
Checkpoint/restart, 47e48
 INDEX 679

Communication collectives, 265e278 GDB cheat sheet, 430

allgather, 272e274 setting a variable, 428e430, 430f
alltoall, 277e278, 277f threads, 430, 431fe432f
broadcast, 268e269 watch points and catch points, 425e426, 427f
collective data movement, 265e267, Valgrind, 430e431, 433t
265fe267f unprotected shared variable assessment, 433e434, 436f
gather, 271e272 Digital logic, HPC architecture, 55e58
reduction operations, 274e276, 274t Directives, OpenMP programming model, 230e231
scatter, 269e271 Discovering oil and gas, 10
Communicators, 255e258 Distributed computation, MapReduce, 584
example, 257e258 Distributing oil and gas, 10
rank, 256e257 Divide and conquer, parallel algorithms, 287e291,
size, 256 288fe291f
Compilation, Linux, 656e659 Domain-specific languages for linear algebra, 329, 330fe332f
Compiler flags for debugging, 439 Dongarra, Jack, 116b
Compilers and compiling, commodity clusters, 112e113 Dynamic random access memory (DRAM), 45, 61e62
Compute architecture, 471e474, 474f
Compute unified device architecture, 485 E
Computer performance, 16e21
Earth Simulator (ES), 64b
Concurrent applications, SLURM job scripting, 167e169
Efficiency, HPC architecture, 46
Connection Machine 2 (CM-2), 454e463
Eigenvalue SoLvers for Petaflop-applications, 328
Connection Machine 5 (CM-5), 252f
Elastic computing, SLURM scheduling, 150
Control Data Corporation (CDC) 6600, 51e55
Electronic numerical integrator and computer (ENIAC), 51f
Control flow, Linux, 654e656
Emacs, 645e646
Conversion between streams and file descriptors, 561e562
Embarrassingly parallel, parallel algorithms, 292e293, 293f
Coprocessors, 456e461
Enabling technology, HPC architecture, 51e62
Intel 8087, 457e459
digital logic, 55e58
Motorola MC68881, 459e461
memory technologies, 58e62
Cray, Seymour, 30e33
early memory devices, 59e61
Cray-1 Supercomputer, 32f
modern memory technologies, 61e62
Critical synchronization directive, OpenMP programming
roles of technologies, 55
model, 242
technology epochs, 51e55
Environment variables
D OpenMP programming model, 229e230
Data access with explicit offset, 556 SLURM job scripting, 169e171
Data compression and archiving, Linux, 662e663 Environment variables of interest, 185e186
Data management, 497e501 Essential bash, Linux, 647e656
Data reuse and locality, symmetric multiprocessor arithmetic on variables, 654
architecture, 204e205 command substitution, 654
Dataflow, 617e618 control flow, 654e656
Deadlock, 434e439 input/output, 649e650
Debugger, commodity clusters, 101 path expansion, 647e648
Debugging, 421e422 pipelines, 649e650
compiler flags for debugging, 439 redirection, 649e650
debugging MPI example, deadlock, 434e439 special character handling, 649
outcomes, 445 variables, 650e653
system monitors to aid debugging, 441e445 Essential gperftools, 391e398
tools, 423e432 Essential Linux
commercial parallel debuggers, 431e432 command-line utilities, 659e663
GNU debugger, 423e430 data compression and archiving, 662e663
back trace, 427e428, 428fe429f process management, 661e662
break points, 424e425, 425f, 425t, 426f text tools, 659e661
680 INDEX

Essential Linux (Continued ) overview of collective calls, 262e263

compilation, 656e659 user-defined data types, 281e283
essential bash, 647e656 Essential OpenMP, 225e226
arithmetic on variables, 654 outcomes, 245e246
command substitution, 654 overview of OpenMP programming model, 226e231
control flow, 654e656 runtime library and environment variables, 228e231
input/output, 649e650 directives, 230e231
path expansion, 647e648 environment variables, 229e230
pipelines, 649e650 runtime library routines, 230
redirection, 649e650 thread parallelism, 226e228
special character handling, 649 thread variables, 228
variables, 650e653 parallel threads and loops, 231e240
files editing, 645e647 parallel “for”, 233e238
Emacs, 645e646 parallel threads, 231e232
Gedit, 646 private, 232e233
Kate, 646 sections, 239e240
Nano, 646 reduction, 244
Vi, 645 synchronization, 241e243
logging In, 637e638 barrier directive, 243
navigating the file system, 641e645 critical synchronization directive, 242
remote access, 639e640 master directive, 242e243
Essential MPI, 250 single directive, 243
communication collectives, 265e278 Essential portable batch system, 172e187
allgather, 272e274 PBS cheat sheet, 186e187
alltoall, 277e278, 277f PBS commands, 174e183
broadcast, 268e269 pbsnodes, 183
collective data movement, 265e267, qdel, 180
265fe267f qstat, 180e182
gather, 271e272 qsub, 174e179
reduction operations, 274e276, 274t tracejob, 182e183
scatter, 269e271 PBS job scripting, 184e186
communicators, 255e258 environment variables of interest, 185e186
example, 257e258 MPI jobs, 185
rank, 256e257 OpenMP jobs, 184e185
size, 256 portable batch system architecture, 173e174, 173f
message-passing interface basics, 253e255 portable batch system overview, 172e173
message-passing interface example, 254e255 Essential POSIX file interface, 554e562
MPI_Finalize, 254 buffered File I/O, 559e562
mpi.h, 253 buffer flush, 560
MPI_Init, 253 conversion between streams and file descriptors,
message-passing interface standards, 251e253 561e562
nonblocking point-to-point communication, file open and close, 559
279e281 offset update and query, 560
outcomes, 283 sequential data access, 560
point-to-point messages, 258e262 system calls for file access, 554e558
example, 260e262 data access with explicit offset, 556
message-passing interface data types, 259, 260t file length adjustment, 556e557
MPI Recv, 259e260 file offset manipulation, 556
MPI send, 259 file open and close, 554e555
synchronization collectives, 262e265 file status query, 557e558
barrier synchronization, 263, 263f sequential data access, 555e556
example, 264e265 synchronization with storage device, 557
 INDEX 681

Essential resource management SLURM cheat sheet, 171e172

essential portable batch system, 172e187 SLURM job scripting, 166e171
PBS cheat sheet, 186e187 concurrent applications, 167e169
PBS Commands, 174e183 environment variables, 169e171
pbsnodes, 183 MPI scripts, 167
qdel, 180 OpenMP scripts, 167
qstat, 180e182 script components, 166e167
qsub, 174e179 SLURM scheduling, 149e150
tracejob, 182e183 elastic computing, 150
PBS job scripting, 184e186 gang scheduling, 149
environment variables of interest, 185e186 generic resources, 150
MPI jobs, 185 high-throughput computing, 150
OpenMP jobs, 184e185 preemption, 149e150
portable batch system architecture, 173e174, 173f trackable resources, 150
portable batch system overview, 172e173 workload organization, 148, 148f
essential SLURM, 146e172 Ethernet, 213e214
architecture overview, 147, 147f Exascale computing, 610e612
commands, 151e166 accelerated approach, 612
sacct, 164e165 capacity, 611e612
salloc, 160e161 challenges to, 611
sbatch, 161e162 lightweight cores, 612
scancel, 163e164 Execution pipeline, symmetric multiprocessor architecture,
sinfo, 165e166 200e201
squeue, 162e163 Expanded parallel programming models,
srun, 151e160 606e607
SLURM cheat sheet, 171e172 advance in message-passing interface,
SLURM job scripting, 166e171 606e607
concurrent applications, 167e169 advances in OpenMP, 607
environment variables, 169e171 MPI+X, 607
MPI scripts, 167 Extended high performance computing architecture,
OpenMP scripts, 167 608e609
script components, 166e167 field programmable gate arrays, 609
SLURM scheduling, 149e150 lightweight architectures, 608e609
elastic computing, 150 world’s fastest machine, 608
gang scheduling, 149 External I/O interfaces, symmetric multiprocessor
generic resources, 150 architecture, 213e222
high-throughput computing, 150 JTAG, 218e219
preemption, 149e150 network interface controllers, 213e215
trackable resources, 150 Ethernet, 213e214
workload organization, 148, 148f InfiniBand, 215
managing resources, 142e146 serial advanced technology attachment,
outcomes, 187e189 215e218
Essential SLURM, 146e172 universal serial bus, 220e222
architecture overview, 147, 147f
commands, 151e166 F
sacct, 164e165 Fast Fourier transform (FFT), 628e631
salloc, 160e161 Field programmable gate arrays, 609
sbatch, 161e162 File length adjustment, 556e557
scancel, 163e164 File management, 350
sinfo, 165e166 File offset manipulation, 556
squeue, 162e163 File open and close, 554e555, 559
srun, 151e160 File status query, 557e558
682 INDEX

File systems, 86 GNU scientific library, 326, 327f

essential POSIX file interface, 554e562 Gnuplot, 365e368, 366fe370f
buffered File I/O, 559e562 gperftools, 391e398
buffer flush, 560 Graph algorithms, 329
conversion between streams and file descriptors, Graph500, 132e135, 133fe134f, 134t, 135fe137f
561e562 Graphics processing units, 464e466, 465f
file open and close, 559 functionality, evolution of, 466e471, 468fe469f,
offset update and query, 560 470t
sequential data access, 560 modern architecture, 471e477, 472t
system calls for file access, 554e558 compute architecture, 471e474, 474f
data access with explicit offset, 556 interconnects, 475e476
file length adjustment, 556e557 memory implementation, 474e475
file offset manipulation, 556 programming environment, 476e477
file open and close, 554e555 Gropp, William D., 251f
file status query, 557e558 Gyrokinetic Toroidal code, 9f
sequential data access, 555e556
synchronization with storage device, 557 H
general parallel file system, 565e569, 566f, 566t
Hadoop, 585e588
Lustre file system, 569e575
Halo exchange, parallel algorithms, 294e299, 294f
network file system, 562e565, 563f
advection equation using finite difference, 295e297,
outcomes, 575e576
296fe298f
role and function, 549e553
sparse matrix vector multiplication, 297e299,
Files editing, Linux, 645e647
299fe300f
Emacs, 645e646
Hard-disk drives, 514e520, 515f, 517f, 519t
Gedit, 646
Hardware architecture, commodity clusters, 93e95
Kate, 646
commercial systems summary, 95
Nano, 646
node, 93e94
Vi, 645
secondary storage, 95
Flops, 46
system area networks, 94e95
Flynn’s taxonomy, 48e50, 50f
Hardware fault, 47e48
Forkejoin model
Heterogeneous computer structures, 78
master/worker threads, 228f
Heterogeneous system architecture, 477e478
parallel algorithms, 286e287, 287fe288f
High performance computing
Forwarding, symmetric multiprocessor architecture, 202
computational science research areas, 314f
Foundational visualization concepts, 364
disciplines, 3e9
application programming, 8e9
G application programs, 4
Gang scheduling, SLURM scheduling, 149 definition, 3e4
Gara, Alan, 610e612 high performance computing systems, 5e7
Gather, 271e272 performance and metrics, 4e5
Gedit, 646 supercomputing problems, 7e8
General parallel file system, 565e569, 566f, 566t programming languages, commodity clusters, 97
Generic resources, SLURM scheduling, 150 systems, 5e7, 7f
Gigabit Ethernet High performance linpack, 120e123, 122f
network interface card, 214f High performance conjugate gradients, 126e128
switch, 214f High throughput computing, SLURM scheduling, 150
Global address space, 614e615 Historic perspective, of accelerator architecture, 454e463
GNU debugger, 423e430 accelerators in processor I/O space, 461
back trace, 427e428, 428fe429f accelerators with industry-standard interfaces,
break points, 424e425, 425f, 425t, 426f 462e463
GDB cheat sheet, 430 coprocessors, 456e461
setting a variable, 428e430, 430f Intel 8087, 457e459
threads, 430, 431fe432f Motorola MC68881, 459e461
watch points and catch points, 425e426, 427f Hopper, Grace Brewster Murray, 421f, 423f
 INDEX 683

Host node, 86 Interconnects, 475e476

HPC architecture I/O system management, 351
enabling technology, 51e62 Isosurfaces, 364, 365f
digital logic, 55e58
memory technologies, 58e62 J
early memory devices, 59e61
JTAG, 218e219, 219f
modern memory technologies, 61e62
roles of technologies, 55
technology epochs, 51e55 K
heterogeneous computer structures, 78 Kate, 646
key properties, 44e48 Kernels construct, 495e496
efficiency, 46 Key properties, HPC architecture, 44e48
parallelism, 45e46 efficiency, 46
power, 46e47 parallelism, 45e46
programmability, 48 power, 46e47
reliability, 47e48 programmability, 48
speed, 45 reliability, 47e48
multiprocessors, 73e78 speed, 45
commodity clusters, 77e78 K-means clustering, 582e584
massively parallel processors, 76
shared-memory multiprocessors, 74e76 L
outcomes, 78e79 Lapack, 324, 325f
overview, 44 driver routines, 324t
parallel architecture families, Flynn’s taxonomy, 48e50 Libraries, 313e315, 315t
single-instruction, multiple data array, 69e73 graph algorithms, 329
Amdahl’s law, 70e73 linear algebra, 315e329, 316f
single-instruction, multiple data architecture, 69e70 basic subprograms, 317e322
vector and pipelining, 64e68 domain-specific languages for, 329, 330fe332f
pipeline parallelism, 65e68 eigenvalue SoLvers for Petaflop-applications, 328
vector processing, 68 GNU scientific library, 326
von Neumann sequential processors, 62e64 Hypre, scalable linear solvers and multigrid methods,
HPC challenge benchmark suite, 123e124 328e329
Hybrid redundant array of independent disks variants, package, 324e326
539e541 portable extensible toolkit for scientific computation,
Hypre, scalable linear solvers and multigrid methods, 327e328
328e329 scalable library for eigenvalue problem computations,
328
I scalable linear algebra, 326
IBM Blue Gene supercomputers, 610e612 supernodal LU, 326e327
ILLIAC (Illinois Automatic Computer), 454e463 mesh decomposition, 333e334, 340f
Independent disks variants, hybrid redundant array of, outcomes, 342e343
539e541 parallel input/output, 330e332, 335f
InfiniBand, 215 parallelization, 334
port, 216f partial differential equations, 329
Input/output , Linux, 649e650 performance monitoring, 341e342
Instruction interface, 70 signal processing, 334e341
Instruction set architecture (ISA), 44, 54 visualization, 334
Instruction-level parallelism, 29e30 Lightweight architectures, 608e609
symmetric multiprocessor architecture, 201 Linear algebra, 315e329, 316f
Integrated performance monitoring toolkits, 407e410, 410f basic subprograms, 317e322
Intel 8087, 457e459, 457fe458f domain-specific languages for, 329, 330fe332f
Intel Touchstone Delta, 35e36 eigenvalue SoLvers for Petaflop-applications, 328
684 INDEX

Linear algebra (Continued ) storage area networks, 541e542

GNU scientific library, 326 tertiary storage, 544, 544f, 545t
Hypre, scalable linear solvers and multigrid methods, history, 512, 513f
328e329 outcomes, 545e546
package, 324e326 storage device technology, 514e532
portable extensible toolkit for scientific computation, hard-disk drives, 514e520, 515f, 517f, 519t
327e328 magnetic tape, 524e528, 527fe528f, 529t
scalable library for eigenvalue problem computations, 328 optical storage, 529e532, 530f
scalable linear algebra, 326 solid-state drive storage, 520e524, 521fe522f, 525t,
supernodal LU, 326e327 529t
Linpack, 116b, 120e123 Massively parallel processor (MPP), 55, 76f
Local registers, 70 HPC architecture, 76
Logging in, Linux, 637e638 Master directive, OpenMP programming model, 242e243
Logging on, commodity clusters, 104e105 Matplotlib, 369e372, 371fe372f
Loop scheduling, 501e504 Mellanox IB cards, 215f
Lower/upper decomposition, 624e628 Memory block, 70
Lustre file system, 569e575 Memory hierarchy, symmetric multiprocessor architecture,
204e209, 204f
data reuse and locality, 204e205
M
memory system performance, 207e209
Magnetic tape, 524e528, 527fe528f, 529t
Memory implementation, 474e475
Magnetism, 60e61
Memory management, 350, 358e361
Makefiles, 111
virtual address translation, 359e361
Managereworker, parallel algorithms, 291e292, 292f
virtual memory, 359
Managing oil and gas, 10
virtual page addresses, 359
Managing resources, 142e146
Memory system performance, symmetric multiprocessor
Manufacturing, accelerating innovation in, 10e11
architecture, 207e209
MapReduce, 579
Memory technologies, HPC architecture, 58e62
distributed computation, 584
early memory devices, 59e61
Hadoop, 585e588
modern memory technologies, 61e62
Map and Reduce, 579e584
Mesh decomposition, 333e334, 340f
K-means clustering, 582e584
Mesh tessellation, 364, 366f
shared neighbors, 581e582
Message-driven computation, 613e614
word count, 581
Message passing interface (MPI), 253e255
outcomes, 588e589
advance in, 606e607
Market data analytics, 10
example, 254e255
Mass storage, 509e512
jobs, 185
aggregated storage, 534e544
message passing interface example, 254e255
network attached storage, 543e544
MPI_Finalize, 254
redundant array of independent disks, 532f, 534e541
mpi.h, 253
hybrid redundant array of independent disks variants,
MPI_Init, 253
539e541
scripts, SLURM job scripting, 167
RAID 0: striping, 534e535
standards, 251e253
RAID 1: mirroring, 535
Metrics, 4e5
RAID 2: bit-level striping with hamming code, 536
Miniapplications as benchmarks, 135e138, 137t
RAID 3: byte-level striping with dedicated parity,
Mirroring (RAID 1), 535
536e537
Modern graphics processing unit architecture, 471e477,
RAID 4: block-level striping with dedicated parity,
472t
537
compute architecture, 471e474, 474f
RAID 5: block-level striping with single distributed
interconnects, 475e476
parity, 538
memory implementation, 474e475
RAID 6: block-level striping with dual distributed
programming environment, 476e477
parity, 539
Other documents randomly have
different content
 "You will kindly understand that I resent your interference with my
finances and my daughter's."

"You may resent," said the prospective son-in-law, "till you're black in
the face, and I shan't lose sleep over it."

Bang went something outside, and Slade started. "Good Lord," he said,
"there's somebody firing at us. Sit down, man, on the floor."

"Nothing of the kind," said Carter testily. "My boy Trouble has got the
engines going to try to work us off this bank, and with his usual cleverness
he has contrived a back fire, that's all. There—you can smell it. Now, I don't
think you are a quarrelsome man as a general thing?"

"Not I. Too much trouble to quarrel with people."

"Well, I'll just ask you to give Laura and myself your benediction, and
leave the rest to us."

Slade let off his limp laugh. "If a wedding present of such dubious value
will please you, I'm most pleased to give it. Especially as I see you're
inclined to stick to my little girl. To tell the truth, I'd heard you were after
somebody else and it made me rather mad. You know how rumors float
about in the bush."

Carter's lips tightened. "Who's the other person, please?"

"Oh, just my present employer—and your late one. But I've no doubt it's
all a mistake."

"If you'll apply to her, I've no doubt she'll endorse that sentiment most
thoroughly. I don't think Miss O'Neill's a person to throw herself away on
one of her own ex-servants."

Slade chuckled. "If you put it that way, I'm sure she isn't. By the way, do
you know who she is?"

"What do you mean?"

 "Well, I suppose you've discovered by this time that the late Godfrey
O'Neill was a bachelor, and Kate's no relation to him at all. He and his sister
Jane, who married a hopeless blackguard called Craven, adopted her
between them and brought her up. I've never fagged myself to find out how
she was bred, but you're one of these energetic fellows that like to dig into
pedigrees, and I thought probably you'd know."

"I don't know, and I shan't inquire."

"All right, don't get excited about it, neither shall I. D'ye know I think if
you could soften that genial manner without straining yourself, it would be
an improvement. I'm led to believe that fathers-in-law expect a civility and
even at times a certain mild amount of deference."

"Did you defer to your father-in-law?" asked Carter brutally.

The tone was insulting and the meaning plain, and ninety-nine men out
of a hundred in a similar place would have resented it fiercely. But Slade
merely yawned. His sallow face neither twitched nor changed its tint. He
got up and stretched himself lazily. "So that's the trouble, is it? Well, you
didn't ask me to consult you when I chose a wife, and I didn't ask you to fall
in love with my daughter." He turned his head and eyed Carter thoughtfully
—"You are in love with her, I suppose?"

"Can you suggest any other possible reason why I should ask her to
marry me?"

"Well, I can hardly imagine you did it for the honor of an alliance with
me. I suppose if I were an energetic man I should try and worry out what it
is you're so sore about. It must be something beyond the detail that Laura's
got a touch of color in her, because of course you knew that from the first
moment you met her. But I guess the something else will show itself in its
own good time. In the meanwhile if you'll give me an account of what you
advanced to Laura for this Grand Canary trip, I'll give you an I.O.U. for it. I
don't care to be indebted to anyone for things like that."

"I'll perhaps send in the bill when I hear there's a possibility of getting
cash payment," said Carter dryly.
 And then for the first time Slade lost his temper, and he cursed his future
son-in-law with all an old Coaster's point and fluency. Every man has his
tender point, and here was Owe-it Slade's. Throughout all his life he had
never paid a bill if he could help it, and he had accepted the consequent
remarks of injured parties with an easy philosophy. But it seemed he owned
a nice discrimination; some items were "debts of honor," and these he had
always sooner or later contrived to settle. And the account which he decided
he owed Carter for Laura's maintenance in Grand Canary he set down as
one which no gentleman could leave unpaid without besmirching his
gentility.

CHAPTER XIX
SENHOR CASCAES

Now, as the servant of O'Neill and Craven, Carter had done his work
well and indeed enthusiastically, and after he had left the firm's employ he
had neither competed with them in business nor done them harm in any way
whatever. It is true that at his memorable interview with the King of Okky
with a little persuasion he could have got that grateful monarch to take off
the embargo which he had laid on the factories at Monk, Malla-Nulla, and
Smooth River, though the fact that he did not put forward pressure on this
point could hardly have reached the ear of Miss O'Neill. Indeed it is to be
doubted if she ever knew that any reference to her name or affairs cropped
up at all.

But be that as it may, she certainly from the date of sending her cable to
Cascaes began to interest herself in opposing Carter's schemes.

The first he knew of it was a typewritten letter from Liverpool on the

firm's note-paper beginning "Dear sir," and ending "O'Neill & Craven, per
K. O'Neill." In arid business sentences it understood he had "a tin-mining
proposition up Smooth River," it pointed out that "our firm for many years
has had very far-reaching interests in this neighborhood," and it suggested
that O'Neill and Craven should buy the mine "to prevent any clash of
interests."

Carter replied to this curtly enough that Tin Hill was not in the market,
and took the next boat home to Liverpool. He had picked up a distressed
merchant skipper named Kettle, and put him in charge of the motor boat,
and the canoes, and the mining work generally, and though in their short
interview he decided that Kettle was the most tactless man in Africa, he
believed him to be honest, and instinctively knew him to be capable.

"One thing I must ask," he said at the end of their talk, "and that is that
you do not try any proselytizing up here. Your creed, I have no doubt, is
very excellent at home, but out here where they are either Moslemin or
nothing it will only stir up disputes, and that I won't have. Is that quite
agreed?"

"I have learned, sir," said the sailor, "to obey orders to the letter even
though I know them to be against an owner's best interests."

"Um," said Carter, and stared at him thoughtfully. "Well, Captain, I think
it would be safest if you went on those lines. You will find your chief
engineer, who carries the name of White-Man's-Trouble, beautifully
unreliable in most things, but he understands the launch's engines
wonderfully, and I like him. I'd take it as a favor if you'd deal with him as
lightly as possible."

"I'll bear your words in mind, sir, though, as a man who has handled
everything colored that serves afloat, I'd like to point out that pampering
spoils them."

"The only other point to remember is that I've made my name up these
rivers mainly by being known as a ju-ju man—sort of wizard, in fact. You'll
have no difficulty, I suppose, in following up that line now I've given you
the hint?"

"You'll pardon me, sir, but if that's made an essential, I must chuck up
the job, sorely in need of employment as I am at the moment. I have my
conscience to consider. And besides as a liar I am the poorest kind of
failure."

"Pooh, man, it's only a little acting that's required."

"Mr. Carter," said the sailor still more stiffly, "you see in me a man who's
sunk very low, but I've never descended yet to working as a theatrical.
According to our Persuasion, we hold that play acting is one degree less
wicked than bigamy, and indeed often leads to it."

"Well," said Carter, "that mail-boat sails in half an hour's time, and I've
got to go by her. I've been building on you, Captain, as the most trustworthy
man now knocking about in West Africa."

"I'm all that, sir."

"So I shall have to respect your scruples and give you the billet."

"You shall never regret it for one minute, sir. You'll find the address of
Mrs. Kettle on this slip of paper, and if you'll post three-quarters of my
wages to her as they fall due, I'd take it as a favor. I've been out of—well, I
won't pester you with domestic matters, sir, but the fact is I'm afraid she
must be in very poor circumstances just at the moment."

"She shall have a check posted the day after I land in Liverpool. I give
you my word for that."

"I thank you, Mr. Carter. Now, if you wanted another officer, there's a
Mr. McTodd, an engineer who's just now at Akassa, that I could get."

"Thanky, Captain, but not for me."

"I believe I could persuade him to take a low wage."

"Not for me, Captain. I know McTodd. He's far too thirsty and far too
cantankerous. You'd find him a ugly handful."

"Me! By James, sir, I can handle that swine in a way that would surprise
you. He's had a bad up-bringing; he belongs to the Free Kirk; but after I've
had the manipulation of Mr. McTodd for a week, I can make him as mild as
Norwegian Swiss milk."

"Well, we'll say 'not for the present,' at any rate. With the organization
I've got together, and the backing from the King of Okky that I've told you
about, you'll be able to haul down all the available ore if you follow out my
instructions, and when it comes to bonus, Captain, if you've been
successful, you'll find me a generous paymaster. I don't toil for nothing
myself. I work about ten times as hard as my neighbors, and draw in about
seventeen times as much pay. I like a man who has got the same ambitions."

The little sailor sighed. "I've always done ten times the normal whack of
work, sir, but somehow I've missed fingering the dibs. I tell you flat,
fourteen pounds a month has been good for me, and month in and month
out I've not averaged ten."

"Then, if that's the case," said Carter briskly, "just here should come the
turn in your fortunes. Shake hands, Captain. Good-bye to you, good health
and good luck. Here's my surf boat. The steamer's heaving short."

"Good-bye, sir," said Kettle, "I'm sure you'll remember to send that
check."

* * * * *

The mail-boat called as usual at Las Palmas and was boarded on arrival
by the usual batch of invalids and Liverpool trippers for the run home.
Carter landed as soon as the port doctor gave clearance papers, rowed to the
mole and chartered a tartana, between whose shafts there drooped a mouse-
colored mule. In it he bumped over the badly laid tram lines from the Isleta
to the city, and then left the city by the Telde road.

Las Palmas is the meeting place of all West Africa, and if one is there
long enough, one expects to meet sooner or later every man who has
business or other interests on the Coast. Carter waved his hand to a Haûsa
constabulary officer in the gateway of the Catalina, and to a Lagos branch
boat skipper who was standing on the steps of the Elder Dempster office.
Coming down from the telegraph station he saw one of the Germans who
had been frightened out of Mokki, and under a café awning by the dry river
bed no less a personage than Burgoyne of Monk River waved a hospitable
hand and invited him to try a glass of Bass.

But further on, where the Telde road leaves the city, he saw a man whose
walk he knew, and instinctively leaned out from the tartana's awning to
show himself, and to wave a greeting. The man was Cascaes. But the
Senhor Cascaes stared him coolly in the face, and—cut him dead.

The tartana rattled on, and Carter nodded after the Portuguese
thoughtfully. "You have always hated me pretty tenderly," he mused. "I
wonder why. I've hammered you a dozen times, but it's only been in the
ordinary way of business, and what any half-baked Portuguese has got to
expect. You surely can't be up against me for that."

Laura was not living in the convent, but lodged in the house of a banana
farmer just beyond. Carter found her in the garden. She was sitting on the
end of a bench overhung with great lavender clots of wistaria at one end
and shaded by a purple mass of bougainvillea at the other. He noted with a
queer thrill that there was something cold in the outward form of her
greeting.

She returned his kiss accurately enough, but without enthusiasm. Still,
from the moment she saw him, the light came into her eyes that he had
grown to know so well. The two things did not seem somehow or other to
tally. Carter sat himself on the bench and took a good hold on his nerves.
Then he slid an arm round her waist and drew her to him. "Well," he said,
"out with it. What's the trouble?"

She dropped her head on his shoulder contentedly enough. "Oh, the
usual. When you're away from me, dear, I never feel quite certain if I ought
to marry you."

"Now, that's awkward, isn't it? But as I have been up country colloguing
with your other suitor, old Kallee, you couldn't very well have been with me
there."

"I wish you hadn't gone."

 "How delightfully unreasonable! We'd nothing to boil the pot on before,
and now we've plenty, and neither of us is a bit the worse. What's broke
since I've been away?"

"The world, I think," said Laura miserably.

"Then I hope I'm the sticking plaster that will mend it. Now, I want to
hear all about Las Palmas, and what you have been doing. I see most of
West Africa's here. Great Christopher! but it is fine to smell even the
outside edge of civilization once more. My mother used to get tired of
Wharfedale occasionally—ah, well, but that wouldn't interest you."

"No, you always cut yourself short when you begin to talk about your
people."

"Do I? Well, what's sauce for the gander's sauce for the goose and you're
the goose. Did you ever speak to me about your folk? Not one word, unless
I dragged it out. Look here, Laura, are you trying to wrangle? Because if so,
and if it's my fault, just say what's the crime, and give me my licking and
get it over. I've got a clear conscience, and I'll be as penitent as you please."

"My dear, you've been perfect."

"Oh, I say," said Carter, "not too sudden. That sort of thing brings on
heart attacks."

"I know your temptations, and you've been an honorable gentleman all
through."

"I wish," said Carter whimsically, "you could persuade other people to
look at me in that light. A missionary on the steamer yesterday called me a
gin-selling ruffian because I happened to be sitting in his deck chair; one of
the Protectorate officials a week ago accused me of being a smuggling gun-
runner, because I've been up country and happened to get on with the native
local headmen instead of scrapping with them, and Miss K. O'Neill, of our
mutual acquaintance, has given me to understand that if I don't quit
poaching on what she's pleased to call O'Neill and Craven's territory, she'll
run me out of business. To give her her due I gather she proposes to pay me
something to clear out."

"And you're going to take it from her?"

"Don't say 'her' so tragically. I'm not going to take anything from her, or
from anyone else. I've got a mine, and it's a nailing good mine, and I'm
going to run it by my lone or bust. It isn't a thing you could sell to a
company, and besides it isn't one of those mines one would care to sell. It's
too good for that. It's just a fortune for two people, and one of them is
presently going to sign herself Laura Carter."

"George, you're quite the best man on earth."

"I doubt it myself at times. By the way, who should I see down in Las
Palmas just now but Cascaes. He did me the honor of ignoring my
existence. It wasn't the unshaved Coast Cascaes either; he'd got a clean blue
chin, and the rest of him was dressed fit to kill. Now, what is the mysterious
Cascaes doing here?"

"He's O'Neill and Craven's agent for Grand Canary. I thought you'd
heard."

"No, it's news to me. It's news, moreover, that they had any business here
that required an agent."

"They haven't."

"Hum," said Carter. "Miss O'Neill doesn't pay a salary without getting
value for it. Now this is one of her deep-laid schemes."

Laura looked at him queerly. "Yes," she said, "this is one of Kate's deep-
laid schemes, George. I wonder if you can see through it."

The sun above them scorched high, and the cool white buildings of the
banana farmer threw the shortest of purple shadows. The fresh breath of the
trade rustled the ferns and the palm leaves of the garden, and stirred the
great masses of the bougainvillea into rhythmical movement. "It's grand to
be in a place like this after a spell on the Coast," said Carter.

"Do you prefer it to England?" Laura asked pointedly.

Carter held down a sigh. "I believe I do," he said steadily. "Come, now,
old lady, what do you say? Shall we buy a property here in Grand Canary,
and settle down, and grow the finest flower garden in the island?"

"But roses are your favorite flower and they don't do well here in the
South."

"Oh, it's roses that my father cares for, at least he and the mater together
run the roses at home. But I think my taste runs more to bougainvillea, say
—and great trees of scarlet geranium with stalks as thick as one's leg, and
palms, and tree ferns. Besides, a garden means irrigation here, and I've
never had a real water-works scheme of my own to play with since I was a
kid and worked out a most wonderful system by the old smelt mill at home.
Yes, we should have great times gardening out here."

They had never said so in words, but both of them knew that George
Carter would never take Laura back to England when once he had married
her, and the girl through all her fierce tropical love for him recognized what
this self-denial must cost and valued it to the full. But presently she brought
him back to the matter they had been talking of before.

"Can't you see why Kate sent Senhor Cascaes here, George?"

"I haven't given him another thought. Besides, although Miss O'Neill is
seeing fit to interfere with me, I don't intend to meddle with her."

"I think you ought to defend what's your own."

"Certainly I shall. Can anyone accuse me of not doing so? But I don't see
why you keep harping on Cascaes. The man is an open admirer of Miss
O'Neill's, and I suppose she's tickled thereby. Anyway that's the only reason
I can see why she should have provided him with a job."
 "Do you mean to say you think it is Kate the Senhor Cascaes is running
after?"

"Certainly I do. Who else was there at Mokki?"

"Do you think I've so few attractions then?"

"But, my good girl, you're engaged to me, and he knew it all along.
There was no secret about our engagement. Everybody about the factory
knew of it."

"And because a girl is engaged, or even married, do you think that's any
bar to another man admiring her?"

Carter whistled. "I've been a blind ass, and I must say I did refuse to
listen to the highfalutin' nonsense Cascaes wanted to pour into my
sympathetic ear. How often have you seen him here in Grand Canary?"

"He has called every day."

"That's not answering my question."

"George, dear, give me credit for loyalty. He told me one day when you
were building that fort at Mokki that he liked me, and that if the Okky-men
came he would die cheerfully before any harm should come to me; and I
told him that he had no right to say such things to a girl who was engaged to
you."

"Why wasn't I told of this?"

"Because he said to me he had nearly shot you once, and I was afraid
that if there was any trouble, dear, you might be hurt."

"You could have trusted me," said Carter dryly, "to keep my end up with
a dago like that. Besides, if you'd given me the tip, I could have seen to it
that I had the drop on him first."

Laura shivered. "You are rather mediæval. I don't want to be fought for."
 "Still, I gather from what you say that you've been seeing the fellow
here?"

"Never when I could help it. Each day I've refused to see him when he
came to the house. But he has waited for me when I went out into the
country, and once he was here in the garden, sitting on this very seat, when
I came out after lunch."

"Does he always tell the same old tale?"

"He says always he wants to marry me."

"I thought you said you refused to listen to him?"

"George, don't be unreasonable. I've told him over and over again it's no
use; I've gone away every time we've met; but it seems to be the one
occupation of his life."

"Except for running after you, I can imagine he does have plenty of time
on his hands out here."

"Don't you think, George, he was sent to the island to have nothing to do
except that?"

"Sent here who by? By Miss O'Neill, do you mean? Great Christopher!
Laura, what morbid idea will you have in your head next? I don't flatter
myself that outside business Miss O'Neill cares whether I'm alive or dead,
and as for you, well, the pair of you may be friendly enough when you were
kids, but you seemed to have outgrown any past civilities last time I saw
you together on the Coast. Don't you go and run away with any wild cat
notions about Miss O'Neill. She's got one amusement in the world, and
that's business, and if she's sent Cascaes here to Las Palmas, you can bet
your best frock the only job he's got in view so far as she's concerned is
dividend hunting. Apropos of which, I nearly forgot. Here's something to
practise your autograph in."

"Why, it's a check-book."

 "Clever girl. Guessed it in once. I just opened a credit for you down at
the bank in Las Palmas for £500 to be going on with. That's for chocolate,
and hairpins, and a mantillina, and the latest thing in Spanish slippers. I say,
Laura, you must get a pair of those tan ones, with the laces tied in a bow
just down over the toe. And if you don't go through the lot whilst I'm away
squaring mine matters up in England, I shall take you solemnly round the
shops when I come back here, and buy you a trousseau of all the ugliest and
most unbecoming garments they have in stock."

"You are good to me, dear. But I can never spend all that."

"If you've any balance you find unwieldy, buy Cascaes a smile with it, if
you can find one that will fit. No, seriously, old lady, you will be marrying a
rich man, although you did not know it when you took him, and you may as
well get used to spending. It's no use for us preparing to save."

"No use preparing to save," poor Laura repeated miserably to herself.

"There will be no—no one except ourselves to look forward to." But she
said nothing of this aloud. She just thanked him, and snuggled in to his
shoulder and patted his sleeve.

Far away over the corner of the isle a steamer hooted in the harbor of the
Isleta, and the sound came to them dimly through the foliage plants. Carter
looked at his watch. "Hullo, I must go, or the criminal who drives my
tartana will flog that poor beast of a mule to death in his effort to catch the
boat. So now, Miss Slade, just please give me a sample of your best good-
bye."

Twilight does not linger in the summer months, even so far north as
Grand Canary. The sun was balanced in lurid splendor on the rocky
backbone of the isle as Carter said his last words of farewell, making the
dead volcanoes look as though at a whim they could spring once more into
scarlet life. It was dark when he got on the road, and the evening chill rode
in on the Trade. The mouse-colored tartana mule sneezed as he pressed his
galled shoulders into the collar.

Carter wedged himself in a corner of the carriage and resolutely looked

on life with a reckless gayety. After all, what was this ache called Love? To
the devil with it! Hereafter he would eat, and drink, and work, especially
work, and—well, Laura was a good sort, and he intended to play the game,
and please her. He had given his word to Laura, he forgot exactly why, but
he had given it, and that was enough. For good or evil he was one of those
dogged Englishmen who keep to a promise that had once been given.

Then with an equal doggedness he thrust all these things from his mind,
and resolutely clamped down his thoughts to Tin Hill and the details of its
working. No news had reached him of the importance which the freakish
British public had placed upon his little arrangement about that detail of the
human sacrifices. He saw himself merely as an unknown business man who
in the near future would be able to sway a thing which at present he knew
nothing about, and that was the tin market. The idea unconsciously
fascinated him. He had no enmity against the present producers of tin, did
not know indeed who they were, but he smiled grimly as he thought of the
way in which presently he would govern them. It was the lust for power,
which is latent in so many men, leaping up into life.

The brilliant stars shone down on him from overhead, and the cool Trade
carried to him salt odors of the sea, but they got from him no attention. His
mind was journeying away in the African bush, on spouting river-bars, in
offices, on metal exchanges....

He was roused from these dreams with much suddenness. In his up

country journeying in Africa he had developed that animal instinct for the
nearness of danger which is present in us all, but in nine hundred and
ninety-nine men out of the thousand becomes atrophied for want of use. In
the river villages the natives had given him a name which means Man-with-
eyes-at-the-back-of-his-head.

It was this slightly abnormal sense that sprang into quick activity, and
Carter made so sudden a stoop that his face smacked against the shabby
cushions on the opposite side of the tartana. But simultaneously he turned
and clutched through the night, and seized a wrist, and held it with all his
iron force. A moment later he found with his other hand that the wrist was
connected with a long bright-bladed knife, so he twisted it savagely till that
weapon fell onto the dirty carpet on the floor. And all the time, be it well
understood, no sounds had been uttered, and the mouse-colored mule
jogged steadily on with the tartana through the dust and the night.

Then Carter began to haul in on the wrist, and the man to whom it was
attached came over into the body of the vehicle, bumping his knees
shrewdly against the wheel-spokes en route.

"Ah, Cascaes, that's you, is it? And I thought once you claimed to be a
gentleman, and agreed not to go at me from behind? Well, I'm afraid there's
only one kind of medicine that will suit you, and that's the kind one gives to
dogs that turn treacherous. Have you got any suggestions to make?"

The Portuguese held his tongue.

"Ready to take your gruel, are you? Well, I propose to give you a full
dose. Hi there, driver, pull up. Wake, you sleepy head! What is it? Why, I've
picked up a passenger whilst you've been nodding, and now we want to get
down for a minute. Here, give me your whip."

* * * * *

Carter's arm was lusty and his temper raw. Moreover, the whip, being the
property of a Las Palmas tartana driver, was made for effective use.

"I may not cure you," said Carter between thumps, "of a taste for cold-
blooded assassination, but I'm going to make the wearing of a coat and
breeches an annoyance to you for the next three weeks at any rate." After
which statement, as the whip broke, he flung the patient into the aloe hedge
at the side of the road, got back into the tartana and told the driver to hurry
on to the Isleta, or they'd miss the boat.

CHAPTER XX
MAJOR MEREDITH
 "The Liverpool Post," said Mrs. Craven, "allows itself to hint gently that
you've been rather persecuting Mr. Carter, Kate. Now, I don't call the Post a
sensational paper, nor is it given to introducing personal matters, as a rule."

"I wish it would mind its own business and leave mine alone," said Kate
crossly.

"'The oppression of nations or individuals,'" read Mrs. Craven, "'may

begin by being a matter of merely domestic importance, but when it
assumes sufficient dimensions it forces itself into public notice.'"

"Do they couple my name with that?"

"They leave you to do that yourself," said the old lady dryly.

"Well, I don't mind. They may say what they like. I'm entirely within my
rights."

"The Post admits that. Here, I'll read you what it says, my dear. 'Mr.
George Carter, whose name has been so prominently before the public of
late in connection with his splendid efforts in winning over the King of
Okky to the side of humanity, has himself been the victim of some very
high-handed oppression. He has discovered a most valuable vein of tin in a
part of the back country where no European explorer had ever trod before,
and with toil and care, and in fact with genius, had brought cargo after
cargo of the valuable ore down mysterious African creeks and rivers to a
spot where the ocean steamers could conveniently ship it. To be precise, he
hired from Messrs. Edmondson's small factory on the Smooth River a piece
of waste-cleared ground, dumped his ore on that as he towed it tediously
down those unknown creeks in a string of dugouts, and there let it
accumulate so as not to flood the markets, and cause ruin to the tin
industries in England—' Shall I go on?"

"Please do, Aunt."

"'But presently an interviewer arrived in the shape of a well-known firm

of West African merchants and financiers, who bought out Messrs.
Edmondson's interest in their Smooth River factory, found that Mr. Carter
had no lease, and gave him notice to quit within forty-eight hours. As an
alternative to removal they demand an annual rent which amounts to more
than fifteen per cent. of the value of the ore stacked there. In other words,
they are endeavoring, in a manner that almost smacks of piracy, to force
themselves into partnership with him.'"

"Sneak," said Miss O'Neill, "to go and tittle-tattle to the papers like that."

Mrs. Craven looked at the girl over her spectacles, and then said she,
"Wait a minute till I read you a little more. 'We should add that what gives
these proceedings a more unpleasant flavor than would appear at first sight
is the fact Mr. Carter is unable to defend himself. He had left West Africa
when action was first taken, and it has been discovered that he was still in
ignorance of what had occurred when his steamer called at Las Palmas. The
whole thing will be sprung upon him with a shock of unpleasant surprise
when he lands in Liverpool to-morrow."

"Ah," said Kate.

Mrs. Craven folded the paper, stood up, and walked towards the door.
"As usual, my dear, you have carried out your plan very perfectly."

"What plan?" asked Kate incautiously.

"Of treating Mr. Carter so badly," said Mrs. Craven, turning the handle,
"that presently when he hits you back you will be able to bring yourself to
hate him. But then you are always successful, Kitty dear, in everything you
set your hand to—tryingly successful sometimes," Mrs. Craven added, and
went out, and shut the door softly behind her.

Kate nodded at the door. "Aunt Jane," she said viciously, "there are
moments when you are a perfect cat. But I will make him detest me for all
that, and then I can truly and comfortably hate him. It's all very well their
calling him a martyr. Why should everybody's feelings be consulted except
mine?"

All the same, Kate bowed in a certain degree to public sentiment. One
thinks also that she had not toughened herself sufficiently to meet Carter
face to face. Anyway, she discovered that urgent affairs called her to
London, and whirled off Aunt Jane to her flat that very night. She left
Crewdson to fight the invader when he landed in Liverpool, and gave the
old man definite instructions in writing that he was not to budge an inch
from the firm's rights. "Show Mr. Carter this letter," she ordered, "if there is
the least occasion for it."

But it seemed that West Africa pursued her. The telephone rang as soon
as she got to the flat.

"That London? That Miss Head? This is Liverpool, Crewdson. London's

just been calling you up. Will you ring Four-owe-seven-three Pad. What's
that? No. Four-naught-seven-three Pad. Yes, that's it. Good-night, Miss."

Kate had more than half a mind to let 4,073 Pad alone. She was tired,
and somehow in spite of all her successes she was a good deal dispirited.
The British public had bought no less than four great rubber companies that
she had offered them; the shares were all at a premium; everybody was
pleased; and she had transferred her own profits safely into land and trustee
securities. Since her first burst of success, money had simply rolled in on
her, and already it had ceased to give her amusement. Success lay sour in
her mouth. She asked Fortune for just one thing more. Because she was a
woman she could not go and get it for herself. She told herself that it was
only a convention that held her back—but she shuddered and chilled all
over at the thought of breaking that convention.

She sat in a deep soft chair, twisting her long gloves into a hard string,
and staring into the glow of the fire, and then with a "Faugh" at her own
weakness, she threw the gloves onto the fender, and walked across to a
telephone that stood on a side-table.

"Four-owe-seven-three Pad, please. No, Forty-seventy-three Paddington.

Yes. Hullo? Hullo? Is that Four-nought-seven-three? This is Miss O'Neill.
Liverpool rang up to say you wanted to speak to me. Who is that, please?"

"No one you know," came in the small clear voice of the telephone. "One
of those sort of people who writes letters to the papers above some such
signature as 'Well-Wisher.'"
 "If you don't give me your name," said Kate sharply, "I shall ring off."

"I don't think you will when I tell you I'm going to give you some news
about your father."

"My father unfortunately is dead. You've got hold of the wrong Miss
O'Neill."

The telephone laughed. "Not a bit of it, it's the lady who is known
generally as Kate O'Neill I'm speaking with, but whose real name is
Katherine Meredith."

Now Kate knew that Mrs. Craven was only "Aunt Jane" by courtesy and
adoption, and had naturally wondered many times over who her real people
might have been. She had always been a very practical young woman, and
had not worried herself unduly over the matter; but still being human, she
had her share of curiosity, and though the subject had always been strictly
taboo at the house in Princes' Park, still that did not hinder her from
discussing it with her own thoughts. And now, "Katherine Meredith!"

"I think you had better tell me who you are," she said to the telephone.

"I prefer anonymity. Do you know Day-Pearce?"

"No. Yes, perhaps I do, if you mean Sir Edward Day-Pearce, the West
African man. I don't know him personally."

"All the better," rasped the telephone. "Anyway, he is lecturing to-night

in a non-Conformist temple in Westbourne Grove—the Athenæum, they
call it. Begins at eight. He's certain to say something about Meredith. I
should try to go if I were you."

"I shouldn't dream—" Kate began, when whizz went the bell, and she
was cut off. She rang again, got the inquiry office, found that 4,073 was a
hairdresser's shop, once more got 4,073, spoke to the proprietor, learned
that the telephone had been hired for an hour by a gentleman who had some
business to transact. No, the gentleman had just gone. No, they didn't know
who he was: never seen him before—Miss O'Neill's ring off had a touch of
temper in it.

She went back to the deep soft chair and tried to bring her thoughts once
more to the subject that had been in hand before the interruptions came. She
was a business woman, and had trained herself to concentrate the whole of
her mind on any matter she chose. But somehow those two little words "My
father" kept cropping up; and presently she began trying to picture what her
mother was like. She went to the telephone and called up a theatre agency.
She had to say three times over "Athenæum—Westbourne Grove" before
the young man at the other end grasped the name, and she was rewarded by
hearing him laugh as he said he had no seats for Sir Edward Day-Pearce's
lecture that evening.

"Where can I get one?" she demanded.

"At the door, madam," was the polite response. "I believe the prices of
entrance are threepence, sixpence, and one shilling, unless you happen to be
a subscriber."

Supposing the whole thing were a hoax to draw her there, and by some
means to make her look ridiculous? It was quite likely. She was a successful
woman, and had already learned that one of the prices of success is the
spitting of spite and envy. But difficulties did not often stay long in the path
of Miss Kate O'Neill. She picked up a telephone directory, turned the pages,
found a number, called it up, and made certain arrangements. Thereafter she
dressed, dined, and took Mrs. Craven to laugh over the new piece at the
Gaiety.

But poor Kate found even the Gaiety dull that night. There was a man on
the stage with a red head. He was not in the least like Carter either in looks,
speech, or manner, but—well, it must have been the hair which persisted in
calling up that unpleasant train of thought which kept her vaguely irritated
throughout all the evening.

There was a bundle of type script waiting for her when she got back to
the flat, which happened to be the verbatim report of Sir Edward Day-
Pearce's lecture which she had arranged that two stenographers should go