Front cover

Introduction Guide to the

IBM Elastic Storage System

John Sing
Lee Nee Helgeson

Redpaper
International Technical Support Organization

Introduction Guide to the IBM Elastic Storage System

February 2021

REDP-5253-02
 Note: Before using this information and the product it supports, read the information in “Notices” on
page vii.

Third Edition (February 2021)

This edition applies to IBM Spectrum Scale and IBM Elastic Storage Systems, and IBM Elastic Storage
Servers.

This document was created or updated on March 23, 2021.

© Copyright International Business Machines Corporation 2016, 2021. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

Summary of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

March 2021, Third Edition (minor updates) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
February 2021, Third Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
July 2020, Second Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Chapter 1. Introduction to the IBM Elastic Storage System . . . . . . . . . . . . . . . . . . . . . . 1

1.1 IBM Elastic Storage System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 IBM Elastic Storage System and IBM Spectrum Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.1 IBM Spectrum Scale overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 Protocol nodes overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3 IBM Elastic Storage System storage building block solution for IBM Spectrum Scale
8
1.3 IBM Elastic Storage System solution value. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3.1 Integrated and reliable storage building block. . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.2 Predictable, consistent high performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.3 New simplified IBM ESS installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.4 Supported as a solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 IBM Elastic Storage System models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 2. IBM Elastic Storage System architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.1 IBM Elastic Storage System architecture overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 IBM Elastic Storage System hardware components . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 IBM Elastic Storage System models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.2 IBM Elastic Storage System 3000. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.3 IBM Elastic Storage System 5000. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.4 Positioning various IBM Elastic Storage System models . . . . . . . . . . . . . . . . . . . 21
2.2.5 IBM Elastic Storage System Management Server . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.6 Data Server hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.7 Storage enclosures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.2.8 Protocol nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2.9 Networking switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.2.10 Racking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.3 IBM Elastic Storage System software components . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3.1 Software solution stack overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3.2 Operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.3 IBM Spectrum Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.3.4 IBM Spectrum Scale RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.3.5 IBM Elastic Storage System solution installation and management scripts . . . . . 41

Chapter 3. IBM Elastic Storage System planning and integration . . . . . . . . . . . . . . . . 45

© Copyright IBM Corp. 2016, 2021. All rights reserved. iii

 3.1 IBM Elastic Storage System planning overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.1.1 Roles and responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2 IBM Elastic Storage System hardware planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2.1 Planning for network switches and networking . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2.2 Planning for servers that are used in IBM Elastic Storage System. . . . . . . . . . . . 52
3.2.3 Planning for storage configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2.4 Planning for IBM Spectrum Scale protocol nodes. . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.5 Planning for physical, electrical, and cooling installation . . . . . . . . . . . . . . . . . . . 56
3.3 IBM Elastic Storage System software planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3.1 Planning for IBM Spectrum Scale file system . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3.2 Planning for IBM Spectrum Scale RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.4 Ordering IBM Elastic Storage System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.5 IBM Elastic Storage System installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.5.1 Physical installation, racking, and cabling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.5.2 Integrating into client environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.5.3 Reliability, Availability, Serviceability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.6 Monitoring IBM ESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.6.1 GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.7 Maintaining and repairing IBM ESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.7.1 ESS IBM Solution service and support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.8 Upgrading IBM ESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.8.1 ESS hardware and firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.8.2 ESS software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.9 ESS hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Appendix A. IBM Elastic Storage System models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

A.1 IBM Elastic Storage System model specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
A.1.1 First-generation IBM Elastic Storage System specifications . . . . . . . . . . . . . . . . 70
A.1.2 Second-generation IBM Elastic Storage System specifications . . . . . . . . . . . . . . 70
A.1.3 Second-generation IBM Elastic Storage System Hybrid models . . . . . . . . . . . . . 72
A.1.4 Third-generation IBM Elastic Storage System 3000 specifications . . . . . . . . . . . 73
A.1.5 Third-generation IBM Elastic Storage System 5000 specifications . . . . . . . . . . . 74
A.2 First-generation IBM Elastic Storage System models overview . . . . . . . . . . . . . . . . . . 75
A.2.1 First-generation IBM Elastic Storage System models . . . . . . . . . . . . . . . . . . . . . 75
A.2.2 First-generation IBM Elastic Storage System GSx models . . . . . . . . . . . . . . . . . 76
A.2.3 First-generation IBM Elastic Storage System GLx models. . . . . . . . . . . . . . . . . . 77
A.2.4 First-generation IBM Elastic Storage System Data Servers . . . . . . . . . . . . . . . . . 77
A.2.5 First-generation IBM Elastic Storage System storage enclosures . . . . . . . . . . . . 78
A.3 Second-generation IBM Elastic Storage System overview. . . . . . . . . . . . . . . . . . . . . . 79
A.3.1 IBM Elastic Storage SystemGSxS models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
A.3.2 IBM Elastic Storage SystemGHxy Hybrid models with SSD and HDD. . . . . . . . . 80
A.3.3 IBM Elastic Storage System GLxS models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
A.3.4 IBM Elastic Storage System GLxC models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
A.4 Differences between the IBM Elastic Storage System generations . . . . . . . . . . . . . . . 84
A.4.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
A.4.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
A.4.3 IBM Elastic Storage System data server IO node . . . . . . . . . . . . . . . . . . . . . . . . 87
A.4.4 IBM Elastic Storage System Management Server . . . . . . . . . . . . . . . . . . . . . . . . 87
A.4.5 Hardware Management Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
A.5 IBM Spectrum Scale for IBM Elastic Storage System licensing . . . . . . . . . . . . . . . . . . 88
A.5.1 Socket licensing on first-generation IBM Elastic Storage System . . . . . . . . . . . . 88
A.5.2 Capacity Licensing on second- and third-generation IBM Elastic Storage System 89
A.6 IBM Spectrum Scale RAID technical overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

iv Introduction Guide to the IBM Elastic Storage System

 A.6.1 IBM Spectrum Scale RAID erasure codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
A.6.2 End-to-end checksum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
A.6.3 Declustered RAID arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
IBM Redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

Contents v
vi Introduction Guide to the IBM Elastic Storage System
Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such
provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION
PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR
IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of
express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring
any obligation to you.

Any performance data contained herein was determined in a controlled environment. Therefore, the results
obtained in other operating environments may vary significantly. Some measurements may have been made
on development-level systems and there is no guarantee that these measurements will be the same on
generally available systems. Furthermore, some measurements may have been estimated through
extrapolation. Actual results may vary. Users of this document should verify the applicable data for their
specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to the names and addresses used by an actual business
enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs.

© Copyright IBM Corp. 2016, 2021. All rights reserved. vii

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation in the United States, other countries, or both. These and other IBM trademarked terms are
marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US
registered or common law trademarks owned by IBM at the time this information was published. Such
trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States,
other countries, or both:
Redbooks (logo) ® IBM FlashSystem® POWER®
IBM® IBM Spectrum® POWER8®
IBM Elastic Storage® PartnerWorld® POWER9™
IBM FlashCore® Passport Advantage® Redbooks®

The following terms are trademarks of other companies:

The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.

Ansible, Red Hat, are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the United
States and other countries.

Other company, product, or service names may be trademarks or service marks of others.

viii Introduction Guide to the IBM Elastic Storage System

Preface

This IBM® Redpaper® publication provides an overview of the IBM Elastic Storage® Server
and IBM Elastic Storage System. These scalable, high-performance data and file
management solution, are built on proven IBM Spectrum® Scale technology. Providing
reliability, performance, and scalability, IBM Elastic Storage Servers and IBM Elastic Storage
Systems can be implemented for a range of diverse requirements.

This publication helps you to understand the solution and its architecture. It describes
ordering the best solution for your environment, planning the installation and integration of the
solution into your environment, and correctly maintaining your solution.

The solution is created from the following combination of physical and logical components:
򐂰 Hardware
򐂰 Operating system
򐂰 Storage
򐂰 Network
򐂰 Applications

Knowledge of the IBM Elastic Storage Server and IBM Elastic Storage System components is
key for planning an environment.

This paper is targeted toward technical professionals (consultants, technical support staff,
IT Architects, and IT specialists) who are responsible for delivering cost effective cloud
services and big data solutions. The content of this paper can help you to uncover insights
among client’s data so that you can take appropriate actions to optimize business results,
product development, and scientific discoveries.

Authors
This paper was produced by a team of specialists from around the world working with
International Technical Support Organization (ITSO), Tucson Center.

John Sing is Offering Evangelist for IBM Spectrum Scale, Elastic Storage Server. In his over
25 years with IBM, John has been a world-recognized IBM speaker, author, and strategist in
Enterprise Storage, File + Object Storage, internet scale workloads and data center design,
big data, cloud, IT strategy planning, high availability, business continuity, and Disaster
Recovery. He has spoken at over 40 IBM conferences worldwide and is the author of eight
IBM Redbooks® publications and nine IBM Redpaper publications.

Lee Nee Helgeson is the World Wide Quality Program Manager for the Elastic Storage
Systems. She ensures IBM ESS continuously meets IBM quality standards from design,
manufacturing, and supplier to the field and back. Lee has been working with IBM ESS and
IBM POWER® Systems at IBM for over 16 years. She has a Bachelor of Science degree in
Computer Science from Michigan Technological University and a Master of Business
Administration from Augsburg College.

© Copyright IBM Corp. 2016, 2021. All rights reserved. ix

 Thanks to the following people for their contributions to this project:

Larry Coyne
International Technical Support Organization, Tucson Center

Puneet Chaudhary, Chiahong Chen, Matthew Drahzal, Dan Dumarot, Steve Duersch, Isaiah
Eaton, Rezaul Islam, Wesley Jones, Christopher Maestas, Don Mathisen, Jim O'Connor, J D
Zeeman
IBM Systems

Steven Daniels, Par Hettinga, Richard Rupp, Lindsay Todd

IBM Global Markets

Thanks to the authors of the first edition:

Dino Quintero, Puneet Chaudhary, Brian Herr, Steven Normann, Marcelo Ramos, Richard
Rosenthal, Robert Simon, Marian Tomescu, Richard Wale

Now you can become a published author, too!

Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an ITSO residency project and help write a book in your
area of expertise, while honing your experience using leading-edge technologies. Your efforts
will help to increase product acceptance and customer satisfaction, as you expand your
network of technical contacts and relationships. Residencies run from two to six weeks in
length, and you can participate either in person or as a remote resident working from your
home base.

Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html

Comments welcome
Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about this paper or
other IBM Redbooks publications in one of the following ways:
򐂰 Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
[email protected]
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

x Introduction Guide to the IBM Elastic Storage System

Stay connected to IBM Redbooks
򐂰 Look for us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks
weekly newsletter:
https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
򐂰 Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html

Preface xi
xii Introduction Guide to the IBM Elastic Storage System
 Summary of changes

This section describes the technical changes made in this edition of the paper and in previous
editions. This edition might also include minor corrections and editorial changes that are not
identified.

Summary of Changes
for Introduction Guide to the IBM Elastic Storage System
as created or updated on March 23, 2021.

March 2021, Third Edition (minor updates)

This revision includes the following new and changed information.

Changed information
Updated to reflect announcement of additional IBM ESS 5000 models SL7 and SC9,
enhanced POWER9 ESS Management Server support for 2nd Gen ESS, ESS 5000
model/capacity upgrades, HDR 100Gbit network cards.

February 2021, Third Edition

This revision includes the following new and changed information.

Changed information
Added IBM ESS 5000 model with POWER9 data servers with 5147-092 or 5147-106 storage
enclosures.

July 2020, Second Edition

This revision includes the following new and changed information.

Changed information
Updated correct 5148-21L model number on pages 22, 29, and 47.

© Copyright IBM Corp. 2016, 2021. xiii

xiv Introduction Guide to the IBM Elastic Storage System
 1

Chapter 1. Introduction to the IBM Elastic

Storage System
This chapter introduces the IBM Elastic Storage Server (ESS) and IBM Elastic Storage
System solutions. In this chapter, we describe their characteristics, value, how they integrate
with IBM Spectrum Scale, and where they fit in today’s business environments.

This chapter also describes some of the software and hardware characteristics of the IBM
Elastic Storage Server and IBM Elastic Storage System, IBM Spectrum Scale RAID
(Redundant Array of Independent Disks) software, and the storage building block concepts of
the solution.

This chapter includes the following topics:

򐂰 1.1, “IBM Elastic Storage System” on page 2
򐂰 1.2, “IBM Elastic Storage System and IBM Spectrum Scale” on page 4
򐂰 1.3, “IBM Elastic Storage System solution value” on page 10
򐂰 1.4, “IBM Elastic Storage System models” on page 13

Note: In this IBM Redpaper publication, IBM ESS can refer to either of the following
products:
򐂰 Elastic Storage Server (the first and second generations of this solution)
򐂰 Elastic Storage System (the third generation of this solution). For more information
about the IBM ESS 3000 and IBM ESS 5000, see the following publications:
– Implementation Guide for IBM Elastic Storage System 3000, SG24-8443
– Implementation Guide for IBM Elastic Storage System 5000, SG24-8498

The general capabilities of these solutions are the same. Where suitable, we differentiate
between the models and generations.

© Copyright IBM Corp. 2016, 2021. All rights reserved. 1

1.1 IBM Elastic Storage System
The IBM Elastic Storage System and IBM Elastic Storage Server (both referred to as IBM
ESS) are integrated IBM Spectrum Scale storage solutions for high-performance, reliable file,
and object storage. Scalable to petabyte levels, they are suitable for analytics, big data,
artificial intelligence, and enterprise data ocean requirements. IBM ESS provides scalable
storage for IBM Spectrum Scale (see Figure 1-1), which is IBM’s proven parallel clustered file
system.

Figure 1-1 Elastic Storage Systems integrated with IBM Spectrum Scale solutions

IBM ESS is a pair of cross-connected IBM Spectrum Scale Data Servers, which are deployed
in a fully integrated, tested storage building block solution (see Figure 1-2 on page 3). Instead
of hardware-based disk RAID controllers, IBM ESS uses IBM Spectrum Scale RAID, which
uses declustered erasure code technology to deliver consistent high performance. This
feature mitigates storage drive failures and RAID rebuild overhead.

2 Introduction Guide to the IBM Elastic Storage System

Figure 1-2 IBM ESS 3000 and IBM ESS 5000 integrated with IBM Spectrum Scale

IBM ESS with IBM Spectrum Scale RAID also provides superior data protection by reducing
rebuild times to a fraction of the time that is needed with conventional hardware-based RAID
controllers.

Data access to IBM ESS storage is available by using one of the following methods:
򐂰 IBM Spectrum Scale clients on user workstations, which provide a POSIX file system
interface to data on IBM ESS.
򐂰 IBM Spectrum Scale protocol nodes, which allow IBM ESS data access through Server
Message Block (SMB), Network File System (NFS) file protocols, or object data access
through IBM Spectrum Scale’s implementation of OpenStack Swift. In addition, Hadoop
clusters can access IBM ESS data by using the IBM Spectrum Scale Hadoop
Transparency Connector.

The IBM Elastic Storage System or IBM Elastic Storage Server is a software-defined storage
building block that combines IBM Spectrum Scale high-performance parallel clustered file
system (suitable for petabyte scale data storage) and in most models of IBM ESS, the CPU
and I/O capability of the IBM POWER® server architecture.

IBM Elastic Storage Systems and IBM Elastic Storage Servers integrate IBM Spectrum
Scale, IBM Storage enclosures, IBM POWER Systems, or x86 servers (depending on the
IBM ESS model), into a complete management and installation software stack. It also
provides an integrated tested IBM Spectrum Scale storage building block solution. IBM
Service supports IBM ESS as an integrated solution.

An IBM ESS base building block consists of a 1 GbE network switch, servers for IBM ESS
Management nodes and I/O nodes or data server nodes, and I/O storage enclosures. Multiple
IBM ESS building blocks can be integrated into an IBM Spectrum Scale cluster.

Chapter 1. Introduction to the IBM Elastic Storage System 3

1.2 IBM Elastic Storage System and IBM Spectrum Scale
In this section, we describe how IBM ESS is part of an IBM Spectrum Scale cluster.

Within the IBM Spectrum Scale cluster, IBM ESS is an integrated IBM Spectrum Scale
storage building block that reads and writes data for IBM Spectrum Scale users. IBM ESS
combines storage hardware with IBM Spectrum Scale software to manage data in storage.

In the following sections, we provide a brief overview of the following topics:

򐂰 What is an IBM Spectrum Scale cluster
򐂰 What is an IBM Spectrum Scale Client and IBM Spectrum Scale Data Server
򐂰 How IBM ESS is an integrated IBM Spectrum Scale storage building block

Note: IBM ESS requires an IBM Spectrum Scale cluster around it; you cannot use IBM
ESS stand-alone without other IBM Spectrum Scale nodes.

Here, we provide a high-level IBM Spectrum Scale and IBM ESS solution overview. For more
information, see Chapter 2, “IBM Elastic Storage System architecture” on page 15.

1.2.1 IBM Spectrum Scale overview

IBM Spectrum Scale is IBM’s strategic high-performance parallel file system shared storage
platform for end-to-end collaborative common enterprise, data platform, big data analytics,
and AI workflows (see Figure 1-3).

Figure 1-3 IBM Spectrum Scale overview

4 Introduction Guide to the IBM Elastic Storage System

IBM Spectrum Scale
IBM Spectrum Scale is designed to provide the following major value propositions:
򐂰 Simplifies data management by supporting enterprise workflows on a single common
enterprise data platform
򐂰 Supplies a single global namespace that supports enterprise-level data over
high-performance networks
򐂰 Enables intelligent automatic tiering of data between storage pools, and externally to tape,
object and cloud resources, which deliver cost-effective storage economics by
automatically managing and tiering data to different classes of storage

Although multiple types of IBM Spectrum Scale cluster configurations are available, the
configuration into which IBM ESS is commonly deployed is the IBM Spectrum Scale Network
Shared Disk (NSD) configuration, as shown in Figure 1-4.

Figure 1-4 IBM Spectrum Scale cluster

IBM ESS is a pair of IBM Spectrum Scale NSD Data Servers, which are configured together
as a tested, integrated, highly available, and reliable IBM Spectrum Scale storage building
block solution.

Chapter 1. Introduction to the IBM Elastic Storage System 5

 As shown in Figure 1-4 on page 5, eight IBM Spectrum Scale nodes are application
workstations, servers, or users. Four nodes also are IBM Spectrum Scale data server nodes.
The user workstations are running the IBM Spectrum Scale client in this IBM Spectrum Scale
configuration. The IBM Spectrum Scale client provides multi-threaded, highly parallel access
data on the IBM Spectrum Scale Data Servers, as shown in Figure 1-5.

IBM Spectrum Scale clients and servers

IBM Spectrum Scale IBM Spectrum Scale

client
cluster

High speed
TCP/IP
or InfiniBand
network

IBM Spectrum Scale

data servers

Storage Storage

IBM Spectrum Scale client does real-time parallel I/O

to all IBM Spectrum Scale data servers and storage volumes/NSDs
(Network Shared Disks)

Figure 1-5 IBM Spectrum Scale clients and servers

The IBM Spectrum Scale client achieves high performance by performing simultaneous
real-time parallel I/O to all IBM Spectrum Scale data servers and storage volumes and NSDs
simultaneously.

6 Introduction Guide to the IBM Elastic Storage System

 An IBM Spectrum Scale cluster can grow by adding nodes, whether they are IBM Spectrum
Scale clients or IBM Spectrum Scale data servers, as shown in Figure 1-6.

Figure 1-6 IBM Spectrum Scale grows by adding scale-out nodes

An IBM Spectrum Scale cluster can provide 1 - 256 logical POSIX file systems to users and
workstations. The IBM Spectrum Scale client provides the appearance of a mountable POSIX
file system to the applications and users on the workstation where the IBM Spectrum Scale
client is installed.

IBM Spectrum Scale users are unaware of the physical distribution of data in the IBM
Spectrum Scale data server physical storage pools. The automatically balanced data
distribution is seamlessly determined by the IBM Spectrum Scale policy engine at the time
the data is imported. The policy engine can also transparently move data from one storage
pool to another storage pool while the data is accessed and active.

The IBM Spectrum Scale parallel file system provides an enterprise capability for data
management of large amounts of data, while also performing constant auto-balance of
workload and storage by equally distributing I/O and data within a storage pool or among
different storage pools.

The preferred method of accessing IBM Spectrum Scale data is to install the IBM Spectrum
Scale client on every workstation or server that accesses IBM Spectrum Scale data. The IBM
Spectrum Scale client provides the multiple threads and communication with multiple data
servers to provide enterprise high-performance parallel throughput. While doing so, IBM
Spectrum Scale also manages full read/write data integrity between multiple users who are
working with the data in the file system.

1.2.2 Protocol nodes overview

Protocol nodes are part of the IBM Spectrum Scale solution. Many valid scenarios exist in
which it is also wanted to access IBM Spectrum Scale data by way of standard SMB, NFS, or
Object protocols. To provide this access, IBM Spectrum Scale provides the protocol node
functions.

Chapter 1. Introduction to the IBM Elastic Storage System 7

 The value of protocol nodes is that applications, workstations, and users that do not have the
IBM Spectrum Scale client can still access IBM Spectrum Scale data through a standard
SMB, NFS, or Object protocol. IBM Spectrum Scale is often used as an enterprise data lake
or central enterprise data repository.

Protocol nodes are IBM Spectrum Scale nodes that are designed, configured, and set up to
provide the following benefits:
򐂰 Standard SMB, NFS, or Object protocols on the client facing side
򐂰 A full IBM Spectrum Scale client that provides parallel access to IBM Spectrum Scale data
on the storage-facing side

Figure 1-7 shows protocol nodes that were added to the IBM Spectrum Scale cluster.

Figure 1-7 IBM Spectrum Scale protocol nodes

When IBM Spectrum Scale data is accessed through protocol nodes, the limiting factor on
performance almost always is the single-threaded SMB, NFS, or Object protocol. These
protocols cannot deliver the highly parallelized performance as the IBM Spectrum Scale
client. However, the value of accessing the IBM Spectrum Scale data through standard SMB,
NFS, or Object protocols is a highly desirable and flexible means for enterprise-wide user
access to IBM Spectrum Scale data.

1.2.3 IBM Elastic Storage System storage building block solution for IBM
Spectrum Scale
IBM ESS with IBM Spectrum Scale is a pair of IBM Spectrum Scale Data Servers that are
cross-configured together into a tested, integrated, highly available, and reliable IBM
Spectrum Scale storage building block. IBM ESS provides a fully tested, integrated, and
supported IBM solution to deploy, manage, and maintain IBM Spectrum Scale data and
storage by using a building block approach.

8 Introduction Guide to the IBM Elastic Storage System

Two IBM ESS models are deployed to provide most of the IBM Spectrum Scale storage, as
shown in Figure 1-8.

Figure 1-8 Elastic Storage Systems are IBM Spectrum Scale storage building blocks

IBM ESS can coexist with other IBM Spectrum Scale data servers in the same IBM Spectrum
Scale cluster. IBM ESS provides a powerful, flexible, fully integrated, and supported option for
deploying IBM Spectrum Scale storage in the most efficient manner possible.

An IBM ESS storage building block is designed as a unit of storage expansion for an IBM
Spectrum Scale cluster. If more storage is needed, an IBM ESS I/O enclosure or IBM ESS
building block can be dynamically added to an IBM Spectrum Scale cluster.

To provide high reliability and consistent high performance, IBM ESS runs IBM Spectrum
Scale RAID erasure coding, which is designed to maintain high performance even while
mitigating performance effects and transparently recovering from storage media or storage
data server failures. IBM Spectrum Scale RAID also provides checksum and disk hospital
functions to prevent silent data corruption issues and maintain high availability of petabyte
scale file systems.

As part of an IBM ESS order, you also can order more POWER servers (machine type
5148-22L) to be used as IBM Spectrum Scale protocol nodes. These protocol nodes are
managed by the IBM ESS solution software stack and the IBM ESS GUI. They also provide a
complete integrated IBM solution for an IBM Spectrum Scale cluster that includes IBM ESS
and IBM Spectrum Scale protocol nodes.

For more information about planning for the use of protocol nodes, see 2.2.8, “Protocol
nodes” on page 33.

Chapter 1. Introduction to the IBM Elastic Storage System 9

1.3 IBM Elastic Storage System solution value
The follow sections describe why the IBM ESS brings added value advantages for deploying
IBM Spectrum Scale storage.

Today’s AI and big data applications require large, high-performance, manageable, and
flexible storage growth.

The IBM Elastic Storage System is designed to provide an integrated (see Figure 1-9) and
tested IBM Spectrum Scale storage building block that provides this storage growth.
Incorporating IBM ESS storage servers flexibly adds to the overall IBM Spectrum Scale
capacity, bandwidth, and performance, all within a single global namespace.

Figure 1-9 Elastic Storage System integrated solution

As shown in Figure 1-9, IBM ESS is an integrated IBM Spectrum Scale solution that
significantly accelerates the time-to-value of deploying IBM Spectrum Scale environments.
IBM ESS provides the following benefits:
򐂰 An integrated and tested IBM Spectrum Scale storage building block
򐂰 Predictable, consistent high performance at any level of scale
򐂰 Tested and integrated with the many functions of IBM Spectrum Scale
򐂰 Supported as an integrated solution worldwide by IBM Service and Support

For more information about the Elastic Storage Server, see the following web pages:
򐂰 IBM Marketplace
򐂰 IBM Knowledge Center

10 Introduction Guide to the IBM Elastic Storage System

1.3.1 Integrated and reliable storage building block
Every part of the IBM ESS solution is engineered to provide high performance and enterprise
data availability and reliability, from entry system to petabyte scale.

IBM engineering and testing teams worked cohesively on designing, building, testing, and
delivering an end-to-end IBM ESS solution that was integrated and tested for reliability. The
complete IBM ESS solution is verified again during manufacturing.

In the deployment stage, IBM ESS-specific tools and scripts are used for deployment. IBM
Systems Lab Services are available and recommended to provide installation of the IBM ESS
solution and to assist you in integrating IBM ESS into your client environment.

IBM ESS provides standardized, optimized IBM Spectrum Scale storage configurations that
include the following benefits:
򐂰 Optimum adapters that are strategically placed on the servers
򐂰 High availability and redundancy for disks, drawers, and adapters
򐂰 Optimal cabling performance
򐂰 Tested and integrated firmware and software versions
򐂰 Policy-managed optimized placement of data
򐂰 High availability access to data
򐂰 Automated storage management

IBM ESS offers scalability from entry TBs to hundreds of petabytes. IBM ESS supports
high-speed data networks, including 10-Gigabit Ethernet (10 GbE), 40-Gigabit Ethernet
(40 GbE), 100-Gbit Ethernet (100 GbE), and InfiniBand.

1.3.2 Predictable, consistent high performance

IBM ESS uses IBM Spectrum Scale RAID, which is a declustered RAID erasure code
technology that recovers from multiple disk failures in minutes, versus hours or days as in
older technology. This reduced recovery time gives the IBM ESS solution predictable
performance and data protection, 8+2 and 8+3 RAID protection, and platter-to-client data
protection.

A high-performance parallel file system environment, such as IBM Spectrum Scale, can run
only as fast as its slowest component. IBM ESS that uses IBM Spectrum Scale RAID is
designed to provide consistent high performance by masking and mitigating performance
effects of storage hardware failures, even if multiple drive failures occur. In this way, IBM ESS
assures that the larger IBM Spectrum Scale parallel file system always runs at optimum
efficiency and consistently delivers expected high performance, even if storage hardware or
media failures occur.

Chapter 1. Introduction to the IBM Elastic Storage System 11

1.3.3 New simplified IBM ESS installation
The newest generation IBM ESS systems provided significantly enhanced ease of use and
faster installation and upgrade procedures. Enhanced IBM ESS technology deployment
methods use Red Hat Ansible-driven container orchestration, which results in faster
installation, upgrades, and less effort and skills required (see Figure 1-10).

Figure 1-10 New simplified IBM ESS installation

As shown in Figure 1-10, IBM Elastic Storage System provides a fully tested and integrated
solution that is composed of Red Hat Enterprise Linux, IBM Spectrum Scale, and IBM ESS
integration code. Installation modules are driven by Ansible-driven container orchestration
and Ansible playbook bundles (APBs), which creates a standard, fast, and manageable
mechanism of automated, faster deployment of IBM ESS and Spectrum Scale storage.

1.3.4 Supported as a solution

The IBM Elastic Storage System is supported as a solution. Clients who call IBM Service and
Support for assistance on IBM ESS and IBM Spectrum Scale RAID receive support from a
dedicated IBM team that is cross-trained on all IBM ESS components (hardware and
software) for the IBM ESS solution. This specialized IBM ESS Service and Support team can
debug and advise on all aspects of IBM ESS.

If more detailed support is needed by an IBM ESS component, IBM ESS service and support
records show and document the problem resolution progress for the overall IBM Service and
Support team. The resulting fast time to resolution contributes to the added value of
purchasing an integrated, tested IBM Spectrum Scale IBM ESS storage solution from IBM.

For more information about IBM Service/Support of the Elastic Storage System, see this web
page (log in required).

12 Introduction Guide to the IBM Elastic Storage System

1.4 IBM Elastic Storage System models
IBM ESS is an IBM Spectrum Scale storage building block that is available in various models.
These models offer a flexible portfolio of IBM Spectrum Scale storage building block options
with NVMe, SSD, or HDD storage.

The general physical arrangement for an IBM ESS is an IBM rack that features the following
components:
򐂰 A pair of IBM Spectrum Scale I/O Data Servers
򐂰 Storage enclosures
򐂰 Network switches

An example of the various models of IBM ESS over time is shown in Figure 1-11.

Figure 1-11 Various Elastic Storage System and Elastic Storage Server models

Chapter 1. Introduction to the IBM Elastic Storage System 13

14 Introduction Guide to the IBM Elastic Storage System
 2

Chapter 2. IBM Elastic Storage System

architecture
This chapter describes the architecture, hardware, and software of the IBM Elastic Storage
System (ESS), including the following components:
򐂰 IBM ESS Management Server (EMS)
򐂰 I/O/data servers
򐂰 Storage enclosures

This chapter also describes the different building block models and software components and
features, such as:
򐂰 Operating system
򐂰 Ansible playbooks and container-based software upgrades
򐂰 eXtreme Cluster Administration Toolkit (xCAT)
򐂰 IBM Spectrum Scale
򐂰 IBM Spectrum Scale RAID

Note: IBM ESS is an integrated packaged solution. The user should not install different
kernel levels or drivers into the IBM ESS system. The user also should not run any
non-IBM ESS (client) application or non-IBM ESS workloads on the I/O nodes or IBM ESS
Management Server.

This chapter includes the following topics:

򐂰 2.1, “IBM Elastic Storage System architecture overview” on page 16
򐂰 2.2, “IBM Elastic Storage System hardware components” on page 16
򐂰 2.3, “IBM Elastic Storage System software components” on page 36

© Copyright IBM Corp. 2016, 2021. All rights reserved. 15

2.1 IBM Elastic Storage System architecture overview
In this section, we describe the overall hardware and software architecture of the IBM ESS
solution.

An IBM ESS is defined as a combine set of hardware and software that are tightly coupled
together and tested as a single unit. The major components of an IBM ESS solution release
include the following items:
򐂰 Server hardware (ESS 5000 uses POWER9 servers, ESS 3000 uses x86 servers)
򐂰 Storage hardware
򐂰 IBM Spectrum Scale software
򐂰 Embedded Red Hat Enterprise Linux (RHEL) operating system

The minor components, which are integration-tested with the major components, include (but
not limited to) the following items:
򐂰 POWER server firmware
򐂰 Server HBAs and related firmware
򐂰 Mellanox network interface cards, drivers, and network switches
򐂰 Storage enclosure firmware
򐂰 IBM racks, power distribution units, and cabling management

In addition, many of the hardware and software components of the IBM ESS solution
communicate to each other over an IP network. IBM ESS requires all networking connections
to be in place and provisioned to install the system. This IP network can use Mellanox
network switches, which are tested with IBM ESS by IBM, or the network switches can be
provided by the client. These IP network components consist of the following switches:
򐂰 Low speed 1 GbE network switches for management and service networks
򐂰 High-speed network switches for data read and write over a high-speed data network

2.2 IBM Elastic Storage System hardware components

In this section, we describe the following hardware components of the IBM ESS:
򐂰 Solution models
򐂰 Server hardware (ESS 5000 uses POWER9 servers, ESS 3000 uses x86 servers)
򐂰 Storage enclosures
򐂰 Network interface cards and network switches
򐂰 Rack

IBM ESS is a rack-mounted IBM Spectrum Scale storage solution. The initial IBM ESS
storage building block in an IBM Spectrum Scale cluster includes the following components:
򐂰 One IBM ESS Management Server (one is required for every IBM Spectrum Scale or IBM
ESS cluster)
򐂰 Two I/O data servers
򐂰 A total of 1 - 9 storage enclosures (depending on the IBM ESS model)

IBM ESS includes various models that provide SAS, NL-SAS, SSD, or NVMe storage.
Different sizes of disks, SSDs, and NVMes are available.

16 Introduction Guide to the IBM Elastic Storage System

 A first- and second-generation IBM ESS storage building block contains of a pair of IBM
POWER8® IBM Spectrum Scale Data servers, which are cross-connected to all of the
storage for high availability.

The new IBM ESS 3000, which is part of the third generation of IBM ESS models, were
announced and delivered in 4Q2019. IBM ESS 3000 is a fully integrated 2U storage building
block that has 12 or 24 NVMe drives in the front of the 2U24 enclosure, and a pair of x86
server canisters in the back of the 2U enclosure (these servers are cross-connected to all the
NVMe storage for high availability).

The new IBM ESS 5000, which is also part of the third generation of IBM ESS models, were
announced and delivered in 3Q2020. IBM ESS 5000 is a fully integrated IBM Spectrum Scale
HDD storage building block with 1 - 9 storage enclosures (depending on the model) and pair
of POWER9 IBM Spectrum Scale Data servers, which are cross-connected to all the storage
for high availability.

2.2.1 IBM Elastic Storage System models

This section describes the different generations and various of IBM ESS. This IBM Redpaper
publication provides information about the IBM ESS building blocks from the first, second, and
third generation of IBM ESS models.

All IBM ESS model generations can coexist with an IBM Spectrum Scale cluster.

First-generation IBM ESS

The first generation of IBM Elastic Storage Server was announced in October 2014. It
includes a solution IBM machine type 5146.

New orders and upgrades were withdrawn from marketing and were no longer available
effective January 19, 2018. These machines are still in full service and supported by IBM until
the announced end of service (EOS) date for these first-generation ESSs, which will be
effective on December 31, 2021.

For more information about first-generation IBM ESS models, see Appendix A, “IBM Elastic
Storage System models” on page 69.

Second-generation IBM ESS

The second generation of IBM Elastic Storage Server was announced in April 2017.

The second-generation IBM ESS does not include a solution IBM machine type. The IBM
ESS server and storage hardware components feature the following ESS-unique machine
types:
򐂰 IBM machine type 5148 for the IBM ESS IBM POWER8 servers
򐂰 IBM machine type 5147 for the IBM ESS IBM Storage enclosures

These machine types uniquely identify these hardware components as part of an IBM ESS
solution. The models of second-generation IBM ESS are available in the following categories:
򐂰 SSD flash storage IBM ESS models
򐂰 HDD IBM ESS models
򐂰 Hybrid IBM ESS models that consist of enclosures of SSD flash drives and HDDs.

Chapter 2. IBM Elastic Storage System architecture 17

 New orders for new second-generation IBM ESS building blocks were announced to be
withdrawn from marketing effective December 20, 2020. The following components can still
be ordered:
򐂰 IBM 5147-084 HDD storage enclosures to add to a second-generation IBM ESS GLxS
model until June 30, 2021.
򐂰 IBM 5147-024 SSD storage enclosures to add to a second-generation IBM ESS GSxS or
GH model until December 31, 2021.
򐂰 IBM 5147-106 HDD storage enclosures to add to a second-generation IBM ESS GLxC
model until December 31, 2021.

Note: The second-generation IBM ESS is still in full service and supported by IBM. As of
this writing, no EOS date was announced yet.

For more information about second-generation IBM ESS models, see Appendix A, “IBM
Elastic Storage System models” on page 69.

Third-generation ESS
The third generation of IBM Elastic Storage Server was announced starting in October 2019
with the IBM ESS 3000 NVMe Flash storage system. This announcement was followed by the
announcement of the IBM ESS 5000 HDD storage systems in July 2020.

The third-generation ESS's have the following ESS-unique IBM machine types:
򐂰 For the IBM ESS 3000: IBM machine type 5148-AF8 for the 2U24 form factor IBM ESS
with embedded NVMe Flash storage.
򐂰 For the IBM ESS 5000:
– IBM machine type 5105-22E for the IBM ESS IBM POWER9™ data servers,
management server, and protocol nodes
– IBM machine type 5147-092 or 5147-106 for the IBM ESS IBM Storage enclosures

These machine types uniquely identify these hardware machines as part of an IBM ESS
solution.

2.2.2 IBM Elastic Storage System 3000

The new IBM ESS 3000 is part of the third generation of IBM ESS. IBM ESS 3000 was
announced in October 2019 and includes the following features:
򐂰 Based on proven IBM FlashSystem® 9150 storage enclosure and server canister
technology.
򐂰 Uses standard enterprise class NVMe drives:
– Can be ordered half-populated with 12 NVMe drives or fully populated with 24 NVMe
drives.
– If ordered half-populated, you can add the remaining 12 NVMe drives non-disruptively
(the additional drives must be of the same size as the first 12 drives).
򐂰 Designed to provide:
– High-performance NVMe storage for High Performance Tier (HPT) for IBM Spectrum
Scale in a compact, energy-efficient 2U24 form factor with two highly available,
cross-connected x86-based IBM Spectrum Scale data servers
– The necessary bandwidth to fully use compute GPUs for AI and analytics workloads

18 Introduction Guide to the IBM Elastic Storage System

 – Improved ease of use, faster time to installation, and faster time to value
򐂰 Deployed by way of containerized Red Hat Ansible playbooks that provide significantly
improved ease of use and orchestration of complex IBM ESS administration tasks, such
as cluster configuration, file system creation, and code update.

The third-generation IBM ESS 3000 model provides high-performance NVMe storage.
Figure 2-1 shows the IBM ESS 3000 NVMe storage building block.

ESS 3rd Generation – ESS 3000 with NVMe

2U24 Enclosure
12 or 24 NVMe drives

Figure 2-1 Third-generation IBM ESS 3000 models

A primary use of the IBM ESS 3000 models with NVMe is for fast IBM Spectrum Scale
metadata, or to provide a dense, high-speed, potentially mobile NVMe storage tier.

Note: IBM ESS performance is available upon request from IBM or IBM Business Partner
representative. They use the IBM File and Object Solution Design Engine to estimate
performance that is based on your workload and network environment.

Optimum IBM ESS performance is derived from unconstrained IOR benchmark for 100%
sequential read numbers by using unconstrained InfiniBand networks. Other networks
(such as 100 GbE, 40 GbE, and 10 GbE) have more overhead than InfiniBand and typically
lower aggregate bandwidth capabilities result.

For more information, contact your IBM or IBM Business Partner representative.

For ease of management and upgrade, IBM ESS 3000 provides a simplified, faster
installation experience compared to previous generation ESS. IBM ESS 3000 hardware is set
up by the IBM SSR. If the client is unfamiliar with IBM Spectrum Scale and ESS, IBM
recommends that IBM System Lab Services be used to assure high satisfaction with your
initial IBM ESS 3000 installation.

2.2.3 IBM Elastic Storage System 5000

IBM Elastic Storage System 5000 is IBM’s newest HDD-based IBM Spectrum Scale storage
platform. This storage platform provides high capacity and high-performance IBM Spectrum
Scale storage by using HDD storage drives.

Chapter 2. IBM Elastic Storage System architecture 19

 IBM Elastic Storage System 5000 can include 6 TB, 10 TB, 14 TB, and 16 TB HDD drives in
various models.

For more information about the full collection of manuals and documentation for IBM ESS
5000, see IBM Knowledge Center.

IBM ESS 5000 is based on IBM Spectrum Scale and IBM Spectrum Scale RAID, as is all
other ESS. IBM ESS 5000 includes the following new features:
򐂰 IBM POWER9-based data servers, which provide the latest advances in IBM POWER
high memory and memory bandwidth, PCI Gen4-based internal bus transfer speeds for
higher internal transfer rates, and support of faster network interface cards.
򐂰 Containerized Red Hat Ansible playbooks that provide significantly improved ease of use
and orchestration of complex IBM ESS administration tasks, such as cluster configuration,
file system creation, and code update.
򐂰 Higher density and better HDD performance per rack than previous HDD-based IBM ESS
models.
򐂰 ESS 5000 supports nondisruptive capacity upgrade. You may add 5147-092 to an SLx
model or a 5147-106 to a SCx model. Therefore, an installed SC1 to SC8 model, or SL1 to
SL6 model, can be upgraded to any model configuration up to SC9 or SL7 respectively,
without causing downtime or an interruption in service. (the additional HDD drives must be
of the same size as the previous drives).

Figure 2-2 shows the IBM ESS 5000 SL models.

Figure 2-2 IBM ESS 5000 SL models

20 Introduction Guide to the IBM Elastic Storage System

 Figure 2-3 shows the IBM ESS 5000 SC models.

Figure 2-3 IBM ESS 5000 SC models

2.2.4 Positioning various IBM Elastic Storage System models

The different models of IBM ESS storage building blocks provide varying levels of
performance and capacity according to the needs of the IBM Spectrum Scale storage pool in
which they are deployed. The various models of IBM ESS can provide NVMe, SSD, SAS, or
NL-SAS storage. Different sizes of HDD disk, SSD, and NVMe are available.

Chapter 2. IBM Elastic Storage System architecture 21

 Figure 2-4 shows a general positioning of the available models of ESS.

Figure 2-4 General positioning of available IBM ESS models

IBM ESS models are available in five categories. The “x” in the models (see Figure 2-5)
denotes the number of storage enclosures in that model. Consider the following points:
򐂰 IBM ESS 3000 is built on NVMe flash storage for high performance, high density, small
footprint.
򐂰 IBM ESS 5000 SL model is built on high capacity 5U92 HDD storage enclosures.
򐂰 IBM ESS 5000 SC model is built on extreme high capacity 4U106 HDD storage
enclosures.

22 Introduction Guide to the IBM Elastic Storage System

 The available IBM ESS models are shown in Figure 2-5.

Figure 2-5 Elastic Storage System models

IBM ESS usable capacity is a percentage of the raw capacity. The usable capacity varies,
depending on the IBM Spectrum Scale RAID parity that is selected for use. Generally,
approximately 73% of the raw capacity is the usable capacity when the default 8+2P parity is
used, which is the most commonly selected parity for most scenarios.

IBM can provide the exact use capacities of the various models upon request by using the
tool IBM File Object Solution Design Engine (FOS DE) to calculate the exact usable capacity.
See your IBM or IBM Business Partner representative for assistance in determining the
usable capacity that your IBM ESS model and implementation provides.

For more information about the various IBM ESS models specifications, see Appendix A,
“IBM Elastic Storage System models” on page 69.

2.2.5 IBM Elastic Storage System Management Server

New with the third-generation IBM ESS 3000 and the IBM ESS 5000 is a new model of IBM
ESS Management Server (EMS), IBM machine type 5105-22E, and is based on IBM
POWER9 (P9).

P9-based EMS server features the following specifications:

򐂰 1x DD2.3 20 Small Cores, 190W/225W, 2.5 GHz/2.9 GHz
򐂰 128 GB default memory, no NVDIMMs (non-volatile dual in-line memory module)
򐂰 No SAS host bus adapters (HBAs)
򐂰 The same network interface card options as the IBM ESS POWER9 Data Server

Chapter 2. IBM Elastic Storage System architecture 23

 Every IBM Spectrum Scale cluster with IBM ESS requires a minimum of one IBM ESS
Management Server (EMS). This IBM POWER server is used to manage and deploy the IBM
ESS I/O data server nodes. This server performs the following tasks:
򐂰 Runs xCAT management software that deploys IBM ESS software, patches, and ongoing
maintenance
򐂰 Runs IBM ESS GUI server for managing ESSs in the cluster
򐂰 Can optionally provide other IBM Spectrum Scale functions, such as quorum node
򐂰 Runs xCat management software that deploys first- and second-generation IBM ESS
software, patches, and ongoing maintenance
򐂰 Runs newer management and ease of installation enhancements by using Red Hat
Ansible orchestration when managing the new third-generation IBM ESS 3000 and IBM
ESS 5000

The initial IBM ESS storage building block in an IBM Spectrum Scale cluster requires an IBM
ESS Management Server. The purpose of the EMS is a central control point to manage
multiple IBM ESS storage building blocks in this cluster.

The first-generation IBM ESS models used an IBM ESS Management Server (IBM machine
type 8247-21L) running Red Hat Enterprise Linux Big Endian. This Big Endian IBM ESS
Management Server can manage only first-generation ESSs.

The second-generation IBM ESS models use an IBM ESS Management Server (IBM
machine type 5148-21L) running Red Hat Enterprise Linux Little Endian. This Little Endian
IBM ESS Management Server can manage second-generation ESSs and IBM ESS 3000, but
cannot manage first-generation ESS or third-generation IBM ESS 5000s.

The third-generation IBM ESS 3000 and IBM ESS 5000 models prefer to use the newest
POWER9 IBM ESS Management Server (IBM machine type 5105-22E) running Red Hat
Enterprise Linux Little Endian. This Little Endian IBM ESS Management Server can manage
ESS 3000, ESS 5000, and second-generation ESSs. (The ability for POWER9 5105-22E
ESS Management Server to manage 2nd generation ESSs, was announced on February 23,
2021).

If you have a mix of generations of ESSs in the same IBM Spectrum Scale cluster, you must
have a suitable number of EMSs to manage the IBM ESS generations.

Consider the following points:

򐂰 First-generation IBM ESS is managed by an IBM 8247-21L POWER8 EMS
򐂰 Second-generation IBM ESS is managed by an IBM 5148-21L POWER8 EMS.
Announced on February 23, 2021, the 2nd generation ESS can also be managed by
POWER9 5105-22E EMS
򐂰 IBM ESS 3000 can be managed by a 5148-21L EMS or a POWER9 5105-22E EMS
򐂰 IBM ESS 5000 is managed by a POWER9 5105-22E EMS

IBM ESS GUI and Call Home

The IBM ESS Management Server also runs the IBM ESS graphical user interface (GUI) and
is the central Call Home control point for the IBM ESS hardware. IBM ESS Call Home
monitors server and storage hardware. If a hardware failure event occurs, the IBM ESS
Management Server starts a Call Home service action to IBM Service and Support.

24 Introduction Guide to the IBM Elastic Storage System

2.2.6 Data Server hardware
The IBM ESS solution integrates a pair of IBM Spectrum Scale Data Servers that are
cross-connected to all storage in the ESS, which provides an IBM Spectrum Scale storage
building block. Most models of ESS (except for the IBM ESS 3000) use IBM POWER servers
for the data servers. The IBM ESS 3000 uses an x86-based pair of servers for data servers.

The server hardware, firmware, and drivers are all tested, integrated, and supported as a
solution by IBM and as part of the overall IBM ESS solution.

In the following sections, we describe the server specifics by generation of ESS.

First-generation ESS
The first generation of IBM Elastic Storage Server was announced in October 2014. It uses
POWER8 IBM 8247-2xL data servers running Red Hat Enterprise Linux Big Endian operating
system. New orders and upgrades were withdrawn from marketing and are no longer
available, effective January 19, 2018. These machines will be in full service and supported by
IBM until the announced EOS date of December 31, 2021.

For more information about first-generation IBM ESS models, see Appendix A, “IBM Elastic
Storage System models” on page 69.

First-generation IBM ESS will continue to receive IBM ESS solution software stack upgrades
until the announced EOS date of December 31, 2021. Until then, IBM will keep these models
current with supported software levels for RHEL Big Endian, IBM Spectrum Scale,
management stack software levels, and firmware.

Second-generation IBM ESS servers

All second-generation ESSs use a pair of IBM POWER8 servers for the integrated IBM
Spectrum Scale Data Servers. The second-generation IBM ESS uses IBM POWER8
(machine type 5148-22L) servers as IBM Spectrum Scale Data servers, which are
cross-connected to all storage for high availability. These 5148-22L servers all run Red Hat
Enterprise Linux Little Endian operating system.

The second-generation IBM ESS POWER servers do not require an IBM Hardware
Management Console (HMC), which was required with the first-generation ESS. The
functions that were performed by the HMC were integrated into the IBM Electronic Service
Agent running on the second-generation IBM ESS Management Server (machine type
5148-21L).

The machine types 5148-22L and 5148-21L are unique to the ESS, and identify these
POWER8 servers as an integrated server component of an IBM ESS solution.

Third-generation IBM ESS servers

With IBM ESS 5000, IBM introduced a new POWER9-based IBM ESS server: IBM Machine
Type 5105-22E. This server is used in the IBM ESS 5000 only.

The 5105-22E is configured in three separate ways to provide of one of the following server
functions in the IBM ESS 5000 system:
򐂰 IBM ESS Management Server
򐂰 Protocol node
򐂰 IBM ESS Data Server

Chapter 2. IBM Elastic Storage System architecture 25

 All these servers are based on the IBM POWER9 architecture and include the same MTM
(5105-22E). In the IBM ESS 5000 configuration, they are differentiated by feature codes that
specify which of the three possible roles this 5105-22E is performing.

The 5105-22E POWER9 servers feature increased memory and backplane bandwidth. In
particular, the use of PCI Gen4 bus allows each of the ports on the dual port network interface
cards to run at full rated speed.

Note: Document the serial number and role of each 5105-22E POWER9 server in an IBM
ESS 5000 system so that operations and service personnel can identify each 5105-22E
and correlate what role that POWER9 server is performing.

The third-generation IBM ESS 3000 does not use separate POWER servers that act as IBM
Spectrum Scale data servers for the IBM ESS 3000 storage.

The IBM ESS 3000 2U24 form factor includes two x86 server canisters in the rear of the 2U24
storage enclosure, which provides a small footprint and a dense and integrated NVMe flash
IBM Spectrum Scale storage building block. These two x86 server canisters run IBM
Spectrum Scale RAID and are the pair of IBM Spectrum Scale data servers, which are cross
connected to all storage in the 2U24 enclosure for high availability.

Note: The IBM ESS 3000 can be supported by a POWER9 5105-22E IBM ESS
Management Server, or a POWER8 5148-21L IBM ESS Management Server. This
management sever can be an IBM ESS 5148-21L that is used to manage other ESSs in
this same IBM Spectrum Scale cluster. If this cluster is a new IBM Spectrum Scale cluster,
order a POWER9 5105-22E IBM ESS Management Server to manage your IBM ESS 3000
or IBM ESS 5000 IBM Spectrum Scale cluster.

Network interface cards

The IBM ESS solution features the following types of networks:
򐂰 Management/service networks that require basic, low-speed 1 GbE
򐂰 High-speed data networks (HDR 100 InfiniBand, EDR InfiniBand, 100 GbE, 40 GbE,
25 GbE, or 10 GbE) over which user data is read and written

The network interface cards (NICs) on the IBM ESS data servers and the IBM ESS
management server service these various network types.

Every IBM ESS node has a 1 GbE NIC for connection to the management and service
networks.

Every IBM ESS management server, in addition to the 1 GbE NIC, must have one of each of
the high-speed network NICs being used on the IBM ESS data servers. This configuration
allows the ESS management server to monitor, manage, and measure the activity on the
high-speed data networks.

On the IBM ESS data server nodes, a sufficient quantity of high-speed data network NICs
must be configured to provide the desired IBM ESS bandwidth.

Various high-speed data network NICs are available to support the following high-speed
network speeds. Where possible, InfiniBand is always the preferred high-speed network. If
InfiniBand cannot be used, 100 GbE Ethernet is preferred:
򐂰 10 GbE
򐂰 25 GbE

26 Introduction Guide to the IBM Elastic Storage System

 򐂰 40 GbE
򐂰 100 GbE
򐂰 EDR InfiniBand/100 GbE network card is newly available on IBM ESS 3000. This is a
dual-port InfiniBand-EDR and Ethernet VPI network adapter based on the Mellanox
ConnectX-5 card with VPI (Virtual Protocol Interconnect) enabled. Each of the two ports
on the card can be independently configured for 100 GbE or EDR-100.
򐂰 HDR InfiniBand/100 GbE network card is newly available on IBM ESS 5000. This is a
dual-port InfiniBand-HDR and Ethernet VPI network adapter based on the Mellanox
ConnectX-6 card with VPI (Virtual Protocol Interconnect) enabled. Each of the two ports
on the card can be independently configured for 100 GbE or HDR-100.

Note: Consider the following points:

򐂰 IBM ESS 5000 POWER9 data servers use PCIe Generation 4 bus, which fully supports
both ports of the 100 GbE or InfiniBand NICs to run at full rated speed.
All ESSs other than IBM ESS 5000 use PCIe Generation 3 bus; therefore, only one port
of the high-speed 100 Gbit Ethernet or InfiniBand dual-port NICs can run at full speed.
The second port on these NICs is typically configured and attached for availability.
IBMers and IBM Business Partners use the IBM File and Object Design Engine tool to
estimate and configure an appropriate amount of network interface card bandwidth.
Experience shows throughput on IBM ESS is often limited to the amount of network
NIC bandwidth available. The best performance is achieved over InfiniBand. Other
networks (such as 100 GbE, 40 GbE, 25 GbE, and 10 GbE) have more overhead than
InfiniBand and typically lower aggregate IBM ESS bandwidth capabilities result.
򐂰 Suitable network cables and connectors must be specified and ordered for your specific
network requirements. Different cables and connectors are usually required for 10 GbE
versus 40 GbE versus 100 GbE versus EDR Infiniband versus. HDR Infiniband. If you
use the VPI feature to reconfigure your high speed network ports, remember to make
sure that your ESS Management Server also can access the newly configured network.
򐂰 RoCE (RDMA over Converged Ethernet) may be available on ESS 3000 and ESS 5000
for specific configurations, networks and environments only. Contact your IBM or
Business Partner representative to submit an RPQ to IBM if you have this requirement.
򐂰 For more information about estimating and sizing your performance, or for specifying
the suitable network cables and connectors, contact your IBM or your IBM Business
Partner representative.

You can intermix high-speed data network NIC types if enough data server slots are available.
Data servers in an IBM ESS must have same number and type of NICs.

Configure your IBM ESS to have the suitable type and number of NICs for your throughput
requirements. It is supported and a common practice on IBM ESS data servers to bond
multiple ports for increased availability and throughput.

2.2.7 Storage enclosures

A storage enclosure is a specialized storage drawer holds and powers Flash or HDD storage,
while providing the connections to allow them to communicate to one or more separate
firmware service modules that run the entire enclosure.

Chapter 2. IBM Elastic Storage System architecture 27

 First-generation IBM Elastic Storage Server
The first generation of IBM ESS was announced in October 2014. It was available in 2U24
storage enclosure (GS models) with SSD or HDD storage, or high capacity GL storage
models that used IBM 1818-80E 4U60 HDD storage enclosures. New orders and storage
enclosure upgrades are withdrawn from marketing and no longer available, effective January
19, 2018.

The first-generation IBM ESS is fully supported by IBM until the announced EOS date of
December 31, 2021. Contact IBM or your IBM Business Partner representative for more
information.

For more information about the first-generation IBM ESS storage enclosures, see
Appendix A, “IBM Elastic Storage System models” on page 69.

Second-generation IBM Elastic Storage Server

The second generation of IBM ESS was announced in April 2017 and was available for order
until the last order date of December 20, 2020.

The following components are still available for order:

򐂰 IBM 5147-084 HDD storage enclosures to add to a second-generation IBM ESS GLxS
model until June 30, 2021.
򐂰 IBM 5147-024 SSD storage enclosures to add to a second-generation IBM ESS GSxS or
GH model until December 31, 2021.
򐂰 IBM 5147-106 HDD storage enclosures to add to a second-generation IBM ESS GLxC
model until December 31, 2021.

Note: The second-generation IBM ESS is still in full service and supported by IBM. No
EOS date was announced as of this writing.

Third-generation IBM Elastic Storage Server

The following third generation of IBM Elastic Storage System versions were announced and
use the associated storage enclosures:
򐂰 IBM ESS 3000 was announced and delivered in the fourth quarter of 2019. It is a
self-contained 2U24 form factor storage enclosure with NVMe flash storage in the front,
and x86-based IBM Spectrum Scale data servers in the back.
򐂰 IBM ESS 5000 was announced and delivered in the third quarter of 2020. It uses 5U92
(IBM machine type 5147-092) or 4U106 (IBM machine type 5147-106) HDD storage
enclosures.

In the following sections, we describe storage enclosures that you are likely to encounter in
working with second-or third-generation ESS. Machine type 5147 is used for the IBM ESS
IBM Storage enclosures. This machine type 5147 uniquely identifies these storage enclosure
hardware components as part of an ESS.

Note: For more information about storage enclosures that are used in the first generation
of IBM ESS, see A.2, “First-generation IBM Elastic Storage System models overview” on
page 75.

2U24 (5147-024)
The 5147-024 2U storage enclosure is used for SSDs in the second-generation IBM ESS
GSxS and second-generation GH hybrid models. The following sizes of SSDs are available:

28 Introduction Guide to the IBM Elastic Storage System

򐂰 3.84 TB
򐂰 15.36 TB

The 5147-024 in the IBM ESS is available in a fully populated 24-drive configuration only.

The IBM 5147-024 Storage Enclosure is shown in Figure 2-6.

IBM 2U24 Storage Enclosure

Figure 2-6 IBM 5147-024 Storage Enclosure

Chapter 2. IBM Elastic Storage System architecture 29

 5U84 (5147-084)
The 5147-084 5U storage enclosure is available to hold HDDs in the second-generation IBM
ESS GLxS models and the GH hybrid models. The following sizes of HDDs are or were
available:
򐂰 4 TB
򐂰 8 TB
򐂰 10 TB
򐂰 14 TB

The IBM 5147-084 storage enclosure is shown in Figure 2-7.

IBM 5U84 Storage Enclosure

Figure 2-7 IBM 5147-084 Storage Enclosure

The 5147-084 in the second-generation IBM ESS is available in a fully populated

configuration only.

The first 5147-084 storage enclosure in an IBM ESS also contains two SSDs that hold
internal IBM Spectrum Scale RAID metadata. The capacity of these two SSDs is not visible or
available as user capacity for the IBM Spectrum Scale or IBM ESS file system.

4U106 (5147-106)
The 5147-106 4U storage enclosure is an extreme density storage enclosure that holds
HDDs and available in second-generation IBM ESS GLxC and third-generation IBM ESS
5000 SC models. The following sizes of HDDs are or were available:
򐂰 10 TB
򐂰 14 TB
򐂰 16 TB (IBM ESS 5000 only)

The 5147-106 in the IBM ESS is available in a fully populated configuration only.

The first 5147-106 in an IBM ESS also contains two SSDs that hold internal IBM Spectrum
Scale RAID metadata. The capacity of these two SSDs is not visible or available as user
capacity for the IBM Spectrum Scale IBM ESS file system.

30 Introduction Guide to the IBM Elastic Storage System

The IBM 5147-106 storage enclosure is shown in Figure 2-8.

Figure 2-8 IBM 5147-106 Storage Enclosure

The IBM 5147-106 storage enclosure is an unusually deep enclosure. Physical planning for
this enclosure must be performed. Also, the racking, aisle widths, and rear clearances all
must be reviewed and approved by the IBM Installation Planning representative.

Note: Because of the depth of this 5147-106 enclosure, the IBM ESS models that use
5147-106 must be installed in the IBM 7965-S42 rack. Therefore, these enclosures (and
associated IBM ESS) cannot be installed in the IBM 7014-T42 rack.

Second-generation IBM ESS Hybrid models storage enclosures

Second-generation Elastic Storage Server Hybrid models combined 5147-024 storage
enclosures with flash SSD storage and 5147-084 enclosures with spinning HDDs and in a
single IBM ESS building block. These IBM ESS Hybrid models provided a combination of
drawers of 2U24 SSD and 5U84 HDDs, which combines the speed and low latency of flash
SSD storage with the density and affordability of spinning HDD.

This configuration provides a flexible, smaller footprint than using a separate flash SSD
storage building block and disk HDD storage building block. Where the IBM Spectrum Scale
IBM ESS configuration indicates the usage of the IBM ESS GH hybrid models, these IBM
ESS hybrid storage building blocks were used to give better storage density and a smaller
footprint at a lower cost.

Third-generation IBM ESS 3000 Storage Enclosure

The third-generation IBM ESS 3000 with NVMe is available with one type of storage
enclosure, a 2U24, with slots for 24 NVMe storage drives in the front of the enclosure. Each
IBM ESS 3000 is a self-contained integrated 2U24 enclosure with storage in the front, and
two x86-based IBM Spectrum Scale Data Servers in the back of the enclosure.

This storage enclosure is based on the proven IBM FlashSystem 9150 storage enclosure
2U24 form factor and x86 server technology.

Chapter 2. IBM Elastic Storage System architecture 31

 IBM ESS 3000 uses Samsung enterprise NVMe drives with the following capacity options,
with 12 or 24 drives installation options:
򐂰 1.92 TB
򐂰 3.84 TB
򐂰 7.68 TB
򐂰 15.36 TB

The IBM ESS 3000 is packaged with 12 or 24 NVMe drives in the front of the enclosure. If
initially ordered with 12 NVMe drives, 12 NVMe drives of the same drive size can be
non-disruptively added to the IBM ESS 3000 later. Two x86-based IBM Spectrum Scale data
servers are in the back of the 2U24 storage enclosure.

Figure 2-9 shows the IBM ESS 3000.

IBM ESS 3000

NVMe 2U24 Storage Enclosure

Figure 2-9 IBM ESS 3000 NVMe 2U24 Storage Enclosure

Note: IBM ESS 3000 does not use the IBM FlashCore® Module NVMe drives. Standard
enterprise class NVMe drives are used, controlled, and monitored by IBM Spectrum Scale
RAID.

IBM ESS 3000 uses a mirrored set of internal 800 GB SSDs as the boot disks for the
x86-based IBM Spectrum Scale Network Shared Disk (NSD) Data servers.

5U92 (5147-092)
The 5147-092 5U storage enclosure is available to hold HDDs in the IBM ESS 5000 SL
models. The following sizes of HDDs are available:
򐂰 6 TB
򐂰 10 TB
򐂰 14 TB
򐂰 16 TB

32 Introduction Guide to the IBM Elastic Storage System

 The IBM 5147-092 storage enclosure is shown in Figure 2-10.

Figure 2-10 IBM 5147-092 Storage Enclosure

The 5147-092 in the IBM ESS 5000 SL model is available in a fully populated configuration
only. The first 5147-092 in an IBM ESS 5000 also contains two SSDs that hold internal IBM
Spectrum Scale RAID metadata. The capacity of these two SSDs is not visible or available as
user capacity for the IBM Spectrum Scale or IBM ESS file system.

2.2.8 Protocol nodes

The IBM ESS IBM Spectrum Scale protocol node feature code makes it possible to order
POWER9 5105-22E server nodes with a specific hardware configuration, which are tested
and tuned by IBM to provide IBM Spectrum Scale Protocol Node (also known as Cluster
Export Services [CES]) services. The machine type and model (MTM) for protocol nodes that
are ordered with an IBM ESS is IBM machine type 5105-22E.

The default protocol node hardware features the following configuration:

򐂰 5105-22E POWER9 model with two 10 core POWER9 processors
򐂰 Runs RHEL Little Endian
򐂰 192 GB or greater memory in default configuration
򐂰 1 GbE 4-port network adapter that is required for management network
򐂰 Up to seven network adapters

All NICs that are available for other 5105-22E data servers are for 5105-22E servers that are
ordered as protocol nodes.

Note: The 5105-22E protocol nodes are supported for management by the POWER9
5105-22E IBM ESS Management Server node only.

It is fully supported for 5105-22E protocol nodes to coexist in an IBM Spectrum Scale cluster
with previous generation IBM ESS 5148-22L protocol nodes and with other non-5148-22L
POWER Little Endian servers that are also protocol nodes.

Chapter 2. IBM Elastic Storage System architecture 33

 Note: IBM Spectrum Scale requires that in any one IBM Spectrum Scale cluster, all
protocol nodes must be of the same architecture and of the same Endian. The following
types of IBM Spectrum Scale protocol nodes architecture and Endian are supported:
򐂰 POWER servers running RHEL Little Endian, including the IBM 5105-22E POWER9
Protocol Node, and the previous generation POWER8 5148-22L protocol node
򐂰 POWER servers running RHEL Big Endian (for example, IBM 8247-2xL servers)
򐂰 x86 servers running supported Linux operating systems

For more information, see the IBM Spectrum Scale Functional Support Matrices in the IBM
Spectrum Scale FAQ at IBM Knowledge Center

If you are ordering POWER 5105-22E servers as protocol nodes, you might have other
POWER RHEL Little Endian protocol nodes in the IBM Spectrum Scale cluster that are not
IBM 5105-22E protocol nodes or previous generation 5148-22L protocol nodes. These
other protocol nodes are installed separately and deployed, upgraded, and managed by
the customer. The IBM ESS management server tool sets (such as xCAT) are not
supported and cannot be used for the management of operating system, kernel, network
manager, systemd, OFED, or firmware on the non-5105-22E or non-5148-22L protocol
nodes.

2.2.9 Networking switches

The IBM ESS solution requires IP or InfiniBand networking switches to interconnect the IBM
ESS hardware components.

IBM recommends that the client can purchase and use Mellanox networking switches. These
network switches are tested by IBM to work with IBM ESS.

Clients also can provide their own IP or InfiniBand networking switches, which are supported
by the client by using their own standard networking practices and vendor support.

Types of networks that are used by IBM ESS

The IBM ESS features the following types of networks that are used to interconnect the IBM
ESS components:
򐂰 Management and service networks, which must have separate 1 GbE networks for
management and service traffic
򐂰 Data network, which is a high-speed network for the data that is written into and read out
of the IBM Spectrum Scale IBM ESS file systems

Regardless of the source and type of network switches between the ESS, the IBM Spectrum
Scale cluster and the users, the networking must be solid, robust, reliable, and provide
consistent latency and response time. Non-blocking, high-speed switches are highly
recommended.

Experience shows that IBM Spectrum Scale and IBM ESSs can easily generate throughput
rates that can overwhelm an over-subscribed network. IBM Spectrum Scale provides a
network load generation testing tool (NSDPERF), which is available to anyone by way of open
source on GitHub. This tool should be used to test network performance, network latency,
network topology, and network readiness level before IBM ESS is installed.

For more information about the NSDPERF tool, see this web page.

34 Introduction Guide to the IBM Elastic Storage System

 Management network switches
IBM ESS components require a 1 GbE Ethernet IP connection for management and service
network traffic. As part of an IBM ESS order, IBM can supply the Mellanox 1 GbE
management network switch, IBM machine type 8831-S52.

IBM recommends ordering and using this 1 GbE network management switch that is provided
by IBM because it is integrated into the IBM ESS solution at the manufacturing stage. This
integration allows the entire IBM ESS to be installed and initially tested without external
network dependencies.

Alternatively, the client can provide any standard 1 GbE network switch management
infrastructure. In this case, the client is responsible to provide the suitable management IP
networking port counts and network switch configuration and definitions, including VLANs.

High-speed data network switches

IBM ESS components also require a high-speed data network IP or InfiniBand connection for
the reading and writing of user data. As part of IBM ESS testing, IBM tests with Mellanox data
network switches.

IBM recommends ordering and using the Mellanox data network switches where possible.
These switches are tested and integrated with the IBM ESS solution.

Mellanox high-speed network switch drivers and firmware is tested and delivered integrated
within the IBM ESS software solution stack.

Alternatively, the client can provide their own 1 GbE management network and high-speed
data network switch infrastructure. In this case, the client is responsible for provisioning
networking port counts, firmware, network configuration, and definitions. The client also is
responsible for maintaining and troubleshooting any networking issues.

2.2.10 Racking
The IBM ESS solution can be ordered with or without the building block being integrated into
an IBM rack.

An IBM ESS solution that is ordered with feature codes that specify integration into the rack in
IBM manufacturing result in a fully integrated, tested, and pre-cabled solution.

The following IBM racks are available for ESS:

򐂰 IBM 7014-T42 Enterprise Rack
򐂰 IBM 7965-S42 Enterprise Slim Rack

Both racks provide 42 EIA (42U) of usable space for IBM ESS components and network
switches. A choice of various electrical Power Distribution Units (PDUs) is available to be
specified for inclusion in the rack side areas. The difference between the racks is the width
and depth.

It is advantageous to order IBM ESS as a racked system because IBM ESS Manufacturing
delivers an IBM ESS with the rack mostly pre-cabled and pre-tested. This configuration
provides faster time to install with higher quality.

The IBM 7014-T42 is the default for most models of ESS, except for the IBM ESS GLxC
models and IBM ESS 5000 SC models.

Chapter 2. IBM Elastic Storage System architecture 35

 For the IBM ESS models that use the 5147-106 storage enclosure, IBM highly recommends
the use of the IBM 7965-S42 rack, which is designed to accommodate the deep IBM
5147-106 storage enclosure. The IBM 7965-S42 rack is narrower and fits on a 24-inch floor
tile. This rack might better fit clients with tight racking requirements and space. It can be
ordered for other IBM ESS models where the client wants an alternative to the IBM 7014-T42
rack.

It is also possible to order IBM ESS without a rack. In this case, the IBM ESS components are
assembled and tested by IBM Manufacturing. Then, the IBM ESS components (servers,
storage, networking switches, and cables) are uninstalled from the manufacturing rack,
labeled, and shipped. In this case, it is the client’s responsibility to correctly specify all
physical planning for electrical power, connectors, cables, and racking locations.

Clients also must plan the physical installation and cabling of the IBM ESS components in the
client-supplied rack. An IBM TSS Services contract is available and recommended to perform
physical planning, specification, and physical installation of the IBM ESS components in a
client rack on behalf of the client.

For the rackless IBM ESS order, the solution must be assembled and integrated into the rack
at the client site. Extra IBM Technical Service and Support (TSS) installation hours for
performing this installation might be required. For more information, see your geography IBM
TSS representative.

For the purposes of cable management and serviceability, each IBM ESS building block must
be integrated within the same rack.

2.3 IBM Elastic Storage System software components

This section describes the software components that are used on the IBM ESS solution.

2.3.1 Software solution stack overview

The IBM ESS solution provides an integrated and tested stack of these operating system,
adapter drivers, firmware, IBM Spectrum Scale software, management software, and
installation scripts into a full IBM ESS software stack. This software solution stack is
supported as an integrated IBM Spectrum Scale storage building block solution by IBM
Service and Support.

In the following sections, we briefly describe the following IBM ESS solution stack software
components:
򐂰 Red Hat Enterprise Linux operating system
򐂰 IBM Spectrum Scale high-performance parallel file system
򐂰 IBM Spectrum Scale RAID software-defined storage
򐂰 IBM ESS solution installation and management software, which includes (but not limited
to) the following components:
– ESS-specific documentation for installation and upgrade scripts
– eXtreme Cluster Administration Toolkit (xCAT)
– Other tools for the IBM SSR to use installing ESS, such as ESSUTILS/GSSUTILS
򐂰 Third-generation IBM ESS systems deploy a newer container-oriented management
software stack in the IBM ESS Management Server that includes Red Hat Ansible
playbooks for significantly enhanced ease of use for installation and orchestration.

36 Introduction Guide to the IBM Elastic Storage System

 IBM preinstalls this complete integrated, tested IBM ESS solution stack on the IBM ESS
servers in IBM Manufacturing.

The IBM ESS solution stack levels are released as a version, release, and modification level.

For more information about the release levels of the IBM ESS software solution and the levels
of the software components for that IBM ESS release level, see IBM Knowledge Center.

The IBM ESS solution stack components are periodically upgraded to newer release levels,
tested as an integrated solution, and released as a new level of IBM ESS solution software.
IBM recommends that clients plan to upgrade their IBM ESS to the current level of IBM ESS
solution software stack at least once a year.

2.3.2 Operating system

The IBM ESS solution runs Red Hat Enterprise Linux (RHEL) as the operating system on the
IBM Spectrum Scale Data Servers.

Each IBM ESS solution release level integrates and tests a suitable current level of RHEL,
including any necessary RHEL fixes and errata that are required for the successful operation
of the IBM ESS solution stack. IBM periodically provides new IBM ESS solution release levels
that incorporate newer levels of RHEL. These releases are provided often enough to assure
that a current level of RHEL is always available.

Consider the following points:

򐂰 First-generation ESSs use Red Hat Enterprise Linux Big Endian.
򐂰 Second-generation and subsequent generation ESSs use Red Hat Enterprise Linux Little
Endian.
򐂰 All generations of IBM ESS can coexist in the same IBM Spectrum Scale cluster.

Embedded RHEL licensing on IBM ESS 3000 and IBM ESS 5000
Starting with the IBM ESS 3000 and IBM ESS 5000 generations, IBM uses a new embedded
Red Hat Enterprise Linux license. This license is ordered and included with every IBM ESS
3000 and IBM ESS 5000 order. Embedded RHEL comes preinstalled on IBM ESS 3000 and
IBM ESS 5000 systems. These newest generation IBM ESS systems no longer need a
separate Red Hat subscription for the RHEL on ESS. IBM handles all RHEL subscriptions,
licensing, and support. In this way, IBM provides an easier to manage, integrated Spectrum
Scale IBM ESS storage solution.

Note: Embedded RHEL does not require or provide an RHEL subscription key for the
customer. Embedded RHEL does not entitle or require the customer to download RHEL
security patches from the Red Hat Portal. IBM handles all RHEL security patches for IBM
ESS 3000 and IBM ESS 5000. IBM Development monitors the Red Hat security patch
stream and provides IBM ESS updates according to severity. IBM provides required
security patches in an update of the IBM ESS software stack by way of IBM Fix Central.

First- and second-generation IBM ESS Red Hat Enterprise Linux

subscriptions
For all first- and second-generation IBM ESS models, Red Hat requires the following separate
subscriptions to be in effect for each POWER server:
򐂰 Red Hat Enterprise Linux Level 3 support subscription
򐂰 Red Hat Linux Extended Update Support (EUS) subscription

Chapter 2. IBM Elastic Storage System architecture 37

 IBM includes the following required Red Hat subscriptions (with RHEL and EUS feature
codes) in every IBM ESS order:
򐂰 First-generation IBM ESS models: IBM Program ID 5639-RH7 - Red Hat Enterprise Linux,
Big Endian
򐂰 Second-generation IBM ESS models: IBM Program ID 5639-RLE - Red Hat Enterprise
Linux, Little Endian

Client responsibilities for RHEL licensing on ESS

For first- and second-generation IBM ESS models, specific client responsibilities exist for
registering the required IBM ESS RHEL subscriptions. For more information about the IBM
ESS client responsibilities, see the question: “How do I register the (required) Red Hat
Enterprise Linux subscription keys for my Elastic Storage Server (ESS)?” in IBM Knowledge
Center.

Note: For the first- and second-generation ESS, it is the client’s responsibility to receive
the RHEL subscription keys and register them on the Red Hat Portal according to the
procedure that is documented in IBM Knowledge Center. It is also the IBM ESS client’s
responsibility to maintain the RHEL subscription as current with Red Hat when the
subscription is due for renewal. For more information, contact your IBM representative.

2.3.3 IBM Spectrum Scale

In this section, we describe IBM Spectrum Scale, the high-performance parallel clustered file
system software that is used in IBM ESS and is proven in enterprise-level high performance
computing environments, university and research environments, and commercial business
environments worldwide, including finance and healthcare. IBM Spectrum Scale runs on
many of the world’s largest supercomputers, including the 2018-delivered Sierra and Summit
supercomputers.

Formerly known as General Parallel File System (GPFS), IBM Spectrum Scale is highly
distributed, clustered file system software that provides high-speed concurrent data access to
applications that run on multiple nodes and clusters. In addition to providing parallel
high-performance file storage capabilities at petabyte scale, IBM Spectrum Scale provides
tools for tiering, management, administration, and archiving of that enterprise-level data. IBM
Spectrum Scale is IBM’s strategic software defined storage for enterprise big data, analytics,
and AI applications.

In the most common IBM Spectrum Scale deployment architecture, IBM Spectrum Scale data
is accessed by IBM Spectrum Scale clients and users over a LAN network, accessing disk
volumes that are known as NSDs that are attached to IBM Spectrum Scale nodes that are
known as NSD Data Servers”. In this IBM Redpaper publication, these nodes also are
referred to as Data Servers.

IBM Spectrum Scale and IBM ESS

IBM Elastic Storage Systems and IBM ESS are pre-integrated, pre-tested IBM Spectrum
Scale storage building blocks. Most models of IBM ESS consist of a pair of IBM POWER
servers with attached IBM storage.

The IBM ESS 3000 uses a pair of x86-based servers. These servers run RHEL and IBM
Spectrum Scale, and are defined as IBM Spectrum Scale NSD Data Servers. In all IBM
ESSs, the NSD Data Servers are cross-connected to provide failover and redundancy. If one
of the NSD Data Servers fails, the IBM ESS fails over the storage and data to the other NSD

38 Introduction Guide to the IBM Elastic Storage System

 Data Server, which assures continued availability of the data in the IBM Spectrum Scale
cluster.

IBM Spectrum Scale for IBM ESS licensing

All nodes in an IBM Spectrum Scale cluster run a copy of the IBM Spectrum Scale software.
Instead of using the normal IBM Spectrum Scale software licensing, an IBM ESS typically is
licensed for IBM Spectrum Scale by using the following specific IBM Spectrum Scale for IBM
ESS IBM program IDs:
򐂰 5765-DAE IBM Spectrum Scale for IBM ESS Data Access Edition
򐂰 5765-DME IBM Spectrum Scale for IBM ESS Data Management Edition

This specific IBM Program ID with a “Per Disk” metric is normally used for licensing IBM
Spectrum Scale for IBM ESS software on an IBM Elastic Storage Server. IBM Spectrum
Scale for IBM ESS software licenses include IBM Spectrum Scale RAID license entitlement.
The license price for IBM Spectrum Scale for IBM ESS is tiered as Flash and HDDs have
different list prices per TB. You need only count the physical number of Flash drives or HDDs
for the "Per Disk" metric.

An advantage of the IBM Spectrum Scale for IBM ESS “Per Disk” metric is that the size of the
SSD or HDD does not affect the license list price. For example, if your IBM ESS model has
550 HDDs, your IBM Spectrum Scale for IBM ESS license list price is the same, regardless of
whether you are specifying 6 TB HDDs or 16 TB HDDs.

IBM Spectrum Scale for IBM ESS licensing helps to contribute in building a complete IBM
hardware and software solution by integrating the IBM Spectrum Scale and IBM ESS
solution.

For more information about IBM Spectrum Scale and IBM Spectrum Scale for IBM ESS
software licensing, options, and considerations (such as the use of IBM ESS in a
Socket-licenses IBM Spectrum Scale cluster, or for IBM Spectrum Scale capacity licenses, or
for using IBM ESS in an IBM Spectrum Scale Enterprise License Agreement environment),
see the following resources:
򐂰 IBM Knowledge Center
򐂰 IBM Spectrum Scale IBM Elastic Storage System - Licensing Information

2.3.4 IBM Spectrum Scale RAID

IBM Spectrum Scale RAID is a software-defined storage controller that performs all of the
storage controller functions that are normally associated with hardware storage controllers.
IBM Spectrum Scale RAID integrates all high availability and features of an advanced storage
server into IBM Spectrum Scale software-defined storage.

IBM Spectrum Scale RAID runs on the IBM ESS NSD data servers. IBM Spectrum Scale
RAID provides sophisticated data placement and error correction algorithms to deliver high
levels of storage reliability, availability, serviceability, and performance.

IBM Spectrum Scale RAID implements a declustered erasure code parity schema,
distributing data, redundancy information, and spare space across all disks of the IBM ESS
enclosures. With this approach, a significant improvement is realized on the application
performance. Also, storage rebuild time overhead is reduced (disk failure recovery process)
compared to conventional RAID controllers.

Chapter 2. IBM Elastic Storage System architecture 39

 IBM Spectrum Scale RAID implements large cache for performance by using memory on the
IBM ESS NSD Data Servers. The large cache intelligently improves read and write
performance, particularly for small block I/O operations.

IBM Spectrum Scale RAID mitigates performance effects of storage

rebuilds
If storage failures occur, IBM Spectrum Scale RAID reconstructs lost or erased stripes for I/O
operations dynamically. By using the highly distributed erasure coding, IBM Spectrum Scale
RAID mitigates the performance effect of storage media failures.

IBM Spectrum Scale RAID end to end checksums

IBM Spectrum Scale RAID includes integrated end-to-end checksums that detect data
corruption that might otherwise go undetected by a conventional storage controller. Unlike
conventional storage controllers, IBM Spectrum Scale RAID is integrated with the IBM
Spectrum Scale file system and performs end-to-end checksum comparison all the way out to
the IBM Spectrum Scale client code on the workstations. This feature ensures data integrity
at a file system level, detecting and automatically correcting data corruption errors that might
occur in conventional storage environments.

In an environment where a customer experienced excessive file system checks and suffered
downtime to repair file systems, the use of IBM Spectrum Scale RAID end-to-end checksums
mitigates file system check problems. This feature assures availability of data and removes
application outages that are caused by file system checks.

IBM Spectrum Scale RAID disk hospital

One of the key features of IBM Spectrum Scale RAID is disk hospital. This powerful function
asynchronously diagnoses errors and faults in the IBM ESS storage media, down to the level
of the individual drive and the individual performance of each drive. IBM Spectrum Scale
RAID is fully aware of and tracks the performance of each individual drives because all drives
do not perform equally. IBM Spectrum Scale RAID uses the individual performance history of
each drive to make intelligent data allocation and data retrieval decisions.

Extensive health metrics down to the level of the individual drive are maintained by the disk
hospital. Performance variation is continually monitored. If or when a disk metric exceeds a
threshold, the storage media is marked for replacement according to the disk maintenance
replacement policy for the declustered array.

As an example, disk hospital features the following metrics:

򐂰 relativePerformance, which characterizes response times. Values are compared to the
average speed. If the metric falls below a specific threshold, the hospital adds “slow” to the
pdisk state, and the disk is prepared for replacement.
򐂰 dataBadness, which characterizes media errors (hard errors) and checksum errors.

For more information about IBM Spectrum Scale RAID implementation and best practices,
see the Administering IBM Spectrum Scale RAID manual at IBM Knowledge Center.

40 Introduction Guide to the IBM Elastic Storage System

2.3.5 IBM Elastic Storage System solution installation and management
scripts
In this section, we provide an overview of the components of the IBM ESS solution installation
and management scripts. This overview includes the following information:
򐂰 ESS-specific documentation for installation and upgrade scripts
򐂰 Ansible playbooks and container-based software installation
򐂰 eXtreme Cluster Administration Toolkit (xCAT)
򐂰 GSSUTILS/ESSUTILS tools for the IBM SSR and administrators to use while
installing/maintaining IBM ESS

The third-generation IBM ESSs enhance these installation and management tools with newer
container-based deployment methods, which use Red Hat Ansible playbooks for installation
and orchestration.

For more information about these IBM ESS solution components, see the IBM ESS solution
release-specific level information at IBM Knowledge Center.

Installation, upgrade, and administering guides

IBM provides manuals and documentation for deploying and administering IBM ESS,
including the following publications:
򐂰 Quick Deployment Guide (QDG), which documents IBM ESS-specific scripts for installing,
deploying, and upgrading IBM ESS for experienced users.
򐂰 IBM Spectrum Scale RAID Administration, which focuses on administering IBM Spectrum
Scale RAID on IBM ESS.
򐂰 Problem determination guide, which provides more information about monitoring,
troubleshooting, and maintenance procedures.

The IBM ESS product documentation is available at IBM Knowledge Center.

Ansible playbooks and container-based software installation

The ESS 3000 and ESS 5000 implement a new ESS software update process and
methodology based on Ansible playbooks. The Ansible playbooks provide powerful, flexible
orchestration tools that drive containerized software deployment methodology, that speeds
and simplifies the process of installing ESS. These new features reduce specialized storage
skill requirements, and reduces need for IBM System Lab Services.

The POWER9 IBM ESS Management Server is also the control Management node where the
Ansible software and containers are installed and resident. The POWER9 EMS is used as the
single point to perform system management over all ESS 3000, ESS 5000, and (with the
announcements of Feb 23, 2021 any second-generation IBM ESSs) in the IBM Spectrum
Scale cluster. On this EMS node, Ansible is configured to store the container and cluster
configuration and definitions for all of the managed IBM ESS Data Servers. Network services
(such as DHCP, SSH) are enabled to allow the IBM ESS Management Server to deploy IBM
ESS solution stack components to all IBM ESS nodes in the cluster.

eXtreme Cloud Administration Toolkit

The 1st and 2nd generation ESS machines are managed by IBM implementations of the
eXtreme Cloud Administration Toolkit (xCAT), an open source distributed computing
management software that was originally developed by IBM. xCAT is commonly used for the
deployment and administration of high-performance computing, Linux-based clusters - and

Chapter 2. IBM Elastic Storage System architecture 41

 thus xCAT was also used for that purpose in administering all first-generation and
second-generation IBM ESSs.

For more information, see the xCAT website.

The IBM ESS Management Server is also the xCAT Management node where the xCAT
software is installed and used as the single point to perform system management over all first
and second-generation IBM ESSs in the IBM Spectrum Scale cluster. On the EMS node, an
xCAT database is configured to store the xCAT node definitions for all managed IBM ESS
Data Servers. Network services (such as DHCP, tftp, and http) are enabled to all the IBM ESS
Management Server to deploy IBM ESS solution stack components to all IBM ESS nodes in
the cluster.

For information about the xCAT architecture, see this web page.

As part of every IBM ESS order, IBM includes the IBM SupportLine for xCAT offering, which
allows IBM Service/Support to service any solution issues that are related to open source
xCAT. As of the writing of this Redpaper, there are still some ESS functions that are based on
xCAT; thus the need for the IBM Supportline for xCAT continues with all ESSs.

The IBM Program ID for this Support Line offering is 5641-CTx, where “x” is 1, 3, or 5 years.

ESSUTILS and GSSUTILS

ESSUTILS (and its predecessor GSSUTILS) are IBM ESS installation and deployment
toolkits that are primarily designed to facilitate IBM System Support Representative (SSR)
hardware setup, installation, deployment, and upgrade tasks. Any authorized IBM ESS
system administrator also can use these tools. The ESSUTILS GSSUTILS main menu is
shown in Figure 2-11.

Figure 2-11 ESSUTILS and GSSUTILS main menu

ESSUTILS/GSSUTILS provides a set of task menus that are related to install and deploy
activities. When a task is selected from the menu, a command is issued to the system for that
task. ESSUTILS and GSSUTILS can be run on the IBM ESS Management Server (EMS)
node only.

For more information about ESSUTILS and GSSUTILS, see the Quick Deployment Guide
and the IBM ESS Command Reference, which are available at IBM Knowledge Center.

42 Introduction Guide to the IBM Elastic Storage System

Chapter 2. IBM Elastic Storage System architecture 43
44 Introduction Guide to the IBM Elastic Storage System
 3

Chapter 3. IBM Elastic Storage System

planning and integration
This chapter provides guidelines and considerations for the correct planning, installation, and
configuration of the IBM Elastic Storage System (ESS).

This chapter also describes configurations and integration considerations for a smooth IBM
ESS deployment into an existing or a new IT environment. In this chapter, we describe the
following elements:
򐂰 Roles and responsibilities
򐂰 Planning
򐂰 Networking
򐂰 Installation
򐂰 Maintaining hardware
򐂰 Maintaining software
򐂰 Upgrades
򐂰 Technical Design Assessment (TDA), File Object Solution Design Engine (FOS DE)

This chapter includes the following topics:

򐂰 3.1, “IBM Elastic Storage System planning overview” on page 46
򐂰 3.2, “IBM Elastic Storage System hardware planning” on page 50
򐂰 3.3, “IBM Elastic Storage System software planning” on page 57
򐂰 3.4, “Ordering IBM Elastic Storage System” on page 57
򐂰 3.5, “IBM Elastic Storage System installation” on page 59
򐂰 3.6, “Monitoring IBM ESS” on page 63
򐂰 3.7, “Maintaining and repairing IBM ESS” on page 63
򐂰 3.8, “Upgrading IBM ESS” on page 64
򐂰 3.9, “ESS hints and tips” on page 66

© Copyright IBM Corp. 2016, 2021. All rights reserved. 45

3.1 IBM Elastic Storage System planning overview
In this section, we make recommendations and highlight best practices for various roles and
responsibilities when an IBM ESS is deployed.

3.1.1 Roles and responsibilities

In this section, we discuss best practices were learned from successful IBM ESS installations
regarding who is responsible for each of the areas of expertise.

Overall planning
An experienced project leader should be appointed to coordinate and manage the IBM ESS
installation project. Deploying an IBM ESS requires coordination across different IT
departments and involves various team members (IBM and client).

The various client and IBM teams that the project manager must coordinate include (but are
not limited to):
򐂰 Physical planning for the data center, including the networking team
򐂰 Networking setup, switch installation, cabling, and verification
򐂰 Hardware setup, installation, and verification
򐂰 Software planning and integration of the file system into the environment
򐂰 Acceptance testing and criteria for moving into production

Networking
ESS components are connected by using IP or InfiniBand networking. IBM ESS data users
access IBM ESS by way of high-speed IP or InfiniBand networking.

The networking planning tasks that are required include (but are not limited to) include the
following examples:
򐂰 Suitable networking switches must be ordered or allocated, including cable planning,
ordering, racking, and proper connectors.
򐂰 Network addressing must be planned for IBM ESS and the integration of that network into
existing client IP or InfiniBand network.
򐂰 After the network is designed and approved, IP addresses must be implemented and
tested.
򐂰 Multiple VLANs must be designed, configured, and tested on the management network
switches.
򐂰 Suitable networking authentication and firewall rules must be configured and tested to
assure that they comply with client network audit and security requirements.
򐂰 Testing of the network before IBM ESS is installed (by using tools, such as NSDPERF or
Network Readiness Tools that are provided by IBM Spectrum Scale as free open source
tools) must be done to identify and verify the network’s ability to provide IBM ESS the
required network bandwidth, latency, and consistent performance.

These tasks require the client IBM ESS infrastructure implementation team to complete the
following tasks:
򐂰 Identify all IP or InfiniBand networking-related requirements.
򐂰 Engage early with the infrastructure networking team.

46 Introduction Guide to the IBM Elastic Storage System

򐂰 Document and communicate all required IP or InfiniBand network provisioning,
implementation, load-testing, and network implementations, including all QA time frames,
implementation lead times, testing windows, and acceptance criteria.
򐂰 Monitor these tasks closely to assure that the networking is in place and load tested
before IBM ESS is installed. Experience shows that networking addressing mismatches,
VLAN issues, configuration issues, and a failure to remedy these issues are the major
cause of unforeseen delays in IBM ESS installation progress.

The client IP or InfiniBand networking team is responsible for the following tasks:
򐂰 Receive the specified networking requirements.
򐂰 Consult and collaborate with the IBM ESS implementation team to design, optimize, and
agree upon a suitable IP or InfiniBand networking design and implementation plan.
򐂰 Implement the agreed-upon networking changes.
򐂰 Run requested network load tests by using tools that are provided by IBM Spectrum Scale
before IBM ESS is installed, including agreeing to and implementing any necessary quality
assurance and acceptance criteria.
򐂰 Adjust and tune the network based on pre-installation tests or during the IBM ESS
implementation.

The following best practices from successful IBM ESS installations assure success:
򐂰 Engage early with the client networking team.
򐂰 Obtain collaborative buy-in and well-defined ownership of networking-related task
completions.
򐂰 Operatively running network performance tests by using IBM Spectrum Scale tools far
enough in advance of the IBM ESS installation allows time for necessary tuning
adjustments to be made.
򐂰 Ongoing IBM ESS implementation team and client networking team dialog and interaction
during the IBM ESS installation process quickly remedies any networking-related
installation issues.

Installation
In this phase, the following IBM and client teams must be involved and coordinated:
򐂰 Physical site administration and engineering
򐂰 Networking team and network engineering
򐂰 Hardware implementation team, including client IT administrators, IBM System Services
Representative (SSR) for IBM ESS physical installation and setup of physical hardware
򐂰 Software implementation team, including IBM System Lab Services consultants, and
client operating system and software administrators

These teams collaborate to implement the following phases:

򐂰 Physical site planning
This planning includes the receipt and staging of IBM ESS components after shipment,
physical floor space planning and racking; electrical power planning and provisioning,
including specifying suitable electrical power, cables, connectors, and Power Distribution
Units; and assuring proper airflow and air conditioning.

Chapter 3. IBM Elastic Storage System planning and integration 47

 򐂰 Software planning and integration
This planning includes defining the IBM Spectrum Scale file system implementation
parameters, requirements, integration testing, and acceptance with the client applications
and procedures.
򐂰 Hardware physical setup, installation, and verification
This planning and scheduling includes the IBM ESS hardware setup with the IBM System
Services Representative and the IT physical installation teams.
򐂰 Acceptance testing
Running well-defined criteria for acceptance testing and moving the IBM ESS system into
production are featured in this phase.

Maintaining hardware
After the IBM ESS is installed and running in production, many parties are responsible for
various ESS hardware tasks, including the following examples:
򐂰 Client ESS administrators:
– Monitoring IBM ESS physical hardware by using provided tools and the GUI
– Basic hardware problem determination
– Optional basic maintenance, such as replacing a failed SSD or HDD
򐂰 IBM System Services Representative: Responsible for physical IBM ESS hardware
maintenance and repair.

IBM Learning Services or IBM Business Partners can provide education courses about the
customer topics for maintaining IBM ESS hardware. More geography resources might be
available. For more information, contact your IBM representative or IBM Business Partner.

Maintaining software
After the IBM ESS is installed and running in production, many parties are responsible for
various IBM ESS software tasks, including the following examples:
򐂰 Client Red Hat Linux administrators:
– Assuring RHEL licensing and subscriptions are in place for first- and
second-generation IBM ESS
– Applying required RHEL security patches as recommended by IBM
򐂰 Client IBM Spectrum Scale administrators:
– Monitoring and tuning IBM Spectrum Scale performance
– Data management, which includes monitoring data and storage usage patterns, writing
and running policies that manage data, backups, and snapshots

IBM Learning Services or IBM Business Partners can provide education courses about the
customer topics for maintaining IBM Spectrum Scale and IBM ESS hardware. More
geography resources might be available. For more information, contact your IBM
representative or IBM Business Partner.

48 Introduction Guide to the IBM Elastic Storage System

Upgrades
The IBM ESS provides an integrated software solution stack, which is composed of specific
software release levels and the following patch levels:
򐂰 Red Hat Enterprise Linux kernel levels and components
򐂰 POWER firmware
򐂰 Mellanox NIC drivers
򐂰 IBM Spectrum Scale and IBM Spectrum Scale RAID software

Note: It is a client responsibility to plan and upgrade their IBM ESS solution software.

IBM recommends upgrading the IBM ESS software solution stack at least once a year and to
keep the IBM ESS current on levels of RHEL, firmware, and IBM Spectrum Scale.

IBM System Lab Services is available and it is recommended to engage Lab Services to
assist and perform these software upgrade actions on behalf of the client, if required.

Technical design assessment and file object solution design engine

An IBM Technical Design Assessment (TDA) is a technical inspection of a completed solution
design. Technical Subject Matter Experts (SMEs) who were not involved in the solution
participate to assure the client’s satisfaction.

In a pre-sales TDA, which is performed before the IBM ESS solution is ordered, the File
Object Solution Design Engine (FOS DE) tool and process is used to examine and approve
the IBM ESS solution design to determine the following information:
򐂰 Will the IBM ESS solution work as expected?
򐂰 Is the configuration and implementation plan sound?
򐂰 Will the IBM ESS solution meet client requirements and expectations?

In a pre-installation TDA, SMEs also evaluate the client’s environment for readiness to install,
implement, and support the proposed solution.

IBM and IBM Business Partner teams are responsible to conduct these pre-sale and a
pre-installation processes. Contact your IBM representative or IBM Business Partner to
schedule and perform these TDA and FOS DE processes.

For more information about the IMPACT and pre-installation TDA tools, which help your IBM
account team, client technical specialist, and IBM Business Partner select the best IBM ESS
and IBM Spectrum Scale solutions to match your business requirements and needs, see the
following web pages:
򐂰 IBM Solution Assurance and TDA
򐂰 TDA and FOS DE processes for IBM Elastic Storage Server

These resources are available to IBMers or Business Partners after logging in by using their
IBM ID or IBM PartnerWorld® ID.

Chapter 3. IBM Elastic Storage System planning and integration 49

3.2 IBM Elastic Storage System hardware planning
In this section, we describe the necessary planning and implementation steps that must be
followed as part of installing the IBM ESS hardware.

3.2.1 Planning for network switches and networking

Suitable network planning and configuration are one of the most important success factors for
an IBM ESS installation. This planning is crucial because the IBM Spectrum Scale and IBM
ESS cluster components are connected by way of an extensive set of IP addresses and
associated networking switches and cabling.

The following networks are required to install an IBM ESS:

򐂰 Service network
This network connects the flexible service processor (FSP) on the management server
and I/O server nodes (with or without the Hardware Management Console [HMC],
depending on the IBM ESS generation).
򐂰 Management and provisioning network
This network connects the management server to the I/O data server nodes (and HMC if
the first-generation IBM ESS machine type 5146 models are used). The management
server runs the Dynamic Host Configuration Protocol (DHCP) on the management and
provisioning network. If an IBM ESS management server is not included in the solution
order, it is assumed that POWER IBM ESS management server exists in the same IBM
Spectrum Scale cluster that can be used.
򐂰 Clustering network
This high-speed data network is used for IBM ESS data transfer to and from the rest of the
IBM Spectrum Scale cluster and client node access. It can be a 10-Gigabit Ethernet
(GbE), 25 GbE, 40 GbE, 100 GbE, or InfiniBand network.
Higher-speed InfiniBand or 100 GbE networks are preferred because the best
performance is achieved over InfiniBand. Ethernet networks (such as 100 GbE, 40 GbE,
25 GbE, and 10 GbE) have more overhead than InfiniBand and typically lower aggregate
bandwidth capabilities result.

Note: This network must provide consistent, congestion-free network packet delivery
with consistent low latency. IBM can supply tools before the installation to test the
network. The network might be provided by Mellanox switches that are ordered through
IBM at order time, or the network might be provided by the customer.

For more information about for estimating and sizing suitable network performance, see
your IBM or IBM Business Partner.

򐂰 External and campus management network

This public network is used for external and campus management of the IBM ESS
management server, the HMC (if available), or both.

50 Introduction Guide to the IBM Elastic Storage System

Figure 3-1 shows an example network topology for IBM ESS.

Figure 3-1 ESS network topology example

An overview of the many networking steps that are required in the installation of an IBM ESS
are described next. This example is representative (not all inclusive) of the necessary network
planning and implementation steps that must be completed before installing the IBM ESS
hardware. For more information about network planning, see the IBM ESS Quick Deployment
Guide that is available at IBM Knowledge Center.

Example: Network IP addressing, planning, and implementation for IBM

ESS
In this section, we present an example of typical networking-related items for installing an IBM
ESS. The following template can be used for the installation process:
1. Confirm Private IP range for DHCP server.
2. Confirm Private Service network with defined IP addresses and private management
network with defined IP addresses. Separate the networks by using switches or VLAN.
3. Confirm Public network connections for IBM ESS Management Server. Multiple IPs are
needed.
4. Confirm Host → IP mappings for:
– HMC (if present because of first-generation IBM ESSs).
– ESS Management Server (EMS).
– ESS Data Servers: IO server 1, IO server 2, IO server 3, IO server 4, and so on.
– 10 GbE, 25 GbE, 40 GbE, and 100 GbE by clicking InfiniBand hostname → IP
mappings.
5. Set up domain names for management private network.
6. Set up domain names for high-speed interconnect.
7. Set up Data I/O partition and partition profile names.
8. Confirm Server names.

Chapter 3. IBM Elastic Storage System planning and integration 51

 9. Confirm that 10 GbE, 25 GbE, 40 GbE, or 100 GbE InfiniBand switches in place and
cabled.
10.Set up Bonding if it is used.
11.Set up Public network in place and cabled to EMS.
12.Confirm that all building block components are correctly cabled in the rack (Data I/O
servers, EMS, and switches).
13.Set up and confirm dual feed power to racked IBM ESS components.
14.Set up IMB ESS Management Server console or terminal, including network connections.
15.Prepare for and install Red Hat Enterprise Linux errata.
16.Register Red Hat subscription keys for all IBM ESS servers, as needed (if you have first or
second-generation IBM ESS).
17.Define and specify IBM Spectrum Scale installation parameters and requirements. For
example, how many file systems, what are the block sizes, how do we allocate and split
metadata, and what replication factor is used.
18.Confirm all physical NVMe, SSD, or HDD disks are in place. IBM ESS installation checks
with scripts.
19.Confirm that all physical cabling in place. IBM ESS installations are checked by scripts.
20.Confirm that WiFi access exists in lab to set up Webex (for access to and by support
teams).

This template shows the processes that IBM Systems Lab Services can perform as part of a
contract to install an IBM ESS System. As part of that contract, IBM Systems Lab Services
provides and works with the customer on a detailed IP networking planning document.

3.2.2 Planning for servers that are used in IBM Elastic Storage System
In this section, we review planning information for the physical servers that are used in an IBM
ESS. We also describe specific recommendations for memory, network interface cards, and
cabling and connectors.

IBM ESS Management Server

Each IBM Spectrum Scale cluster that uses an IBM ESS requires at least one IBM ESS
Management Server (EMS). The EMS acts as the central management focal point for all IBM
ESSs that are in the cluster; runs specific functions, such as the IBM ESS graphical user
interface (GUI); and stores and distributes the master boot image for all the IBM ESSs that
are in the cluster by using the xCAT software (first- and second-generation IBM ESS) or by
using Red Hat Ansible playbooks (third-generation IBM ESS 3000 and IBM ESS 5000).

For the first-generation IBM ESS (machine type 5146), a POWER8 machine type 8247-21L
server is used as the EMS. This EMS runs Red Hat Enterprise Linux Big Endian. Only
first-generation IBM ESSs are supported by the 8247-21L EMS.

For the second-generation IBM ESS, a POWER8 server with machine type 5148-21L is used
as the EMS. This EMS runs Red Hat Enterprise Linux Little Endian. In an IBM Spectrum
Scale cluster, a 5148-21L EMS can simultaneously support second-generation IBM ESSs
and IBM ESS 3000. 5148-21L EMS cannot support a first-generation IBM ESS.

52 Introduction Guide to the IBM Elastic Storage System

For the third-generation IBM ESS (IBM ESS 3000 and IBM ESS 5000), a POWER9 server
with machine type 5105-22E is used as the EMS. This EMS runs Red Hat Enterprise Linux
Little Endian. If the IBM ESS 3000 is deployed into a cluster that features a 5148-21L EMS,
the IBM ESS 3000 can be supported by that 5148-21L EMS.

ESS 3000 and IBM ESS 5000 introduce a new EMS software deployment mechanism that is
based on containers and Red Hat Ansible orchestration playbooks.

An IBM Spectrum Scale cluster can simultaneously support all three generations of IBM ESS.

All first-generation IBM ESSs are supported by an IBM POWER8 8247-21L IBM ESS
Management Server running RHEL Big Endian. All second-generation and IBM ESS 3000s
are supported by an IBM POWER8 5148-21L IBM ESS Management Server running RHEL
Little Endian. All IBM ESS 3000s and IBM ESS 5000s are supported by IBM POWER9
5105-22E IBM ESS Management Server running RHEL Little Endian. Announced on
February 23, 2021, this POWER9 5105-22E ESS Management Server can now also support
2nd generation ESSs.

IBM ESS data servers

Every IBM ESS is an integrated IBM Spectrum Scale storage building block that is consists of
two IBM Spectrum Scale data servers and each data server is cross-connected to all the
storage for high availability. In normal operation, half of the IMB ESS storage is running
through the first data server; the other half of the IBM ESS storage is running through the
second data server. If a planned or unplanned outage occurs on one of the data servers, the
storage is failed over to the remaining data server in the IBM ESS, which provides high
availability and resiliency.

Each first-generation IBM ESS (machine type 5146) storage building block uses two
POWER8 servers (machine type 8247-22L) each acting as an IBM Spectrum Scale Data
Server. These 8247-22Ls run Red Hat Enterprise Linux Big Endian. Only first-generation IBM
ESSs use the 8247-22L RHEL Big Endian-based data servers.

In the second-generation IBM ESS, two POWER8 servers with machine type 5148-22L are
used as the IBM ESS Data Servers. These servers are physically identical to 8247-22L
servers, but carry an IBM ESS-specific machine type of 5147 to assure that IBM Service and
Support always knows that these servers are part of an overall IBM ESS-integrated solution.
These 5148-22L POWER8 servers run Red Hat Enterprise Linux Little Endian.

An IBM Spectrum Scale cluster fully supports a mix of all generations of IBM ESS in the same
cluster. A 5148-21L EMS can simultaneously support second-generation IBM ESSs and IBM
ESS 3000. A 5148-21L EMS cannot support a first-generation IBM ESS.

For the third-generation IBM ESS 3000, the 2U24 form factor uses an integrated pair of
x86-based servers (no separate machine type) as IBM Spectrum Scale Data Servers.

For the third-generation IBM ESS 5000, two POWER9 servers with machine type 5105-22E
are used as the IBM ESS Data Servers. These servers are specific POWER9 servers for IBM
ESS 5000. They carry an IBM ESS-specific machine type of 5105-22E to assure that IBM
Service and Support always knows that these servers are part of an overall IBM
ESS-integrated solution. These 5105-22E POWER9 servers run Red Hat Enterprise Linux
Little Endian.

An IBM Spectrum Scale cluster can simultaneously support all three generations of IBM ESS.
All first-generation IBM ESSs in the cluster are supported by an 8247-21L IBM ESS
Management Server running RHEL Big Endian. All second-generation and IBM ESS 3000s in
the cluster are supportable by the 5148-21L IBM ESS Management Server or 5105-22E IBM

Chapter 3. IBM Elastic Storage System planning and integration 53

 ESS Management Server, both of which run RHEL Little Endian. All third-generation IBM
ESS 5000s require a 5105-22E POWER9 IBM ESS Management Server that runs RHEL
Little Endian. Announced on February 23, 2021, the POWER9 5105-22E ESS Management
Server can now also support 2nd generation ESSs.

Memory
Standard POWER9 DIMM memory cards (specified with standard POWER9 server memory
feature codes) are used in the 5105-22E servers. The default memory chip is a 32 GB DIMM.

The standard memory size for the 5105-22E when used as a Data Server is 384 GB,
consisting of 32 GB DIMMs with quantity of 12. When 5105-22E is used as IBM ESS
Management Server, the default memory is 128 GB, consisting of 16 GB DIMMs with quantity
of 8.

When 5105-22E is used as IBM ESS Protocol node with single socket, the default memory is
128 GB, consisting of 16 GB DIMMs with quantity of 8. When 5105-22E is used as IBM ESS
Protocol node with two sockets, the default memory is 192 GB, consisting of 16 GB DIMMs
with quantity of 12. This memory is enough for most IBM ESS installations.

Standard POWER8 DIMM memory cards (specified with standard POWER8 server memory
feature codes) are used in the 8247-21L, 8247-22L, 5148-21L, and 5148-22L servers. The
standard memory size for the 8247-21L and the 5148-21L EMS was originally 32 GB (two
16 GB DIMMs). Recent IBM ESS 5148-21L IBM ESS Management Servers moved to larger
32 GB memory DIMMs and the default memory is now 64 GB for the EMS.

The default IBM ESS Management Server memory size is enough for most IBM ESS
installations. If many IBM ESSs are used in your Spectrum Scale IBM ESS configuration,
check with your IBM representative to see whether larger IBM ESS Management Server
memory sizes might be required for your installation. More EMS memory can be specified at
order time or added later as a field MES.

The standard memory for the IBM ESS data servers is enough for most IBM ESS usage
cases. If you have a large workload configuration, or you have a specific workloads type,
larger memory might be needed on your IBM ESS data servers. Check with your IBM
representative to see whether you have specific workloads that might benefit from more main
memory.

Standard POWER memory upgrades for IBM ESS data servers can be ordered and installed
without disruption to access to the IBM Spectrum Scale file system. In the installation
process, the storage in one of the two IBM ESS Data Servers to be upgraded, is failed over to
the other IBM ESS Data Server. This procedure allows the first IBM ESS data server to be
gracefully brought down and then upgraded. Then, the same process is repeated for the other
IBM ESS data server to be upgraded.

Note: Both data servers must have the same amount of memory in each IBM ESS building
block.

ESS 3000 uses a pair of x86-based servers that are integrated into the back of the 2U24
storage enclosure. The standard memory for each of these IBM ESS 3000 data servers is
384 GB. This memory is enough for most of IBM ESS 3000 usage cases. You can specify
another 384 GB of main storage per data server at order time (or later by field upgrade),
which results in a total of 768 GB of memory per IBM ESS 3000 data server.

54 Introduction Guide to the IBM Elastic Storage System

 Network interface cards
The IBM ESS features the following types of networks that are used to interconnect the IBM
ESS components:
򐂰 Management and service
Mandatory, separate 1 GbE network for management and service traffic.
򐂰 Data
A high-speed network for the user data being written into and read out of the IBM
Spectrum Scale and IBM ESS file system.

Standard POWER server network interface cards (NICs) are used to provide the high-speed
network connectivity in an IBM ESS. The network cards that are available vary depending on
the IBM ESS generation.

Make sure that the network bandwidth requirements are clearly studied, documented, and
communicated. Specify a suitable number of NICs to meet expected IBM ESS performance
requirements. Plan and specify the correct cables and connector types.

Note: IBM ESS POWER Data Servers do not support every possible POWER PCI
adapter.

Cabling and connectors

In addition to specifying the correct number of NICs to provide bandwidth requirements, make
sure that physical planning for cable lengths, adapter connector types, and switch connector
types, are clearly studied, documented, and communicated.

Physical machine room placement of IBM ESS components relative to the physical location of
the IP network switches must be clearly understood and documented. Physical cable orders,
including wanted colors, connector types, and required cable routing must be studied,
specified, and verified with network teams, server teams, physical site teams, and
infrastructure teams.

Many network connector types cable types are available. Requirements must be identified for
the required networking components, including the following examples:
򐂰 Transceivers
򐂰 Optical versus copper
򐂰 Connector form factor
򐂰 Cable lengths

3.2.3 Planning for storage configuration

ESS storage configuration for ordering purposes is relatively straightforward.

First, determine your IBM Spectrum Scale storage requirements. Request that your IBM
representative or IBM Business Partner use the FOS DE tool to help identify usable capacity
requirements and to estimate wanted performance.

After storage capacity and performance requirements are known, identify which models of
IBM ESS feature the suitable type and quantity of storage media available. Within each
model, you choose the suitable drive size. Request that your IBM Representative or IBM
Business Partner use the FOS DE tool to verify the performance of your chosen IBM ESS
model with your requirements.

Chapter 3. IBM Elastic Storage System planning and integration 55

 Part of the value of IBM ESS is that after storage performance and capacity requirements are
known, IBM ESS models provide known performance and usable capacity characteristics
because these factors were considered when IBM designed the IBM ESS model.

Each IBM ESS model includes predefined storage configurations and requirements, which
are pre-tested and supported as a solution by IBM. Cable lengths, installation parameters,
adapters, drive preparation, and location, all are pre-specified within the IBM ESS
architecture. This demonstrates the value of the use of IBM ESS as an optimum, proven,
tested method to deploy IBM Spectrum Scale storage by using an efficient building block
approach.

3.2.4 Planning for IBM Spectrum Scale protocol nodes

First announced in 2014, IBM Spectrum Scale protocol nodes provide external SMB, NFS,
Object, and Hadoop access to IBM Spectrum Scale data.

From a planning perspective, IBM Spectrum Scale requires that all Protocol Nodes in an IBM
Spectrum Scale cluster must be all x86, POWER Little Endian, or POWER Big Endian.

No Protocol Node intermixing of these nodes is allowed in the same IBM Spectrum Scale
cluster.

ESS coexists in an IBM Spectrum Scale cluster with protocol nodes. IBM ESS also has no
effect on which type of Protocol Node can be used in an IBM Spectrum Scale cluster.

With the IBM ESS 3000 and IBM ESS 5000, IBM provided the option of ordering IBM
Spectrum Scale protocol nodes based on the POWER9 5105-22E server. These Protocol
Nodes are supported by the IBM ESS Management Server and can be managed by the IBM
ESS GUI and by IBM ESS installation tools, which provides greater ease of use and
management capability for ordering a complete IBM Spectrum Scale or IBM ESS or protocol
node solution from IBM.

The 5105-22E IBM Spectrum Scale Protocol Nodes can coexist with any other POWER Little
Endian protocol nodes in the same IBM Spectrum Scale cluster.

For more information about 5105-22E Protocol Nodes, see Quick Deployment Guide, which is
available at IBM Knowledge Center.

3.2.5 Planning for physical, electrical, and cooling installation

Preparation for installation of your IBM ESS system is as important as the physical installation
of the system. To have a smooth and successful installation, preparation processes, and
Technical Design Assessments must be followed.

FOS DE and pre-installation meetings with IBM TSS and IBM Service personnel must be
completed. Also, IBM Elastic Storage Server environmental specifications provide detailed
information about your system, including dimensions, electrical, power, temperature,
environmental requirements, and noise emissions. By following the preparation process, IBM
service personnel can efficiently install the physical hardware and also deployment of the
software.

The specifications can be found in IBM Knowledge Center by searching for “System
specifications for the IBM Elastic Storage Server (5147 and 5148)” and “Individual building
block specifications”.

56 Introduction Guide to the IBM Elastic Storage System

3.3 IBM Elastic Storage System software planning
In this section, we highlight key items to be aware of for planning the installation of the IBM
ESS software stack, in particular, IBM Spectrum Scale and IBM Spectrum Scale RAID.

3.3.1 Planning for IBM Spectrum Scale file system

The IBM Spectrum Scale file systems that are placed into the IBM ESS must be thoroughly
planned, documented, and communicated. Make sure that user application, authentication,
firewall, and backup and restore requirements are understood and included.

Before the arrival and installation, the IBM Knowledge Center manual for IBM Spectrum Scale
and IBM Spectrum Scale RAID Administration should be consulted. These manuals contain
detailed instructions and best practices recommendations for implementing the requirements
of the client’s environment.

3.3.2 Planning for IBM Spectrum Scale RAID

The IBM Spectrum Scale RAID Administration manual provides best practices and
recommendations for setting up IBM Spectrum Scale RAID. Consult this manual and follow
the recommendations.

The IBM Elastic Storage System and IBM Elastic Storage Server implementation of IBM
Spectrum Scale RAID uses JBOD arrays, provides the required redundancy protection, and
delivers usable disk capacity, required spare capacity, and maintenance strategy.

In IBM ESS, the IBM Spectrum Scale RAID implementation best practices are enforced as de
facto standards by the IBM ESS installation scripts, which provide configuration parameters
that are proven for achieving the best storage performance.

Each JBOD array is connected to each of the two NSD Data Servers in the IBM ESS to
protect against server failure. Each NSD Data Server has two independent paths to each
physical disk to protect against path failure and provide higher throughput to the individual
disks.

3.4 Ordering IBM Elastic Storage System

IBM ESS is always part of a larger IBM Spectrum Scale system. Because IBM Spectrum
Scale and IBM ESS are sophisticated technologies to assure high client satisfaction, IBM
requires that a Technical Design Assessment and the IBM File Object Solution Design Engine
tools are used to elicit requirements, assure that designs are reviewed, assessed, and
approved.

The IBM ESS solution process engages a pre-sale and a pre-delivery technical delivery
assessment (TDA). The purpose of these two assessments is to match the client’s
requirements and needs. It also evaluates the client’s current IT environment to create the
best proposed IBM ESS solution that matches the client’s needs.

Before confirming the IBM ESS order, the IBM team or the Business Partner uses the IBM
FOS DE tool. FOS DE is a question and answer tool that elicits and documents the various
technical requirements for a solution that uses IBM ESS. Contact your IBM representative to
request that a pre-order IBM ESS FOS DE design evaluation is performed.

Chapter 3. IBM Elastic Storage System planning and integration 57

 IBM eConfig is used by the IBM or Business Partner team to configure the IBM ESS for
ordering. eConfig is a tool that helps the IBM or Business Partner team to configure the
correct combination of IBM storage enclosures, servers, network interface cards, and
associated feature codes to make up the wanted IBM ESS.

Note: An approved pre-installation TDA and an approved FOS DE run must be performed
by the IBM account team or IBM Business Partner to have an IBM ESS order to be entered
and shipped.

As part of the order, various decisions regarding hardware must be made. Be prepared to
supply requirement information to the IBM Business Partner. The following requirements
often are included in the information that is sent:
򐂰 Storage requirements (total capacity, HDD or SSD sizes, and wanted performance).
򐂰 Server memory and network interface card requirements.
򐂰 Wanted network cables, cable lengths, connector types, and network switches.
򐂰 Physical electrical power requirements (phase and voltage).
򐂰 Any other more requirements (your IBM or Business Partner representative works with
you to determine these requirements).

IBM or the Business Partner then configures the IBM ESS. IBM ESS components can be
installed in an IBM rack (preferred) or installed in customer supplied racks.

Figure 3-2 shows an example of some possible IBM ESS configurations in one rack. Shown
on the left in Figure 3-2 are the size of various IBM ESS components in rack units. Shown on
the right of Figure 3-2 are examples of four IBM ESS SL models. These configurations are
examples only; your configuration is tailored to your requirements.

Figure 3-2 Elastic Storage Server physical racking examples

Note that if a ESS 5000 is upgraded with additional storage enclosures, it is not necessary to
reposition the POWER9 5105-22E data servers. IBM will install the additional storage
enclosures above the POWER9 data servers in the rack.

58 Introduction Guide to the IBM Elastic Storage System

3.5 IBM Elastic Storage System installation
In this section, we describe the process that is used to physically install the IBM ESS and
deploy it into the client’s IBM Spectrum Scale environment.

The installation process starts with the client preparing their environment to be ready for the
IBM ESS. After the client’s environment is prepared, the IBM expert service personnel install
the hardware. Depending on the solution, the deployment of IBM ESS also is performed by
IBM specialized service personnel.

3.5.1 Physical installation, racking, and cabling

IBM Technical Support Services (TSS) should be engaged to help you plan for the physical
installation and racking of the IBM ESS components.

IBM TSS works with you to help you plan for the many items to consider for racking,
installation, and servicing of the IBM ESS. Your TSS representative can help you plan for
many factors, including the following examples:
򐂰 Electrical power, power connectors, PDU, and amperage
򐂰 Network adapter and connectors
򐂰 Electrical power cables, network cables, and cable lengths
򐂰 Airflow considerations (especially IBM ESS SCx models), which use the deep IBM
5147-106 storage enclosures
򐂰 Shipping location of the system
򐂰 Location of systems that are installed in customer racks (if rackless; that is, not installed in
IBM rack)
򐂰 IP addresses, networking, and switch configuration that is needed to install and connect
the IBM ESS components
򐂰 Lift tools
򐂰 Ladders
򐂰 IBM System Support Rep (SSR) physical installation instructions (WCII) and SSR
scheduling and confirmation

If you have an IBM ESS HDD model, allow time for the physical installation of the many HDDs
into the storage enclosures. HDD IBM ESS models are shipped with the storage enclosures
empty to mitigate shipping damage. Each HDD is shipped in a sealed protected package. In a
large IBM ESS, many HDDs are used and it takes time for the IBM SSR to install each HDD
into its slot in the storage enclosures.

When are you planning for physical installation, remember to consider for future expansion.
Some factors include the following examples:
򐂰 Whether to reserve space in a rack for future expansion
򐂰 Location of rack in data center for purposes of future cable lengths and electrical power
supply expansion
򐂰 Location of network switches, IBM ESS Management Server for purposes of cable lengths
for networking

Chapter 3. IBM Elastic Storage System planning and integration 59

3.5.2 Integrating into client environment
Integrating the IBM ESS into the clients’ environment consists of many actions. In this section,
we provide an overview of key best practices, practical advice, and experiences and lessons
learned for a successful IBM ESS deployment.

Integrating into customer networking environment

In any IBM Spectrum Scale or IBM ESS solution, the network interconnect always plays a
central role. The goal of this section is to guide you in lessons that are learned from an IBM
Spectrum Scale or IBM ESS network readiness discussion.

Networking is a vast expertise area, and you should not expect to need to become a
networking expert. The following questions guide you through initial discussions you must
have with your networking infrastructure team to assure that your IP network is ready for your
IBM Spectrum Scale or IBM ESS solution to integrate IBM ESS into your environment.

Will an existing network be used, or will this network be new?

Document if you are installing IBM Spectrum Scale or IBM ESS on an existing network or if
this installation is on a new network that is designed specifically for IBM Spectrum Scale or
IBM ESS.

IBM Spectrum Scale or IBM ESS is a high-performance, high-data rate big data solution. Best
practices include providing the IBM Spectrum Scale or IBM ESS cluster with a dedicated
network infrastructure that is co-designed with the solution. If this configuration is possible,
designing that new network assures the high bandwidth and low latency that is required for
IBM Spectrum Scale or IBM ESS is available.

However, especially in a commercial environment, many (if not most) circumstances are such
that you cannot implement a dedicated network for IBM Spectrum Scale or IBM ESS. Instead,
the IBM Spectrum Scale or IBM ESS traffic must coexist on an IP network. Especially in this
case, consider the following points to validate if the network can provide the high bandwidth
and consistent low latency IP networking that IBM Spectrum Scale or IBM ESS solution
requires:
򐂰 IP Network vendors and switch models
Document and provide a list of the IP network switch vendors and switch models that are
to be used to provide the network infrastructure for this IBM Spectrum Scale or IBM ESS
cluster. IBM Spectrum Scale or IBM ESS depends upon IP networking for all of the IBM
Spectrum Scale nodes and users to connect and communicate with each other.
Mellanox is the preferred high-performance IP networking switch vendor for IBM. If you are
designing a net-new IP network for the IBM Spectrum Scale or IBM ESS cluster, consider
the use of Mellanox.
򐂰 Provide a Network Topology diagram
Provide a copy of the network topology and document where the IBM ESSs are to be
located and how they are to be connected.
򐂰 Total number of nodes in IBM Spectrum Scale or IBM ESS cluster?
Document the number and type of IBM Spectrum Scale and IBM ESS nodes that are
planned for your solution. Provide the number of protocol nodes and the number of IBM
Spectrum Scale NSD Data Servers.

60 Introduction Guide to the IBM Elastic Storage System

Are there sufficient port counts and port speeds available for the network
switches?
Document the number of ports and port speeds that are required by your IBM ESS and
assure that number of ports and speeds are available within the networking infrastructure.

What are the interconnects between switches?

Document the inter-switch links, where they are located, and what traffic traverses them.

Can existing networking links and switches be used?

IBM Spectrum Scale or IBM ESS can generate unusually high workload on a network. If IBM
Spectrum Scale or IBM ESS is to be placed on an existing network, document how heavily
used the network is, and what the anticipated usage will be after IBM ESS is added.

If it is necessary to validate whether the client's network can sustain the intended IBM
Spectrum Scale or IBM ESS workload, use the IBM network performance tool NSDPERF
(written by IBM Spectrum Scale Development). NSDPERF mimics IBM Spectrum Scale
workloads and provides an open source, freely available method to test the network before
installation.

For more information, see this web page.

What other network traffic types and amounts must coexist on these network
links?
Document what other traffic is to compete for bandwidth on the IP network. IBM Spectrum
Scale depends on low latency, consistent delivery of many small metadata messages to work
properly. If any congestion exists in the IP network, IBM Spectrum Scale can encounter
problems. Therefore, the best practice is to have a dedicated network infrastructure for the
IBM Spectrum Scale or IBM ESS, if possible.

Communicate this need for low latency and consistent, congestion free delivery of IBM
Spectrum Scale metadata messages to the network infrastructure team.

What are the basic workload types?

Document what other traffic IBM Spectrum Scale is to compete with for network bandwidth.
Identify usage cases that are to be used, looking to check the workload mix (small block IOPs
versus large block sequential, for example). Engage a qualified IBM Spectrum Scale architect
with networking skills to project how the IBM Spectrum Scale or IBM ESS network
infrastructure performs in your specific environment.

Sustained bandwidth required in GBps?

Determine performance expectations. In addition to documenting GBps sequential streaming
workload requirements, what other specific workloads are to be running? What tools or
criteria are to be used to quantify that performance? What specific criteria determines “good”
or “not so good” performance?

Is validation required to confirm that the existing network can provide

necessary GBps and IOPs?
If it is deemed necessary to examine the network further and validate that it is ready for an
IBM Spectrum Scale or IBM ESS solution after reviewing these questions and answers, use
the NSDPERF tool. For more information about this tool, see this web page.

Chapter 3. IBM Elastic Storage System planning and integration 61

 Integrating into customer software environment
IBM Spectrum Scale and IBM ESS provide various file or object interfaces to the user
applications. Determine, inventory, and document the user applications and their required
interfaces that the applications require. From that information, design the file systems,
directory structures, and data architecture that must be supported by the IBM Spectrum Scale
or IBM ESS solution.

Design the IBM Spectrum Scale or IBM ESS cluster to deploy the wanted POSIX/Linux/UNIX
file systems. If necessary, design and deploy the IBM Spectrum Scale protocol nodes and
their associated shares, mount points, and IP addresses that the software applications
require.

Integrating into customer security and authentication environment

ESS is a part of an IBM Spectrum Scale cluster. IBM ESS and IBM Spectrum Scale support
security and authentication by using various interfaces, including LDAP and Active Directory.
Determine, document, and inventory the required security, authentication, and access control
requirements. Then, use that information to design the IBM Spectrum Scale or IBM ESS
solution.

For more information about IBM Spectrum Scale authentication and security, see IBM
Spectrum Scale Security, REDP-5426.

3.5.3 Reliability, Availability, Serviceability

Elastic Storage System uses servers and design that have a rich tradition of Reliability,
Availability, Serviceability (RAS).

ESS’s reliability (and the availability of the applications it supports) is a function of much more
than just the reliability of the processors, or even of the entire system hardware. A full
description of a system design for RAS must include all of the hardware, firmware, operating
system, applications, operating environment, duty cycle, and so on.

Reliability in hardware is all about how often a hardware fault requires a system to be
serviced; the less frequent the failures, the greater the reliability.

Availability, then, is how infrequently such a failure affects the operation of the system or
application. For high levels of availability, correct system operation must not be adversely
affected by hardware faults. That is, a highly available system design ensures that most
hardware failures do not result in an application outage.

Serviceability is concerned with identifying what fails and ensuring an efficient repair (of that
component, firmware, or software).

IBM Call Home: IBM ESS hardware

The IBM ESS Management Server and I/O server hardware includes IBM Call Home
capability.

In the first-generation IBM ESS, the Hardware Management Console (HMC) provided the Call
Home focal point for service receiving, logging, tracking system errors, and, if enabled,
forwarding problem reports to IBM Service for hardware events.

When a serviceable event occurs on one of the monitored components (such as servers or
storage), the Call Home generates a Call Home event.

62 Introduction Guide to the IBM Elastic Storage System

 In the second and subsequent generation of IBM ESS, the HMC is no longer required. In
these newer IBM ESSs, the IBM ESS Management Server provides the focal point for
hardware service receiving, logging, tracking system errors, and, if enabled, forwarding
problem reports to IBM Service for hardware events for the EMS, I/O nodes, and storage.

IBM Call Home: IBM Spectrum Scale software

IBM Spectrum Scale software also has its own separate Call Home capability for purposes of
automatically collecting and securely uploading data that is related to IBM Spectrum Scale to
IBM Service and Support. These functions are separate from the IBM ESS Call Home
functions.

3.6 Monitoring IBM ESS

Monitoring the IBM ESS system includes system health, performance, and capacity
monitoring. You can monitor the system through IBM ESS GUI or with the help of CLI.

For more information, search for the topic “Monitoring IBM Spectrum Scale RAID” at IBM
Knowledge Center.

3.6.1 GUI
The IBM ESS graphical user interface (GUI) provides robust facilities to monitor and manage
the system. IBM Knowledge Center provides more information about how you can configure
the IBM ESS GUI to monitor the performance of the following functional areas in the system:
򐂰 Network
򐂰 System resources
򐂰 IBM Spectrum Scale RAID
򐂰 NSD server
򐂰 IBM Spectrum Scale client
򐂰 NFS
򐂰 SMB
򐂰 Object
򐂰 Clustered Trivial Data Base (CTDB), which is used for SMB
򐂰 Transparent cloud tiering
򐂰 Waiters
򐂰 Active File Management (AFM)

For more information, see Monitoring and Managing the Elastic Storage Server using the
GUI, REDP-5471.

3.7 Maintaining and repairing IBM ESS

IBM Knowledge Center for IBM ESS features a full library of documentation for maintaining
the hardware and software of the IBM ESS solution. At the website, select the IBM ESS
solution release level in which you are using.

3.7.1 ESS IBM Solution service and support

IBM provides service and support for IBM ESS as a solution. IBM ESS-specific keywords and
procedures are in place so that you, as a client, when calling or electronically requesting IBM

Chapter 3. IBM Elastic Storage System planning and integration 63

 ESS support, are routed to an IBM ESS cross-component solution support team that
understands the interaction of all of the IBM ESS components. They also can diagnose and
respond to your query in a holistic fashion, and where necessary, direct your query to suitable
Level 2 or Level 3 resources.

For more information, see IBM Elastic Storage Server (ESS) Support Reference Guide here.

3.8 Upgrading IBM ESS

Maintaining and upgrading your IBM ESS software solution stack is a customer’s
responsibility.

You also can choose to engage IBM System Lab Services or an IBM Business Partner to
perform the upgrades for you.

The IBM ESS solution is an integrated package of hardware, firmware, and software. All
components of an IBM ESS are integrated, tested, and supported by IBM as a solution. IBM
periodically provides a refreshed or up-leveled IBM ESS software solution stack.

Existing IBM ESS customers are entitled to download the newer IBM ESS solution stack from
IBM Fix Central. IBM recommends that you plan to upgrade your IBM ESS solution stack at
least once a year.

For hardware upgrades, you should contact and work with your IBM or Business Partner
specialist to plan and run the IBM ESS upgrades when necessary.

3.8.1 ESS hardware and firmware

ESS hardware and firmware updates are included in the IBM ESS solution stack. Upgrading
to a new level of IBM ESS solution stack also upgrades the server and storage firmware
levels, if necessary.

Typically, the IBM ESS upgrades are done non-disruptively. During a planned maintenance
window, fail over the storage to one of the I/O data servers, upgrade the I/O data server, and
then, fail over the storage to this newly upgraded server and upgrade the other server.

Storage enclosure firmware can typically be applied non-disruptively.

ESS also supports nondisruptive addition of more storage enclosures, which allows an ESS
to be non-disruptively upgraded to a larger capacity. For example, with ESS 5000, this
process supports the addition of a 5147-092 to an SLx model or a 5147-106 to a SCx model.
Therefore, an installed SC1 to SC8 model, or SL1 to SL6 model, can be upgraded to any
model configuration up to SC9 or SL7 respectively, without causing downtime or an
interruption in service.

An ESS 3000 which was ordered half-populated with 12 NVM drives, can be upgraded to fully
populated 24-drive ESS 3000 in a similar manner, by simply adding the NVMe drives in the
front of the ESS 3000 storage enclosure. No downtime or interruption in service is required.

The client is responsible for keeping the IBM ESS operating system, software, firmware, and
GUI current with recommended levels of software. For more information about upgrade
instructions, see IBM Knowledge Center.

64 Introduction Guide to the IBM Elastic Storage System

 Note: IBM and IBM Business Partners might offer IBM Services to assist or perform these
upgrades on behalf of the client.

For all hardware upgrades, clients work with the IBM or their Business Partner to match
their needs with the IBM ESS upgrades and services offerings that are available.

3.8.2 ESS software

ESS provides a fully integrated and tested solution stack of software that consists of the
following components at a specific release and fix level (including all prerequisites) that were
fully integrated and tested:
򐂰 Red Hat Enterprise Linux
򐂰 IBM Spectrum Scale
򐂰 Cluster export services code (CTDB, Samba for SMB, Ganesha for NFS, OpenStack Swift
for Object)
򐂰 xCAT (eXtreme Cloud Administration Tool)
򐂰 POWER or x86 server firmware
򐂰 Red Hat Enterprise Linux kernel at specific level (for example: 3.10.0-957.21.3)
򐂰 Red Hat Enterprise Linux system errata at specific level (for example: 219-62.el7_6.6)
򐂰 Red Hat Enterprise Linux Network Manager at specific level (for example:
1.12.0-10.el7_6)
򐂰 Red Hat Enterprise Linux device drivers (IPR)
򐂰 Mellanox OFED network adapter drivers
򐂰 Electronic Service Agent (ESA)
򐂰 Hardware Management Console (HMC) if this installation is first-generation IBM ESS
򐂰 For IBM ESS 3000 and IBM ESS 5000, Red Hat Ansible playbooks and software is used
to manage and provide significantly improved ease of use for upgrading IBM ESS software
components

Keeping the IBM ESS software solution stack current is a customer responsibility. IBM
recommends that clients plan to upgrade their IBM ESS software solution stack at least once
a year. Typically, the IBM ESS hardware and software solution stack upgrades are performed
at the same time.

For more information about the levels of the software components for each IBM ESS software
solution release level, see IBM Knowledge Center.

Upgrading the IBM ESS solution software is done by a full replacement of the software
solution stack with a newer level.

New levels of IBM ESS solution software stack are available at IBM Fix Central. For more
information about these newer levels of IBM ESS software, log on to IBM Fix Central by using
your IBM ID. At the website, in the Product Selector field, enter IBM Elastic Storage Server.

Then, select your wanted IBM ESS version or release level, your wanted platform (for current
IBM ESS, this platform often is Linux POWER PC 64 Little Endian).

A list of available IBM ESS solution software fix packs for download is shown. When you start
the download, IBM FixCentral checks your IBM ID’s authentication profile to assure that you

Chapter 3. IBM Elastic Storage System planning and integration 65

 set up your IBM ID with the suitable authentication and client customer number information.
This information authenticates that you are authorized to download IBM ESS solution
software.

Note: If you confirmed that you are authorized to download fix packs but receive a
message that says, “No applicable support agreement found”, this message indicates that
your IBM ID profile does not match the IBM entitlement database. In this case, click
Request Help in the download window. An IBM Help ticket is opened and IBM contacts
you to help resolve the matter.

For more information about applying the IBM ESS Fixpack, see the ESS Quick Deployment
Guide, which is available at IBM Knowledge Center.

IBM System Lab Services also is available to perform the IBM ESS software upgrade for you.
If you are new to IBM ESS, we recommend that you take advantage of these services. For
more information, contact your IBM representative.

3.9 ESS hints and tips

Consider the following hints and tips:
򐂰 Physically plan ahead and think through the following issues:
– Where the IBM ESS servers and components are to go in the rack. W
– Where to place the network switch and the server locations into the rack’s position.
– Cabling and type of adapter connectors. Always follow the recommendations from IBM.
򐂰 IBM recommends that you use defaulted tuning parameters, unless your IBM or Business
Partner subject matter expert reviewed your environment and recommended other
suitable choices.
򐂰 Always test and qualify your IP network before installing IBM ESS. IBM Spectrum Scale
and IBM ESS can achieve high performance and use all available network bandwidth.
Tune your network specifically for optimum IBM Spectrum Scale and IBM ESS
performance.
򐂰 Have all networks planned, implemented, and tested before IBM ESS hardware is
installed. For example, identify and implement network configuration, VLANs, IP
addresses, and authentication and security passwords. Make sure that your IP network
environment meets IBM’s ESS network criteria and is verified by an IBM subject matter
expert.
򐂰 Define clear IBM ESS performance criteria expectations and test plans to verify
performance.
򐂰 Use only IBM ESS and IBM Spectrum Scale certified machines and components in your
IBM ESS hardware and software environments.
򐂰 Ensure that every IBM ESS cluster includes at least one IBM ESS Management Server
and one management network switch.
򐂰 Plan your education and skill sets for IBM Spectrum Scale and IBM ESS so that you can
understand and use this technology to its fullest capabilities.
򐂰 IBM recommends that each IBM ESS building block has the same IBM ESS code level.
For migration or upgrade purposes, adhere to requirements for IBM ESS code levels that
are allowed to coexist during the migration or upgrade within each cluster. For example,
ensure that a code level must be no more than two different OFED levels apart.

66 Introduction Guide to the IBM Elastic Storage System

򐂰 Ensure that each IBM ESS building block is integrated into the same rack; it cannot be
split into multiple racks.

Chapter 3. IBM Elastic Storage System planning and integration 67

68 Introduction Guide to the IBM Elastic Storage System
 A

Appendix A. IBM Elastic Storage System

models
In this appendix, we describe information and differences between the three generations of
IBM Elastic Storage Systems (ESS) that are used today. Understanding the nomenclature
and the differences between these generations is helpful when you are designing your IBM
ESS environment, especially if different generations of IBM ESS coexist in the same IBM
Spectrum Scale cluster.

This appendix includes the following topics:

򐂰 A.1, “IBM Elastic Storage System model specifications” on page 70
򐂰 A.2, “First-generation IBM Elastic Storage System models overview” on page 75
򐂰 A.3, “Second-generation IBM Elastic Storage System overview” on page 79
򐂰 A.4, “Differences between the IBM Elastic Storage System generations” on page 84
򐂰 A.5, “IBM Spectrum Scale for IBM Elastic Storage System licensing” on page 88
򐂰 A.6, “IBM Spectrum Scale RAID technical overview” on page 89

© Copyright IBM Corp. 2016, 2021. All rights reserved. 69

A.1 IBM Elastic Storage System model specifications
This section describes the specifications of the various IBM ESS models.

A.1.1 First-generation IBM Elastic Storage System specifications

The first-generation IBM ESS models were first announced and shipped in the fourth quarter
of 2014. These models, including upgrades, were withdrawn from marketing in January 2018.

The first-generation IBM ESSs are still fully supported by IBM Service and Support for
hardware and software maintenance. First-generation IBM ESSs also continue to receive IBM
ESS solution stack hardware, firmware, and software updates.

The first-generation IBM ESS consisted of the models that are listed in Table A-1.

Table A-1 First-generation IBM ESS building block models

ESS model Enclosure type /U/ Enclosures Number of drivesa
IBM machine type -
model

GS1 EXP24S 1 24 Solid-state drives (SSDs)

2U
GS2 2 46 SAS + 2 SSD or 48 SSD
No machine/type;
GS4 4 94 SAS + 2 SSD or 96 SSD
specified as feature
GS6 code on IBM ESS 6 142 SAS + 2 SSD
POWER
servers

GL2 DCS3700 2 116 NL-SAS + 2 SSD

4U
GL4 4 232 NL-SAS + 2 SSD
Machine type
GL6 6 348 NL-SAS + 2 SSD
1818-80E

a. On the hard disk drive (HDD) configurations, the two SSDs are not visible user capacity. These
SSDs hold data that is used internally for IBM Spectrum Scale RAID metadata only.

Consider the following points regarding network interface cards (NIC):

򐂰 The IBM ESS GSx and GLx models support three high-speed data NICs per POWER8
data server.
򐂰 Because two POWER8 data servers are used per IBM ESS building block, a total of six
high-speed data NICs are used per IBM ESS GSx or GLx building block.

A.1.2 Second-generation IBM Elastic Storage System specifications

This section describes the second-generation IBM ESS specifications.

The second-generation IBM ESS models were announced in April 2017 and are still
available, including upgrades, from IBM. They also are in full service and supported by IBM.

70 Introduction Guide to the IBM Elastic Storage System

The second-generation IBM ESS does not include a solution IBM machine type. The
second-generation IBM ESS server and storage hardware components feature the following
IBM ESS-unique machine types:
򐂰 IBM machine type 5148 for the IBM ESS IBM POWER8 servers
򐂰 IBM machine type 5147 for the IBM ESS IBM Storage enclosures

These machine types uniquely identify these hardware components as part of an IBM ESS
solution.

The second-generation IBM ESS models are available in the following types:
򐂰 GSxS models for all SSD storage
򐂰 GHxx models hybrids of some SSD storage and some HDD storage
򐂰 GLxS models for high-capacity HDD storage
򐂰 GLxC models for ultra high-capacity and density HDD storage

These GLxS and GSxS models were first announced and available in the second quarter of
2017. They are fully supported by IBM Service and Support for hardware, firmware, software
updates, and maintenance.

The SSD flash storage second-generation IBM ESS models GSxS specifications are listed in
Table A-2.

Table A-2 Second-generation IBM ESS SSD Flash building block models
ESS model Enclosure U/Number drives Enclosures Number of drives
IBM machine type - model

GS1S 2U24 1 24 SSD

Machine type 5147-024
GS2S 2 48 SSD

GS4S 4 96 SSD

The HDD disk drive storage second-generation IBM ESS models “GLxS” specifications are
shown in Table A-3.

Table A-3 Second-generation IBM ESS HDD hard disk building block models
ESS model Enclosure U /Number drives Enclosures Number of drivesa
IBM machine type - model

GL1S 5U84 1 82 NL-SAS + 2 SSD

Machine type 5147-084
GL2S 2 166 NL-SAS + 2 SSD

GL3S 3 250 NL-SAS + 2 SSD

GL4S 4 334 NL-SAS + 2 SSD

GL5S 5 418 NL-SAS + 2 SSD

GL6S 6 502 NL-SAS + 2 SSD

a. On the HDD configurations, the two SSD are not visible user capacity. These SSDs hold data
that is used internally for IBM Spectrum Scale RAID metadata only.

Appendix A. IBM Elastic Storage System models 71

 Consider the following points regarding NICs:
򐂰 The IBM ESS GSxS and GLxS models support three high-speed data NICs per POWER8
data server.
򐂰 Because two POWER8 data servers are used per IBM ESS building block, a total of six
high-speed data NICs are used per IBM ESS GSxS or GLxS building block.

The ultra-high capacity HDD storage second-generation IBM ESS models GLxC
specifications are as listed in Table A-4.

Table A-4 Second-generation IBM ESS HDD hard disk building block models
ESS model Enclosure U /Number drives Enclosures Number of drivesa
IBM machine type - model

GL1C 4U106 1 104 NL-SAS + 2 SSD

Machine type 5147-106
GL2C 2 210 NL-SAS + 2 SSD

GL3C 3 316 NL-SAS + 2 SSD

GL4C 4 422 NL-SAS + 2 SSD

GL5C 5 528 NL-SAS + 2 SSD

GL6C 6 634 NL-SAS + 2 SSD

GL8C 8 846 NL-SAS + 2 SSD

a. On the HDD configurations, the two SSD are not visible user capacity. These SSDs hold data
that is used internally only for IBM Spectrum Scale RAID metadata.

Consider the following points regarding NICs:

򐂰 The IBM ESS GLxC models include another SAS adapter per POWER8 server, compared
to the GSxS and GLxS models. This addition provides enough SAS connections for the
GL8C model to be built.
򐂰 The GLxC models support two high-speed data NICs per POWER8 data server.
򐂰 Because two POWER8 data servers are used per IBM ESS building block, a total of four
high-speed data NICs are used per IBM ESS GLxC building block.

When the ultra-high-capacity IBM ESS GLxC models are configured, ensure that the NIC
capacity and bandwidth is sufficient for your workload. If you need higher network bandwidth
capability than four high-speed data NICs per POWER8 data server can provide, choose one
of the other IBM ESS models that has three high-speed data NICs per POWER8 data server.

A.1.3 Second-generation IBM Elastic Storage System Hybrid models

A primary use of the GHxy hybrid models is for a less expensive option to combine an IBM
Spectrum Scale SSD storage tier with an IBM Spectrum Scale HDD capacity storage tier.

The IBM ESS GHxy Hybrid models combined one or two drawers of SSD with two or four
drawers of 5U84 HDD storage. You can think of these models as a combination of the GSxS
models and the GLxS models.

72 Introduction Guide to the IBM Elastic Storage System

 ESS Hybrid models are IBM ESS storage building blocks that consist of SSD Flash 5147-024
storage enclosures and HDD 5147-084 storage enclosures. The IBM ESS Hybrid models
(model GHxy) are a blend of 2U24 SSD enclosures and 5U84 HDD enclosures, where “x” is
the number of 2U24 SSD enclosures, and “y” is the number of 5U84 HDD enclosures. The
following models are available:
򐂰 GH12
򐂰 GH22
򐂰 GH14
򐂰 GH24

The IBM ESS GHxy models use the same POWER8 data server. They also feature the same
NIC capacity as the IBM ESS GSxS and GLxS models.

A.1.4 Third-generation IBM Elastic Storage System 3000 specifications

The new IBM Elastic Storage System 3000 is part of the third generation of IBM ESS.

The third-generation IBM ESS 3000 model with NVMe flash storage was announced in
October 2019 and is available, including upgrades, from IBM. It also is fully supported by IBM.

The third-generation IBM ESS 3000 features an IBM ESS-unique machine type of IBM
machine type 5141-AF8 for the IBM ESS 3000.

The specifications of the IBM ESS 3000 model are listed in Table A-5.

Table A-5 Third-generation IBM ESS NVMe Flash building block model
ESS model Enclosure U/Number drives Enclosures Number of drives
IBM machine type - model

ESS 3000 2U24 1 12 or 24 NVMe

Machine type 5141-AF8 Available NVMe drive sizes
are: 1.92 TB, 3.84 TB, 7.68 TB,
or 15.36 TB

Appendix A. IBM Elastic Storage System models 73

A.1.5 Third-generation IBM Elastic Storage System 5000 specifications
The new IBM Elastic Storage System 5000 is part of the third generation of IBM ESS.

The third-generation IBM ESS 5000 model with POWER9 data servers and 5147-092 or
5147-106 storage enclosures, was announced in July 2020 and General Availability was
August 2020.

The specifications of the IBM ESS 5000 SL model are listed in Table A-6.

Table A-6 Third-generation IBM ESS 5000 SL model - HDD hard disk building block
ESS model Enclosure U/Number drives Enclosures Number of drivesa
IBM machine type - model

SL1 5U92 1 90 NL-SAS + 2 SSD

Machine type 5147-092
SL2 2 182 NL-SAS + 2 SSD
HDD Drive sizes available:
SL3 3 274 NL-SAS + 2 SSD
6 TB
SL4 10 TB 4 366 NL-SAS + 2 SSD
14 TB
SL5 16 TB 5 458 NL-SAS + 2 SSD

SL6 6 550 NL-SAS + 2 SSD

SL7 7 642 NL-SAS + 2 SSD

a. On the HDD configurations, the first storage enclosure has two slots that are taken up by SSDs,
which are not visible user capacity. These SSDs hold data that is used internally by IBM
Spectrum Scale RAID metadata only.

The specifications of the IBM ESS 5000 SC model are listed in Table A-7.

Table A-7 Third-generation IBM ESS 5000 SC model - HDD hard disk building block
ESS model Enclosure U/Number drives Enclosures Number of drivesa
IBM machine type - model

SC1 4U106 1 104 NL-SAS + 2 SSD

Machine type 5147-106
SC2 2 210 NL-SAS + 2 SSD
HDD Drive sizes available:
SC3 3 316 NL-SAS + 2 SSD
10 TB
SC4 14 TB 4 422 NL-SAS + 2 SSD
16 TB
SC5 5 528 NL-SAS + 2 SSD

SC6 6 634 NL-SAS + 2 SSD

SC7 7 740 NL-SAS + 2 SSD

SC8 8 846 NL-SAS + 2 SSD

SC9 9 952 NL-SAS + 2 SSD

a. On the HDD configurations, the first storage enclosure has two slots that are taken up by SSDs,
which are not visible user capacity. These SSDs hold data that is used internally by IBM
Spectrum Scale RAID metadata only.

74 Introduction Guide to the IBM Elastic Storage System

A.2 First-generation IBM Elastic Storage System models
overview
You can encounter IBM Spectrum Scale clusters that include the first-generation IBM ESS
models within them.

IBM fully supports an IBM Spectrum Scale cluster that has a mix of first-, second-, and
third-generation IBM ESS models.

In this section, we describe the first-generation IBM ESS models so that you can understand
and manage your IBM Spectrum Scale environment that includes a mix of IBM ESS model
generations.

A.2.1 First-generation IBM Elastic Storage System models

The first-generation IBM ESS models were first announced and shipped in the fourth quarter
of 2014. These models, including upgrades, were withdrawn from marketing in January 2018.
The first-generation IBM ESSs are still fully supported by IBM Service and Support for
hardware and software maintenance. First-generation IBM ESSs also continue to receive IBM
ESS solution stack hardware, firmware, and software updates.

The first-generation IBM ESS models were available in the following types:
򐂰 GSx models
򐂰 GLx models

Depending on the first-generation IBM ESS model, the following configurations were
available:
򐂰 2U24 or storage enclosures hold SSDs or 10K RPM HDDs storage drives.
򐂰 4U60 of storage enclosures to hold high capacity Nearline SAS HDD storage drives.

The first-generation IBM ESS has a solution IBM machine type: 5146. This machine type
applies to the first-generation IBM ESSs only. Each first-generation IBM ESS building block
consists of two IBM POWER8 servers and a specific number of storage enclosures (1, 2, 4, or
6) depending on the model.

The POWER8 data servers were model S822L (IBM machine type 8247-22L). The storage
enclosures can be one of the following configurations:
򐂰 IBM EXP24S SFF (small form factor) Gen2-bay drawer (2U 24 drives), which are used in
models GS1, GS2, GS4, or GS6.
򐂰 IBM DCS3700 (IBM machine type 1818-80E) are 4U and contain 58 HDD drives each,
which is used in models GL2, GL4, and GL6.

The first-generation IBM ESS GS models can use 2.5-inch 10 K rpm HDD (1. 2 TB) or
2.5-inch SSD (400 GB or 800 GB).

The first-generation IBM ESS GL models can use 3.5-inch NL-SAS HDDs (2 TB, 4 TB, or
6 TB).

All IBM ESS models can be mixed in an IBM Spectrum Scale cluster to meet the specific
needs of client applications that use IBM ESS storage. For example, a common best practice
is to use SSDs for IBM Spectrum Scale metadata and NL-SAS for IBM Spectrum Scale data
storage.

Appendix A. IBM Elastic Storage System models 75

A.2.2 First-generation IBM Elastic Storage System GSx models
The GSx models that used 2U24 storage enclosures were available with all SSD or all 10K
RPM, high-speed SAS HDD storage.

The first-generation IBM ESS GSx models are shown in Figure A-1.

ESS 1st gen - GSx

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

FC 5887
FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
FC 5887
FC 5887

FC 5887
FC 5887
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

GS1 building block GS2 building block GS4 building block GS6 building block
1 enclosure, 6U 2 enclosure, 8U 4 enclosure, 12U 6 enclosure, 12U
24 SSD data disks 46 SAS ad 2 SSD – or 94 SAS ad 2 SSD – or 142 SAS ad 2 SSD
48 SSD data disks 96 SSD data disks
400GB or 800GB 400Gb or 800GB SSD
400GB or 800GB SSD 400GB or 800GB SSD
10GbE or 40GbE 1.2TB SAS
InfiniBand 1.2TB SAS 1.2TB SAS
10GbE or 40GbE
10GbE or 40GbE 10GbE or 40GbE
InfiniBand InfiniBand
InfiniBand

Figure A-1 First-generation IBM ESS GSx models

A primary use of the GSx models with SSD was for IBM Spectrum Scale metadata, or for
high-speed HDD storage tier.

76 Introduction Guide to the IBM Elastic Storage System

A.2.3 First-generation IBM Elastic Storage System GLx models
The HDD models of the first-generation IBM ESSs (see Figure A-2) used 4U60 Storage
enclosures (IBM 1818-80E).

ESS 1st Generation - GLx

GL2 building block GL4 building block GL6 building block

2 enclosures, 12U 4 enclosures, 20U 6 enclosures, 28U
116 NL-SAS data disks 232 NL-SAS data disks 348 NL-SAS data disks
2, 4 or 6TB 2, 4 or 6TB 2, 4 or 6TB
2 SSD caching disks 2 SSD caching disks 2 SSD caching disks
10GbE or 40GbE 10GbE or 40GbE 10GbE or 40GbE
InfiniBand InfiniBand InfiniBand

Figure A-2 First-generation IBM ESS GLx models

The primary usage of the GSx models with NL-SAS HDDs was for IBM Spectrum Scale
capacity tiers.

A.2.4 First-generation IBM Elastic Storage System Data Servers

The first-generation IBM ESS uses a pair of IBM POWER8 (machine type 8247-22L) servers
as IBM Spectrum Scale Data servers, which are cross-connected to all storage for high
availability. These 8247-22L servers run Red Hat Enterprise Linux Big Endian operating
system.

The first-generation IBM ESS Management Server is also an IBM POWER8 (machine type
8247-21L) server. This 8247-21L server also runs Red Hat Enterprise Linux Big Endian
operating system.

The first-generation IBM ESS POWER servers were managed by an IBM Hardware
Management Console (HMC), which is a hardware component that is commonly used to
manage IBM Power server environments.

Appendix A. IBM Elastic Storage System models 77

 HMC for first-generation IBM Elastic Storage System
The first-generation IBM ESS used an IBM HMC to manage the POWER server portion of the
first-generation Elastic Storage Server environment. This HMC can be used to manage other
IBM Power Systems servers that are outside of the Elastic Storage Server environment. Also,
clients can opt to use an HMC within their environment to manage the POWER servers in a
new first-generation IBM ESS environment.

HMC provides reliability, availability, and serviceability (RAS) features for the POWER8
servers in a first-generation IBM ESS.

Note: IBM ESSs required an HMC for the first-generation only. In subsequent IBM ESS
generations, the function of the HMC was integrated into the IBM Electronic Service Agent
(ESA) running on the second-generation IBM ESS Management Server (EMS).

A.2.5 First-generation IBM Elastic Storage System storage enclosures

In this section, we describe the storage enclosures that were available on the first-generation
IBM ESSs.

2U24 (feature code on 8247-22L)

The first-generation IBM ESS 2U24 storage enclosures can be ordered with SSDs or HDDs.

These storage enclosures do not have a separate IBM machine type; they were feature codes
of POWER8 8247-22L data servers in the first-generation IBM ESS.

The IBM 2U24 storage enclosure is shown in Figure A-3.

IBM 2U24 Storage Enclosure

Figure A-3 IBM 2U24 Storage Enclosure

The following storage drives are available in these 2U24 enclosures:

򐂰 2.5-inch 10 K rpm HDD (1.2 TB)
򐂰 2.5-inch SSD (400 GB or 800 GB)

All 2U24 enclosures within one first-generation IBM ESS building block must have the same
SSD or HDD size. Different IBM ESS building blocks can have different storage enclosures or
drive sizes.

78 Introduction Guide to the IBM Elastic Storage System

 4U60 (1818-80E)
The first-generation IBM ESS HDD models (GL2, GL4, and GL6) used IBM machine type
1818-80E storage enclosures for HDD storage. All 1818-80E storage enclosures are 4U60 in
size (4 rack units, 60 HDD slots available) and were populated with 58 HDDs (no partial
population was available).

The first 1818-80E in a first-generation IBM ESS also contained two SSDs that hold internal
IBM Spectrum Scale RAID metadata. The capacity of these two SSDs is not visible nor
available as user capacity for the IBM Spectrum Scale and IBM ESS file system. The IBM
1818-80 4U60 storage enclosure is shown in Figure A-4.

IBM 4U60 Storage Enclosure

Figure A-4 IBM 1818-80E 4U60 Storage Enclosure

Each IBM 1818-80E used in the first-generation IBM ESS, which held 58 HDDs.

All HDDs in a first-generation IBM ESS building block with 4U60 must have same HDD size.
In the 1818-80E, 2 TB, 4 TB, and 6 TB HDDs were available.

A.3 Second-generation IBM Elastic Storage System overview

The second generation of IBM Elastic Storage Server was announced in April 2017 and is
available (including upgrades) from IBM and is supported by IBM.

The second-generation IBM ESS does not include a solution IBM machine type. The IBM
ESS server and storage hardware components feature the following IBM ESS-unique
machine types:
򐂰 IBM machine type 5148 for the IBM ESS IBM POWER8 servers
򐂰 IBM machine type 5147 for the IBM ESS IBM Storage enclosures

These machine types uniquely identify these hardware components as part of an IBM ESS
solution.

The available models of second-generation IBM ESS are available in the following categories:
򐂰 SSD flash storage IBM ESS models
򐂰 HDD IBM ESS models
򐂰 Hybrid IBM ESS models that consist of enclosures of SSD flash drives and HDDs.

Appendix A. IBM Elastic Storage System models 79

A.3.1 IBM Elastic Storage SystemGSxS models
The second-generation IBM ESS GSxS models are all-SSD models. GSxS models are not
available with HDD storage.

Figure A-5 shows the three IBM ESS GSxS models. A primary usage of the GSxS models
with SSD is for IBM Spectrum Scale metadata, or to provide a high-speed SSD storage tier.

ESS 2nd Generation - GSxS Models with SSD

Model GS4S
96 SSD

Model GS2S 8

9
16

17 EXP3524

48 SSD
Model GS1S 8

9
16

17 EXP3524

24 SSD 8 16

EXP3524
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M4

9 17

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M4

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M4 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 System x3650 M4

8 16

9 17 EXP3524

8 16 8 16 8 16

9 17 EXP3524 9 17 EXP3524 9 17 EXP3524

Figure A-5 Second-generation IBM ESS GSxS models

Note: IBM ESS performance is available upon request to IBM or your IBM Business
Partner representative. They use the IBM File and Object Solution Design Engine to
estimate performance that is based on your workload and network environment.

Optimum IBM ESS performance is derived from unconstrained IOR benchmark for 100%
sequential read numbers by using unconstrained InfiniBand networks. Other networks
(such as 100 GbE, 40 GbE, and 10 GbE) have more overhead than InfiniBand and typically
lower aggregate bandwidth capabilities result. For more information, contact IBM or IBM
Business Partner representative.

A.3.2 IBM Elastic Storage SystemGHxy Hybrid models with SSD and HDD
The IBM ESS GHxy Hybrid models combined 1 - 2 drawers of SSDs with 2 - 4 drawers of
HDD storage. A primary usage of the GHxy hybrid models is for a less expensive option to
combine an IBM Spectrum Scale SSD storage tier with an IBM Spectrum Scale HDD capacity
storage tier.

Second-generation IBM Elastic Storage System Hybrid models

ESS Hybrid models are IBM ESS storage building blocks that consist of SSD Flash 5147-024
storage enclosures and HDD 5147-084 storage enclosures. The IBM ESS Hybrid models
(model GHxy) are a blend of 2U24 SSD enclosures and 5U84 HDD enclosures; “x” is the
number of 2U24 SSD enclosures, and “y” is the number of 5U84 HDD enclosures.

80 Introduction Guide to the IBM Elastic Storage System

 The following models are available:
򐂰 GH12
򐂰 GH22
򐂰 GH14
򐂰 GH24

Figure A-6 shows the IBM ESS GHxy models.

ESS 2nd Generation – GH Hybrid Models

Model GH24:
2 2U24 Enclosure SSD
Model GH14: 4 5U84 Enclosure HDD
1 2U24 Enclosure SSD 334 NL-SAS, 48 SSD
4 5U84 Enclosure HDD
334 NL-SAS, 24 SSD

Model GH22:
2 2U24 Enclosure SSD
Model GH12: 2 5U84 Enclosure HDD
1 2U24 Enclosure SSD 166 NL-SAS, 48 SSD ESS 5U84
Storage
ESS 5U84
Storage
2 5U84 Enclosure HDD
166 NL-SAS, 24 SSD

ESS 5U84 ESS 5U84

Storage Storage

ESS 5U84 ESS 5U84

Storage Storage

ESS 5U84 ESS 5U84

Storage Storage

ESS 5U84 ESS 5U84 ESS 5U84

ESS 5U84 Storage Storage
Storage Storage

Figure A-6 Second-generation IBM ESS GHxy models

Note: IBM ESS performance is available upon request to IBM or your IBM Business
Partner representative. Use the IBM File and Object Solution Design Engine to estimate
performance that is based on your workload and network environment.

Optimum IBM ESS performance is derived from unconstrained IOR benchmark for 100%
sequential read numbers by using unconstrained InfiniBand networks. Other networks
(such as 100 GbE, 40 GbE, and 10 GbE) have more overhead than InfiniBand and typically
lower aggregate bandwidth capabilities result. For more information, contact IBM or IBM
Business Partner representative.

A.3.3 IBM Elastic Storage System GLxS models

The IBM ESS GLxS models provided high capacity HDD storage by using a 5U84 storage
enclosure. A primary use of the GLxS models is to provide a high capacity IBM Spectrum
Scale HDD storage tier.

Appendix A. IBM Elastic Storage System models 81

 Figure A-7 shows the various models available.

ESS 2nd Generation – GLxS HDD Capacity Models

Model GL6S:
6 Enclosures, 34U
502 NL-SAS, 2 SSD

Model GL5S:
5 Enclosures, 29U ESS 5U84
Storage
418 NL-SAS, 2 SSD
Model GL4S:
4 Enclosures, 24U ESS 5U84 ESS 5U84
334 NL-SAS, 2 SSD Storage Storage

Model GL3S:
3 Enclosures, 19U ESS 5U84 ESS 5U84 ESS 5U84
250 NL-SAS, 2 SSD Storage Storage Storage

Model GL2S:
2 Enclosures, 14U ESS 5U84 ESS 5U84
166 NL-SAS, 2 SSD Storage Storage

Model GL1S: ESS 5U84 ESS 5U84

1 Enclosures, 9U ESS 5U84
Storage Storage

82 NL-SAS, 2 SSD Storage

ESS 5U84 ESS 5U84 ESS 5U84 ESS 5U84

Storage Storage Storage Storage
ESS 5U84
Storage

ESS 5U84 ESS 5U84 ESS 5U84 ESS 5U84 ESS 5U84
Storage Storage Storage Storage Storage

Figure A-7 Second-generation IBM ESS GLxS models

Note: IBM ESS performance is available upon request to IBM or IBM Business Partner
representative. They use the IBM File and Object Solution Design Engine to estimate
performance that is based on your workload and network environment.

Optimum IBM ESS performance is derived from unconstrained IOR benchmark for 100%
sequential read numbers by using unconstrained InfiniBand networks. Other networks
(such as 100 GbE, 40 GbE, and 10 GbE) have more overhead than InfiniBand and typically
lower aggregate bandwidth capabilities result. For more information, contact IBM or IBM
Business Partner representative.

A.3.4 IBM Elastic Storage System GLxC models

ESS GLxC models use the 4U106 storage enclosure that was pioneered and designed for
use in the Sierra and Summit supercomputers at the US Department of Energy1.

1 https://www.ibm.com/thought-leadership/summit-supercomputer

82 Introduction Guide to the IBM Elastic Storage System

Figure A-8 shows the IBM ESS GLxC models.

ESS 2nd Generation – GLxC Ultra High Capacity Models

Model GL8C
8 Enclosures, 36U
846 NL-SAS, 2 SSD

4U106
Model GL6C Storage

6 Enclosures, 28U
Model GL5C 634 NL-SAS, 2 SSD 4U106
Storage

5 Enclosures, 28U
Model GL4C 528 NL-SAS, 2 SSD 4U106 4U106
4 Enclosures, 16U Storage Storage

432 NL-SAS, 2 SSD

Model GL3C 4U106
Storage
4U106
Storage 4U106
Storage
2 Enclosures, 12U
316 NL-SAS, 2 SSD 4U106
Storage
4U106
Model GL2C 4U106
Storage Storage

2 Enclosures, 12U
Model GL1C 210 NL-SAS, 2 SSD 4U106
Storage
4U106
Storage

1 Enclosure, 8U 4U106
Storage

104 NL-SAS, 2 SSD 4U106

Storage
4U106 4U106
4U106
Storage Storage
Storage

4U106
Storage 4U106
4U106 Storage
Storage 4U106 4U106 4U106
Storage Storage Storage

4U106 4U106
4U106 4U106 4U106
Storage Storage
Storage Storage Storage 4U106
Storage

1.46 PB raw 2.9 PB 4.4 PB 5.9 PB 7.3 PB 8.8 PB 11.8 PB raw

Figure A-8 Second-generation IBM ESS GLxC models

A primary use of the GLxC models is to provide an ultra high-capacity, high-density IBM
Spectrum Scale HDD storage tier.

You can expect performance of the IBM ESS GLxC models to be similar to the corresponding
IBM ESS GLxS model.

If you desire to upgrade an existing ESS GLxC model to an equivalent ESS 5000 SC model,
submit an RPQ to IBM to replace the POWER8 data servers with POWER9 data servers. If
the RPQ is granted, this will be a destructive upgrade. Data must be backed up off of the ESS
GLxC, the upgrade and replacement of the POWER8 data servers with POWER9 is
performed, and then the ESS SW stack must be reloaded and the IBM Spectrum Scale file
system redefined on the newly upgraded ESS 5000 SC model.

For more information about comparison tables with IBM ESS model specifications for SSD
and HDD drive sizes and counts by model, see Appendix A, “IBM Elastic Storage System
models” on page 69.

Appendix A. IBM Elastic Storage System models 83

A.4 Differences between the IBM Elastic Storage System
generations
The following section discusses the component differences between the first-, second-, and
third-generation IBM ESSs. It is helpful if these different generations of IBM ESS are in the
same IBM Spectrum Scale cluster.

A.4.1 Hardware
The IBM ESS generations feature the following major hardware differences:
򐂰 First-generation:
– Uses POWER8 8247-21L for IBM ESS Management Server and POWER8 8247-22L
for the IBM ESS Data Servers, all running Red Hat Enterprise Linux Big Endian. This
generation IBM ESS has an overall solution machine type 5146.
– The first-generation IBM ESS for 2U24 storage used the IBM EXP24 2U24 storage
enclosure, which is specified as feature codes on the 8247-22L Data Server. The HDD
storage used IBM 1818-80E storage enclosures for large capacity.
– It required an IBM POWER HMC to manage the IBM POWER servers.
– This generation requires an RHEL Big Endian-based IBM ESS Management Server
(8247-21L). One RHEL Big Endian-based IBM ESS Management Server is required
per IBM Spectrum Scale cluster that includes first-generation IBM ESS.
򐂰 Second-generation:
– Uses POWER8 5148-21L for IBM ESS Management Server and POWER8 5148-22L
for the IBM ESS Data Servers, all running Red Hat Enterprise Linux Little Endian.
These machine types are unique to the IBM ESS and uniquely identify these POWER8
servers as part of an IBM ESS solution. The second-generation IBM ESS does not
have an overall solution machine type.
– This generation uses IBM 5147-024 storage enclosures for SSD storage. For high
capacity HDD storage, it uses IBM 5147-084 storage enclosures or 5147-106 storage
enclosures. These machine types are unique to IBM ESS and uniquely identify these
storage enclosures as part of an IBM ESS solution.
򐂰 Third-generation
– The IBM ESS 3000 has machine type 5141-AF8, and can be managed by a 5148-21L
POWER8 IBM ESS Management Server or a 5105-22E POWER9 IBM ESS
Management Server.
– The IBM ESS 5000 has a common server IBM machine type of 5105-22E POWER9
servers for Data Servers, Protocol Nodes, and IBM ESS Management Server.
– An IBM ESS 5000 storage building block can use 5147-092 storage enclosures (SL
model) or 5147-106 storage enclosures (SC model). You cannot intermix 5147-092
and 5147-106 in the same IBM ESS 5000 storage building block.
– ESS 5000 must be managed by a 5105-22E POWER9 IBM ESS Management Server.
The second-generation 5148-21L POWER8 management server cannot support the
IBM ESS 5000.

Consider the following points:

򐂰 Only the first-generation IBM ESS requires an HMC for the POWER servers. In the
second- and third-generation IBM ESS, the HMC function was integrated into the EMS’s
Electronic Service Agent (ESA).

84 Introduction Guide to the IBM Elastic Storage System

򐂰 IBM fully supports all three multiple generations of IBM ESS in the same IBM Spectrum
Scale cluster. If doing so, make sure you satisfy the requirement that the appropriate IBM
ESS Management Server generation also is attached to the IBM Spectrum Scale cluster.

Note: All IBM ESS generations can all coexist in a IBM Spectrum Scale cluster.

A typical first-generation IBM ESS deployment is shown in Figure A-9.

Figure A-9 Typical first-generation IBM ESS components and deployment

A typical IBM ESS second-generation deployment is shown in Figure A-10.

Appendix A. IBM Elastic Storage System models 85

 Figure A-10 ESS second-generation components and deployment

The components and deployments of IBM ESS first generation is compared to IBM ESS
second generation in Figure A-11. Consider the following major differences that are featured
in the second generation:
򐂰 Does not have a solution machine type that is equivalent to “5146”. Instead, the
second-generation IBM ESS components are uniquely identified as machine type 5148
servers or machine type 5147 storage enclosures.
򐂰 Runs Red Hat Enterprise Linux Little Endian.
򐂰 Does not require an HMC for the POWER servers.

1st Generation ESS 2nd Generation ESS

Protocol Node is mandatory for external client Protocol Node is mandatory for external client connection
connection if not using an IBM Spectrum Scale client If not using an IBM Spectrum Scale client
2nd generation
ESS does not
Protocol Node Protocol Node have a
“solution”
machine type
5146
ESS Building block
5146 family
ESS Building block
5147 / 5148 family X
ESS Internal ESS Internal
Management switch Management switch 2nd generation ESS
uses RHEL Little
EMS Big Endian (entitlement)
8247-21L
EMS Little Endian (entitlement)
5148-21L
EMS X
Big Endian
Endian, does not
use RHEL Big Endian

HMC
X
HMC 2nd generation ESS
does not require a
IO node IO node Hardware
8247-22L 5148-22L Management
Console
IO node IO node
8247-22L 5148-22L

IO enclosure(s) IO enclosure(s)

Figure A-11 Differences between IBM ESS first- and second-generation components

86 Introduction Guide to the IBM Elastic Storage System

 IBM fully supports first-, second-, and third-generation IBM ESSs co-existing in an IBM
Spectrum Scale cluster.

A.4.2 Software
The IBM ESS generations feature the following major software differences and nomenclature:
򐂰 First-generation IBM ESS runs Red Hat Enterprise Linux Big Endian
򐂰 Second- and third-generation IBM ESS runs Red Hat Enterprise Linux Little Endian
򐂰 Second-generation IBM ESS software stacks are under the nomenclature “ESS 5.x.x”
򐂰 The various IBM ESS software stack component levels for second-generation IBM ESS
5.x.x, are documented in IBM Knowledge Center
򐂰 Third-generation IBM ESS software stacks are under the nomenclature “ESS 6.x.x”
򐂰 The various IBM ESS software stack component levels for third-generation IBM ESS 6.x.x,
are documented in IBM Knowledge Center
򐂰 For more information about the first- and second-generation IBM ESS, see IBM
Knowledge Center
򐂰 For more information about the third-generation IBM ESS 3000 and IBM ESS 5000, see
IBM Knowledge Center

Note: An IBM Spectrum Scale cluster can coexist nodes that run Big Endian and Little
Endian operating systems.

A.4.3 IBM Elastic Storage System data server IO node

The IBM Spectrum Scale data server machine type and model for first-generation IBM ESS is
IBM machine type 8247-22L. This machine type and model is the same as the equivalent
POWER8 compute servers.

The second-generation IBM ESS uses IBM machine type 5148-22L for the IBM ESS Data
Servers. This machine type is unique to IBM ESS and is a different machine type from the
stand-alone POWER8 compute servers.

Two identical data servers always must be used in each IBM ESS building block.

The third-generation IBM ESS 3000 data server is based on an x86 server and is integrated
in a single 2U24 enclosure system with the NVMe storage also within the 2U24 IO
enclosures.

The third-generation IBM ESS 5000 uses IBM machine type 5105-22E for the IBM ESS Data
Servers. This POWER9 server is unique to IBM ESS

A.4.4 IBM Elastic Storage System Management Server

The EMS machine type and model for first generation is IBM machine type 8247-21L. This
machine type and model is the same as the equivalent POWER8 compute servers.

The second-generation IBM ESS uses IBM machine type 5148-21L for the IBM ESS
Management Server. This IBM machine type is unique to IBM ESS to differentiate it from
standard 8247-21L POWER8 compute servers.

Appendix A. IBM Elastic Storage System models 87

 An IBM Spectrum Scale cluster with IBM ESS must have at least one IBM ESS Management
Server for each “Endianness”.

If you have first-generation IBM ESS in an IBM Spectrum Scale cluster, you must have a Big
Endian IBM ESS Management Server. This Big Endian EMS 8247-21L supports the
first-generation IBM ESS systems only.

If you have second- and third-generation IBM ESS systems in an IBM Spectrum Scale
cluster, you must have at least one Little Endian IBM ESS Management Server. Use the
suitable Little Endian EMS to support your second- and third-generation IBM ESS systems.

Consider the following points”

򐂰 The IBM ESS 3000 has machine type 5141-AF8 and can be managed by a 5148-21L
POWER8 IBM ESS Management Server or a 5105-22E POWER9 IBM ESS Management
Server.
򐂰 The IBM ESS 5000 has a common server IBM machine type of 5105-22E POWER9
servers for data servers, protocol nodes, and IBM ESS Management Server.
򐂰 An IBM ESS 5000 storage building block can use 5147-092 storage enclosures (SL
model) or 5147-106 storage enclosures (SC model). You cannot intermix 5147-092 and
5147-106 in the same IBM ESS 5000 storage building block.

ESS 5000 must be managed by a 5105-22E POWER9 IBM ESS Management Server. The
second-generation 5148-21L POWER8 management server cannot support the IBM ESS
5000.

A.4.5 Hardware Management Console

An HMC is used by the first-generation IBM ESS systems only. For the second- and
third-generation IBM ESS, the HMC features are integrated into the Little Endian IBM ESS
Management Server.

A.5 IBM Spectrum Scale for IBM Elastic Storage System

licensing
In this section, we describe some of the differences between the first, second, and third
generation of IBM Spectrum Scale licensing. For more information about IBM Spectrum Scale
licensing (with and without IBM ESS), see IBM Knowledge Center.

The IBM Spectrum Scale IBM Elastic Storage System - Licensing Information documentation
is available at this web page.

A.5.1 Socket licensing on first-generation IBM Elastic Storage System

The first-generation IBM ESS used IBM Spectrum Scale socket licensing. Each
first-generation IBM ESS POWER8 server was licensed with a specific number of IBM
Spectrum Scale sockets. No difference existed in IBM Spectrum Scale licensing costs
between the different-sized first-generation IBM ESS models (regardless of capacity or
storage type) because all of the first-generation IBM ESS used the IBM Spectrum Scale
socket licensing on POWER8 servers.

88 Introduction Guide to the IBM Elastic Storage System

 The first-generation IBM ESS featured a separate IBM Spectrum Scale RAID software
license, which was licensed per server. These software licenses were licensed against the
8247-22L IBM ESS Data Servers. An IBM Spectrum Scale RAID license was not needed on
the IBM ESS Management Server.

Because your first-generation IBM ESSs become due for software renewal, contact IBM or
IBM Business Partner for more information. IBM ESS solution software component levels
moved up to IBM Spectrum Scale Version 5. In the process of IBM Spectrum Scale software
maintenance renewal, options are available to you to convert your IBM Spectrum Scale
socket licensing to a support IBM Spectrum Scale V5 licensing mode on your first-generation
IBM ESS.

A.5.2 Capacity Licensing on second- and third-generation IBM Elastic Storage

System
All IBM ESS second- and third-generation IBM ESSs are licensed by capacity. Instead of the
standard IBM Spectrum Scale software license with a “Per TiB” metric, a specific IBM
Spectrum Scale for IBM ESS license with a specific IBM Program ID and with a “Per Disk”
metric is used.

The “Per Disk” metric licenses the IBM Spectrum Scale for IBM ESS by the number of SSDs
or HDDs in the IBM ESS. No difference exists in the license charge for the size of the SSD or
HDD. Therefore, the licensing is the same for an IBM ESS model, regardless if it has small 4
TB HDDs or large 14 TB HDDs.

Note: IBM Spectrum Scale for IBM ESS (current IBM Program IDs are 5765-DAE for Data
Access Edition and 5765-DME for Data Management Edition) is a specific IBM Program ID
for use on IBM ESS. It uses a different metric than the IBM Spectrum Scale software-only
capacity license (5641-DAx or 5641-DMx), which is licensed by TiB. (The “x” indicates 1, 3,
or 5 years of IBM software service and support.)

Also, unlike IBM Spectrum Scale software (which IBM clients can manage through IBM
Passport Advantage®), IBM Spectrum Scale for IBM ESS does not have a Passport
Advantage part number. Renewals with 5765-DAE and 5765-DME can be managed with
by working with your IBM Technology Support Services (TSS) representative.

A.6 IBM Spectrum Scale RAID technical overview

The following section provides a high-level, technical overview of the IBM Spectrum Scale
RAID that is used in all IBM ESS models.

IBM Spectrum Scale RAID software that is used in the IBM ESS solution runs on standard
serial-attached SCSI (SAS) disks in just a bunch of disks (JBOD) arrays. IBM Spectrum Scale
RAID on IBM ESS provides significant JBOD cost reduction while simultaneously providing
enterprise class reliability.

Different IBM ESS models provide solid-state drives (SSDs or NVMe) when more
performance is needed. The IBM ESS solution does not require or use any kind of external
RAID controller or acceleration.

IBM Spectrum Scale RAID supports multiple RAID codes and distributes client data,
redundancy information, and spare space across the disks in such a way that if a physical
disk loss or even a group of physical disk loss occurs, it does not affect data availability.

Appendix A. IBM Elastic Storage System models 89

 Instead of relying on the disks or conventional RAID controller only, IBM Spectrum Scale
RAID implements erasure coding, which is embedded within the IBM Spectrum Scale file
system to provide reliability and high performance. This configuration detects and reports
storage media faults and read or write errors, and resolves other data integrity problems.

IBM Spectrum Scale RAID also implements an end-to-end checksum from the storage media
all the way out to the IBM Spectrum Scale client.

A.6.1 IBM Spectrum Scale RAID erasure codes

IBM Spectrum Scale RAID in the Elastic Storage Server supports different data protection
algorithms and can detect and correct up to two or three concurrent storage media faults per
erasure code array.

The options for RAID configuration are eight stripes of data plus two or three parity stripes
that use Reed-Solomon codes or one stripe of data plus two or three replica stripes. The data
plus parity or replica stripes, which are known as tracks, are shown in Figure A-12.

Figure A-12 IBM Spectrum Scale RAID Tracks

A.6.2 End-to-end checksum

If you use the IBM Spectrum Scale software client to access data on the IBM ESS, the IBM
Spectrum Scale client is aware if the IBM Spectrum Scale file system is based on IBM
Spectrum Scale RAID Network Shared Disks. If this configuration is used, an 8-byte
checksum is calculated during a write operation, which is appended to the data, and sent over
the network to the IBM Spectrum Scale RAID server. The checksum is verified and then IBM
Spectrum Scale RAID writes the data along with its checksum on the disks and logs the
version number on its metadata.

When a read operation is requested, IBM Spectrum Scale RAID verifies checksum and
version on its metadata. If it is OK, it sends the data to the client. If it is not OK, the data is
rebuilt based on parity or replication and then sent to the client along with newly generated
checksum.

The end-to-end checksum feature provides a robust means to prevent and correct silent disk
errors or missing disk writes. This feature provides an IBM ESSential level of reliability to
assure data integrity and prevent file system checks, which assures reliability and uptime for
ever larger petabyte-scale file systems.

90 Introduction Guide to the IBM Elastic Storage System

A.6.3 Declustered RAID arrays
IBM Spectrum Scale RAID implements its own data and spare disk layout scheme that
reduces overhead and mitigates the performance impact to users when recovering from disk
failures. IBM Spectrum Scale RAID spreads or declusters user data, redundancy information,
and spare space across all the disks of the array instead of leaving all spare space in a single
disk. A conventional 1+1 RAID layout is compared to a declustered array in Figure A-13.

Figure A-13 Declustered array versus 1+1 array

Consider an example where seven stripes of data are on each disk. Figure A-13 shows the
left three arrays of two disks in a replicated 1+1 configuration and a spare. On the left, you
can see the data stripes spreading all over the seven disks of the declustered array.

In case of failure of one disk, all data from the remaining disks of the array must be replicated
to the spare disk on the traditional 1+1 array. On the declustered array, the replication occurs
on spare space of all the remaining disks, which can decrease the rebuild impact from three
to four times.

Appendix A. IBM Elastic Storage System models 91

 The IBM Spectrum Scale RAID overhead reduction is shown in Figure A-14.

1+1 array rebuild operation vs. IBM Spectrum Scale RAID

failed disk failed disk

Rebuild overhead
reduced by 3.5x

time
time

Rd Wr Rd-Wr

Rebuild activity confined to just a Rebuild activity spread across many

few disks – slow rebuild, disrupts disks, less disruption to user
user programs programs

Figure A-14 Array rebuild operation

On the IBM ESS, using RAID 8+2 or 2-way replication (1+2), if one disk loss occurs, the
rebuild operation starts with low priority with even lower impact for the clients. With this array
configuration, the rebuild is considered critical and run on high priority only if two concurrent
disk losses occur.

By using 8+3 RAIDs or 3-way replication (1+3), the rebuild operation becomes critical only if
three concurrent disk losses occur in the same declustered array.

92 Introduction Guide to the IBM Elastic Storage System

Related publications

The publications that are listed in this section are considered particularly suitable for a more
detailed discussion of the topics that are covered in this paper.

IBM Redbook
Many IBM Redbooks publications and IBM Redpapers are available about IBM Spectrum
Scale and ESS. For more information, see the following search results on the IBM Redbooks
publications website:
򐂰 IBM Spectrum Scale
򐂰 Elastic Storage Server
򐂰 Elastic Storage System

You can search for, view, download or order documents and other Redbooks, Redpapers,
Web Docs, draft, and additional materials at the following website:
ibm.com/redbooks

Online resources
The following websites are also relevant as further information sources:
򐂰 IBM Knowledge Center for IBM Spectrum Scale:
https://www.ibm.com/support/knowledgecenter/en/STXKQY/ibmspectrumscale_welcome.
html
򐂰 IBM Knowledge Center Frequently Asked Questions for IBM Spectrum Scale:
https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html
򐂰 IBM Knowledge Center for Elastic Storage Server:
https://www.ibm.com/support/knowledgecenter/en/SSYSP8/sts_welcome.html
򐂰 IBM Knowledge Center Frequently Asked Questions for Elastic Storage Server:
https://www.ibm.com/support/knowledgecenter/SSYSP8/gnrfaq.html
򐂰 ESS IBM Spectrum Scale RAID Quick Deployment Guide:
https://www.ibm.com/support/knowledgecenter/SSYSP8_5.3.5/ess_qdg.pdf?view=kc
򐂰 IBM ESS manuals and publications on IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/SSYSP8_5.3.5/bl8xx_library_prodoc.h
tm
򐂰 IBM Spectrum Scale Protocol Node Quick Deployment Guide as part of ESS solution:
https://www.ibm.com/support/knowledgecenter/SSYSP8_5.3.5/ess_protocols_qdg.pdf?
view=kc
򐂰 IBM Rack:
https://www.ibm.com/support/knowledgecenter/en/POWER8/p8had/p8had_oemrack.htm

© Copyright IBM Corp. 2016, 2021. All rights reserved. 93

 򐂰 IBM Network Switches sourced from Mellanox:
– https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hdx/p9hdx_network_swi
tches.htm
– https://www.ibm.com/support/knowledgecenter/en/POWER8/p8hdx/p8hdx_network_swi
tches.htm
򐂰 IBM Sales Manuals for Elastic Storage Servers:
– https://www.ibm.com/common/ssi/printableversion.wss?docURL=/common/ssi/rep_sm
/1/897/ENUS5146-_h01/index.html&request_locale=en
– https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_sm/1/897/EN
US5148-_h01/index.html&lang=en&request_locale=en
– https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_sm/3/897/EN
US5147-_h03/index.html&lang=en&request_locale=en
– https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_sm/2/897/EN
US5147-_h02/index.html&lang=en&request_locale=en
– http://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_sm/4/897/ENU
S5147-_h04/index.html&request_locale=en
– https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_sm/1/872/EN
US5141-_h01/index.html&lang=en&request_locale=en
򐂰 IBM ESS Problem Determination Guide:
https://www.ibm.com/support/knowledgecenter/SSYSP8_5.3.5/ess_pdg.pdf?view=kc
򐂰 IBM ESS System Specifications:
https://www.ibm.com/support/knowledgecenter/P8ESS/p8ehb/p8ehb_sysreq_5148.htm
򐂰 IBM ESS classes from IBM Learning Services: Introduction to Elastic Storage Server
(no-charge course):
https://www.ibm.com/services/learning/search?query=DL08017G
򐂰 IBM education on Elastic Storage Server or on IBM Spectrum Scale:
https://www.ibm.com/services/learning
At this web page, search for:
– Elastic Storage Server
– IBM Spectrum Scale

Help from IBM

IBM Support and downloads
ibm.com/support

IBM Global Services

ibm.com/services

94 Introduction Guide to the IBM Elastic Storage System

Back cover

REDP-5253-02

ISBN 0738459445

Printed in U.S.A.

®
ibm.com/redbooks